Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Server keep dying after ~7-10days
#21
4.2.16

I think I found source of the problem, however I can't find it in the driver source

it looks like my pandora driver leaks memory... however i don't see anything in the driver which can do that, unless there are issue with json parser.

Code:
12/30 23:02:24-hpc, CQCServer, CQCDrv_PandoraThread13
{
    CQCKit, MEng.System.CQC.Runtime.CQCLogger.636, Status/App Status
    Error getting status URL: Could not allocate system memory buffer. Size=524288
}
12/30 23:02:25-hpc, CQCServer, CQCDrv_PandoraThread13
{
    CQCKit, MEng.System.CQC.Runtime.CQCLogger.636, Status/App Status
    Error getting status URL: Could not allocate system memory buffer. Size=524288
}
12/30 23:02:26-hpc, CQCServer, CQCDrv_PandoraThread13
{
Reply
#22
On that version, this issue would be expected still. You'll need to move up to the latest beta to get rid of it.

On the memory thing, that would be pretty wierd. If that's happening, then you should be able to see in the task manager that CQCServer has gotten very big.

One sort of suspicious thing is that that number comes out to exactly 0x80000, which is unlikely to be a coicidence. I wonder if that's a limit somewhere in the underlying code to avoid accidental allocation of really large buffers unless explicitly increased. I'll look for such a constant...

The HTTP class uses a local buffer of that max size but it's for the outgoing header lines, which isn't the issue here and would never be a problem.

The web image widget uses that as a max size for the images it downloads, which might be a little small, but it's not the issue here.

OK, so it doesn't look like that is the issue. Do you see a lot of memory usage on the CQCServer process? Are you being sure to reset things, i.e. you aren't just continuously appending to some list instead of resetting it are you?
Dean Roddey
Explorans limites defectum
Reply
#23
Dean Roddey Wrote:The most important thing is just grab the error info that shows up if a client gets an error that the server didn't respond.

Had another recurrence of one of my ongoing issues come up today. I noticed that the Pandora driver was not being responsive so I went to pause/ restart the driver (this is something that seemingly needs to be done with every server reboot due to something in the way that the driver connects with CQC through sproxy). I received the following error when trying to pause the driver. I tried pausing others and received the same error.

Facility: CQCKit
Severity: Failed
Line: CQCKit_ThisFacility.cpp/2387
Error#: 6004/0/0

The CQC security Server could not be contacted

Tried to test some things locally on the server by bringing up the IV and noticed that things were taking a looooooong time to load. Task manager shows that CQCServer is using 1.9GB of memory...

I'll hold off on rebooting in case you want me to report on anything else.
Reply
#24
What version are you on? There was a blantant leak in the new JSON stuff that was fixed a drop or two back, which could add up fast in a JSON based driver doing queries fairly often. And there appears to be one in the text input stream, which would be used by many drivers that parse JSON or XML received as text, which was just reported a day or so ago, so I've yet to tackle that one (though it's considerably smaller.)
Dean Roddey
Explorans limites defectum
Reply
#25
Dean Roddey Wrote:What version are you on? There was a blantant leak in the new JSON stuff that was fixed a drop or two back, which could add up fast in a JSON based driver doing queries fairly often. And there appears to be one in the text input stream, which would be used by many drivers that parse JSON or XML received as text, which was just reported a day or so ago, so I've yet to tackle that one (though it's considerably smaller.)


I am on 4.2.17
Reply
#26
Yeh, the leak was found after that, in one of the betas. If it doesn't look like we can get a super-tight beta out soon, and after I find the other leak, I'll do a 4.2.18 just to fix those issues in the meantime.
Dean Roddey
Explorans limites defectum
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Event Server Error Bugman 4 128 06-05-2022, 08:54 AM
Last Post: Bugman
  Trigger not Working after Move to new Server & V6.0.7 kblagron 4 366 03-02-2022, 05:35 AM
Last Post: sic0048
  Migrating to New Server gReatAutomation 1 877 06-12-2020, 05:40 PM
Last Post: Dean Roddey
  Global Variables in the Event Server context lleo 1 1,008 05-16-2020, 11:21 AM
Last Post: Dean Roddey
  Is there a way to change the listening port for the CQC server? ghurty 3 1,165 04-23-2020, 05:54 PM
Last Post: Dean Roddey
  Hosted CQC server - Latency zra 4 1,919 01-06-2020, 05:24 AM
Last Post: zra
  CQC Server Closing Connection gReatAutomation 1 1,142 12-19-2019, 01:53 PM
Last Post: Dean Roddey
  Moving the CQSL server Dean Roddey 28 7,262 09-03-2019, 05:27 AM
Last Post: Dean Roddey
  SubSonic Media Server API Support? gReatAutomation 4 2,375 06-16-2019, 02:45 PM
Last Post: gReatAutomation
  Logic Server not sending triggered events znelbok 7 3,150 05-08-2019, 08:13 AM
Last Post: Dean Roddey

Forum Jump:


Users browsing this thread: 1 Guest(s)