Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Event Order ...
#1
Hey Dean,

  I'm chasing a strange problem, and my working theory is that it is caused by events being processed out of order.  Is that possible?  Here's my scenario:

Boolean Field X has a bidirectional IsTrue trigger.  I have a triggered event which calls me via SendDrvCmd
T=0; X becomes True
T=30; X becomes False
T=31; X becomes True

My code seems to occasionally receive (not sure about delay times)
True (30secs) True (1sec) False

Instead of the correct
True (30secs) False (1sec) True

I haven't been able to capture this yet, but I'm writing some code to try to prove my conjecture.

Based on the CQC architecture, is my theory even possible?  Is there any way to force the events to be processed in the correct order?  Can I get the EventId and re-sort the events after receipt?  CQCEventSrv has 25 threads on my system, so I suspect this may be possible...

Thanks -- Bob
Reply
#2
The event server has multiple threads to process triggers and it grabs them and hands them out to the threads as they come in. Otherwise, one longer running event would hold up all of the others and make them slow.

So it's possible if they come in very close together, that they would both be grabbed by separate threads and processed at basically the same time, and one might win since order of thread execution is not deterministic.

I've been thinking for a long time about being able to mark events as singleton type, so another event of the same type (say, same event type for the same field), could be held back until the previous one processes. That is something that would take some significant thought as to what really constitutes the 'same type of event.

I guess it could be at the filter level. So, mark any events that pass this filter with a monotonic counter value and force them to process in order. Something like that.
Dean Roddey
Explorans limites defectum
Reply
#3
Grrr.  I typed/posted a lengthy reply yesterday, but it seems to be lost.  I'll try again.

If I understand correctly, EventServer has one thread which processes filters and many worker threads to execute trigger macros for event/triggers that satisfy the filter.

If so, it is possible for a trigger macro to be executing simultaneously.  Consider a simple counter: ReadFieldValue; add 1; WriteFieldValue.  Concurrency could lead to missed counts.

I like the counter idea.  The when an event passes a trigger filter the master thread could increment a pending counter for the trigger and queue up the event/trigger/counter for the workers.  When a worker gets started, it checks its counter against the (execute counter - 1) for that trigger.  If it is a match, execute the trigger and set the execute counter for that trigger.  If it is not a match either sleep or go to next item in worker queue...

Another possible solution would be to dedicate triggers to specific threads.  TriggerA can only be executed by Thread4.  The downside is imperfect load balancing.  How about one worker thread per trigger?  Ha.

Do you think this could be added soon?  I'm trying to decide whether to code a work-around or not.

Thanks -- Bob
Reply
#4
Let me cypher on it a little bit. That's not something to do lightly since it might have unforeseen consequences. I think that the per-filter counter is probably the most reasonable scenario. That doesn't mean of course that the same trigger won't get through another filter, but that's something I can't control.
Dean Roddey
Explorans limites defectum
Reply
#5
Oh, I was missing the obvious fact that there's a one to one relationship between filters and triggered events. Duh! So the path of the triggered event itself is the unique id. So there's no need for a sequence number.

I think that, to make it actually efficient (since threads are always blocked on the input queue and would just continuously awake and maybe have to ignore some queued event), is to let each thread have a sub-queue of its own. When the next thread wakes up because there's an event in the main input queue to process, it will first see if any other thread is already processing one of this type. If so, it will push that one out to that thread's private queue, and then check the next one. It might end up just distributing them all and going back to sleep.

Each thread will empty its own queue before going back to the main input queue to get another. That would make for a reasonably efficient way to serialize events of the same type and not have to worry about numbering them or anything. Their order in the thread's sub-queue is enough.

If no other one of that type is processing, then the next that gets awakened when something is in the main queue will just grab it and process it normally.

Ultimately that shouldn't be too big a deal. The synchronization won't be particularly difficult. And it just requires adding a single boolean field to the triggered events, and checkbox to the filter screen.
Dean Roddey
Explorans limites defectum
Reply
#6
The thought of multiple workers writing into each other's private queues is a bit scary!  A worker thread would need to lock the other queue recheck that the trigger is still running over there and then write/unlock.  The worker itself would need to lock before removing finished tasks, etc..  Seems like the private queue can only have events of the same TrigType.  Also, when a worker is moving an event from the main queue to a private queue it must keep the main queue locked until it has finished writing to the private queue otherwise it could get swapped out and when it wakes up the event could end up out of sequence...

I was thinking that you wouldn't need to add a field to triggered events or the filter screen.  This should be the behavior for all triggers.  Keep the code as simple as possible.

Its hard to imagine there would be a need for anything to run parallel/out of sequence.  Maybe a trigger that is so slow and frequently called that a single thread can't keep up?  Seems pretty unlikely...  Actually, keeping the trigger running in the same thread would probably speed things up anyway rather than swapping from worker to worker...
Reply
#7
I'm a little worried about breaking folks who might have been depending on some sort of side effects of there being no serialization. It's hard to think when it would be a good thing, but it's always hard to think how such things could affect existing systems.

The synchronization is really messy. I've thought of and rejected a number of scenarios because they don't quite work.

Some that might work a bit better wouldn't preserve ordering of the other events. So, if thread A is processing an X type event, he could have access to the main queue which is already protected. He could check for others of his type and grab them. But that would let those jump others in the queue which we don't want to happen. If the other one is back in the main queue and the thread processing the current one is done, then it's already not an issue.

It only works if we can deal with the next available one, and for all threads. We could sync getting one into any given thread, but there's no way to know if, in the meantime, another one didn't grab one off the queue of the same type and is now processing it.

Avoiding that means synching with all threads, but that can't be done because they are out there processing a triggered event for some arbitrary period of time and cannot be synced with.

It's not remotely feasible to have a thread dedicated per event type, unless they started and stopped as required, then all of the above issues come back into play again just with more confusion.

Anyhoo, I'll have to think more on that tomorrow when my caffeine levels are higher.
Dean Roddey
Explorans limites defectum
Reply
#8
I think synchronization is not bad, but the new step is each worker has to lock the main queue when it has finished executing its event/trigger.

Worker waits on Main Q.
Something shows up: Worker locks Main Q
Worker removes top entry.
Worker checks trigger types array for "running flag".
  If set it adds event to trigger types Wait Q (or could get ID of processing thread and add to private thread Wait Q).  Unlock Main Q and go back to sleep. 
  Else Worker SETs "running flag" * unlock Main Q and process trigger for event.  When finished it locks main Q and checks Wait Q.
    If more work, remove it and go to *
    Else CLEARs "running flag" unlock Main Q and goes back to sleep

The other easy solution is to provide a configuration parameter limiting number of event worker threads to ONE.  It may not be viable on all systems, but I suspect it would run just fine on mine (and hopefully all systems that require guaranteed event serialization)…

--Bob
Reply
#9
I think I have it worked out. We just need an 'active serial events' list, and a separate queue for those types of events. So, the main thread will check to see if the next event that passes the filters is in that active list. If so, it will dump it into a separate queue, because there is a thread responsible for these and it will keep handling them until there are no more. The other threads are still working on the main queue. When one that is processing a serialized ones doesn't find another in the second queue, it just goes back to blocking on the main queue.

So the gotcha is that serialized ones can 'jump the queue'. If ten others come in, then another of an active serialized type, that new serialized ones goes to the other queue and the thread processing that type will take it first, possibly ahead of the other ten non-serialized ones.

But this gets rid of the issue of threads just spinning because there are events in the main queue but they can't process them.

The queue's mutex will be used to synchronize access to both the queue and the active serialized events list. When a thread completes a serialized event, it has to lock to check the second queue. If it doesn't find anything, it removes that event path from that active list, then unlocks and goes back to listening on the main queue. But, this guarantees that the main thread won't have to worry about a race condition. If there is a thread that is processing an event of a serialized type, it will either get a new one into the second queue before that thread can remove it from the active list (which means that thread will then grab it and continue working on that serialized type) or after the thread has already removed it which means that the new event can go into the main queue to be grabbed naturally by the next thread (which will then put it into the active list.)

So, with the one gotcha of jumping the queue, it works well. It does raise the specter of a pathological scenario where all threads are working on serialized events and there is a fast stream of enough types of them to keep all the threads busy, so they never are able to process the next non-serialized one. 

But, if that was happening, most likely in the current situation that would just saturate the event queue and events would get dropped anyway. And the same would happen here if they were coming way too fast to process them.
Dean Roddey
Explorans limites defectum
Reply
#10
Hmmm, okay. Well, you know your code best.

The solution I provided in post #8 does not have the 'jump the queue' problem. It also doesn't have a worker threads spin because there is something in the main queue they cannot process problem.

I would also encourage you to treat all events as serial. As you say the current method is non-deterministic. Frankly it is probably causing many bugs that users don't even know they are having.
Reply


Possibly Related Threads...
Thread Author Replies Views Last Post
  Triggered Event Server not triggering EST 52 7,378 08-10-2018, 11:23 AM
Last Post: Dean Roddey
  Triggered Event Questions RichardU 3 1,073 07-14-2018, 03:53 PM
Last Post: Dean Roddey
  Triggered event w/ multiple triggers lleo 16 2,706 06-06-2018, 06:21 PM
Last Post: lleo
  Event entry not responding! Deane Johnson 4 1,447 12-11-2017, 02:10 PM
Last Post: Dean Roddey
  Writing to an external file from trigger event Ira 21 5,878 11-13-2017, 02:08 PM
Last Post: znelbok
  Motion Event Running Twice? zra 2 1,574 07-31-2017, 08:15 PM
Last Post: zra
  EVent Server Monitor znelbok 1 1,343 06-07-2017, 12:36 PM
Last Post: Dean Roddey
  I have an event that I want to happen 5 hours after sunset. ghurty 8 3,597 03-24-2017, 08:40 AM
Last Post: Dean Roddey
  Scheduled Event - Yearly Basis Bugman 2 1,504 11-07-2016, 04:44 PM
Last Post: Dean Roddey
  Vizia RF scene controller event traps? IVB 64 7,394 08-16-2016, 01:58 PM
Last Post: IVB

Forum Jump:


Users browsing this thread: 1 Guest(s)