Resolve L2SS-188 "2021 05 27 branched from master udp recv sst client"

@mol @jurges Replaced deque qith Queue and tested it. My main concern is now wether to use the put or put_nowait function. For now I've assumed that the put function works best, since we're dealing with UDP.

Good point, @snijder I am making an exception and read the Queue doc.

class queue.Queue(maxsize=0)
Constructor for a FIFO queue. maxsize is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If maxsize is less than or equal to zero, the queue size is infinite.

OK. Understood. This tells me that, if the Queue happens to be full, the put will block regardless. But there's the option that you, @snijder , mention:

Queue.put(item, block=True, timeout=None)
Put item into the queue. If optional args block is true and timeout is None (the default), block if necessary until a free slot is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Full exception if no free slot was available within that time. Otherwise (block is false), put an item on the queue if a free slot is immediately available, else raise the Full exception (timeout is ignored in that case).

Queue.put_nowait(item)
Equivalent to put(item, False).

Considreing this and - as I already did yesterday - assuming that this is the blue print for all of our fututre "receive this data over UDP from that IP:PORT"-devices, I would immediately opt for the non-blocking put. But in addition I would then want to have an MP in the device that indicates the "fullness" of the queue, i.e. occupancy of the Queue in 0%-100%. Set the warning/alarm values accordingly (warning >=90%, alarm 100%) which will allow Tango to automatically send out an alarm event to interested parties.

Here's my reasoning: We run this software on an multiuser/multiclient/multiprocess system and we do not have full control over what the system does behind the scenes. Therefore I actually expect that the system cannot keep up with storing the content of an incoming data stream. Why my software actually is not able to keep up is for the handling of the new situation without significance. If it is due to my software not being able to clear the queue of the imcoming stuff fast enough or because the OS is busy doing some system maintenance, ragrdless my software cannot keep up. Is it then smart to block the incoming stream to slow down the rate at which we need to store data? No because we cannot reconfigure how often the SSTs/XSTs/whatever are computed. Does it make sense to speed up the computation thread? Yes, but at the time of the incident, there is nothing that can actually speed up the computations. We can only make code faster after it showed that it is too slow. This leaves the storing of the received data as the only place where a proper action can take place. Long story made short: queue full? discard the packet, the alarm is already active since it got triggered during the last cycle when the last empty space in the queue was used up and the queue's occupancy reached 100%. This will keep the system stable under unexpected but unavoidable high load.

added 1 commit

94945a28 - changed back to put_nowait and added queue percentage MP

Compare with previous version

added 1 commit

65c8c87d - exposes buffersize and queuesize to the sst_client init function with default parameters

Compare with previous version

merged

mentioned in commit d0b217f5

Resolve L2SS-188 "2021 05 27 branched from master udp recv sst client"

Merge request reports

Activity