diff --git a/applications/lofar2/doc/prestudy/desp_hdl_design_article.txt b/applications/lofar2/doc/prestudy/desp_hdl_design_article.txt
index 371ad15eed5ba49576b4206d2dc9a7977b0e0039..96f97f9f40737ee07d827c4c43b9b8c3cf46f0ee 100644
--- a/applications/lofar2/doc/prestudy/desp_hdl_design_article.txt
+++ b/applications/lofar2/doc/prestudy/desp_hdl_design_article.txt
@@ -1,5 +1,6 @@
-. Distinguish beteen state registers and pipeline registers.
- The state registers keep the state of the function and the function itself is programmed in combinatorial logic.
+Idea / rule: Distinguish beteen state registers and pipeline registers.
+
+. The state registers keep the state of the function and the function itself is programmed in combinatorial logic.
In this way the pipelining that is needed to achieve timing closure can be added independent of the function.
This approach could be described in a paper, because it is quite significant and differs from the well known
Gailser approach (that uses RL=1 and does not separate state from pipeline). AXI uses RL=0 but need to check
@@ -28,5 +29,5 @@
Ref:
- tools/oneclick/doc/desp_firmware_dag_erko.txt : Secion 6) VHDL design
- tools/oneclick/doc/desp_firmware_overview.txt
\ No newline at end of file
+ $RADIOHDL/tools/oneclick/doc/desp_firmware_dag_erko.txt
+ $RADIOHDL/tools/oneclick/doc/desp_firmware_overview.txt
\ No newline at end of file
diff --git a/applications/lofar2/doc/prestudy/station2_sdp_hdl_components.txt b/applications/lofar2/doc/prestudy/station2_sdp_hdl_components.txt
index 10479b07685d6f294f7a6adac0be7a990fd18479..0b1cc193b482232c63a9efc238e1546e766aef8a 100644
--- a/applications/lofar2/doc/prestudy/station2_sdp_hdl_components.txt
+++ b/applications/lofar2/doc/prestudy/station2_sdp_hdl_components.txt
@@ -2,13 +2,9 @@
* DP encoder
*******************************************************************************
- dp_packet_enc
- . Current dp_packet_enc encodes sosi fields into: CHAN (32b), BSN (64b), DATA (>= 1 b), ERR (32b).
+ . Current dp_packet_enc encodes sosi fields into: CHAN (32b), sync & BSN (64b), DATA (>= 1 b), ERR (32b).
. Use new dp_packet_enc with CRC to mitigate false positive ETH CRC --> dp_packet_enc_crc:
- CHAN (32b), BSN (64b), DATA (>= 1 b), ERR (32b), CRC (32b)
- . Use variant of dp_packet_enc with only BSN to move BSN through a FIFO, without having to use a
- wider FIFO --> dp_packet_enc_bsn. The BSN has to be transported via one data word, such that it
- is avaiable at the sop.
- BSN (32b), DATA (>= 32b)
+ CHAN (32b), Sync & BSN (64b), DATA (>= 1 b), ERR (32b), CRC (32b)
- RSP RAD frame:
. uses: FSI, FSN, DATA, CRC.
@@ -20,7 +16,7 @@
in 4.3e9. The packet rate is f_sub. Per T_sub interval the ring carries about 10 - 20 packets between N = 16
nodes. Hence in total the packet rate of the ring for one LBA station is about 195312.5 * 20 * 16 ~= 60 M
packets / s. With 50 stations and LBA and HBA this become about a factor 100 more, so about 6 G packets / s.
- If 0.01 % of the packets have errors, then the packet error rate is 0.6 M, so then once every 1 / (0.6e6 * 2.3e-10)
+ If 0.01 % of the packets have errors, then the packet error rate is 0.6 M /s, so then once every 1 / (0.6e6 * 2.3e-10)
~= 2 hours somewhere in LOFAR there will occur a false positive CRC. If such an error occurs, then it must not
cause the entire processing to stall. Therefore some additional check is necessary using a CRC. It is not
sufficient to check e.g. the ETH type and the expected packet length, because these do not cover the other
@@ -34,52 +30,205 @@
Design decisions:
-- Use CHAN (32b), Local BSN (32b), DATA (>= 1 b), ERR (32b), CRC (32b) to transport data between FPGAs
+- Use CHAN (32b), Sync & BSN (64b), DATA (>= 1 b), ERR (32b), CRC (32b) to transport data between FPGAs
without false positive CRCs during the lifetime of LOFAR 2.0 to garantuee that only correct packets
enter the FPGA internal processing. The internal FPGA processing must be robust to lost packets, but
it does not have to be robust against corrupted packets (wrong contents, wrong length).
-- Use Local BSN (32b), DATA (>= 32b) for BSN aligners, to avoid extra parallel FIFOs for BSN
+
*******************************************************************************
* BSN aligner
*******************************************************************************
-Aspects to consider:
-. time out:
- - sop time out per input or align timeout for all inputs
- - sop timeout based on interpacket time > T_sub
- - sop timeout based on no packet detected during T_sub grid
- - align time out based on maximum latency difference between all inputs (= g_sop_timeout in dp_bsn_align.vhd)
- - when FIFO full
- - when number of packets in FIFO --> avoids possibility of fragmented packets in FIFO
- Use FIFO packet count per input or align timeout? Align timeout requires knowing the T_sub period. Using
- packet counts is easier than using timeouts in seconds or T_sub periods or FIFO size.
-
-. Input flush
- - until empty or flush one packet
- - flush per input per input or flush all inputs
- - flush by reading or by moving a read pointer
-
-. lost packet
- - causes all streams to drop this packet
- - could still output and flag the other (enabled) streams, but only if there is time left, which is not certain in case
- multiple packets got lost
- - for BF drop all (because beam is affected), for XC pass on (becomes visibilities of active inputs are still oke).
- - output enabled inputs that are aligned, fill in unaligned input with zero data or flagged data
- . flag lost packets via header flag or via most negative integer in the data.
- . flag via can make use of the fact that the data is typically complex, so e.g. use real as flag and imag as cause identifier.
- - Lost packets can be detected by:
- . comparing BSN between inputs using sosi.bsn(c_bsn_align_w-1:0) max = min and the align_sync
- . checking per input that the align bsn increments +1 within the align_sync interval.
- - If multiple packets on a remote input get lost, then the other inputs fill up. Maintain FIFO packet count per
- input and flush all inputs empty if one of them got filled up. Flush empty to avoid
- that at some moment all inputs may have multiple packets pending in the FIFO, that will then be output
- in a burst. The pending packets that corresponded to the lost packet will need to be discarded anyway,
- because there is no time to output them still.
+Usage schemes:
+. General data transports:
+ - N = 2 input aligner with 1 local data and 1 remote data
+ - N > 2 input aligner with 1 local data and N-1 remote data
+ - N >=2 input aligner with only N remote data (not used on ring, but was used in APERTIF)
+
+. Ring data transports:
+ - beamlets on ring: l --> r+l --> r+l --> ... --> r+l
+ . on each node align two inputs: l,r
+ . output filler data if remote got lost, to preserve nominal output rate to CEP
+
+ - crosslets on ring: rrrrrrrr,l --> rrrrrrrr,l --> ... --> rrrrrrrr,l
+ . on each node separately align N/2 pairs of inputs l,r, have one pair per XC cell
+ or
+ . on each node first align all inputs l,N/2*r, and then split into N/2 pairs of l,r to have one pair per XC cell
+ . discard output data if remote got lost, to count number of active blocks per integration sync interval
+ or
+ output filler data if remote got lost, and use zero to not disturb the intergation and count unflagged blocks
+ to know the number of active blocks per integration sync interval
+
+ - subbands on ring: l, rl, rrl, rrrl, ..., rrrrrrrrrrrrrrrl
+ . on final node align all l,(N-1)*r inputs
+ . output filler data if remote got lost, to preserve nominal output rate to AARTFAAC
+
+ - transient buffer readout: l, r, r, ..., r
+ . no align, readout from one node at a time
+
+
+Assumptions:
+. Per input the blocks arrive in order
+. Only allow correct Rx packets to enter the FPGA processing, therefore discard Rx packets per input if:
+ - if CRC is wrong,
+ . this requires using store and forward per node, because the enitre packet is needed to calculate the CRC
+ - if the BSN at sync in the Rx packet is not equal to the local Station BSN at sync.
+ . if the sync BSN is not equal then discard all subsequent blocks until the sync BSN is equal again.
+ - sync monitor component: checks that received Station BSN ar sync equals local BSN at sync
+ . if the sync BSN is equal, then:
+ - assume that the BSN for all subsequent blocks will be correct too. This
+ is a save assumption, because the BSN is determined inside the remote FPGA (so error free, because the
+ logic is error free) and transported using a CRC (so error free, because the CRC detects errors.)
+ - BSN based Wr pointer and the align_sync are sufficient in BSN aligner to ensure that all inputs are
+ aligned
+- Lost packets:
+ . should not cause subsequent packets to get lost too
+ - in APERTIF the BSN aligner does loose more packets due to flush and realign
+ - in APERTIF the sync_checker looses entire sync intervals to ensure filled sync intervals
+ - therefore recovery at the 1 ssync takes too long, because an incidental loss of packet should not
+ cause an entire second of data to get lost.
+ . must not create a burst of filler packets
+- BSN latency
+ . If all hops on the ring are active then the total latency will be (N-1)*(d + 1) where d is the transport
+ latency of each hop and 1 is due to store-and-forward at each node. Typically the total transport latency
+ on the ring is (N-1)*d < 1, so less than one block period.
+ . The total ring latency is covered by g_bsn_latency > (N-1)*(d + 1).
+ . programmable g_bsn_latency is not needed
+ . the input latencies are fixed by design, so inputs cannot have more letency than g_bsn_latency and the
+ Rx packet CRC and BSN at sync checking ensure that only good packets enter the FPGA.
+- If one input packet is lost then it is acceptable that the corresponding output is lost
+ . there should be an option to have filler data such that the other inputs can still be output
+- Only output correct packets, so do not allow packets with corrupted data or wrong BSN to slip through the BSN aligner.
+- If often packets on one input get lost then it is not acceptable that the output is lost.
+ . Support static and dynamic input enable/disable control, or
+ . support filler data on streams that lost input packets
+- If all inputs of BSN aligner stop, then the output stops.
+
+
+
+Design options:
+- Lost packets and filler packets:
+ . Detected by
+ - in circular buffer the Wr flag for the lost block remains unset
+ - in FIFO by no pending input or pending input with higher BSN then current output BSN
+ - checking per input that the align bsn increments +1 within the align_sync interval
+ - timeout checking per input using a local slot pacer at the block rate
+ - realign after FIFO overflow on other inputs, due to that output waits for lost packet
+ - can we do without timeouts? Yes because the input latencies are static and within a fixed range
+ . replace by a filler block or let all streams to drop this block
+ - for BF drop all inputs, because beam is affected
+ - for XC pass on, because visibilities of active inputs are still oke.
+ . flagging
+ - filler data blocks can be flagged using a sosi.channel bit as flag
+ - filler data can be undefined, forced to zero, noise, most negative integer in the data, for compled e.g. use real as
+ flag and imag as cause identifier.
+
+
+- Input FIFO
+ . packets are stored in arrival order
+ . The BSN of the input packets must differ during a g_bsn_latency interval, but does not have to be incrementing or
+ continuous, because the alignment is based on the BSN being equal
+ . the FIFO must also pass on the 1 s sync
+ . Flushing:
+ - flush per packet or flush until empty?
+ - flush per input per input or flush all inputs?
+ - flush by reading, or by reset or by moving a Rd pointer
+ - Use packet count instead of FIFO full indicator
+ - can we do without flushing the FIFO? Not if we need to realign.
+ - If multiple packets on a remote input get lost, then the other inputs fill up if there is no timeout. Flush
+ all inputs empty if one of them got filled up. Flush empty to avoid that at some moment all inputs may have
+ multiple packets pending in the FIFO, that will then be output in a burst. The pending packets that
+ corresponded to the lost packet will need to be discarded anyway, because there is no time to output them still.
+ - also useful to know BSNs at FIFO inputs? --> No, because FIFO packet count can be used to detect pending FIFO overflow.
+ . Keep FIFOs outside or inside BSN aligner component.
+ - the input of the FIFO is needed to be able to maintain a count of the number of packets in the FIFO, which is
+ relevant for the align timeout. The input eop increments the count and the output eop decrements the count.
+ - inputs with a large latency could use a smaller FIFO, this is easier to control with external FIFOs
+ - if the BSN aligner relies on FIFO input information, then it is better to have the FIFOs inside.
+
+
+- Input circular buffer
+ . can handle data arriving out of order, but this is not needed within SDP
+ . The buffer memory size is g_bsn_latency * g_nof_inputs slots that can store a packet.
+ - the maximum latency between any two inputs must be < g_bsn_latency number of data blocks
+ - For each slot there is a Wr flag that needs to be maintained.
+ - For each slot there is also a sync flag to pass on the 1 s sync
+ . Can handle out-of-order data, because it uses the BSN as an index. However on the ring in SDP all data will be in order.
+ . The circular buffer could be used as a FIFO with internal access and an incrementing Wr pointer. However it
+ seems better to use it with a Wr pointer that is derived from the BSN.
+ . the BSN must be continuous BSN and incrementing, because then the remainder of the BSN divided by the buffer size can
+ be used as Wr pointer.
+ . The buffer size is preferrably a power of two, but can be any size (to save memory):
+ - Using a buffer size that is a power of 2 avoids an integer divsion of the BSN, because it can then use the
+ corrsponding LSbits of the BSN as Wr pointer.
+ - Modulo 2**n - 1 can be calculated efficiently for binary numbers, by adding the n-bit digit parts. Similar as
+ mpdulo 3 (= (10-1)/3) can be calculated by adding the decimal digits.
+ - Modulo n for constnat n can be calculated efficiently suing multiplication by 1/n. The 1/n fraction must be
+ represented with sufficient accuracy to determine the remainder.
+ . The slots in the circular buffer have a Wr flag that is set when the slot is written with an Rx packet and cleared
+ when the slot is read for output.
+ . Flushing:
+ - Clearing a Wr flag or all Wr flags is much faster than flush reading a FIFO.
+ . The Rd pointer increments at every output block period.
+ . The Rd pointer increments after every output slot.
+ . The write pointer always needs to be ahead of the Rd pointer. The minimum distance between the Wr and Rd pointer
+ is g_bsn_latency. The size of the circular buffer is the same for all inputs and must be > g_bsn_latency (for wr)
+ + 1 (for rd). The circular buffer read can occur when the write pointer exceeds rd pointer + g_bsn_latency.
+ . the circular buffer is part of the BSN aligner component
+ . On CEP the beamlet data is written into a circular buffer based on the time stamp. A flag indicates whether data in the
+ circular buffer is valid. The size of the circular buffer is in the order of hundreds of ms to cover the distance latency
+ of the international stations. An array of tupples lists the lenght of continuous blocks in the circular buffer, and
+ therefore also to the gaps. A local timer determines when the circular buffer is read. The local timer has ms accuracy
+ compared to UTC, so the size of the circular buffer dominated by the largest latencies. The channel filterbank in CEP
+ also flags the initial channel data that is disturbed after a gap.
+
+
+- static input enable/disable via M&C
+ - align modes local only, remote only, combined, disabled can be achieved via input enable/disable M&C
+. dynamic input enable/disable in case of lost packets
+ - Scheme:
+ . Fine per packet scheme:
+ - Input packets arrive every block period, remote packets can arrive anywhere within a block period,
+ - If one input stops, then g_sop_timeout occurs in s_align and then that input could be dynamically disabled in s_xoff.
+ Inputs can dynamically be enabled if they arrive within two block periods, in s_align.
+ - too nervous, too difficult to debug and monitor
+ . Coarse per sync interval scheme:
+ - if an input has no lost packets during one (or more) sync interval then it can be dynamically enabled for the next sync interval
+ - if an input has lost packets during one (or more) sync interval then it can be dynamically disabled for the next sync interval
+ - This is a suitable scheme because it does not react too fast and it can be monitored via M&C.
+ - Define number of sync intervals for dynamic input control as a generic
+ - preferred because it is less active and easier to monitor
+ - Is dynamic input enable/disable necessary if a lost packet does not affect next packets?
+ . If lost data is replaced by filler data, then only static input enable/disable is necessary. During initial input
+ alignment it is important that all active inputs are indeed active, because together they determine the latency
+ difference between inputs. After initial alignment the data output can continue at at a fixed rate, driven by a
+ local reference. The local input could be used as local output block reference, because (N-1)*d << 1.
+ Alternatively another block period pacer with another time offset can be started after achieving
+ input alignment. If an input becomes inactive it will be flagged and the output can still continue.
+ . If lost data causes all inputs to be discarded, then dynamic input enable/disable may be useful to avoid that a
+ single input causes all output to stop.
+
+- BSN output driven by local input or by remote input
+ . in case of 1 remote input, the remote input does not need a FIFO if it drives the output
+ . in case of > 1 remote input, then the remote inputs also requires FIFOs
+ . using local input increases the latency from remote input to output, because fixed to T_sub grid
+ . using local input at T_sub grid avoids bursts, this can also be handled using flow control
+ . with local input driving the output the assumption is that if the local input has M packets, then all remote
+ inputs will have delivered at least one frame, so there should be a sop pending from all.
+ . if there is no local input, then an artifical local input can be derived when BSN is equal on all enabled remote inputs.
+ . if remote input is lost, then entire output is lost if remote drives output, because there is not enough spare time
+ to still output the other input packets
+ . For remote driven output a slot can be output when for all active inputs there is a block. However if one or
+ a series of packets got lost, then the other inputs will overflow. Hence remote driven output needs a timeout
+ to keep the output running, so a form of local driven output. Hence to avoid additional packet loss on other
+ inputs or of subsequent packets in time it is necessary to have a local driven output. Therefore using a remote
+ driven output is not feasible.
+
+
. alignment:
- - per input the blocks arrive in order
- use BSN incrementing or only use that the input BSN (and input sync) must be equal
- use local data stream as reference stream or treat all streams as equal
. the local data stream is considered perfect, because it can not have packet loss.
@@ -87,7 +236,6 @@ Aspects to consider:
. in dp_bsn_align the artifical local data stream was used to ensure that the output block size was correct,
by using extra CRC checking (ETH CRC and DP CRC) and store and forward in Rx it is already certain that only
correct input packets arrive at the BSN aligner input. Therefore an artifical local data stream is not needed.
- - also useful to know BSNs at FIFO inputs? --> No, because FIFO packet count can be used to detect pending FIFO overflow.
- it is not necessary to be able to achieve alignment near the sync. Once alignment is achieved then the near the sync
the alignment is kept, because the BSN from all enabled inputs are still equal, even though in time they are
discontinuous at the sync due to that the local BSN then restarts at 0.
@@ -96,159 +244,86 @@ Aspects to consider:
. Else if all inputs are treated equal, then which input to select is dynamic. Therefore then use any one or all of them
so output bsn = or(input BSNs) provided that the disabled inputs have BSN = 0.
-. static input enable/disable via M&C
- - align mode local only, remote only, combined, disabled can be achieved via input enable/disable M&C
-. dynamic input enable/disable in case of lost packets
- - Fine per packet scheme:
- . Input packets arrive every block period, remote packets can arrive anywhere within a block period,
- . If one input stops, then g_sop_timeout occurs in s_align and then that input could be dynamically disabled in s_xoff.
- Inputs can dynamically be enabled if they arrive within two block periods, in s_align.
- . Too fine and too difficult to debug and monitor.
- - Coarse per sync interval scheme:
- . if an input has no lost packets during one (or more) sync interval then it can be dynamically enabled for the next sync interval
- . if an input has lost packets during one (or more) sync interval then it can be dynamically disabled for the next sync interval
- . This is a suitable scheme because it does not react too fast and it can be monitored via M&C.
- . Define number of sync intervals for dynamic input control as a generic
-
-. always output and flag inactive inputs
- - The BSN aligner could maintain a fixed output grid and output packets with input data if available and
- flag inputs that have no data.
- - First wait for align_sync on any input to find the input with the shortest BSN latency. With the ring this
- will be the local input, which could also be used as fixed reference. Then start filling the ring buffer
- memory using the LSbits of the local BSN as periodic index. When the reference input has filled g_bsn_latency
- number of packets, then start outputting. The output pace can be set by a local BSN period.
- - The flagging may require a dedicated DP header word.
+
+. Initial alignment declaration can be based on:
+ - All active inputs have data pending in the same slot or at the FIFO output
+ - If BSN latency number of slots on all inputs got filled, then set the Rd pointer. This requires that all inputs
+ start filling at the same BSN index, because then the input with the lowest latency will get filled first. The
+ Rd pointer is set at the BSN index.
+ - The same slot is filled on all active inputs, this slot index sets the Rd pointer:
-. use FIFO or ring buffer memory with flags
- - For FIFO packets are stored in arrival order. Lost packets are only noticed by timeout or by the next packet.
- - Using a ring buffer memory requires continuous BSN that are incrementing, because then the LSbits of the BSN
- can be used as write pointer. The local BSN is not continuous because it wraps at sync and moreover it wraps at
- 195312 or 195313 every sync interval and for oversampled filterbank it also wraps at two different values, but
- a pattern that dependents on the R_os ratio.
- - The ring buffer memory can handle out-of-order data, because it uses the BSN as address.
- - The ring buffer memory flag is set on write packet and cleared on read or flush packet. Clearing a flag is
- much faster than flush reading a FIFO. Lost packets leave an unused block in ring buffer memory.
- - The read pointer increments at every T_sub. The write pointer always needs to be ahead of the read pointer. The
- minimum distance between the wr and rd pointer is g_bsn_latency. The size of the ring buffer is the same for all
- inputs and must be > g_bsn_latency (for wr) + 1 (for rd). The ring bufffer read can occur when the write pointer
- exceeds rd pointer + g_bsn_latency. To ease the ring buffer control the ring buffer size should be a power of
- two such that the corresponding BSN bits can be used as write pointer.
- - Both FIFO and ring buffer could discard output if there are empty inputs or output rd data with flags for empty inputs.
- What will happen with the flagged data, it may be better to discard all inputs if one or more inputs fail.
- - On CEP the beamlet data is written into a ring buffer based on the time stamp. A flag indicates whether data in the
- ring buffer is valid. The size of the ring buffer is in the order of hundreds of ms to cover the distance latency
- of the international stations. An array of tupples lists the lenght of continuous blocks in the ring buffer, and
- therefore also to the gaps. A local timer determines when the ring buffer is read. The local timer has ms accuracy
- compared to UTC, so the size of the ring buffer dominated by the largest latencies. The channel filterbank in CEP
- also flags the initial channel data that is disturbed after a gap.
- - With store and forward at each node the latency on the ring is at least (N-1)*T_sub. Each hop also has a small
- transport latency. The total ring latency is covered by g_bsn_latency > N-1.
- - The ring buffer memory size is g_bsn_latency * g_nof_inputs packets. In parallel for each packet location a
- packet valid flag needs to be maintained. The buffer can output if one input has filled the buffer or if
- all active inputs have data pending.
-
-. Ring buffer state machine
- Maintain a buffer that can fit g_nof_inputs * g_bsn_latency number of packets. The g_bsn_latency is a power
- of 2 such that the LSbits of the input BSN can serve as sop address. The input streams have an align_sync
- that has a period > 2 * g_bsn_latency. First wait for the
- . State machine using ring buffer
- To initially align or to realign the input FIFOs are read empty in s_xoff. In s_xoff it is also possible
- to change the static input enable/disable control.
- Then in s_align the sync aligner waits for the align sync on all enabled inputs or an align timeout. The
- align_sync period can be much less then the sync period to ensure quick realignment. The align_sync period
- and align timeout must be larger than the maximum possible BSN latency difference between any two inputs.
- In this way the all align_sync condition in s_align ensures that all inputs then have the same BSN,
- without explicitly having to check that they all have the same BSN. There is simply not enough memory in
- the system that could cause two inputs to be have align_sync within the BSN latency that do not correspond
- to the same instant. If the align timeout occurs then restart acquiring alignment on the next align_sync
- in s_xoff.
- The alignment of the input packets at the align_sync is always checked in s_align. If all input still have
- align_sync active then the packets can be output. Otherwise if not all packets have align_sync, then the
- after some time the align_sync timeout will occur.
- The alignment of the input packets after the align sync is checked in s_sop. If s_sop detects an align_sync
- on any input then it diverts the further control to s_align. By checking any align_sync both the normal case,
- where all inputs have an align_sync, and the lost packet case where only some inputs have an align sync are
- covered. Otherwise if all enabled input BSN incremented by one, then this means that all inputs still have
- the same BSN, because they had so at the previous align_sync. If an input has a wrong BSN increment, then
- try to recover at the next align_sync in s_align. If an input has no data then the align timeout occurs
- and try to recover via s_xoff.
- The actual output of the aligned packets is done per packet in s_output. The BSN of any of the enabled inputs
- is used as output BSN for all outputs. At the end of the packet the next packet is waited for in s_sop, or
- if a M&C event for the static input enable/disable control occured then this is handled in s_xoff.
-
- s_xoff
- accept static and dynamic input enable/disable control
- flush all inputs until they are empty --> s_align
- # flush by reading the FIFOs empty and by dropping new input
- s_align
- wait for align_sync on all enabled inputs
- if all enabled inputs have align_sync --> s_output
- # the align_sync garantuees that all inputs have the same BSN, so
- # use the BSN of one of the enabled input streams for all output streams
- elsif align timeout --> s_xoff
- # the align times out when one or more enabled input FIFO contains more than g_bsn_latency packets,
- # the input FIFOs are too full to recover via s_align, therefore recover via s_xoff
- s_sop
- wait for sop on all enabled inputs
- if any enabled input has a align_sync --> s_align
- # Always at align_sync do output via s_align, this also covers the sosi.sync that can occur out of order
- # check for any input having a align_sync, to detect lost packet at end of align_sync interval.
- # if all enabled inputs have align_sync then s_align will continue output, else s_align will try
- # to recover at next align_sync.
- elsif all enabled inputs have sop:
- # Within align_sync interval do output via s_sop
- if no input lost a packet, so all enabled input BSN incremented by 1 --> s_output
- # all BSN are still the same because for all enabled inputs the BSN incremented by 1, so
- # use the BSN of one of the enabled input streams for all output streams
- else --> s_align
- # out of order align_bsn so there was a lost packet, try to recover at next align_sync
- elsif align timeout --> s_xoff
- # the align times out when one or more enabled input FIFO contains more than g_bsn_latency packets,
- # the input FIFOs are too full to recover via s_align, therefore recover via s_xoff
- s_output
- output one block
- if at end of block:
- if input control event --> s_xoff
- else --> s_sop
+ t=0 t=1 t=2 t=3 t=4 t=5 t=6 t=7 t=8
+ 9 10 11 12
+ t=0 1 2 3 i=0 1 2 3 i=0 1 2 3 i=0 1 2 3 i=0 1 2 3 i=0 1 2 3 i=0 1 2 3 i=0 1 2 3 i=0 1 2 3 i=0 1 2 3
+
+ 0 1 2 3 W . . . 0 W . . 0 1 W . . 1 2 W W . 2 3 0 W . 3 0 1 W . . 1 2 W W . 2 3
+ 3 0 1 2 . . . W W . . 3 0 W . 3 . 1 W 3 . . 2 W W . . 3 0 W . . . 1 W . . . 2 W
+ 2 3 0 1 . . W . . . 2 W W . 2 3 . W 2 3 . . W 3 . . . W W . . . . W . . . . W .
+ R R R R R R R
+
+ If a packet got lost, then the alignment will fail and needs to be restarted.
+ - Align_sync found in same slot on all active inputs, this slot index sets the Rd pointer
+ . Align_sync period:
+ The maximum latency between two inputs is g_bsn_latency. The minimum time between the last align_sync of the
+ previous align_sync interval and the first align_sync in this align_sync interval is align_sync period -
+ g_bsn_latency. Hence if the align_sync period - g_bsn_latency > circular buffer size, then the align_sync in
+ the circular buffer all apply to the same BSN.
+ The period of the align_sync is preferrably a power of two, such that the align_sync can easily be derived
+ from the BSN and such that the align_sync will always occur at first slot of the circular buffer.
+ The 1 s sync interval could be used as align_sync, but in LOFAR2.0 the sync period is not a power of two and
+ differs by 1 per sync interval, so the sync appears at different slots. Furthermore a 1 s period is
+ relatively slow, using a dedicated and much shorter alig_sync period allows fast initial alignment.
+ . Do we need align_sync?
+ - The advantange of using an align_sync is that if the alignment fails in one period, e.g. due to a lost
+ packet, then it will automatically try again in the next interval. Schemes without an
+ align_sync require a restart, because they wait until the buffer has filled sufficiently and need to refill
+ to try again.
+
+ t=0 t=1 t=2 t=3 t=3 t=4 t=5
+ t=0 1 2 3 4 5 6 7 i=0 1 2 3 i=0 1 2 3 i=0 1 2 3 i=0 1 2 3 i=0 1 2 3 i=0 1 2 3 i=0 1 2 3
+
+ 4 5 6 7 0 1 2 3 W . . . 4 W . . 4 5 W . 4 5 6 W W 5 6 7 S W 6 7 S 5 W 7
+ 3 4 5 6 7 0 1 2 . . . W W . . 3 4 W . 3 4 5 W 3 4 5 6 W W 5 6 7 S W 6 7
+ 2 3 4 5 6 7 0 1 . . W . . . 2 W W . 2 3 4 W 2 3 4 5 W 3 4 5 6 W W 5 6 7
+ R
+ - Without align_sync the buffer would need to be twice as large to ensure unambigous detection of aligment
+ - The align_sync is only used within the BSN aligner.
+ - The input streams have an align_sync that has a period > 2 * g_bsn_latency and that is a power of 2
+ . using a power of 2 implies that the data block at the align_sync will be stored at slot 0.
+ . using an align_sync period > 2 * g_bsn_latency ensures that the sync applies to the same BSN for all inputs
+ . using align_sync << 1 s sync allows faster initial alignment
+ . using align_sync instead of 1 s sync allows using an interval that is a power of 2 so that the align_sync
+ always occurs in slot 0 of a circulare buffer. The 1 s sync can occur in any slot, so then the BSN aligner
+ needs to detect which slot the sync occurred to set the initial Rd pointer.
+ - For the ring the latency depends on the number of hops. Therefore require that initial BSN alignment is achieved
+ with all active input as defined by M&C, to ensure that the total input latency at each node on the ring is
+ determined by the nominal operation.
+
+
+. Circular buffer state machine
+ Receive and monitor input
+ Derive align_sync and Wr pointer from input BSN
+ Write the input at the slot indexed by the Wr pointer and set the Wr flag for that slot.
+ s_xoff:
+ Accept static input enable/disable control
+ Clear all Wr flags of the slots to initially align or to realign the inputs.
+ Reset the Rd pointer at the first slot, because the align_sync is defined at slot 0
+ --> s_align
+ s_align:
+ If input control event --> s_xoff
+ If for all active inputs the Wr flag is set in slot 0 and slot 0 contains the align_sync then
+ restart a periodic slot pulse to set the pace for outputting the slots. An offset of the pace period
+ is used to ensure that in subsequent block periods --> s_sop
+ s_sop
+ if input control event --> s_xoff
+ If slot pulse --> s_output
+ s_output:
+ output one block, clear Wr flag of slot and increment Rd pointer --> s_sop
+
-. Keep FIFOs outside or inside BSN aligner
- - the input of the FIFO is needed to be able to maintain a count of the number of packets in the FIFO, which is
- relevant for the align timeout. The input eop increments the count and the output eop decrements the count.
-
-. BSN output driven by local input or by remote input
- - in case of one remote input, the remote input does not need a FIFO if it drives the output
- - in case of > 1 remote input, then the remote inputs also requires FIFOs
- - using local input increases the latency from remote input to output, because fixed to T_sub grid
- - using local input at T_sub grid avoids bursts, this can also be handled using flow control
- - with local input driving the output the assumption is that if the local input has M packets, then all remote
- inputs will have delivered at least one frame, so there should be a sop pending from all.
- - if there is no local input, then an artifical local input can be derived when BSN is equal on all enabled remote inputs.
- - if remote input is lost, then entire output is lost if remote drives output, because there is not enough spare time
- to still output the other input packets
-
-. General data transports:
- - N = 2 input aligner with 1 local data and 1 remote data
- - N > 2 input aligner with 1 local data and N-1 remote data
- - N >=2 input aligner with only N remote data (not used on ring, but was used in APERTIF)
-
-. Ring data transports:
- - beamlets on ring: l --> r+l --> r+l --> ... --> r+l
- . on each node align two inputs: l,r
-
- - crosslets on ring: rrrrrrrr,l --> rrrrrrrr,l --> ... --> rrrrrrrr,l
- . on each node separately align N/2 times two inputs l,r
- or
- . on each node align all inputs l,N/2*r
-
- - subbands on ring: l, rl, rrl, rrrl, ..., rrrrrrrrrrrrrrrl
- . on final node align all l,(N-1)*r inputs
-
- - transient buffer readout: l, r, r, ..., r
- . no align, readout from one node at a time
-
. BSN max/min scheme of dp_bsn_align.vhd core:
- State machine
@@ -307,7 +382,7 @@ Aspects to consider:
o o o o x x x x x x x x x x x x o recover output in next align_sync interval
- Minimum number of blocks per sync interval.
- . the align sync interval has n = 2**c_bsn_align_w and must be > 2*g_bsn_latency
+ . the align_sync interval has n = 2**c_bsn_align_w and must be > 2*g_bsn_latency
. n=1 block per sync interval:
If one packet gets lost then packets from different sync intervals get aligned, as indicated by 1'. This
is not detected because the bsn always remains 0 (so no increment) and if the align timeout is based on
@@ -396,7 +471,7 @@ Aspects to consider:
. State machine using input FIFOs
To initially align or to realign the input FIFOs are read empty in s_xoff. In s_xoff it is also possible
to change the static input enable/disable control.
- Then in s_align the sync aligner waits for the align sync on all enabled inputs or an align timeout. The
+ Then in s_align the sync aligner waits for the align_sync on all enabled inputs or an align timeout. The
align_sync period can be much less then the sync period to ensure quick realignment. The align_sync period
and align timeout must be larger than the maximum possible BSN latency difference between any two inputs.
In this way the all align_sync condition in s_align ensures that all inputs then have the same BSN,
@@ -407,9 +482,9 @@ Aspects to consider:
The alignment of the input packets at the align_sync is always checked in s_align. If all input still have
align_sync active then the packets can be output. Otherwise if not all packets have align_sync, then the
after some time the align_sync timeout will occur.
- The alignment of the input packets after the align sync is checked in s_sop. If s_sop detects an align_sync
+ The alignment of the input packets after the align_sync is checked in s_sop. If s_sop detects an align_sync
on any input then it diverts the further control to s_align. By checking any align_sync both the normal case,
- where all inputs have an align_sync, and the lost packet case where only some inputs have an align sync are
+ where all inputs have an align_sync, and the lost packet case where only some inputs have an align_sync are
covered. Otherwise if all enabled input BSN incremented by one, then this means that all inputs still have
the same BSN, because they had so at the previous align_sync. If an input has a wrong BSN increment, then
try to recover at the next align_sync in s_align. If an input has no data then the align timeout occurs
@@ -457,32 +532,20 @@ Aspects to consider:
. Comparison of align_sync scheme and BSN max/min scheme:
- Both schemes are limited by the fact that the align_bsn wraps if c_bsn_align_w < 32 b
- Deriving BSN max and min for g_nof_input inputs each with c_bsn_align_w bits per BSN requires much logic in case
- of many inputs, an thus pipelining to achieve timing closure. Comparison of BSN increment per signal input and
+ of many inputs, and thus pipelining to achieve timing closure. Comparison of BSN increment per signal input and
combining these results between inputs takes less logic.
- If one remote packets gets lost, then output for all inputs is lost, because there is no time to output them still.
. with BSN max/min scheme the output may recover at the next packet (if not it did not occur near the wrap of the
align BSN), because the other inputs will flush one packet.
. with align_sync scheme the output will recover at the next align_sync, and thus mis the remaining packets in that
- align sync interval.
+ align_sync interval.
-Assumed performance:
-- Only output correct packets, so do not allow packets with corrupted data or wrong BSN to slip through the BSN aligner.
-- If one input packet is lost then it is acceptable that the corresponding output is lost
- . Alternative would be to flag the input with lost data and still output the other inputs, however can be impossible
- to do so in time, because detecting lost data takes time
-- If one input packet is lost then it is acceptable that some more subsequent outputs packets are lost to reacquire alignment
- . However recovery at the sosi.sync takes too long, because an incidental loss of packet should not cause an entire second
- of data to get lost. Therefore the align_sync is defined which is based on the maximum g_bsn_latency.
-- If a burst of input packets is lost, then it is acceptable that some more outputs packets are lost to reacquire alignment
- . A burst of lost packets already indicates a bad link, so lossing some more packets to recover is acceptable.
-- If often packets on one input get lost then it is not acceptable that the output is lost.
- . Support static and dynamic input enable/disable control
-
Design decisions:
-. Use FIFO instead of ring buffer memory to store input packets:
+. Use FIFO instead of circular buffer memory to store input packets:
- avoids need for deriving address from discontinuous local BSN
- - avoids need for memory write and read pointer control and memory used/not used flags
+ - avoids need for memory write and Rd pointer control and memory used/not used flags
+ - avoids need for buffer size that is a power of 2 (or need for modulo address calculation based on entire BSN)
. Support number of inputs >= 2
. Treat all inputs equal, so no special role for a local input
- suits more general usage
diff --git a/applications/lofar2/doc/prestudy/station2_sdp_icd.txt b/applications/lofar2/doc/prestudy/station2_sdp_icd.txt
index 2cb17ac33e85311644ba2321e1569bcd08d23340..541cd7b5907c7246f51099c3059f595f86c146db 100644
--- a/applications/lofar2/doc/prestudy/station2_sdp_icd.txt
+++ b/applications/lofar2/doc/prestudy/station2_sdp_icd.txt
@@ -22,25 +22,32 @@ UDP link control
- UDP/IPv4
. UDP checksum (not used in LOFAR1)
-
+> nslookup <hostname> # e.g. <astron.nl> to find IP address
+> sudo arp
+> ping <IP address> # to find MAC address for IP address ?
LFAA-CSP_Low : OSI (Open Systems Interconnection) layers
-7 Application : Not applicable, this is the level where the STAT and CEP products each perform their allocated functions.
+7 Application : Not applicable, this is the level where the STAT and CEP products each perform their
+ allocated functions.
6 Presentation :
- SPEAD header
header first word:
- magic = 0x53 ='S' 8b, version = 0x4 8b, itemPointerWidth = 0x2 8b, HeapAddrWidth = 0x0 8b, rsvd=0 16b, number of items = 0x8 16b
+ magic = 0x53 ='S' 8b, version = 0x4 8b, itemPointerWidth = 0x2 8b, HeapAddrWidth = 0x0 8b, rsvd=0 16b,
+ number of items = 0x8 16b
header items:
- heap_counter = coarse channel number (1-511) 16b, packet counter 32b # restart at 0 for new observation, 2k samples per packet --> packet counter wraps after few days
+ heap_counter = coarse channel number (1-511) 16b, packet counter 32b # restart at 0 for new
+ observation, 2k samples per packet --> packet counter wraps after few days
pkt_len = packet payload length 48b
sync_time = unix_epoch_time [s] 48b # last time system was syncrhonised by PPS in seconds since 1 Jan 1970
- timestamp = timestamp [ns] 48b # time of center of first sample in packet since sync_time in ADC sample periods of 1.25 ns
+ timestamp = timestamp [ns] 48b # time of center of first sample in packet since sync_time in
+ ADC sample periods of 1.25 ns
center_freq = frequency [Hz] 48b # center frequency of coarse channel (1-511) * 781250 in Hz
csp_channel_info = rsvd 16b, beam_id 16b, freq_id 16b
- csp_antenna_info = substation_id (1-512) 8b, subarray_id (1-16) 8b, station_id 16b, nof_contributing_antenna (typ. 256) 16b
+ csp_antenna_info = substation_id (1-512) 8b, subarray_id (1-16) 8b, station_id 16b, nof_contributing_antenna
+ (typ. 256) 16b
sample_offset = payload_offset = 0x0
data
@@ -68,7 +75,10 @@ LFAA-CSP_Low : OSI (Open Systems Interconnection) layers
L1 ICD 11109 : STAT - CEP
. Beamlet data
. Transient buffer read out
- . Subband offload (for AARTFAAC)
+
+Not included:
+ . SST, BST, XST, because these are for monitoring and calibration, not for science data
+ . Subband offload for AARTFAAC2.0 will have own EICD
STAT-CEP Beamlet data interface:
@@ -82,10 +92,14 @@ STAT-CEP Beamlet data interface:
. 1b critically PFB, oversampled PFB (or p, q for R_os = p/q)
. 4b beamlet width in number of bits (default 8 for W_beamlet = 8 bit, instead of BM = beamlet mode)
. 5b UniBoard2 FPGA id (16 FPGAs for LBA, 16 for HBA in International Station, instead of RSP ID)
+ . ==> Also beamlet scale setting
+ . ==> Number of antenna in beam (core, LBA, HBA inner to make HBA international look like HBA remote)
- CONFIGURATION_ID 8b (used in LOFAR1? intended to refer to the parset that defines this observation)
+ ==> observation ID 32b
- STATION_ID 16b (idem as LOFAR1)
+ ==> or 8b because there are only ~50 stations
- One packet per range of Station beamlets out of 488 beamlets
. Full band : S_sub_bf * W_beamlet * N_complex / W_byte = 488 * 8b * 2 / 8b = 976 octets
@@ -126,6 +140,9 @@ STAT-CEP Beamlet data interface:
- TX_PACKET_COUNT 32b
+ ==> Not useful, because then CEP needs to count Rx packets. Better send filler packets to keep the
+ packet rate at the nominal rate, so that any packet loss is due to the Network and already
+ clear at OSI 2 layer using lower level tools like Wireshark.
. OSI transport layer 4
. Per stream
. Started at Station power up, increments by 1 for every transmitted packet.
diff --git a/applications/lofar2/doc/prestudy/station2_sdp_m_and_c.txt b/applications/lofar2/doc/prestudy/station2_sdp_m_and_c.txt
index 2fd61a275e3c8fc3a7af979f86962cbc381b6346..eaf504b262098fe26e103f193416d3251b4bb7f3 100644
--- a/applications/lofar2/doc/prestudy/station2_sdp_m_and_c.txt
+++ b/applications/lofar2/doc/prestudy/station2_sdp_m_and_c.txt
@@ -2,7 +2,7 @@
* Station Control software:
*******************************************************************************
-The Station contains hardware and software devices that deliver the funnctionality of the application
+The Station contains hardware and software devices that deliver the functionality of the application
[4.1.2.1]. The Station Control software consist of Control and M&C. The Control determines the behaviour
of the devices in time. Via the M&C the Control can control the devices and monitor them. The M&C uses
a standard software interface for the Control to access the devices. For the Station M&C the M&C will
@@ -48,8 +48,12 @@ FPGAs in parallel. The synchronous M&C can be for a single PPS instant or for ev
- Use fixed internal sync aligned to PPS
- . The disadvantage of a fixed interval is that it is inflexible and cannot be controlled by the SCU,
- This could also be considered an advantage, because it is well defined and does not need control.
+ . In LOFAR1 and APERTIF the sync period is used as fixed update interval for periodic monitoring,
+ periodic control (the beamformer weights) and periodic integration intervals (AST, SST, BST and
+ XST).
+ . The advantage of a fixed update interval is that it is well defined and does not need control.
+ This can also be a disadvantage because a fixed interval is inflexible and cannot be controlled
+ by the SCU. Probably only for the XST this flexibility is nice to have.
. With a fixed interval the monitored information may only reflect what happened during the previous
period. Therefore if the monitoring has to be without gaps in time then the SCU needs to monitor
and aggregate the information at every period. Using a configurable period this aggregation in the
@@ -167,4 +171,9 @@ Behaviour of the data points:
-
+Conclusion:
+- Identify casue of error preferrably via a single monitoring point
+- With proper monitoring no test time is needed
+- Support writing status fields in a test mpd for SW - FW interface testing
+- Use 1 s sync interval of PPS to time period M&C events for all. Optionally support a local BSN scheduler
+ for the XST.
diff --git a/applications/lofar2/doc/prestudy/station2_sdp_timing.txt b/applications/lofar2/doc/prestudy/station2_sdp_timing.txt
index 24df15e8d350b282f5a88d6ce25c294c31e0a862..ac488b6e9b194cde7a6d910d3e0060e1ba0ac694 100644
--- a/applications/lofar2/doc/prestudy/station2_sdp_timing.txt
+++ b/applications/lofar2/doc/prestudy/station2_sdp_timing.txt
@@ -4,9 +4,9 @@
The Station needs an external trigger to align all ADCs in the RCU2S and all FPGA procesing nodes in the SDP.
For this trigger a pulse from the pulse per second (PPS) is used. The PPS is aliged to the top of second of the
-UTC time of day (Tod). The PPS is a hardware trigger that is available within the entire SDP at sample clock
-cycle accuracy. Thanks to the TD the PPS trigger is also available as hardware trigger in all Stations.
-Thanks to the Timing Distributor (TD) the PPS is aligned to UTC ToD that is available to the Telescope Manager
+UTC time of day (ToD). The PPS is a hardware trigger that is available within the entire SDP at sample clock
+cycle accuracy. Thanks to the Timing Distributor (TD) the PPS trigger is also available as hardware trigger in
+all Stations. Thanks to the TD the PPS is aligned to UTC ToD, and the ToD is available to the Telescope Manager
(TM) in LOFAR2.0 and to Station Control in each Station. The TM controls, via Station Control, which PPS pulse
is used to start SDP. The PPS is identified by a Seconds Sequence Number (SSN) that counts PPS since a certain
date in the past, e.g. t_epoch = 1 jan 1970, but some other fixed date is possible too.
@@ -16,60 +16,82 @@ The SDP processes the data in blocks of ADC samples that are identified by a Sta
the Station BSN counts blocks since the same t_epoch as the SSN, so the t_epoch defines the common reference
moment in history for the Station BSN grid and for the PPS grid. The PPS grid does not necessarily always
coincide with the Station BSN grid. The BSN period determines whether the Station BSN can start exactly at an
-PPS or not. To be able to start the data processing at any PPS it is necessary that the Station BSN source
-can start at a programmable fraction of a BSN period after the PPS. In this way processing of one Station ADC
-signal input can be restarted at any PPS (with zero phase offset to the other signal input) and an entire
-Station can be restarted at any PPS (with zero delay offset to the other Stations). The Station BSN source
-ensures that the BSN timing is always on the fixed Station BSN time grid.
+PPS or not.
+
+The processing of the ADC inputs in SDP is done by multiple FPGAs in parallel. Each FPGA has a BSN source that
+creates the Station BSN grid. The BSN source is the wall clock of the FPGA. To be able to start the data
+processing at any PPS it is necessary that the BSN source can start at a programmable fraction of a BSN period
+after the PPS. In this way processing of one Station ADC signal input can be restarted at any PPS (with zero
+phase offset to the other signal input) and an entire Station can be restarted at any PPS (with zero phase
+offset to the other Stations). The BSN source ensures that the BSN timing is always on the fixed Station
+BSN time grid. The initial BSN and the offset fraction of a BSN period need to be provided to SDP via the
+M&C interface by Station Control. Both Station Control and SDP know the PPS grid. Station Control also knows
+UTC and with that information Station Control can program and initialize the BSN source in the FPGAs to start
+counting data blocks at the next PPS.
The sample frequency f_adc = 200 MHz is an integer number of Hz and locked to the PPS, therefore the PPS grid
always coincides with the ADC sample period T_adc = 1/f_adc = 5 ns grid. The Station BSN block period is an
integer number of N_blk sample periods T_adc. The Station BSN period is set by the subband rate of the subband
-polyphase filterbank (PFB). For the critically sampled PFB the BSN period is N_blk = N_fft = 1024 [T_adc].
-For an oversampled PFB N_blk = N_FFT / R_os [T_adc], so e.g. N_blk = 864 for R_os = 32/27. In both cases the
-Station BSN period is equal to the subband period is T_sub. The offset between the PPS grid and the BSN grid is
-between 0 and N_blk-1 ADC sample periods. Hence to start the processing at any PPS the Station BSN source has
-to be able to start the Station BSN at an offset of 0:N_blk-1 sample periods. The Station BSN source starts at
-this PPS with an initial Station BSN of:
+polyphase filterbank (PFB), so the Station BSN period is equal to the subband period is T_sub.
+The input to the subband filterbank is the real signal from the ADC. For both the critically sampled PFB and
+the oversampled PFB the data block size of the input data is N_blk = N_FFT = 1024 ADC samples, however for the
+oversampled PFB the blocks overlap by a factor R_os. Hence for the critically sampled PFB the BSN period is
+T_sub = N_blk = 1024 [T_adc] and for the oversampled PFB the BSN period is T_sub = N_blk / R_os = 864
+[T_adc], in case R_os = 32/27. At the output of the subband filterbank a data block contains
+N_sub = N_FFT / N_complex = 512 complex subband samples that all correspond to the same time instant as
+defined by the Station BSN. Each subband sample represents another frequency.
+
+The offset fraction between the PPS grid and the BSN grid is between 0 and N_blk-1 ADC sample periods. Hence
+to start the processing at any PPS the BSN source has to be able to start the Station BSN at an offset
+of 0:N_blk-1 sample periods. The BSN source starts at a PPS with an initial Station BSN of:
initial Station BSN = ceil((SSN * 1 s) / T_sub) = ceil((SSN * f_adc) / N_blk)
-and a BSN fractional offset of:
+and a BSN offset fraction of:
- BSN fractional offset = mod(SSN * 1 s, T_sub) / T_adc = mod(SSN * f_adc, N_blk)
+ BSN offset fraction = mod(SSN * 1 s, T_sub) / T_adc = mod(SSN * f_adc, N_blk)
to make sure that the BSN grid is always relative to t_epoch, independent of at which PPS the BSN source was
-started. The Station BSN increments after every block. The time t at the BSN grid is:
+started. The Station BSN increments after every block. The time ToD_BSN at the BSN grid is:
- t = t_epoch + Station BSN * T_sub
+ ToD_BSN = t_epoch + Station BSN * T_sub
-Note: The BSN fractional offset could also be compensated for by delaying the sample data in the signal input
-buffers. Delaying the data does compensate for phase differences in the subband data, but does not compensate
-for the offset in the BSN grid. The BSN alignment buffers between different inputs then still need to compensate
-for this BSN fractional offset. Hence delaying the data is an indirect and incomplete solution, and therefore it
-is not used.
+Note:
+- The BSN offset fraction could also be compensated for by delaying the sample data in the ADC signal input
+ buffers at the input of SDP. Delaying the data does compensate for phase differences in the subband
+ data, but does not compensate for the offset in the BSN grid. The BSN alignment buffers between signal
+ inputs from different FPGAs then still need to compensate for this BSN offset fraction. Hence delaying the
+ data is an indirect and incomplete solution, and therefore it is not used.
+- In LOFAR1 the Station BSN is divided in a 32 bit seconds sequence number (SSN) that counts PPS intervals and
+ a 32 bit local BSN that counts blocks within each SSN interval. In APERTIF the BSN is a continuous BSN that
+ can start at any PPS, but without support for a BSN offset fraction. In APERTIF f_sub = 1/T_sub = 781250 so
+ the timing of the APERTIF BSN grid does not depend on at which PPS it is started. For LOFAR2.0 the
+ continuously incrementing BSN is preferred, but with the BSN offset fraction to ensure that the BSN grid can
+ be started at any PPS.
*******************************************************************************
* Relation between Station BSN and timestamp in fractional seconds
*******************************************************************************
-The T_sub and the fixed BSN grid provide sufficient timing accuracy to timestamp the data and any
-M&C upon the data, because:
+The T_sub and the fixed BSN grid given by ToD_BSN provide sufficient timing resolution to timestamp the data
+and any M&C upon the data, because:
- It is not necessary to facilitate using an offset 0 < T_sub_o < T_sub to start the BSN grid at an integer
number of T_adc after t_epoch, because the BSN grid is sufficiently fine.
- It is not necessary to represent fine group delays of digital filters or analogue electronics and
- cables in the BSN, because these are all fixed after calibration. Course group delays and cable delay
- differences can be compensated for in steps to T_adc via the signal input buffer of every ADC input in
- SDP. Fine group delay differences within a Station can be calibrated via the subband calibration weights
- and group delay differences between Stations need to be calibrated at CEP.
+ cables in the BSN, because these delays are all accounted for after calibration.
+ . Course group delays and cable delay differences can be compensated for in steps to T_adc via the signal
+ input buffer of every ADC input in SDP.
+ . Fine group delay differences within a Station can be calibrated via the subband calibration weights.
+- Group delay differences between Stations need to be calibrated at CEP, and can be compensated at CEP or
+ at Station via the input buffers and the subband calibration weights.
- For the PFB it does not matter at which Station BSN period it was started. What matters is that the PFB
for all signal inputs (within a Station and between Stations) is started at the same fixed Station BSN
grid, because this ensures that the output from all signal inputs will not get relative phase offsets.
-The resolution T_sub of the Station BSN in LOFAR2 depends on the ADC sample frequency and on the
+The actual resolution T_sub of the Station BSN in LOFAR2 depends on the ADC sample frequency and on the
subband filterbank:
N_blk T_adc T_sub T_sub_i
@@ -82,15 +104,15 @@ subband filterbank:
In LOFAR2 the timestamp should be independent of:
-- using 200 MHz sample rate or 160 MHz sample rate
-- using critically sampled filterbank or oversampled filterbank
+- using 200 MHz sample rate or 160 MHz sample rate,
+- using critically sampled subband filterbank or oversampled subband filterbank
If T_sub was fixed then T_sub could be used as timestamp resolution (like in APERTIF). However T_sub depends
-on the subband filterbank with a resolution of T_adc. If T_adc was fixed then T_adc could be used as timestamp
-resolution. However T_adc depends on the sample clock rate. Therefore the timestamp resolution needs to be
-as fine as the greatest common time resolution of T_adc = 5 ns and T_adc = 6.4 ns, which is 0.2 ns.
+on the type of subband filterbank with a resolution of T_adc. If T_adc was fixed then T_adc could be used
+as timestamp resolution. However T_adc depends on the sample clock rate. Therefore the timestamp resolution
+needs to be as fine as the greatest common time resolution of T_adc = 5 ns and T_adc = 6.4 ns, which is 0.2 ns.
A 64 bit timestamp with 0.2 ns resolution can count 2**64 / (365.25 * 24 * 3600 / 0.2e-9) = 116 years. Hence
-for t_epoch = 1970 this is until 2086, which is sufficient for the lifetime of LOFAR2. Internally in SDP
+for t_epoch = 1970 this is until 2086, which is sufficient for the lifetime of LOFAR2.0. Internally in SDP
firmware use the BSN to count T_sub. Externally at the SDP interface use timestamp values with a resolution
of 0.2 ns such that they are:
@@ -120,149 +142,236 @@ combinations of f_adc = 200M or 160M and R_os = 1 or 32/25, that can be represen
-* SDO
-RSP uses rad_bsn to caluculate continuous BSN from seconds timestamp, odd and even second and local BSN. The
-PPS timestamp is captured by the stream sync (so the sync must not have gone lost). The local BSN is derived
-counting eop and restarting at the sync (so no by an input fsn, an therefore lost packets will cause a
-wrong local BSN). The BSN on UniBoard is thus a continous BSN that counts since some PPS in the past,
-defined by the seconds timestamp. The RSP clock frequency is represented by a bit (0 = 160 MHz, 1 = 200 MHz).
+*******************************************************************************
+* Data timing using BSN and sync
+*******************************************************************************
+
+Together the initial BSN, counting blocks and the order of the data within a block uniquely define the timing
+of the data in a Station. However counting blocks is not sufficient to maintain the data timing, because:
+
+- The data flow at the SDP input may be stopped and restarted,
+- at the external interfaces of the FPGAs in SDP it is possible that blocks of data get lost.
+
+The assumptions are:
+
+- data is transported and processed in blocks,
+- partial blocks cannot occur.
+- the data flow can only stop or continue at block boundaries.
-* Continuous Station BSN
+To recover from gaps in the data flow the BSN can be transported along with every data block.
+For the external FPGA interfaces one or more data blocks get packed into the payload and the BSN is then
+transported via the header. The BSN in the header corresponds to the first data block in the payload, the
+position of a data block in the payload defines the offset to this BSN.
+
+For data transport within the FPGA it is costly from a resource point of view to tranport the 64 bit BSN
+along with every data block throughout each data move and data processing function.
+Instead of the full 64 bit BSN a single bit sync signal can be passed on along with the data inside the FPGA.
+At the BSN source the sync is linked to the BSN. Each
+FPGA in the SDP firmware has a BSN source, that all run synchronously within a Station and also between
+Stations, because they have been started by the external PPS from the Timing Distributor (TD). The sync is a
+periodic signal with period larger than the maximum latency of the data within the SDP, to ensure
+that a sync at any FPGA refers to the same time instant. If somewhere in an FPGA the sync comes along, then
+the BSN can be recreated by directly using the BSN that was held at the sync by the BSN source and start
+continue counting blocks from there.
+
+
+*******************************************************************************
+* Data loss and filler data
+*******************************************************************************
+
+The assumptions are:
+
+- Within an FPGA there occur no logic errors (provided that the code synthesis went ok)
+- Within an FPGA no blocks get lost (so e.g. FIFO overflow must not occur and would be as bad as an logic error)
+- At external FPGA interfaces data can get lost
+- Packets that are received at external FPGA interfaces with a CRC error are discarded
+
+
+For the blocks between sync pulses the Station BSN is incremented with every block. This implies that if the
+BSN needs to be preserved during the sync interval, then lost or discarded blocks must be replaced by filler
+blocks. Whether only the BSN at the data sync is relevant, or whether also the BSN of subsequent data blocks is
+needed depends on the function. For the statistics (AST, SST, BST, XST) the BSN at the data sync is sufficient to
+mark the timing of integration results. For these integration results the number of data blocks within the
+integration interval is relevant to know how many blocks contributed (and thus also how many blocks were lost).
+However for the integration result it is not relevant which blocks got lost, because the statistics do not have
+to keep accurated time centroid information. It is sufficient to use the BSN at data sync to timestamp the
+integration results, as if all blocks contributed. As another example, for the beamformer it is important to
+be able to recreate the BSN at the data sync and all subsequent data blocks, because the beamformer must weight
+and sum the input beamlets that coincide in time. Similar for the beamformer output to CEP and for the subband
+offload to AARTFAAC it is important to accurately timestamp every data output block. For the output via the
+Network to CEP and to AARTFAAC it is also benefical to replace lost data blocks by filler packets such
+that the Station output remains at the nominal rate. In this way the destination can distinguish between data
+blocks that got lost inside Station and packet loss on the Network.
+
+The assumptions are:
+
+- Data blocks from packets that are lost or discarded may or may not be replaced by a filler blocks dependend
+ on the downstream function.
+- Within an FPGA only approved packets are processed (so either correct data blocks or filler blocks).
+- The filler data is flagged and typically the filler data is set to zero.
+
+Note:
+- Internally in the FPGA no data is lost, so then a valid and eop are sufficient to mark all blocks. The first
+ valid and the valids after each eop then identify the start of block (sop). Counting eop (or sop) and adding
+ the initial BSN yield the Station BSN. However it is convenient to do transport a sop and a sync along with
+ the data, because:
+ . using a sop avoids having to derive the sop information from the first valid after an eop
+ . using a sync avoids having to derive the sync information from counting valids or sops
+
+
+*******************************************************************************
+* Creation of data blocks and filler data
+*******************************************************************************
+
+There are two places in the FPGA where data blocks can be created:
+
+- BSN source
+- Input streams BSN aligner
+
+The BSN source creates the original data blocks using the wall clock time. The BSN aligner can only create
+filler blocks in an already active stream of data blocks. Within SDP each FPGA has a BSN source that times
+the local data blocks. In the beamformer and the subband correlator data blocks from all
+signal inputs are combined. Therefore local data from this FPGA and remote data from other FPGAs needs to
+be aligned at the BSN by an input streams BSN aligner. The task of the BSN aligner is to time align
+external or external and internal inputs streams based on their BSN. If at least one input is
+active, then output data can be created by inserting filler data for any inactive inputs. If all inputs of
+a BSN aligner are inactive, then no output can created. Within SDP the local input is always active,
+unless the BSN source is stopped via M&C. The remote input(s) can be inactive due to M&C or due to packet
+loss. If the output has stopped due to that the local input went inactive, then the output can resume as
+soon the local input becomes active again. It is not necessary to wait for a sync to resume the output.
-In APERTIF the BSN is a continuous BSN that can start at any PPS, but without support for a BSN fractional
-offset.
*******************************************************************************
* Internal sync interval
*******************************************************************************
-The BSN source can provide a periodic sync pulse that is passed along with the data, throughout all intermediate
-transport and processing functions within the FPGA. The purpose of the sync pulse can be:
-
-- To pass along the Station BSN from the central BSN source in the FPGA directly to where it is needed in
- the FPGA, without having to pass along the 64 bit Station BSN value itself. For the blocks between sync
- pulses Station BSN is incremented with every block. This implies that during the sync interval lost blocks
- must be replaced by filler blocks to preserve the BSN derivation. If lost blocks cannnot be replaced, then
- one lost block causes the remaining blocks in the sync interval to be discarded too.
-- To provide a timing grid at which downstream functions can start or resynchronise in case of lost data.
-- To provide a fixed update interval for various purposes. In LOFAR1 and APERTIF the sync period is used as
- fixed update interval for periodic monitoring, periodic control (the beamformer weights) and periodic
- integration intervals (AST, SST, BST and XST). The advantage of a fixed update interval is that it is well
- defined and does not need control, the disadvantage can be that it is inflexible.
-- To avoid having to derive the sync interval from the BSN where needed in the FPGA. This would require the
- BSN to be passed along in the FPGA. Deriving the sync interval from the BSN is awkward if the sync interval
- is not a power of 2 number of blocks.
-
-Assumptions:
-- Within an FPGA there occur no logic errors
-- Within an FPGA no packets get lost
-- Within an FPGA only correct packets are processed
-- Packets that are received with a CRC error are discarded
-- Packets that are lost or discarded packets remain discarded or are replaced by a filler packet (TBD)
- . If at least one input of a BSN aligner is still active, then the other inputs can be created if they
- are not active.
- . If all inputs of a BSN aligner are inactive, then no output should be created. This can occur with
- the ring in LOFAR or mesh in APERTIF BF if the BSN source has not been started yet. In APERTIF XC
- this can occur if all dishes have no output.
- . If the output has stopped due to that all inputs went inactive, then the output should only resume
- at a sync (TBD).
- . The BSN source can be regarded as a BSN aligner with one input and a local reference.
- . If lost or discarded data is not replaced, then the packets need to keep their BSN number, because
- the BSN can then not be derived by locally counting packets within a sync interval. This not a
- problem for intergration, but it is for CEP data output.
- . If the received sync does not coincide with the local reference sync then something unexpected has
- happened.
- . If no data is lost then a valid and eop are sufficient to mark all blocks. The first valid and the
- valids after each eop then identify the start of block (sop). Counting eop (or sop) and adding the
- initial BSN yield the Station BSN.
-
-The period of the sync interval can be:
-- Must be longer than the input - output latency within SDP, to ensure that a sync at any FPGA refers to the
- same time instant. On the ring with N = 16 FPGAs and store and forward at every FPGA the latency will be
- in the order of 2*N = 32 blocks, so about 32 * T_sub = 32 * 5.12 us = 0.16 ms
-- A period of 1 s [4.5.2.1]:
- . fits the way in which humans count time
- . aligns with the PPS grid
- . is short enough to meet required periodic M&C rates
- . is long enough for the SCU software to perform time critical M&C in time
- . suits as intergration period for the periodic statistics like ADC power, SST, BST and XST
- . suits as update period for the BF weights
-- In LOFAR1 the sync interval was chosen to be aligned to the external PPS, so a period of 1 s. This resulted
- in having 195313 T_sub for even PPS sync intervals and 195312 T_sub for odd PPS sync intervals.
-- In APERTIF the sync interval was chosen to be an integer number of T_sub, which resulted in 800000 T_sub
- or a period of 1.024 s
-
-
-In LOFAR1 the Station BSN is divided into a 32b seconds sequence number (SSN) that counts PPS intervals and a
-32b local BSN that counts blocks within a sync interval. The MSbit is '1' for to identify the SSN when
-the local BSN is 0. The MSbit is '0' to identify the local BSN. A 31 bit SSN can cover 2**31 *1.024
-/ 24/3600/365.25 = 69 years. Since t_epoch = 1970 this extents to 2039, which could still be in the life time of
-LOFAR, although the requirement is 10 years.
-
-For LOFAR the BSN counts T_sub = 5.12 us periods. The sync interval is 1 second, so there are 195313 T_sub
-periods in even sync intervals and 195312 T_sub periods in odd sync intervals.
-The advantage of transporting 32b BSN is that the BSN is then typically small enough to fit in a data word. The
-data word may be the sosi.data or the concatenation of sosi.im and sosi.re.
-
-Distinguish between BSN within SDP and timestamp external to SDP. Internally in SDP the local BSN is sufficient,
-because the latency within SDP is only tens of T_sub. The local BSN = 0 marks the start of a sync interval,
-which is aligned to the PPS UTC seconds grid. At an external interface the IO component needs to maintain a
-SSN that is initialized via M&C. This SSN belongs to the sync interval that starts when the local
-BSN = 0 arrives. In this way the SSN does not have to be transported within SDP, however it does have
-to be initialized at each IO node that uses it. If the BSN only transports the local BSN then the SSN
-can be defined as 32b unsigned to cover 2**32 *1.024 / 24/3600/365.25 = 139 years. Since 1970 this extents to
-2109, to well beyond 2039.
-
-The SSN does not have to be transported along with the internal data stream. Instead like on RSP the
-SSN can be maintained centrally on each FPGA. The SSN is initalized via M&C. On RSP the M&C
-has to write the timestamp at each PPS interval. For SDP choose to let M&C only initialize the timestamp and
-have the firmware increment the timestamp = SSN at the PPS. The sosi.sync in a data stream captures
-the SSN, like on RSP. Thanks to the data path latency the sosi.sync in the data stream has a
-latency > 0, but <<< 1 s with the PPS. The central SSN can be 32b to cover 139 years until 2109.
-
-What if the sosi.sync is lost? A lost sync is detected by the local BSN being smaller than the previous
-local BSN. This can also be used to capture the SSN, except for the end of the sosi.sync interval
-that covers the latency between PPS and sosi.sync. Alternatively, like in RSP, except that a sosi.sync may
-get lost and cause a timestamp to be wrong for that sync interval. This corrupts the entire sync interval,
-but the chance of a missed sosi.sync is small.
-
-UTC time = SSN * 1 s + local BSN * T_sub + mod(SSN * 1 s, T_sub)
-
-The SDP firmware can derive UTC time from SSN and local BSN. To avoid having to calculate mod() the
-M&C should with the initial BSN also provid mod(initial BSN * 1 s, T_sub) and the increment of
-mod(1 s, T_sub). The SSN and corresponding UTC time in T_sub units can be monitored via M&C.
-
-UTC time = init BSN * 1 s + mod(init BSN * 1 s, T_sub) +
- p * 1 s + mod( p * 1 s, T_sub) +
- local BSN * T_sub
-
-Use central SSN and transport local BSN. However at sync we could still transport the 31 LSbits of
-the SSN instead of the known 0 of the local BSN. This could be used for monitoring purposes and as
-a useful index in simulation.
-
-The CDO in LOFAR 1.0 sends a 32 bit SSN (called timestamp) to CEP and a local BSN. The local
-BSN is based on the 16 bit FSN for bits [15:0] and every time the FSN = -1 the local BSN is incremented
-by 2**16. This scheme allowes using an FSN of only 16 bit, but relies on that the block with FSN = -1 =
-2**16-1 did not get lost. Alternatively the check on FSN = -1 could be replaced by FSN < prev FSN. The
-FSN and local BSN both restart at the sync, so they also both rely on that the block with the sync did not
-get lost.
-
-Can we decouple the FSN used for frame alignment from the BSN used as UTC timestamp? Important aspects:
-- UTC since 1 Jan 1970
- . 2**64 / (365.25 * 24 * 3600 / 5 ns) = 2922 years
-- Start or restart processing at a PPS
-- UTC/PPS provide the common reference to SC and SDP
-- Timestamp unit:
- . sample period T_adc = 5 ns --> 200000000
- . subband period T_sub = 1024 * T_adc = 5.12 us --> 195312.5 = 195312 + 1/2
- . oversampled subband period T_osub = T_sub / R_os = 5.12 / (32/27) = 4.32 us --> 231481.481 = 231481 + 13/27
- . channel period T_chan = 128 * T_sub = 655.36 us --> 1525.87890625 = 1525 + 225/256
- . oversampled channel period T_ochan = 128 * T_osub = 552.96 us --> 1808.449074074074 = 1808 + 97/216
- . integration period, PPS period
- In general the periods do not integer fit with the 1 s PPS grid.
-- Subband block phase
-- Signal input samples buffer
+In LOFAR1 the sync interval was chosen to be aligned to the external PPS, so a period of 1 s. This resulted
+in having 195313 T_sub for even PPS sync intervals and 195312 T_sub for odd PPS sync intervals. In LOFAR1
+the notion of odd and even PPS arises due to that the BSN grid can start at any PPS. This notion is akward,
+because it means that Stations should start at an even sync interval when the PPS grid and BSN grid coincide
+to ensure that all stations remain aligned. Starting only at even PPS ensures that LOFAR1 uses a BSN grid
+that is fixed to t_epoch = 1970. For an oversampled subband filterbank the PPS grid and BSN grid
+coincide every q-th PPS, where R_os = p/q, so then a Station should only start every q-th PPS.
+In APERTIF the sync interval was chosen to be an integer number of fine channel periods T_chan =
+N_Chan * T_sub, which resulted in 12500 T_chan and 800000 T_sub or a period of 1.024 s. This 1.024 s is used
+as unit integration period of the correlator in APERTIF. A sync interval of 1 s would have resulted in
+781250 T_sub and 12207.03125 T_chan. The APERTIF sync interval of 1.024 s is akward too, because it differs
+from 1 s and because this also causes that different dishes should only restart when the PPS grid and sync
+grid coincide, which is once every 125 s, because 128/125 = 1.024.
+LOFAR1 and APERTIF show that in general application periods do not integer fit with the 1 s PPS grid. For
+integration periods the only two options are to either use another integration interval (like 1.024 s in
+APERTIF) or to accept that the number of samples per integration interval can differs by one (like 195313 or
+195312 in LOFAR1).
+Both LOFAR1 and APERTIF cannot start at any PPS without affecting the BSN grid. This needs to be solved for
+LOFAR2.0. Like in LOFAR1, for LOFAR2.0 the PPS grid and BSN grid are fixed to t_epoch = 1970. However,
+instead of waiting until the BSN grid and PPS grid coincide, the BSN source in
+LOFAR2.0 has the capability to adjust the timing by a BSN offset fraction, such that the Station processing
+can start or restart at any PPS, while still preserving the fixed BSN grid that starts at t_epoch = 1970.
+In this way starting a Station or restarting a Station in an already running LOFAR telescope become possible
+at any PPS. Still like in LOFAR1 even sync intervals will have 195313 blocks and odd sync intervals will have
+195312 blocks, but this is inevitable due to that at f_adc = 200 MHz and with N_FFT = 1024 the BSN grid and
+PPS grid only coincide at even PPS.
+
+For the Station the maximum data latency within SDP mainly depends on:
+
+- the number of taps in the subband filterbank,
+- the number of hops and latency per node on the ring.
+
+Therefore the maximum data latency will be less than 100 data blocks, so for T_sub = 5.12 us this is less
+than 1 ms. For LOFAR2.0 choose the same internal sync interval as in LOFAR1, so with a period of 1 s and
+aligned to the PPS grid. The sync interval of 1 s is a suitable choice because it [ADD 4.5.2.1]:
+
+- can serve to pass on the BSN information within the FPGA and within the entire Station
+- is short enough to restart the SDP processing after it had stopped (due to an M&C stop command)
+- fits the way in which humans count time
+- can start at any PPS and thus aligns with the PPS grid
+- is short enough to provide a fixed update interval with sufficient time resolution for periodic M&C
+ . suits as integration period for the periodic statistics like ADC power, SST, BST and XST
+ . suits as update period for monitoring e.g. transport latencies, BSN alignement
+ . suits as update period for controlling the BF weights
+- is long enough to perform time critical periodic M&C in time within the correct sync interval
+
+
+*******************************************************************************
+* Transport of BSN and sync
+*******************************************************************************
+
+Between FPGAs the BSN is transported in the packet header. It is beneficial to also transport the sync
+information in the packet header, to avoid having to derive the sync interval from the received BSN.
+Deriving the sync interval from the BSN is awkward if the sync interval is not a power of 2 number of
+blocks, because then the derivation requires a division. For packets that contain multiple blocks the sync
+will only be transported if it applies to the first block in the payload. A separate header field could
+identify the index of the block within the payload to which the sync applies. However, within SDP the
+packets contain only one data block per payload, so the index field is always 0 and can be omitted.
+
+The Station BSN need 50 bits to count 116 years. In the packet header use 64 bit to transport the Station
+BSN and use the MSbit to transport the sync. The sync in the received packets is used for monitoring
+purposes to:
+
+- Measure the transport latency of the received packet relative to the PPS
+- Verify that for remote input streams the received BSN at the sync agrees with the BSN at the sync of
+ the local BSN source. If the received sync does not coincide with the local reference sync then
+ something unexpected has happened. The action for the input sync monitor is then to make this remote
+ input inactive and report this via its M&C monitoring point.
+
+If for an remote input the packet with a sync got lost, then this should be reported via a received sync
+timeout M&C monitoring point. The M&C monitoring points that rely on receiving the packet with the sync
+should then flag that the reported value is invalid, e.g. by reporting -1 = 0xFFFFFFFF.
+
+To summarize, transporting the sync pulse instead of the BSN saves logic and FIFO resources in the FPGA
+and avoids having to derive the sync interval from the BSN. Therefore:
+
+- Internally in the FPGA transport a sync along with the data for each data block.
+- Externally at the FPGA interface transport the sync and BSN in the packet header of each data packet,
+
+The BSN in LOFAR2.0 is a continuous counter with reference to t_ecpoch = 1970, so the BSN in LOFAR2.0
+does not restart at 0 at every like in LOFAR1 and in APERTIF.
+
+
+
+*******************************************************************************
+* Input streams BSN aligner
+*******************************************************************************
+
+In Station SDP the BSN serves two purposes;
+
+- the entire BSN provides wall clock time on the BSN grid and is thus linked to UTC,
+- the difference in BSN is used to time align input streams
+
+In Station SDP each FPGA has a local BSN source and a local stream that carries the data from its local ADC
+signal inputs. The BSN aligner needs to align the local stream with the remote streams that are received
+from the other FPGA via the ring. The maximum BSN latency on the ring depends on the number of FPGAs in the
+ring. Suppose each FPGA introduces a latency of at least one packet, because it applies store and forward on
+packets, and less than two packets. Furthermroe assume that on the ring each packet contains one data block.
+The maximum number of hops between the first FPGA on the ring and the final FPGA is N_FPGA-1. For the LBA
+ring N_FPGA = 16. Hence the maximum BSN latency that can occur within
+SDP is < (N_FPGA-1) * 2 < 32 block periods. Hence the maximum BSN difference between the local input and a
+remote input of the BSN aligner is < 32. Therefore to align the input streams the BSN aligner only has to
+compare the log2(32) = 5 LSbits of the BSN of all input streams. This implies that for the BSN aligner it
+would be sufficient to only transport these bits in the packet header, however it is convenient and not too
+costly to transport the entire 50 bit Station BSN and the sync bit in the packet header.
+
+Note:
+. In LOFAR1 the CDO sends a 32 bit SSN (called timestamp) to CEP and a local BSN. The local BSN is based
+ on a 16 bit frame sequence number (FSN) for bits [15:0] and every time the FSN = -1 the local BSN is
+ incremented by 2**16. This scheme allowes using an FSN of only 16 bit, but relies on that the block with
+ FSN = -1 = 2**16-1 did not get lost. Alternatively the check on FSN = -1 could be replaced by FSN <
+ previous FSN. The FSN and local BSN both restart at the sync, so they also both rely on that the block
+ with the sync did not get lost.
+- The local BSN per sync interval in LOFAR1 wraps every sync interval and moreover it wraps at two different
+ values 195313 or 195312. Therefore the lcoal BSN is less suitable to be used in a BSN aligner.
+- In LOFAR1 the SDO uses rad_bsn to caluculate continuous BSN from seconds timestamp, odd and even second and
+ local BSN. The PPS timestamp is captured by the stream sync (so the sync must not have gone lost). The
+ local BSN is derived counting eop and restarting at the sync (so no by an input fsn, an therefore lost
+ packets will cause a wrong local BSN). The BSN on UniBoard is thus a continous BSN that counts since some
+ PPS in the past, defined by the seconds timestamp. The RSP clock frequency is represented by a bit (0 =
+ 160 MHz, 1 = 200 MHz).
+
+
+
+
+
Design decisions:
@@ -270,8 +379,7 @@ Design decisions:
- Use a pulse from the PPS to align ADCs via JESD204D
- Use a pulse from the PPS to start the internal sync interval
- Use a pulse from the PPS to start the subband processing, so start a 1 s UTC grid
-- Allow starting the subband processing at any PPS, the subband phase offset can be controlled using the
- signal input samples buffer.
+- Allow starting the subband processing at any PPS, by supporting a BSN offset fraction in the BSN source.
- Use 1 s period of 200M (or 160M) cycles for the internal sync interval, because this fits the integration
time for the statistics and the update rate of the beamlet weights
- The number of samples per sync interval of 200M or 160M must be programmable via M&C. The default is 200M,
@@ -283,3 +391,4 @@ Design decisions:
- Within SDP transport the sync and the local BSN. The sync is transported via the MSbit of the local BSN.
At the sync transport the 31 bit SSN instead of local BSN 0, but only for monitoring purposes.
- Derive 64 bit UTC timestamp in units of T_sub in SDP firmware and use this for data output to CEP
+
diff --git a/applications/lofar2/doc/prestudy/station2_to_do_erko.txt b/applications/lofar2/doc/prestudy/station2_to_do_erko.txt
old mode 100644
new mode 100755
index 29738c64a1565d653734b6098da86b3cb33bba90..96c255608455ed015caa5f8384f4b859802a36f8
--- a/applications/lofar2/doc/prestudy/station2_to_do_erko.txt
+++ b/applications/lofar2/doc/prestudy/station2_to_do_erko.txt
@@ -1,413 +1,534 @@
-*******************************************************************************
-* SKA experience:
-*******************************************************************************
-- Experiences -> believes -> choices -> actions -> results
-- Model noise as chorus of 100 sine waves
-- Phase closure with 3 stations: angle(ab*) + angle(bc*) + angle(ca*) = 0 degrees
-- Amplitude closure with 4 stations: |ab*| * |cd*| / (|ac*| * |bd*|) = 1
-- Correlator efficiency
-- RFI flag using most negative value (in LFAA), but this value must be replaced in subsequent processing
-- signal capture, power statistics, power max hold
-- signal generator sinus, sinus in noise, noise, test sequence, ...
-- Jones matrix
- . orthogonalisation due orientation with rotation of earth and frequency dependence due to changes in ionosfeer?
- . beamforming due to geometrical delay with rotation of earth
- . analogue complex gain calibration (temperature dependent)
- . beam shape tapering? (fixed)
-
-*******************************************************************************
-* LaTeX
-*******************************************************************************
-- \sigma \sqrt{}
-- 4.15 \cdot 10^{15}
-- M =
- \left[ {begin{array}{cc}
- 1 & 2 & 3 & 4\\
- 5 & 6 & 7 & 8\\
- \end{array} } \right]
-
-
-*******************************************************************************
-* Run RadioHDL with SVN
-*******************************************************************************
-echo "Uniboard trunk is selected"
-export SVN=${HOME}/svnroot/UniBoard_FP7
-#Setup RadioHDL environment for UniBoard2 and and new Uniboard1 applications
-. ${SVN}/RadioHDL/trunk/tools/setup_radiohdl.sh
-# Support old UniBoard environment (including Aarfaac and Paasar)
-. ${SVN}/RadioHDL/trunk/tools/setup_unb.sh
-
-
-*******************************************************************************
-* Run RadioHDL with GIT
-*******************************************************************************
-
-> cd ~/git/hdl
-> . ./init_hdl.sh # setup development environment for hdl/
- # hdl/libraries, hdl/boards and hdl/applications are developed simultaneously and therefor in one git hdl/ repository
- # automatically also sources ../radiohdl/init_radiohdl.sh if necessary
-> compile_altera_simlibs unb1 # creates build/unb1/hdl_libraries_ip_stratixiv.txt
- # creates build/quartus/<tool version> simulation models that need to be moved to /home/software/modelsim_altera_libs
-> generate_ip_libs unb1 # creates build/unb1/qmegawiz/
- # creates build/unb1/quartus_sh --> empty dir, why is it there?
-> quartus_config unb1 # creates build/unb1/quartus/<hdllib libraries> for synthesis
- # creates build/unb1/quartus/technology_select_pkg.vhd
-> modelsim_config unb1 # creates build/unb1/modelsim/<hdllib libraries> for simulation
- # creates build/unb1/modelsim/modelsim_project_files.txt for Modelsim commands.do
- # creates build/unb1/modelsim/technology_select_pkg.vhd
-> run_qsys unb1 unb1_minimal_qsys
-
-
-*******************************************************************************
-* GIT workflow
-*******************************************************************************
-
-git add <dir>/<file> # cannot add empty dir, need empty file in it
-git status
-git commit -a <dir> -m "Commit comment"
-
-
-
-
-*******************************************************************************
-* Confluence:
-*******************************************************************************
-- space tools menu links onder om secties the ordenen.
-
-
-
-
-*******************************************************************************
-* To do:
-*******************************************************************************
-- Check that the Expert users (MB, SJW, MN), Maintainers (HM) and Local users are happy with the design decisions
-- H6 M&C loads section
-- H3 Functions mapping
-- H3/4 Timing (1s default, PPS, event message)
- Timing en M&C in ADD --> detailed design sectie doorlezen
- * Use PPS event message from SDP --> SC for:
- 1 Hz --> Set ToD (this requires that PPS is avaiable in SDP independent of data path processing)
- 1 Hz --> Update BF weigths
- 1 Hz --> Monitor AST, SST, BST
- * Use dedicated event message from SDP --> SC for XST readout
- 1 Hz or more --> XST
- All other M&C is not time critical, which means that required rates are average rates.
-- H4/8
- . Subband correlator read out via MM or offload --> let SC do read out via M&C and provide offload to TM
- . One or more subbands per interval
- . Shorter < 1 s intervals and longer > 1 s intervals
-- HBA tile elements individually controlable, trimmable (e.g. in case of new production batch)
-- Discuss GIT, RadioHDL with PD (read radiohdl readme and Ruuds docu to prepare)
-- System enigineering (map, SEMP doc HvdM)
-- System testcases in SEMP doc --> check with AJB, BH --> test plan after PDR (see SEMP doc)
-- S_hba_international = 192
-- nof tx, rx pkt/s, CRC error count, nof pause frames
-- monitor functions publish their data points, other functions can use these for alarms, visualisation, calibration
-- data buffer on output beamlets --> histogram
- . buffer all beamlets per T_sub at sync
- . buffer one beamlet for some T_sub time series after sync
-
-- RCU2S-SDP signal input allocation:
- Decided:
- . LBA and HBA on seperate RCUs and also on seperate subracks and on independent rings, so SDP is independent
- for LBA and HBA.
- . X and Y on the same ring to support dual polarization ACM
- Still open:
- . X and Y together per PN or X and Y on seperate subracks
- - polarization correction via subband weights is not needed, so X and Y can be on different PN
- - EMI between X and Y, but X and Y have only about 40 dB isolation
- - EMI between single pol inputs get suppressed dependent on the station digital beam pointing
- . LBA inner signal inputs and LBA outer signal inputs on different subracks or arbitrary. Inner is not used
- Instead they use sparse odd and sparse even to have two more or less random antenna allocations.
- . HBA core station sub-array inputs on different Uniboard2 to reduce EMI
- . HBA-X power, Y control --> power distribution
- . HBA RCU 3X and 3Y
-
-- EMI:
- . RCU2S-SDP signal input allocation
- . Shielding
- . more cross talk between horizontal RCUs than between vertical RCUs
- . more cross talk between neighbour RCUs
- . AC-DC instead of using AC-DC48-DC
-
-
-*******************************************************************************
-* System engineering:
-*
-*******************************************************************************
-
-- Requirements,
-- Functions have their own hierarchy, independent of requirement levels, and independent of PBS.
- Requirements map to products according to the L0, L1, L2 hierarchy
- Requirements map to functions at any level
-- Products
-- ICDs
-
-Polarion:
-- Maintain hierarchy of functions and link them to products
-- Start with parent function and allocate that to one or more L3 products/ functions in ADD tables,
- after that number the child level functions
-- Start with L3 product an look for potential requirements in L2 SRS contents
- allocate requirement number + keywords
-- Check for unallocated requirements
- Check for unallocated parent functions
- Check for unallocated child functions
-- Number child functions in ADD tables
-- After PDR:
- . Update/add parent functions and child functions in Polarion and in ADD tables
- . Number the child functions in Polarion
-
-* Non compliant requirements
- - 3099 LOFAR 1.0 frequency grid only with critically sampled filterbank
- - 2278 Alias free broadband reconstruction only with oversampled filterbank
-
-* Missing requirements for functions
- - operational: all sky image (maybe 2400)
- - transient buffer: trade inputs for duration
- - transient detection: what algorithm should be used and with what parameters
- - Expert user interface requirements
- - Station Control support for EAA (LORA, NenuFAR)
- - Rack space for NenuFar
-
-* Missing functions for requirements
- - 2404 Control of RFI mitigation processes
- - 2405 RFI process handling
- - 2400
-
-* Unmapped requirements
- -
-
-
-* Mark Ruiter: Missing calibration strategy.
-
- level 1: Analogue input (hba, lba)
- level 2: Beam ( analog)
- level 3: (station)
- level 4: (local ionosphere)
- level 5: (distributed ionosphere lofar 2)
-
- This only mentions digital station calibrating assuming a lot.
- --> Gijs Schoonderbeek: Added this list in chapter 10, where the expected performance is described.
-
-
-Station:
-1) SC-TM: Local mode - Station and TM - Station ICD needed to define SC functionality
- - ICD for SC-TM
-2) SC: SCU for application layers with OPC-UA frame work like EPICS, WinCC
-3) SC-SDP: Separate 'PC / microcontroller platform' to translate OPC-UA - Gemini Protocol of UniBoard2
- - ICD for SC-SDP
- - work starts with supporting the 'hello world' of controlling a UniBoard2
-4) SC-RCU2S : 'Microprocessor / microcontroller platform' to translate OPC-UA - I2C, on/off line, SPI.
- - ICD for SC-PCC
- - PCC connects to RCU2 and UniBoards
-5) SC-STCA
-6) SC-STF
-7) SC-STIN
-
-
-
-*******************************************************************************
-* Terminology
-*******************************************************************************
-
-* Software blocks:
- . server
- . driver
- . handler
-
- . device
-
-* Synonyms for Control:
- . setup
- . configure
- . handle
- . set
- . supply
- . send
- . update
- . boot, run, initialize
- . write
- . store
-
-* Synonyms for Monitor
- . obtain
- . get
- . read
-
-
-
-*******************************************************************************
-* Station Control
-*******************************************************************************
-Remarks:
-- Provide direct TB access to the buffer data is an L4 or L5 function
-- Station beam or station beams? --> Station beams, otherwise call it full band station beam
-- There is no automatic feedback:
- . AST and SST are only for diagnosis and alarms.
- . XST is used in separate function to determine a new set of the subband calibration weigths.
- . The scale beamlets setting is fixed per beamlet mode and nominal signal input levels
-
-+ RCU2-LBA
- * Monitor RCU2-LBA
- . Monitor RCU2 hardware . Sense RCU2 temperatures, voltages and currents
- . Monitor LBA current . Sense LBA current
- * Setup RCU2-LBA
- . Control LBA power . Supply LBA with power
- . Select frequency band . Filter band of interest
- . Select signal level . Set signal level
- . Set ADC mode . Digitize signal
-
-+ RCU2-HBAT
- * Monitor RCU2-HBAT
- . Monitor RCU2 hardware . Sense RCU2 temperatures, voltages and currents
- . Monitor HBA tile current . Sense HBA tile current
- * Setup RCU2-HBAT
- . Control HBA tile power . Send power or control
- . Communicate with HBA tile
- . Select frequency band . Filter band of interest
- . Select signal level . Set signal level
- . Set ADC mode . Digitize signal
-
- * Create HBA tile beam
- . Calculate HBA tile delays
- . Update HBA tile delays
- . Communicate with HBA tile . Convert to HBA tile control protocol
-
-+ SDP
- * Monitor SDP
- . Monitor SDP status * Monitor HW, FW and interface
- * Receive and timestamp ADC input
- . Monitor timing . L4 PPS monitor, BSN monitor (ToD timestamp, alignment)
- . Monitor ADC signal input
- . L4 ADC interface monitor (L4, because only for expert/developer)
- . Take snapshot of input samples
- . Calculate ADC statistics (AST) (mean, power, histogram?)
-
- * Create calibrated subbands
- . Monitor subband statistics . Calculate subband statistics (SST)
- * Form station beams
- . Monitor beamlet statistics . Calculate beamlet statistics (BST)
-
- * Setup SDP
- . Control firmware application * Support and run FW application (upload, store, select, boot, version)
- * Receive and timestamp ADC input
- . Control ADC signal input . Receive ADC samples (L4 including WG and DB)
- . Insert test signal
- . Compensate coarse cable length . Align input samples
- . Set time of day . Time stamp input samples
- * Create calibrated subbands
- . Set subband weights . Equalize subbands
- * Form station beams
- . Select beamlet output bit range . Scale beamlets
- . Set output header . Output beamlets
-
- * Handle subband correlator * Create calibrated subbands
- . Select subband . Correlate subband for all signal inputs (XST)
- . Monitor subband correlations
-
- * Derive subband weights
- . Calculate subband weights from subband correlations
- . Store subband weights
-
- * Create station beams * Form and output station beams
- . Set subband selection . Select subbands
- . Calculate beamlet weights
- . Update beamlet weights . Form beamlets
-
- * Handle transient detection * Detect radio transients
- . Set detection parameters
- . Collect and forward triggers
-
- * Handle transient buffer * Buffer ADC or subband data
- . Control TB data input . Select and record input data
- . Control TB data capture
- . Control TB data read out . Select and read out buffered data
-
- * Handle subband offload * Offload subbands
-
-
-STF
- * Monitor Station TF
- . Monitor status of TD
- . Monitor timing distribution . Distribute timing (10 MHz, PPS)
-
- * Setup Station TF
- . Select sample clock frequency . Generate sample clock (200 MHz or 160 MHz)
- . Receive time of day from L2-TD
- . Set Time of Day timestamp in SDP . Receive and timestamp ADC input
-
-STCA
- * Setup Station Cabinet
- . Setup power
- . Setup cooling
- . Setup network
- * Monitor Station Cabinet
- . Monitor power
- . Monitor cooling
- . Monitor network
-
-STIN
- * Monitor Station Infrastructure
- . Monitor station housing
-
-*******************************************************************************
-* Station ADD
-*******************************************************************************
-
-- H4
- . SDP dynamisch bereik figure versimpelen + formule + simuleer processing gain van FFT voor real input
- . Check timing dither with PK, because this would require detailed timing control for SDP and RCU2
- . In SDP JESD details maybe too detailed design, purpose here is to explain JESD in own words to understand it without
- having to read references.
- . Check that +48 v is mentioned (not -48 V)
- . io_ddr in FN beamformer uses 26 M9K waarvan 10 in de IP. De IP gebruikt ook 1 M144K (~= 16 M9K). De FN beamformer gebruikt 1 DDR3 module.
- --> voor twee DDR modules zou je ongeveer (26 + 10) * 2 = 72 block RAM verwachten, terwijl TBB-base in LOFAR1 maar 6 gebruikt volgens MP
- syntesis report.
- . Timimg fixed op interne sync of op single or periodic timestamp set by SC.
- - let SDP send event for each interne sync, to help SC align its time critical control and monitoring
- - possibly use linear interpolation for BF weights to relieve SC from tigth 1 Hz control and to allow fast tracking of satellites
- - define timestamp as 32b seconds and 32b blocks within second
-
-
-
-- H6
- PK:
- . Write design decision on dithering (meant to make ADC effectively more linear):
- 1) none
- 2) level using additative CW --> suppress RFI harmonics already at ADC by making the ADC sampling more linear by crossing more levels
- 3) time using delays --> does this work anyway, because the digital BF already suppresses RFI from all directions in which it does not point.
- The dithering makes that it looks like the RFI comes from all directions, this does not help, does it?
-
-PDR:
-- Polarization correction at station via subband weights, via BF weights?
-- Station Control SCU (Local mode capabilities, Linux, ..)
-- SC and L2-TD at Station
-- CEP ICD (separate beamlet streams)
-- HBA2 tile control
-- Oversampled filterbank (design, R_os rate)
-- TB for LBAS and HBAS simultaneously
-- TB size
-- S_sub_bf = 488
-- Subband weights update rate > 10 min is sufficient?
-- There are >= 488 independent Station beams per polarization. This also implies that the polarizations can have independent
- Station beams, so different frequencies and pointings. In LOFAR 1.0 a subband and a beamlet are defined as a X,Y tuple.
- For LOFAR 2.0 the beamlets are still output as tuples but the X and Y are treated independently, so beamlet index i for X
- may use a different subband frequency and pointing than beamlet index i for Y. Therefore in LOFAR 2.0 define a subband and
- a beamlet per single polarization.
-- How to continue SE after PDR:
- . Update ADD with OAR answers and internal remarks from team
- . Mapping of function tree, L2 requirements and L3 products in ADD
- . Put function tree in Polarion and update function names
- . Add station functions from ADD to function tree in Polarion?
- - what is the purpose of the function tree, does it provide a check list?
- - how deep will the function tree go, till the lowest level products?
- . Do we need more SE views then that we already have with states/modes, requirements, functions and products?
- . ICDs
- . L4 requirements?
- . L4 products?
- - Uniboard2 hardware is an L4 product?
- - Uniboard2 firmware is an L4 product?
- . Detailed design of the firmware product
-
-
+*******************************************************************************
+* SKA experience:
+*******************************************************************************
+- Experiences -> believes -> choices -> actions -> results
+- Model noise as chorus of 100 sine waves
+- Phase closure with 3 stations: angle(ab*) + angle(bc*) + angle(ca*) = 0 degrees
+- Amplitude closure with 4 stations: |ab*| * |cd*| / (|ac*| * |bd*|) = 1
+- Correlator efficiency
+- RFI flag using most negative value (in LFAA), but this value must be replaced in subsequent processing
+- signal capture, power statistics, power max hold
+- signal generator sinus, sinus in noise, noise, test sequence, ...
+- Jones matrix
+ . orthogonalisation due orientation with rotation of earth and frequency dependence due to changes in ionosfeer?
+ . beamforming due to geometrical delay with rotation of earth
+ . analogue complex gain calibration (temperature dependent)
+ . beam shape tapering? (fixed)
+
+*******************************************************************************
+* LaTeX
+*******************************************************************************
+- \sigma \sqrt{}
+- 4.15 \cdot 10^{15}
+- M =
+ \left[ {begin{array}{cc}
+ 1 & 2 & 3 & 4\\
+ 5 & 6 & 7 & 8\\
+ \end{array} } \right]
+
+
+*******************************************************************************
+* Run RadioHDL with SVN
+*******************************************************************************
+echo "Uniboard trunk is selected"
+export SVN=${HOME}/svnroot/UniBoard_FP7
+#Setup RadioHDL environment for UniBoard2 and and new Uniboard1 applications
+. ${SVN}/RadioHDL/trunk/tools/setup_radiohdl.sh
+# Support old UniBoard environment (including Aarfaac and Paasar)
+. ${SVN}/RadioHDL/trunk/tools/setup_unb.sh
+
+
+*******************************************************************************
+* Run RadioHDL with GIT
+*******************************************************************************
+
+> cd ~/git/hdl
+> . ./init_hdl.sh # setup development environment for hdl/
+ # hdl/libraries, hdl/boards and hdl/applications are developed simultaneously and therefor in one git hdl/ repository
+ # automatically also sources ../radiohdl/init_radiohdl.sh if necessary
+> compile_altera_simlibs unb1 # creates build/unb1/hdl_libraries_ip_stratixiv.txt
+ # creates build/quartus/<tool version> simulation models that need to be moved to /home/software/modelsim_altera_libs
+> generate_ip_libs unb1 # creates build/unb1/qmegawiz/
+ # creates build/unb1/quartus_sh --> empty dir, why is it there?
+> quartus_config unb1 # creates build/unb1/quartus/<hdllib libraries> for synthesis
+ # creates build/unb1/quartus/technology_select_pkg.vhd
+> modelsim_config unb1 # creates build/unb1/modelsim/<hdllib libraries> for simulation
+ # creates build/unb1/modelsim/modelsim_project_files.txt for Modelsim commands.do
+ # creates build/unb1/modelsim/technology_select_pkg.vhd
+> run_qsys unb1 unb1_minimal_qsys
+
+
+*******************************************************************************
+* GIT workflow
+*******************************************************************************
+
+difftool ?
+mergetool ?
+
+* Pro Git book by Scott Chacon: https://git-scm.com/book/en/v2
+* YouTube : David Mahler part 1,2,3
+
+Part 1:
+
+# After GIT install
+git version
+git config --global http://user.name "EricKooistra"
+git config --global http://user.email "erkooi@gmail.com"
+git config --list
+touch .gitignore # create .gitignore if it does not already exist
+.gitignore # file with working tree dirs and files to ignore, must also be commited
+
+# To start a repo
+git init # start new repo at this dir, creates .git/
+git clone # get and start with existing repo
+
+git status # what is in stage area and what is modified
+
+Three areas:
+* working tree # local directory tree
+| git add
+v
+* staging area (index)
+|
+v git commit
+* history # .git repository with entire commit graph
+
+# To use a repo
+git add <dir>/<file> # add to stage area, set for commit. Cannot add empty dir, need empty file in it
+git add . # add all new and modified to stageing area
+git diff # diff between file in working tree and staging area
+git diff --staged # diff between file in staging area and history
+git rm <filename> # remove file from working tree and stage the delete
+git checkout -- <filename> # revert a working tree change
+git reset # clear stage area
+git reset HEAD <filename> # revert staged change
+git log -- <filename> # show history of file
+git checkout <version hash> -- s2 # retrieve file from history into staged area and working tree
+
+
+Part 2:
+
+git commit -m "" # commit what is in stage area
+git commit -a -m "" # add to stage area and commit what is in stage area
+alias graph="git log --all --decorate --oneline --graph"
+git branch <branch name> # creat branch
+git branch # show branches
+git checkout <branch name> # change working tree and stage area to branch
+git checkout master
+git merge <branch name> # Fast forward merge of branch name to master if there is a direct path
+ # by moving master to branch name, this is when there have been no updates
+ # on the master branch since the branch was created.
+ # Three way merge combine the differences of the branch and the master
+ # compared to their common version, this can lead to merge conflicts if
+ # changes on both branches occur at same parts of a file.
+git branch --merged # show branches that ghave been merged to master
+git branch -d <branch name> # remove branch
+git checkout <commit hash> # detached HEAD because it points to a version not a branch
+git branch <branch name> # start a branch from the commit hash, HEAD is attached again
+
+# Stash area to store working tree
+git stash save "comment" # store working tree and stage area to get a clean
+git stash list # show all stashes
+git stash apply <label> # restore stash
+git stash apply # restore last stash
+
+
+Part 3: Remote repositories (Github, Gitlab, Bitbucket, ...)
+
+create repo on Github
+http://README.md # md = mark down
+git clone <url:.../<repo name>.git> # get copy from url
+cd <repo name>
+git config --local http://user.name "EricKooistra"
+git config --local http://user.email "erkooi@gmail.com"
+git remote # origin
+git remote -v # full url
+
+# To align with remote repo
+# update from remote
+git status # shows also origin/master, but not live
+git fetch origin
+git status # shows also origin/master, now with latest remote
+git merge origin/master
+git pull # get latest from remote repo, combines fetch and merge
+
+# upload to remote
+git push
+git push origin master # put local repo to remote repo
+
+# On Github fork is a copy of the a repo in Github to get a repo on your account
+git clone <url of fork> # get copy of fork repo, will be origin
+git remote add upstream <url of original repo on Github> # will be upstream
+git fetch upstream
+git status
+# commit local change on branch
+# git push origin <branch name> # push to my fork repo on Github
+# pull request on Github
+# delete branch and fetch npstream if the pull request was accepted
+git remote remove <remote name> # remove a remote repo
+
+
+
+*******************************************************************************
+* Confluence:
+*******************************************************************************
+- space tools menu links onder om secties the ordenen.
+
+
+*******************************************************************************
+* RadioHDL
+*******************************************************************************
+Open issues:
+- Central HDL_IO_FILE_SIM_DIR = build/sim --> Project local sim dir
+- avs_eth_coe.vhd per tool version? Because copying avs_eth_coe_<buildset>_hw.tcl to $HDL_BUILD_DIR copies the
+ last <buildset>, using more than one buildset at a time gices conflicts.
+
+
+
+*******************************************************************************
+* To do:
+*******************************************************************************
+- Check that the Expert users (MB, SJW, MN), Maintainers (HM) and Local users are happy with the design decisions
+- H6 M&C loads section
+- H3 Functions mapping
+- H3/4 Timing (1s default, PPS, event message)
+ Timing en M&C in ADD --> detailed design sectie doorlezen
+ * Use PPS event message from SDP --> SC for:
+ 1 Hz --> Set ToD (this requires that PPS is avaiable in SDP independent of data path processing)
+ 1 Hz --> Update BF weigths
+ 1 Hz --> Monitor AST, SST, BST
+ * Use dedicated event message from SDP --> SC for XST readout
+ 1 Hz or more --> XST
+ All other M&C is not time critical, which means that required rates are average rates.
+- H4/8
+ . Subband correlator read out via MM or offload --> let SC do read out via M&C and provide offload to TM
+ . One or more subbands per interval
+ . Shorter < 1 s intervals and longer > 1 s intervals
+- HBA tile elements individually controlable, trimmable (e.g. in case of new production batch)
+- Discuss GIT, RadioHDL with PD (read radiohdl readme and Ruuds docu to prepare)
+- System enigineering (map, SEMP doc HvdM)
+- System testcases in SEMP doc --> check with AJB, BH --> test plan after PDR (see SEMP doc)
+- S_hba_international = 192
+- nof tx, rx pkt/s, CRC error count, nof pause frames
+- monitor functions publish their data points, other functions can use these for alarms, visualisation, calibration
+- data buffer on output beamlets --> histogram
+ . buffer all beamlets per T_sub at sync
+ . buffer one beamlet for some T_sub time series after sync
+
+- Use GIT
+- Understand AXI4 streaming (versus avalon, RL =0)
+- Use ARGS to define peripherals
+ . check docs and article
+- Learn how gmi_minimal HDL code works to prepare for porting to unb2b_minimal_gmi
+- Update RadioHDL docs
+- Write RadioHDL article
+- Write HDL RL=0 article - desp_hdl_design_article.txt
+
+
+
+
+
+- RCU2S-SDP signal input allocation:
+ Decided:
+ . LBA and HBA on seperate RCUs and also on seperate subracks and on independent rings, so SDP is independent
+ for LBA and HBA.
+ . X and Y on the same ring to support dual polarization ACM
+ Still open:
+ . X and Y together per PN or X and Y on seperate subracks
+ - polarization correction via subband weights is not needed, so X and Y can be on different PN
+ - EMI between X and Y, but X and Y have only about 40 dB isolation
+ - EMI between single pol inputs get suppressed dependent on the station digital beam pointing
+ . LBA inner signal inputs and LBA outer signal inputs on different subracks or arbitrary. Inner is not used
+ Instead they use sparse odd and sparse even to have two more or less random antenna allocations.
+ . HBA core station sub-array inputs on different Uniboard2 to reduce EMI
+ . HBA-X power, Y control --> power distribution
+ . HBA RCU 3X and 3Y
+
+- EMI:
+ . RCU2S-SDP signal input allocation
+ . Shielding
+ . more cross talk between horizontal RCUs than between vertical RCUs
+ . more cross talk between neighbour RCUs
+ . AC-DC instead of using AC-DC48-DC
+
+
+*******************************************************************************
+* System engineering:
+*
+*******************************************************************************
+
+- Requirements,
+- Functions have their own hierarchy, independent of requirement levels, and independent of PBS.
+ Requirements map to products according to the L0, L1, L2 hierarchy
+ Requirements map to functions at any level
+- Products
+- ICDs
+
+Polarion:
+- Maintain hierarchy of functions and link them to products
+- Start with parent function and allocate that to one or more L3 products/ functions in ADD tables,
+ after that number the child level functions
+- Start with L3 product an look for potential requirements in L2 SRS contents
+ allocate requirement number + keywords
+- Check for unallocated requirements
+ Check for unallocated parent functions
+ Check for unallocated child functions
+- Number child functions in ADD tables
+- After PDR:
+ . Update/add parent functions and child functions in Polarion and in ADD tables
+ . Number the child functions in Polarion
+
+* Non compliant requirements
+ - 3099 LOFAR 1.0 frequency grid only with critically sampled filterbank
+ - 2278 Alias free broadband reconstruction only with oversampled filterbank
+
+* Missing requirements for functions
+ - operational: all sky image (maybe 2400)
+ - transient buffer: trade inputs for duration
+ - transient detection: what algorithm should be used and with what parameters
+ - Expert user interface requirements
+ - Station Control support for EAA (LORA, NenuFAR)
+ - Rack space for NenuFar
+
+* Missing functions for requirements
+ - 2404 Control of RFI mitigation processes
+ - 2405 RFI process handling
+ - 2400
+
+* Unmapped requirements
+ -
+
+
+* Mark Ruiter: Missing calibration strategy.
+
+ level 1: Analogue input (hba, lba)
+ level 2: Beam ( analog)
+ level 3: (station)
+ level 4: (local ionosphere)
+ level 5: (distributed ionosphere lofar 2)
+
+ This only mentions digital station calibrating assuming a lot.
+ --> Gijs Schoonderbeek: Added this list in chapter 10, where the expected performance is described.
+
+
+Station:
+1) SC-TM: Local mode - Station and TM - Station ICD needed to define SC functionality
+ - ICD for SC-TM
+2) SC: SCU for application layers with OPC-UA frame work like EPICS, WinCC
+3) SC-SDP: Separate 'PC / microcontroller platform' to translate OPC-UA - Gemini Protocol of UniBoard2
+ - ICD for SC-SDP
+ - work starts with supporting the 'hello world' of controlling a UniBoard2
+4) SC-RCU2S : 'Microprocessor / microcontroller platform' to translate OPC-UA - I2C, on/off line, SPI.
+ - ICD for SC-PCC
+ - PCC connects to RCU2 and UniBoards
+5) SC-STCA
+6) SC-STF
+7) SC-STIN
+
+
+
+*******************************************************************************
+* Terminology
+*******************************************************************************
+
+* Software blocks:
+ . server
+ . driver
+ . handler
+
+ . device
+
+* Synonyms for Control:
+ . setup
+ . configure
+ . handle
+ . set
+ . supply
+ . send
+ . update
+ . boot, run, initialize
+ . write
+ . store
+
+* Synonyms for Monitor
+ . obtain
+ . get
+ . read
+
+
+
+*******************************************************************************
+* Station Control
+*******************************************************************************
+Remarks:
+- Provide direct TB access to the buffer data is an L4 or L5 function
+- Station beam or station beams? --> Station beams, otherwise call it full band station beam
+- There is no automatic feedback:
+ . AST and SST are only for diagnosis and alarms.
+ . XST is used in separate function to determine a new set of the subband calibration weigths.
+ . The scale beamlets setting is fixed per beamlet mode and nominal signal input levels
+
++ RCU2-LBA
+ * Monitor RCU2-LBA
+ . Monitor RCU2 hardware . Sense RCU2 temperatures, voltages and currents
+ . Monitor LBA current . Sense LBA current
+ * Setup RCU2-LBA
+ . Control LBA power . Supply LBA with power
+ . Select frequency band . Filter band of interest
+ . Select signal level . Set signal level
+ . Set ADC mode . Digitize signal
+
++ RCU2-HBAT
+ * Monitor RCU2-HBAT
+ . Monitor RCU2 hardware . Sense RCU2 temperatures, voltages and currents
+ . Monitor HBA tile current . Sense HBA tile current
+ * Setup RCU2-HBAT
+ . Control HBA tile power . Send power or control
+ . Communicate with HBA tile
+ . Select frequency band . Filter band of interest
+ . Select signal level . Set signal level
+ . Set ADC mode . Digitize signal
+
+ * Create HBA tile beam
+ . Calculate HBA tile delays
+ . Update HBA tile delays
+ . Communicate with HBA tile . Convert to HBA tile control protocol
+
++ SDP
+ * Monitor SDP
+ . Monitor SDP status * Monitor HW, FW and interface
+ * Receive and timestamp ADC input
+ . Monitor timing . L4 PPS monitor, BSN monitor (ToD timestamp, alignment)
+ . Monitor ADC signal input
+ . L4 ADC interface monitor (L4, because only for expert/developer)
+ . Take snapshot of input samples
+ . Calculate ADC statistics (AST) (mean, power, histogram?)
+
+ * Create calibrated subbands
+ . Monitor subband statistics . Calculate subband statistics (SST)
+ * Form station beams
+ . Monitor beamlet statistics . Calculate beamlet statistics (BST)
+
+ * Setup SDP
+ . Control firmware application * Support and run FW application (upload, store, select, boot, version)
+ * Receive and timestamp ADC input
+ . Control ADC signal input . Receive ADC samples (L4 including WG and DB)
+ . Insert test signal
+ . Compensate coarse cable length . Align input samples
+ . Set time of day . Time stamp input samples
+ * Create calibrated subbands
+ . Set subband weights . Equalize subbands
+ * Form station beams
+ . Select beamlet output bit range . Scale beamlets
+ . Set output header . Output beamlets
+
+ * Handle subband correlator * Create calibrated subbands
+ . Select subband . Correlate subband for all signal inputs (XST)
+ . Monitor subband correlations
+
+ * Derive subband weights
+ . Calculate subband weights from subband correlations
+ . Store subband weights
+
+ * Create station beams * Form and output station beams
+ . Set subband selection . Select subbands
+ . Calculate beamlet weights
+ . Update beamlet weights . Form beamlets
+
+ * Handle transient detection * Detect radio transients
+ . Set detection parameters
+ . Collect and forward triggers
+
+ * Handle transient buffer * Buffer ADC or subband data
+ . Control TB data input . Select and record input data
+ . Control TB data capture
+ . Control TB data read out . Select and read out buffered data
+
+ * Handle subband offload * Offload subbands
+
+
+STF
+ * Monitor Station TF
+ . Monitor status of TD
+ . Monitor timing distribution . Distribute timing (10 MHz, PPS)
+
+ * Setup Station TF
+ . Select sample clock frequency . Generate sample clock (200 MHz or 160 MHz)
+ . Receive time of day from L2-TD
+ . Set Time of Day timestamp in SDP . Receive and timestamp ADC input
+
+STCA
+ * Setup Station Cabinet
+ . Setup power
+ . Setup cooling
+ . Setup network
+ * Monitor Station Cabinet
+ . Monitor power
+ . Monitor cooling
+ . Monitor network
+
+STIN
+ * Monitor Station Infrastructure
+ . Monitor station housing
+
+*******************************************************************************
+* Station ADD
+*******************************************************************************
+
+- H4
+ . SDP dynamisch bereik figure versimpelen + formule + simuleer processing gain van FFT voor real input
+ . Check timing dither with PK, because this would require detailed timing control for SDP and RCU2
+ . In SDP JESD details maybe too detailed design, purpose here is to explain JESD in own words to understand it without
+ having to read references.
+ . Check that +48 v is mentioned (not -48 V)
+ . io_ddr in FN beamformer uses 26 M9K waarvan 10 in de IP. De IP gebruikt ook 1 M144K (~= 16 M9K). De FN beamformer gebruikt 1 DDR3 module.
+ --> voor twee DDR modules zou je ongeveer (26 + 10) * 2 = 72 block RAM verwachten, terwijl TBB-base in LOFAR1 maar 6 gebruikt volgens MP
+ syntesis report.
+ . Timimg fixed op interne sync of op single or periodic timestamp set by SC.
+ - let SDP send event for each interne sync, to help SC align its time critical control and monitoring
+ - possibly use linear interpolation for BF weights to relieve SC from tigth 1 Hz control and to allow fast tracking of satellites
+ - define timestamp as 32b seconds and 32b blocks within second
+
+
+
+- H6
+ PK:
+ . Write design decision on dithering (meant to make ADC effectively more linear):
+ 1) none
+ 2) level using additative CW --> suppress RFI harmonics already at ADC by making the ADC sampling more linear by crossing more levels
+ 3) time using delays --> does this work anyway, because the digital BF already suppresses RFI from all directions in which it does not point.
+ The dithering makes that it looks like the RFI comes from all directions, this does not help, does it?
+
+PDR:
+- Polarization correction at station via subband weights, via BF weights?
+- Station Control SCU (Local mode capabilities, Linux, ..)
+- SC and L2-TD at Station
+- CEP ICD (separate beamlet streams)
+- HBA2 tile control
+- Oversampled filterbank (design, R_os rate)
+- TB for LBAS and HBAS simultaneously
+- TB size
+- S_sub_bf = 488
+- Subband weights update rate > 10 min is sufficient?
+- There are >= 488 independent Station beams per polarization. This also implies that the polarizations can have independent
+ Station beams, so different frequencies and pointings. In LOFAR 1.0 a subband and a beamlet are defined as a X,Y tuple.
+ For LOFAR 2.0 the beamlets are still output as tuples but the X and Y are treated independently, so beamlet index i for X
+ may use a different subband frequency and pointing than beamlet index i for Y. Therefore in LOFAR 2.0 define a subband and
+ a beamlet per single polarization.
+- How to continue SE after PDR:
+ . Update ADD with OAR answers and internal remarks from team
+ . Mapping of function tree, L2 requirements and L3 products in ADD
+ . Put function tree in Polarion and update function names
+ . Add station functions from ADD to function tree in Polarion?
+ - what is the purpose of the function tree, does it provide a check list?
+ - how deep will the function tree go, till the lowest level products?
+ . Do we need more SE views then that we already have with states/modes, requirements, functions and products?
+ . ICDs
+ . L4 requirements?
+ . L4 products?
+ - Uniboard2 hardware is an L4 product?
+ - Uniboard2 firmware is an L4 product?
+ . Detailed design of the firmware product
+
+
\ No newline at end of file