Update beamlet header format.

applications/lofar2/doc/prestudy/station2_sdp_icd.txt

@@ -127,7 +127,18 @@ UDP link control
   24 Byte, Flags (0-23, 1 bit per pol/dish)
   
   --> Total 40 Bytes
+  
+# Arts SC1
+  1 Byte, Marker byte
+  1 Byte, Format version
+  2 Byte, Source id
+  2 Byte, Number of channels per block (N channel )
+  2 Byte, Number of blocks per packet (N block )
+  8 Byte, Timestamp
+ 24 Byte, Flags
 
+  --> Total 40 Bytes
+  
 # Arts SC2
 
   1 Byte, Marker (= 120)
@@ -268,7 +279,6 @@ The FPGA has 2713 M20k, so this is 128/2713 ~= 5% of the internal BRAM resources
 The total number of streams to CEP then becomes NOF_BEAMSETS * NOF_LANES.
 
 
-
 - 1 Byte, MARKER
   . Like in APERTIF and ARTS, may be useful to quickly recognize the data packet.
   . Beamlets: 98 = 0x62 = 'b'
@@ -290,57 +300,78 @@ The total number of streams to CEP then becomes NOF_BEAMSETS * NOF_LANES.
 
 - 2 Byte, STATION_ID (idem as LOFAR1)
   ==> or 8b because there are only ~50 stations
-  ==> use 16b to fit number from station name (e.g. CS001, LV614, see list of stations at
-      https://proxy.lofar.eu/array_status/STATIONS/HTML/cs011/index.html)
-
-- 4 Byte, SOURCE_INFO
-    Only include info that can be inserted by SDP, without explicit write by SC. Therefore e.g. RCU mode,
-    Nyquist zone, nof antenna in array are not included.
-    The packet header does not contain all info to derive the RF frequency, because it does not contain the
-    subband-beamlet mapping R_bsub. Therefore it is not necessary to include the Nyquist zone information in 
-    the header. However, just for information the Nyquist zone is useful to have in the header. 
-  . 1b f_adc, sample clock frequency of the ADC, 0 = 160 MHz, 1 = 200 MHz
-  . 2b nyuist_zone
-  . 1b pfb_type, 0 = critically PFB, 1 = oversampled PFB (rather than p, q for R_os = p/q)
-       - The f_adc and t_pfb are in the header, because this information is sufficent to know the subband
-         frequency grid (f_sub = f_adc / N_fft = 195312.5 or 156250 Hz) and the subband sample rate
-         (R_os * f_sub). 
-  . 1b payload_error, 0 payload ok, 1 one or more blocks in payload have data errors
+  ==> Use 16b to fit number from station name (e.g. 1 = CS001, 614 = LV614). The STATION_ID also indirectly reveals whether it is a core, remote or international station.
+      see list of stations at https://proxy.lofar.eu/array_status/STATIONS/HTML/cs011/index.html)
+
+- 2 Byte, SOURCE_INFO
+    Only include info that can be inserted by SDP, without explicit write by SC. The packet header does
+    not contain all info to derive the RF frequency, because it does not contain:
+    . the subband-beamlet mapping R_bsub, because that requires many bytes
+    . the frequency_band: 0 = 10-90, 1 = 30-90, 2 = 110-190, 3 = 170-230, 4 = 210-240, 5 = 210-250 MHz,
+      because these may differ per RCU2 (= RCU modes)
+    . the Nyquist zone information, because that is an implementation detail that is redundant with the
+      frequency_band information
+    . the number nof antenna in array, because that that depends on the beam control which may differ per
+      beamlet.
+    Try use same fields as for statistics
+      
+  . 1b antenna_band: 0 = LBAS, 1 = HBAS [LOFAR2-3098, 6996]
+  . 1b f_adc, sample clock frequency of the ADC, 0 = 160 MHz, 1 = 200 MHz [LOFAR2-3578]
+  . 1b fsub_type: 0 = critically sampled PFB , 1 = oversampled PFB [LOFAR2-2278]
+       - rather than transporting p, q for R_os = p/q
+       - the f_adc and fsub_type are in the header, because this information is sufficent to know the subband
+         frequency grid (f_sub = f_adc / N_fft = 195312.5 or 156250 Hz) and the subband sample rate (R_os * f_sub).
+  . 1b data_flag: 0 = data ok, 1 = data disturbed, due to repositioning of beam [LOFAR2-3123]
+    1b payload_error, 0 = payload data is ok, 1 one or more blocks in payload have data errors, indicating
+       some problem at Station (purpose: fault analysis)
        - a single payload error bit means that all blocks in the payload get discarded if only only block
          has an error.
        - no need for indicator bit per block, assuming errors are rare and will result in loss of
          multiple blocks anyway
-  . 5b beamlet_width in number of bits
-       - Instead of BM = beamlet mode
-       - Default 8 for W_beamlet = 8 bit
-       - Use 5 bit to even fit 16b mode like in LOFAR1)
-  . 6b pn_id = UniBoard2 FPGA ID
+  . 6b reserved (= 0)
+  . 5b pn_id, index of the PN FPGA on UniBoard2 in Station SDP that offloaded the data (purpose: fault analysis)
        - Instead of RSP_ID in LOFAR1
-       - 16 FPGAs for LBA, 16 for HBA in International Station, so maximum 32, but use one bit extra
+       - 16 FPGAs for LBA, 16 for HBA in International Station, so maximum index 31 fits in 5 bits
        - The PN ID is useful for fault diagnoses, to know the PN in SDP from which the data originated.
-       - The pn_id implicitly also reveals the antenna array ID (core station 1 LBA, 2 HBAS, 3 HBA0, 4 HBA1, ...)
-         Therefore it is not necessary to define an explicit antenna ARRAY_ID field that would need to be
-         filled in by SC.
+
+- 2 Byte, BEAMLET_INFO
+  . 4b beamlet_width in number of bits
+       - Instead of BM = beamlet mode
+       - Default 8 for W_beamlet = 8 bit
+       - Use 0 bit to represent 16b mode like in LOFAR1, if necessary
   . 12b beamlet_scale
-    - 18b --> 8b, beamlet_scale = 1 yields lowest bits, beamlet_scale = 1024 (= 11b) yields highest bits
-    - 18b --> 4b, beamlet_scale = 1 yields lowest bits, beamlet_scale = 4096 (= 13b) yields highest bits
-    - beamlet_scale = 1 --> suitable if only one antenna input was used for the beamlet
-    - beamlet_scale = 12, 24, 48, 96 --> to account for number of antennas in beam 
-    - beamlet_scale > 96 --> to have more dynamic range, but less sensitivity. More dynamic range only makes
-                     sense in 8b mode (or 16b mode, but not in 4b or 2b mode), therefore given the
-                     18b beamlet sum the maximum beamlet_scale = 1024.
+    - try to fit beamlet_width and beamlet_scale in 2 Bytes
     - In SDP the beamlet beamlet_scale function extracts the lowest 8b from the 18b beamlet sum, after having
-      multiplied the beamlet sum by 1/beamlet_scale. Internally the beamlet beamlet_scale function uses an 18b 
-      unsigned representation of the 1/beamlet_scale fraction, so 2**18 / beamlet_scale. This yields:
-      beamlet_scale = 1    --> 262144
-      beamlet_scale = 96   -->   2731
-      beamlet_scale = 1024 -->    256
+      multiplied the beamlet sum by 1/beamlet_scale. Same value for all beamlets.
+    - norm = 2**12 / 1024 = 4 to support beamlet_scale values with a resolution of 1/norm = 0.25
+    - 18b --> 8b, beamlet_scale = 1 * norm yields lowest bits, beamlet_scale = 1024 (-1 = 10b) yields highest bits
+    - 18b --> 4b, beamlet_scale = 1 * norm yields lowest bits, beamlet_scale = 4096 (-1 = 12b) yields highest bits
+    - beamlet_scale is typically proportional to sqrt(nof antenna in beam) to maintain sensistivity
+    - larger beamlet_scale values provide more dynamic range, but less sensitivity. More dynamic range only makes
+          sense in 8b mode (or 16b mode, but not in 4b or 2b mode), therefore given the 18b beamlet sum the maximum
+          beamlet_scale = 1024 * norm = 2**10 * 2**2 = 2**12. In practise the dynamic range of beamlet_scale can
+          be less to have more resolution for the norm. The beamlet_scale needs at to be at least sqrt(192) = 13.4
+          to support scaling beamlets for a beam with all antenna inputs. For RFI the beam add coherently, so then
+          beamlet_scale needs to be at least 192. Therefore use 8b.4b for beamlet scale, so norm = 16
+    - beamlet_scale = sqrt  12 * norm = 1    * norm =  16 --> suitable if only one antenna input was used for the beamlet
+                      sqrt  12 * norm = 3.46 * norm =  55
+                      sqrt  24 * norm = 4.9  * norm =  78
+                      sqrt  48 * norm = 6.93 * norm = 111
+                      sqrt  96 * norm = 9.8  * norm = 157
+                      sqrt 192 * norm = 13.4 * norm = 222 --> to account for number of antennas in beam
+    - Internally the beamlet beamlet_scale function uses an 18b unsigned representation of the 1/beamlet_scale fraction,
+      so 2**18 / beamlet_scale. This yields:
+        beamlet_scale = 1       * norm --> 16384
+        beamlet_scale = sqrt 96 * norm -->  1670
+        beamlet_scale = 256     * norm -->    64
+      After mutliplying the 18b.0b beamlet_sum by 1b.12b 1/beamlet_scale the result is a 24b.12b value that is rounded and 
+      clipped to output the 8b beamlets.
+        
 
 - 1 Byte, reserved (= 0)
 
-- 2 Byte, BEAMLET_INDEX = beamset_index * nof_beamlets_per_set
-  . nof_beamlets_per_set = S_sub_bf = 488
-  . nof_beamsets = 1, fixed for LOFAR2.0 stage 1 one beamset per antenna array
+- 2 Byte, BEAMLET_INDEX of first beamlet in the payload
+  . sdp_beamset_index * nof_beamlets_per_set
   . beamset_index = 0 in range(nof_beamsets).
   . global beamlet index of first beamlet in block
     -   0: 487 for beamset 0