Improved ICD for statistics and beamlets.

applications/lofar2/doc/prestudy/station2_sdp_icd.txt

@@ -51,19 +51,9 @@ UDP link control
 > ping <IP address>  # to find MAC address for IP address ?
 
 
-###################################################################################################
-# L1 ICD 11109 STAT-CEP
 
-Included:
- A) Beamlet data
- B) Transient buffer read out
- 
-Not included:
- . SST, BST, XST, because these are for monitoring and calibration, not for science data
- . Subband offload for AARTFAAC2.0 will have own EICD
-
-
-LFAA-CSP_Low : OSI (Open Systems Interconnection) layers
+###################################################################################################
+# LFAA-CSP_Low : OSI (Open Systems Interconnection) layers
 
 7 Application  : Not applicable, this is the level where the STAT and CEP products each perform their
                  allocated functions.
@@ -107,15 +97,98 @@ LFAA-CSP_Low : OSI (Open Systems Interconnection) layers
 1 Physical  :
   - Ethernet standard [IEEE Std 802.3-2015], 40 GbE
 
+###################################################################################################
+# APERTIF / ARTS
+
+# APERTIF BF
+
+  47b reserved
+   1b sync
+  64b BSN
+
+# APERTIF X
+
+  1 Byte, Marker (= 120)
+  1 Byte, Version (= 1)
+  2 Byte, Beamlet index (0-16383)
+  2 Byte, Channel index (0-63)
+  2 Byte, Reserved
+  8 Byte, Timestamp
+  24 Byte, Flags (0-23, 1 bit per pol/dish)
+  
+  --> Total 40 Bytes
+
+# Arts SC2
+
+  1 Byte, Marker (= 120)
+  1 Byte, Version (= 1)
+  1 Byte, Source id (0-15 from 16 UniBoard2)
+  1 Byte, TAB index (0-11)
+  2 Byte, number of channels per block
+  2 Byte, Number of blocks per packet
+  8 Byte, Timestamp
+  24 Byte, Flags (0-23, 1 bit per pol/dish)
+  
+  --> Total 40 Bytes
+
+# Arts SC3,4
+
+  1 Byte, Marker (= D0,1,2,3, E0,1,2,3)
+  1 Byte, Version (= 1)
+  1 Byte, CB index (0-39)
+  1 Byte, TAB index (0-11)
+  2 Byte, Channel index (0-1535)
+  2 Byte, Application payload size (6250, 8000 bytes)
+  8 Byte, Timestamp
+  1 Byte, Sequency number 
+  7 Byte, Reserved
+  24 Byte, Flags (0-23, 1 bit per pol/dish)
+  
+  --> Total 48 Bytes
+
+
+###################################################################################################
+# LOFAR_ASTRON_ICD_009_RSP_CEP
+
+1 Byte, VERSION_ID
+2 Byte, SOURCE_INFO : BM 2b, 160/200M 1b, payload OK 1b, RSP_ID 5b
+1 Byte, CONFIGURATION_ID
+2 Byte, STATION_ID
+1 Byte, NOF_BEAMLETS_PER_BANK
+1 Byte, NOF_BLOCKS
+4 Byte, TIMESTAMP
+4 Byte, BSN
+
+--> Total 16 Bytes
+
+###################################################################################################
+# L1 ICD 11109 STAT-CEP
+
+Included:
+ A) Beamlet data
+ B) Transient buffer read out
+ 
+Not included:
+ . SST, BST, XST, because these are for monitoring and calibration, not for science data
+ . Subband offload for AARTFAAC2.0 will have own EICD
 
 A) STAT-CEP Beamlet data interface:
 
+Do use:
+- marker  like in APERIF, ARTS and SPEAD (magic)
+- version like in APERIF, ARTS and SPEAD
+- source info that relates to SDP setting
+- observation_id to relate to parset with telescope setting
+- station_id
+- nof beamlets per block to allow 
+- nof blocks per packet to optimize use of jumbo frames
+
 LOFAR1 supported beamlet bit modes 16b, 8b and 4b by packing these beamlets into 16b, so 1 16b, 2 8b or 4 4b beamlet
 values per 16b word. This creates 1, 2, or 4 beamsets called banks. Packing the beamlets from different
 beamsets per 16b word made sense because these banks were already distinghuised at the AP and on the ring.
 Default LOFAR2.0 only has 8b mode and 1 beam set of S_sub_bf = 488 beamlets. Future LOFAR2.0 could support more
 beamsets. At the PN and on the ring these beamsets would be treated indepently and will always use W_beamlet_sum
-= 18b indepent of the beamlet bit mode. Therefore in LOFAR2.0 packing different beamsets 8b word in a payload makes
+= 18b indepent of the beamlet bit mode. Therefore in LOFAR2.0 packing different beamsets per 8b word in a payload makes
 less sense. Instead it is better to use more blocks per packet to have sufficiently large payload for the 4b and 2b
 beamlet bit modes. Hence in LOFAR2.0 the extra beamlets that can be transported for the lower beamlet bit modes are
 treated as independent beamsets, similar as an extra 8b beam set. The beamlet index follows from:
@@ -139,7 +212,7 @@ be optimum to have 22 destinations, because it has 22 processing input nodes. Th
 In LOFAR2.0 the number of lanes does not depend on the number of physical lanes on the ring. Therefore the number
 of lanes can be different than 4. With S_sub_bf = 488 = 2*2*2*61 there can be 1, 2, 4 or 8 lanes with equal number
 of beamlets per lane, respectively 488, 244, 122 or 61. With less beamlets per lane the NOF_BLOCK_PER_PACKET needs
-to be increased to have sufficiently large packets (< 9000 octets). However other number of lanes are feasible to,
+to be increased to have sufficiently large packets (< 9000 octets). However other number of lanes are feasible too,
 but will result in different number of beamlets per packet. For example using 22 lanes yields 18 * 22 + 4 * 23 = 
 488, so 18 streams with 22 beamlets per packet and 4 streams with 23 beamlets per packet.
 
@@ -152,65 +225,96 @@ The FPGA has 2713 M20k, so this is 128/2713 ~= 5% of the internal BRAM resources
 
 The total number of streams to CEP then becomes NOF_BEAMSETS * NOF_LANES.
  
-- MARKER 8b
+- 1 Byte, MARKER
   . Like in APERTIF and ARTS, may be useful to quickly recognize the data packet.
+  . Beamlets: 20
   
-- VERSION_ID 8b
+- 1 Byte, VERSION_ID
   . 2,3,4 for LOFAR1
   . 5 first for LOFAR2.0
   
-- STATION_ID 16b (idem as LOFAR1)
-  ==> or 8b because there are only ~50 stations
-  ==> use 16b to fit number from station name (e.g. CS001, LV614, see list of stations at
-      https://proxy.lofar.eu/array_status/STATIONS/HTML/cs011/index.html)
-
-- OBSERVATION_ID 32b
-  Instead of CONFIGURATION_ID 8b (used in LOFAR1? intended to refer to the parset that defines this observation)
-  The observation ID provides the hook to information on:
+- 4 Byte, OBSERVATION_ID
+    Instead of CONFIGURATION_ID 8b (used in LOFAR1? intended to refer to the parset that defines this observation)
+    The observation ID provides the hook to information on:
   . RCU mode
   . f_adc = 200 MHz, 160 MHz
   . Nyquist zone (0, 1, 2)
   . critically PFB, oversampled PFB (or p, q for R_os = p/q)
+  . nof antenna in array
+    - Number of antenna in array (core, LBA, HBA inner to make HBA international look like HBA remote),
+    - Maximum S_ant = 192.
   . etc
 
-- SOURCE_INFO 16b
-  . 1b f_adc = 200 MHz, 160 MHz
-  . 1b critically PFB, oversampled PFB (or p, q for R_os = p/q)
-  . 5b beamlet width in number of bits
-       - default 8 for W_beamlet = 8 bit, instead of BM = beamlet mode
+- 2 Byte, STATION_ID (idem as LOFAR1)
+  ==> or 8b because there are only ~50 stations
+  ==> use 16b to fit number from station name (e.g. CS001, LV614, see list of stations at
+      https://proxy.lofar.eu/array_status/STATIONS/HTML/cs011/index.html)
+
+- 4 Byte, SOURCE_INFO
+    Only include info that can be inserted by SDP, without explicit write by SC. Therefore e.g. RCU mode,
+    Nyquist zone, nof antenna in array are not included.
+  . 1b f_adc = 200 MHz, 160 MHz, sample rate
+  . 1b t_pfb = PBF type, 0 critically PFB, 1 oversampled PFB (rather than p, q for R_os = p/q)
+  . 1b payload_ok, 0 payload ok, 1 one or more blocks in payload have data errors
+       - no need for indicator bit per block, assuming errors are rare and will result in loss of
+         multiple blocks anyway
+  . 5b w_beamlet in number of bits
+       - Instead of BM = beamlet mode
+       - Default 8 for W_beamlet = 8 bit
        - Use 5 bit to even fit 16b mode like in LOFAR1)
-  . 6b PN ID = UniBoard2 FPGA ID
-    - Instead of RSP_ID in LOFAR1
-    - 16 FPGAs for LBA, 16 for HBA in International Station, so maximum 32, but use one bit extra
-    - the PN ID implicitly also reveals the array ID (core station 1 LBA, 2 HBA/HBA0, 3 HBA1, ...)
-
-- NOF_ANTENNA_IN_BEAM 8b
-  Number of antenna in beam (core, LBA, HBA inner to make HBA international look like HBA remote),
-  . maximum S_ant = 192.
-  
-- BEAMLET_SCALE 16b
-  Beamlet scale setting:
-    - 18b --> 8b, scale = 1 yields lowest bits, scale = 1024 yields highest bits
-    - 18b --> 4b, scale = 1 yields lowest bits, scale = 4096 yields highest bits
+  . 6b pn_id = UniBoard2 FPGA ID
+       - Instead of RSP_ID in LOFAR1
+       - 16 FPGAs for LBA, 16 for HBA in International Station, so maximum 32, but use one bit extra
+       - The pn_id implicitly also reveals the antenna array ID (core station 1 LBA, 2 HBAS, 3 HBA0, 4 HBA1, ...)
+         Therefore it is not necessary to define an explicit antenna ARRAY_ID field that would need to be
+         filled in by SC.
+  . 12b beamlet_scale
+    - 18b --> 8b, scale = 1 yields lowest bits, scale = 1024 (= 11b) yields highest bits
+    - 18b --> 4b, scale = 1 yields lowest bits, scale = 4096 (= 13b) yields highest bits
+    - scale = 1 --> suitable if only one antenna input was used for the beamlet
+    - scale = 12, 24, 48, 96 --> to account for number of antennas in beam 
+    - scale > 96 --> to have more dynamic range, but less sensitivity. More dynamic range only makes
+                     sense in 8b mode (or 16b mode, but not in 4b or 2b mode), therefore given the
+                     18b beamlet sum the maximum scale = 1024.
+    - In SDP the beamlet scale function extracts the lowest 8b from the 18b beamlet sum, after having
+      multiplied the beamlet sum by 1/scale. Internally the beamlet scale function uses an 18b 
+      unsigned representaion of the 1/scale fraction, so 2**18 / scale. This yields:
+      scale = 1    --> 262144
+      scale = 96   -->   2731
+      scale = 1024 -->    256
+      
   
 
-- NOF_BEAMLETS_PER_SET 16b = S_sub_bf = 488
-- SET_INDEX 8b
-- LANE_INDEX 8b
-- NOF_LANES 8b
+- 2 Byte, BEAMLET_INDEX = SET_INDEX * NOF_BEAMLETS_PER_SET + bl * NOF_LANES + LANE_INDEX
+  . NOF_BEAMLETS_PER_SET = 488
+  . SET_INDEX in range(number of beamsets, currenlty 1 beamset per antenna array)
+  . NOF_LANES 8b
+  . LANE_INDEX 8b in range(NOF_LANES)
+  . global beamlet index of first beamlet in block
+    -   0: 487 for beamset 0
+    - 488: 975 for beamset 1, etc
+    - can fit maximum 2**16 / 488 = 134 beamsets
 
-  . global beamlet index = SET_INDEX * NOF_BEAMLETS_PER_SET + bl * NOF_LANES + LANE_INDEX
   . stream index = SET_INDEX * NOF_LANES + LANE_INDEX
     - Separate destination address per stream
+    - no need to have an stream index field, because the CEP only needs to know the beamlet index.
+    - the beamlet index for each sample follows from BEAMLET_INDEX, BEAMLET_STEP, NOF_BEAMLETS_PER_BLOCK
+      and NOF_BLOCKS_PER_PACKET
     - LOFAR1 supports 4 streams (4 lanes from RSP ring, staggered so rsp_id identifies lane)
     - LOFAR2.0 preferrably supports >> 4 streams
     - LOFAR2.0 preferrably outputs only 1 stream
     - CEP with N processing nodes would like N streams, Cobalt has N = 22
     - S_sub_bf = 488 = 2*2*2* 61, so only NOF_LANES = 1, 2, 4, and 8 yield a fixed integer number
       of NOF_BEAMLETS_PER_BLOCK.
-    ? Is it useful to support LANE_INDEX and NOF_LANES > 1 at SDP but < 22 which is optimum for CEP?
+
+    
+- 1 Byte, BEAMLET_STEP
+  . Index increment of subsequent beamlets in block
+  . BEAMLET_STEP = NOF_LANES
+  ? Is it useful to support BEAMLET_STEP=NOF_LANES > 1 at SDP but < 22 which is optimum for CEP?
+  
   
-- NOF_BEAMLETS_PER_BLOCK
+- 2 Byte, NOF_BEAMLETS_PER_BLOCK
    . Equals floor or ceil of NOF_BEAMLETS_PER_SET / NOF_BEAM_LANES dependent on LANE_INDEX,
      so redudant if all beamlets are send, but could be used to send less beamlets.
    . Instead of NOF_BEAMLETS_PER_BANK in LOFAR1
@@ -220,7 +324,7 @@ The total number of streams to CEP then becomes NOF_BEAMSETS * NOF_LANES.
      W_beamlet = 4b :                              1952 beamlets
      W_beamlet = 2b :                              3904 beamlets
   
-- NOF_BLOCKS_PER_PACKET 8b
+- 1 Byte, NOF_BLOCKS_PER_PACKET
   . Multiple beamlet time slots in one packet to increase payload efficiency.
   . Maximum NOF_BLOCKS_PER_PACKET is about 4 * NOF_LANES, because:
     NOF_LANES = 1: 4 --> 4 * 1952 = 7808 octets < 9000 Jumbo
@@ -228,16 +332,19 @@ The total number of streams to CEP then becomes NOF_BEAMSETS * NOF_LANES.
   . LOFAR1 has payload ok bit in SOURCE_INFO to indicate that at least one block in the packet
     has incorrect data  
 
-- TIMESTAMP 50b
-  . Instead of 32b seconds TIMESTAMP and 32b BLOCK_SEQUENCE_NUMBER within second)
-  . Block Sequence Number (BSN) used to detect lost blocks and to align blocks from different stations
-  . BSN unit T_sub, 50b yields > 100 year span (1970 - 2070)
-
-- BLOCK_PERIOD 13b
-  . Subband period T_sub in ns resolution, 5120 ns @ 200 MHz, Ros = 1
+- 8 Byte BSN
+  . 50b Blcok Sequence Number
+    - Instead of 32b seconds TIMESTAMP and 32b BLOCK_SEQUENCE_NUMBER within second of LOFAR1
+    - Block Sequence Number (BSN) used to detect lost blocks and to align blocks from different stations
+    - BSN unit T_sub, 50b yields > 100 year span (1970 - 2070)
+    
+- 2 Byte BLOCK_PERIOD
+  . 13b Subband period T_sub in ns resolution, 5120 ns @ 200 MHz, Ros = 1
   
-
-- TX_PACKET_COUNT 32b
+--> Total 1 + 1 + 4 + 2 + 2 + 1 + 2 + 1 + 8 + 2 = 28 Bytes
+  
+Remark:
+  - TX_PACKET_COUNT
   ==> Not useful, because then CEP needs to count Rx packets. Better send filler packets to keep the
       packet rate at the nominal rate, so that any packet loss is due to the Network and already 
       clear at OSI 2 layer using lower level tools like Wireshark.
@@ -308,4 +415,4 @@ d) OPC-UA representation (by OPC-UA servers)
 # L2 ICD 11218 SDP-STCA
 
 
- 
+ 
\ No newline at end of file