Updated planning after SR5 white board meeting.

applications/apertif/doc/apertif_fpga_firmware_planning_erko.txt

@@ -5,30 +5,36 @@ Done) #1274: Arts SC1: Beamlet selection for CB 0 also seems to require CB 1 -->
 Done) #1654: Arts SC1: Run Arts SC1 with MAC controling the Apertif BF using cold Apertif start and 
              warm Arts start in main.py
 
+*) #1658: Implement beamlet or channel offload by Apertif X to Arts on UniBoard2
+  - See section 9 in SP-062: output channels for SC3 or beamlets for SC2
+
 *) #1648: Arts SC1: Support TAB1 weights gain control dependent on number of dishes in main.py --> Daniel
 
 *) #1300: Arts SC4: Subband bandpass correction after the channel filterbank?
+
 *) #1484: Additional fringe stopping for TAB within CB --> not necessary for SC3,4, only for SC2
-   - *) #1593: Arts SC4: Apply TAB weights per channel instead of per beamlet?  --> WontFix, see #1484
-   - *) #1660: Arts SC4: TAB BF driver support in MAC --> not needed anymore
-   - *) #1691: Arts SC4: Provide Python script to control TAB beamformer weights
+   WontFix) #1593: Arts SC4: Apply TAB weights per channel instead of per beamlet?  --> WontFix, see #1484
+   -) #1660: Arts SC4: TAB BF driver support in MAC --> not needed anymore
+   -) #1691: Arts SC4: Provide Python script to control TAB beamformer weights --> Pieter
 
 *) #1515: Make Arts beamlet mapping independent of Apertif
-  - Done) Create single element beam mapping for Arts SC4 and apply in dish_command.sh:
+  Done) Create single element beam mapping for Arts SC4 and apply in dish_command.sh:
        python /home/kooistra/svnroot/UniBoard_FP7/UniBoard/trunk/Software/python/base/unb_apertif.py
-  - Done) #1649: Add R_beamout transpose function in Apertif BF after R_beam but before T_int_x transpose
+       
+  Done) #1649: Add R_beamout transpose function in Apertif BF after R_beam but before T_int_x transpose
     The R_beamout is needed for all Arts SC, because MAC assumes the Apertif setting for R_beam and
     therefore also for the BF weights and the fringe stopping. The R_beamout reorders the beamlets to
     route the CB to the PN for Arts. For Apertif X it does not matter which PN processes the beamlets,
     because they all arrive via the 1/10GbE network at the same DW.
     
-  - #1650 Inverse R_beamout function on beamlet indices at Apertif X output, to transparently 
+  -) #1650 Inverse R_beamout function on beamlet indices at Apertif X output, to transparently 
     support SC3 --> add method in pi_apertif_system.py, add firmware function in X
     
-  - #1862 Add support for Arts SC4 offload balancing in R_beamout setting by R_rotate rotating the selection per FN
-    For SC4 R_rotate is needed to balance the output load per CB on the 40G link (section 7.4.3)
-    For SC3 this R_rotate is not needed (section 10.3.3).
-
+  -) #1862 Add support for Arts SC4 offload balancing in R_beamout setting by R_rotate rotating the selection per FN
+    . For SC4 R_rotate is needed to balance the output load per CB on the 40G link (section 7.4.3), support
+      this by adding a --cmd to pi_apertif_system.py. In the Arts SC4 firmware Jonathan will support this by adjusting
+      the beamlet index in the packet headers.
+    . For SC3 this R_rotate is not needed (section 10.3.3).
     
 
 ###############################################################################
@@ -75,13 +81,13 @@ Done) #1377: Dish output enable per band based on the active correlator boards
   Disable of bands should not be necessary, because we specify list of active --tel with main.y, however
   the MAC software assumes that all active dish ouptuts should participate in the measurement.
 
-Almost done) #1100: ADU lock robustness
+Done) #1100: ADU lock robustness
   - with r 17830 for filterbank and beamformer
   - From FB r17502 (8 aug) the ADU handler once in lock, keeps this ADC sample phase forever
 
-!) #1230, 1294: Support PAF calibrator using BF BST
+Done) #1230, 1294: Support PAF calibrator using BF BST
   - See overview email 27 nov 2017 "PAF calibratie" from Ruud Overeem.
-  - Support dish PAF calibrator using WG in pi_apertif_system.py
+  - Support dish PAF calibrator using WG in pi_apertif_system.py --> supports 64x64 using --cmd 90
   - Meeting 15 feb (#1544):
     Off-line = static calibration uses Cas-A and points each CB to it to measure the PAF. Tracking this
     pattern of CB uses the holog mode of the telescope. It is also necessary to take at least one cold
@@ -119,26 +125,17 @@ Done) #1517: Support beamlet mapping for Aperif X and Arts (20 story points)
         In March implemented the 63x63, because SR3 deadline was closing in. This works OK and can
         be retried using setup_wg.sh and cmd 90.
 
-!) #1543: End to end VHDL simulations using MM file IO (20 story points)
-  - Updated Visio drawings FB, BF, X
-  - VHDL simulation of the full chain (not feasible, instead use end to end WG - DW tests on hardware)
-  - Verify visibility output order and output load to DW, see tb_node_apertif_unb1_correlator_processing
-  
-*) #1163: Flagging for failing links, RFI ? etc.
-  - Support MAC
-  - Failing links during observation can be handled by MAC
-  - RFI is detected by astronomers at visibility level.
-
-* #1598, #1599, #1600 : Robust flashing--> Daniel
+Done) #1598, #1599, #1600 : Robust flashing--> Daniel
   - See also #1098, #1105 of SR2
   - By protecting the factory image
+  - Use latest unb1_minimal as factory image in all UniBoards
 
 Hardware issues/ firmware bugs
 Done) #1488: Replaced wrong rank DDR3 module at RT5 UNB-7 FN-0, still need to check RT-D
-*) #1722: RT4 BSN timeout UNB5-FN1 (node 41)
-*) #1751: Writing application image in central UNB15-BN0 sometimes fails
-*) #1758: FPGA can hang when application image is corrupt
-*) #1870: I2C with RT-D, UNB-7, BN-3, ADU-CD fails and therefore it cannot lock
+Done) #1722: RT4 BSN timeout UNB5-FN1 (node 41)
+Done) #1751: Writing application image in central UNB15-BN0 sometimes fails
+Done) #1758: FPGA can hang when application image is corrupt
+Done) #1870: I2C with RT-D, UNB-7, BN-3, ADU-CD fails and therefore it cannot lock
   10 apr: RT8, X --> UNB-5 link issue?
   20 apr: RT5, X --> UNB-7 BN-1 link 0 issue? The BSN aligner at the input does not pass on the other 2.
 
@@ -151,17 +148,14 @@ Done) #895: BF output Fiber length compensation in FN beamformer
 Done) #895: PPS cable length compensation in BN filterbank
   - This can be set via pi_common_pulse_delay.py in dish_commands.py
 
+Done) #1937: Make all visibility packet header flags writable by MAC
+
 Software issues
-*) #1518, #1654: 135 --> 200 MHz
+*) #1518 : 135 --> 200 MHz
    Kept MAC Rbeam settings in:
      $RADIOHDL/applications/apertif/commissioning/mac_rbeam_coreDout.log
-*) #1510: Integrate main.py into MAC software, purpose is to have MAC control and thus know all settings (one captain).
      
-*) #1542: End to end WG to DW test (20 story points)
-  Relates to: #1517, #1230
-  - Intermediate verification points are the SST, BST, but finally the visibilities should be verified given WG for single element BF
-  - Overwrite statistics before reading to ensure fresh data
-  - Verify visibility output order and output load to DW, see tb_node_apertif_unb1_correlator_processing
+*) #1510: Integrate main.py into MAC software, purpose is to have MAC control and thus know all settings (one captain).
 
 *) #1860: Use beamlet clipping for correlator and TAB
   - dish beamlet output -->
@@ -171,13 +165,45 @@ Software issues
     . advantage of using clip is that it also kind of flags the data, whereas wrap kind of scrambles the data
     . advantage of only using clip is that we do not have to select between clip and wrap per Apertif X and Arts SC
     --> Conclusion: Use clip in node_apertif_unb1_fn_beamformer
+    . however if clipping occurs then there is RFI, so the visibility will not be useful anyway
+    --> Conclusion: Keep wrap in node_apertif_unb1_fn_beamformer
+    
+*) #1931: Confirm complex format of BF weights: is it (Im,Re) or (Re,Im).
 
 
 ###############################################################################
-# SR5 30 Jul: Online calibration (Start of Apertizer observations)
-# - 200 MHz, 37 CB, 12 dishes
-# - Uses the noise source to calibrate for analogue variations, about twice per observation (before and after)
-#   . For online calibration the 63*1 PAF correlator is suffcient
+# SR5 30 Jul: Start of Apertizer observations (enable pre-survey)
+# - goals: science quality data, test operation, test survey strategy
+# - system: 200 MHz, 37 CB, 12 dishes
+# - commisioning:
+#   . Parameter set (parset) defines observation, measurement set contains all meta data to interpret the data.
+#   . Online calibration: Uses the noise source to calibrate for analogue variations, about twice per observation (before and after)
+#     using the 63*1 PAF correlator and SigCtrl. Online calibration is also needed directly after static calibration. In this way
+#     the online calibration takes care that the static calibration weights form the stable reference for the beam model. The online
+#     calibrated weights are applied, such that the static weights still define the primary beam. The online calibrated weigths (or
+#     their differences) need to be stored for monitoring purposes.
+#   . Online delta Tsys monitoring: uses noise source 
+#   . Flag noise source on in visibiltiies to DW, the noise source is on for a few s, so the DW integration interval of 30 s still
+#     is valuable.
+#   . Flux scale:
+#     Every week: tune system (phase zero, attenuations)
+#     Every observation: measure with calibrator source in one beam for ~15 min before and ~15 min after a 12 hour observation and
+#     then use this flux scale to also derive the flux scale for the other beams. This also relates to whether the phase zero of
+#     one CB for all dishes can be transfered to all other CB
+#     (ik denk van wel, want static calibration heeft binnen de PAF alles gecalibreerd, waardoor voor alle CB de onderlinge phase
+#     tenminste constant is en mogelijk zelfs 0. Als je dan phase zero hebt tussen dishes voor een CB dan geldt deze ook voor de
+#     andere CB.)
+#   . Beam model: element beam patterns using holography + compound beam weigths from static calibration using the PAF Correlator. The
+#     Online calibration is used to ensure that the holography and PAF correlator measurement can be combined in phase. For the element
+#     beam pattern subband resolution is sufficient and fits the CB beamlets weights.
+#   . Channel filter and frequency response
+#   . Pointing model update because the PAF is lighter than the MFFE, so the actual pointing varies along the sky due to different
+#     bending.
+#   . Operator: able to start and run an observation (Antonis)
+#   . Subband based delay errors: #1640
+#   . Monitoring & Alarming: WinCC
+#
+#   . SigCtrl, CorrCtrl, DwCtrl
 
 Done) Compare SST with BST
   - Supported by pi_apertif_system.py --cmd 72, run separately per pol to ensure that SST and BST apply to same Tint_x interval 
@@ -185,6 +211,16 @@ Done) Compare SST with BST
 Done) #1276: Plot BST per CB
   - Supported by pi_apertif_system.py --cmd 71
   
+Done) #1163, #397: Flagging for failing links, RFI ? etc.
+  - Support MAC
+  - Failing links during observation can be handled by MAC
+  - RFI is detected by astronomers at visibility level.
+
+!) #1543: End to end VHDL simulations using MM file IO (20 story points)
+  - Updated Visio drawings FB, BF, X
+  - VHDL simulation of the full chain (not feasible, instead use end to end WG - DW tests on hardware)
+  - Verify visibility output order and output load to DW, see tb_node_apertif_unb1_correlator_processing
+  
 *) #1655: Finalize quantisation (8 story points)
   - SST
   - BST
@@ -200,25 +236,29 @@ Done) #1276: Plot BST per CB
   - Fchan SST is already disabled, because there are N_blk sync per t_int_x, so not enough time to read the SST anyway.
   - Fchan reorder needed for SC3, so then do not reorder in Apertif X output.
 
-*) #1868: wpfb_unit_dev force Fchan channel output data to 0 for first N_tap bin samples after sync
-
-*) #1658: Implement beamlet or channel offload by Apertif X to Arts on UniBoard2
-  - See section 9 in SP-062: output channels for SC3 or beamlets for SC2
+*) #1243 Channel ghosts in channel 16 and 48
 
-*) #676, 893: FPGA register map
+*) #676: FPGA register map with number of instances, ready in firmware, to do in MAC.
 
 *) Correlator synthesis @ 200 MHz
   - For r 17830 4 out of 5 synthesis achieved timing > 200 MHz
   - Enable channel FIR filter in Apertif X --> fits and achieves > 200MHz with r 18042
   - Verify channel FIR coefficients in Python
   - 6 bit mode
-  - Restore Apertif beamlet numbering via inverse R_beamout on beamlet indices
+  - #1650) Restore Apertif beamlet numbering via inverse R_beamout on beamlet indices
 
 
 ###############################################################################
 # SR-6 : Dual polarization
 
-*) 
+*) #1542: End to end WG to Correlator output test (8 story points)
+  - Intermediate verification points are the SST, BST, but finally the visibilities should be verified given WG for single element BF
+  - Overwrite statistics before reading to ensure fresh data
+  - Verify visibility output order and output load to DW using a databuffer in the correlator output, see
+    tb_node_apertif_unb1_correlator_processing_output
+
+*) #1868: wpfb_unit_dev force Fchan channel output data to 0 for first N_tap bin samples after sync
+
 
 ###############################################################################
 # SR-7 : Full 300 MHz BW and 40 CB
@@ -243,7 +283,6 @@ Done) #1276: Plot BST per CB
     . #1868: The F_chan filter response is disturbed for the first Ntap output, this could also
              affect the AAF, but only by a factor 8/12500 so probably too small to notice.
 
-*)  Channel gosths? 
 
 ###############################################################################
 # SR-9 : Start of Apertif surveys
@@ -264,11 +303,6 @@ Done) #1276: Plot BST per CB
 
 
 
-###############################################################################
-# Useful commands
-- find . -name gen_hex_files*.py 
-- find . -name *transpose*.py
-
 ###############################################################################
 # Unallocated to do list:
 
@@ -279,19 +313,10 @@ Done) #1276: Plot BST per CB
 ! main.py join tc.unbNrs --> tc.unbStr gebruiken in dish_commands.sh call
 ! stories opdelen in tasks per SR
 ! task wezen onderbrengen in parent task (= story)
-- Validate tb_apertif_unb1_fn_beamformer_nodes on HW:
-  . move to apertif_unb1_fn_beamformer_nodes
-  . add dp_data_force_parallel to BF sopc
-  . make a python script to be able to validate Tint using dp_data_force_paralllel and DB in FN. 
-- Validate tb_apertif_unb1_fn_beamformer_nodes on HW:
-  . use dp_data_force_paralllel in FN and DB in central unb after mesh in PN
-  . make a python script to be able to validate BF 10g link and mesh PN
-
-- pi_apertif_system.py opdelen in aparte files
+
 - support duplicate subbands in Rsub and Rbeam
 - add R_beamout inverse index lookup table generation in R_beamout class of pi_apertif_system.py.
 
-- pi_dp_force_data_parallel.py is ok, add similar pi_dp_force_data_serial.py. There is also pi_dp_force by JH.
 - update or create variant of unb1_board.sdc for qsys regarding get_clocks *u_sopc
 - ppsh.vhd decrease c_pipeline_output : NATURAL := 100;
 - add sync_checker stable monitor?
@@ -304,8 +329,6 @@ Done) #1276: Plot BST per CB
 
 - MEM-199 bijwerking met Fif 10b inclusief channel correlator --> visibility j w_m(tau_d-tau_g) - j phi_offset  phase term
 
-- MAC R_beam == 300MHz_37CB ?
-
 - create apertif_unb1_fn_beamformer_nodes with trans and output
   . use dp_block_gen_valid_arr.vhd to pass on BSN in node transpose
   . move dp_force from beamformer node to transpose node