From 20fe6def37a4bf7dd7aa2e7c0cc6af95ffc1fbd4 Mon Sep 17 00:00:00 2001 From: Eric Kooistra <kooistra@astron.nl> Date: Wed, 17 May 2023 16:22:08 +0200 Subject: [PATCH] Updates after competing ICD and FW design. --- .../station2_sdp_transient_buffer.txt | 190 ++++++++---------- 1 file changed, 88 insertions(+), 102 deletions(-) diff --git a/applications/lofar2/doc/prestudy/station2_sdp_transient_buffer.txt b/applications/lofar2/doc/prestudy/station2_sdp_transient_buffer.txt index ee5e128aed..e82291dea2 100644 --- a/applications/lofar2/doc/prestudy/station2_sdp_transient_buffer.txt +++ b/applications/lofar2/doc/prestudy/station2_sdp_transient_buffer.txt @@ -12,6 +12,7 @@ Detailed design: Transient Buffer (TBuf) function for LIFT project 8) Crossbar 10) Planning 11) Transient detection (TDet) Design +12) Cosmic ray References: @@ -92,7 +93,7 @@ Design decision 16GByte DDR4 na L2SDP-854, 850 - Buffer lengte versus nof antennes - Self trigger -The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW yet. I was n +The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW yet. 3) TBB (Transient Buffer Board) LOFAR1 - From 2.3 in [4] @@ -145,11 +146,11 @@ The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW y - timing . Use sample sequence number (RSN) or mem_bsn: - - RSN increments by c_tbuf_nof_samples_per_page = 8176 - - mem_bsn increments by 1 per page, so per block of c_tbuf_nof_samples_per_page = 8176. + - RSN increments by c_sdp_tbuf_nof_samples_per_block = 8176 + - mem_bsn increments by 1 per page, so per block of c_sdp_tbuf_nof_samples_per_block = 8176. . RSN counts sample periods (5 ns) since t_epoch = 1970, can fit 2**64 / (365.25 * 24 * 3600 / 5e-9) > 2922 years - . TBuf uses sop and eop to mark c_tbuf_nof_samples_per_page = 8176 + . TBuf uses sop and eop to mark c_sdp_tbuf_nof_samples_per_block = 8176 . TBuf does not need sync ? . Start RSN or mem BSN at same time as SDP BSN by FPGA_processing_enable_RW. @@ -207,10 +208,10 @@ The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW y that DDR4-I can wrap without a gap or extend to DDR-II without a gap. - RSN = 64b = 8B - 14b packed data - . c_tbuf_nof_samples_per_page = 8K - 8 - 8 = 8192 - 16 = 8176 B / 14b = 4672 + . c_sdp_tbuf_nof_samples_per_block = 8K - 8 - 8 = 8192 - 16 = 8176 B / 14b = 4672 . 4672 * 5 ns = 23.36 us per page, so ~42.8 pages / ms - CRC = 64b = 8B, use 64b CRC to match 64b words, no need to reduce stored - CRC to less bits, because c_tbuf_nof_samples_per_page fits preferrably a multiple + CRC to less bits, because c_sdp_tbuf_nof_samples_per_block fits preferrably a multiple of 64b words. - Memory storage overhead is 16 / 8192 = 0.2% @@ -270,10 +271,8 @@ The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW y . SDPFW: no need for MM interface: use nof_clk_per_sync from dp_bsn_source also for dp_rsn_source start RSN source when dp_bsn_source is (re)started, derive RSN from BSN - . c_tbuf_nof_samples_per_block = - c_tbuf_nof_samples_per_page = - c_tbuf_nof_samples_per_packet = 2000 - . nof_block_per_sync = nof_clk_per_sync / c_tbuf_nof_samples_per_packet + . c_sdp_tbuf_nof_samples_per_block = 2000 + . nof_block_per_sync = nof_clk_per_sync / c_sdp_tbuf_nof_samples_per_block = 100k for sync interval 1 s at FPGA_sdp_info_f_adc_R = 200 MHz = 80k for sync interval 1 s at FPGA_sdp_info_f_adc_R = 160 MHz @@ -306,31 +305,28 @@ The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW y * Dumping: . FPGA_tbuf_dump_inter_packet_time_RW Wait time in seconds between packets send to CEP. SDPTR translates float time into - T_adc units. + T_adc units --> reg_dump_inter_packet_time_RW . FPGA_tbuf_dump_start_timestamp_RW[pn] - FPGA_tbuf_dump_start_timestamp_R[pn] = reg_dump_start_rsn_RW * T_adc + FPGA_tbuf_dump_start_timestamp_R[pn] = dump_start_rsn * T_adc in SDPTR Timestamp of first packet that will be dumped. Equals FPGA_tbuf_dump_start_timestamp_RW if it exists in the buffer. . FPGA_tbuf_dump_end_timestamp_RW[pn] - FPGA_tbuf_dump_end_timestamp_R[pn] + FPGA_tbuf_dump_end_timestamp_R[pn] = reg_dump_end_rsn_RW * T_adc Timestamp of last packet that will be dumped. Equals FPGA_tbuf_dump_end_timestamp_RW if it exists in the buffer. . FPGA_tbuf_dump_nof_packets_R[pn] = reg_dump_nof_pages_RW - Amount of packets that will be dumped for range that is requested via FPGA_tbuf_dump_start_timestamp_RW - and FPGA_tbuf_dump_end_timestamp_RW. Forced to 0 if requested tange is not in buffer. - Each packet contains c_tbuf_nof_samples_per_packet = 2000 raw samples. This corresponds - to c_tbuf_nof_samples_per_packet * T_adc = 2000 * 5 ns = 10 us or 2000 * 6.25 ns = 12.5 us - of raw sample data per packet. Hence it takes about 100 packets to transport 1 ms of - raw sample data for one antenna input, and about 100000 packets to transport 1 s. + Amount of packets that will be dumped per antenna input for range that is requested via + FPGA_tbuf_dump_start_timestamp_RW and FPGA_tbuf_dump_end_timestamp_RW. Forced to 0 if + requested range is not in buffer. . FPGA_tbuf_dump_enable_RW[pn][ai] FPGA_tbuf_dump_enable_R[pn][ai] = value of FPGA_tbuf_dump_enable_RW in SDPFW True = start / keep dumping packets for the requested ai, dumping stops when all packets have been dumped, it is not necessary to explicitely stop the dumping. - when busy then FPGA_tbuf_memory_read_nof_packets_R increments until it reaches value of - FPGA_tbuf_dump_nof_packets_R - - when finished then FPGA_tbuf_memory_remaining_nof_packets_R = 0 + A_en * FPGA_tbuf_dump_nof_packets_R + - when finished then FPGA_tbuf_memory_read_nof_packets_R == FPGA_tbuf_dump_nof_packets_R False = RW: stop dumping packets immediately, R: dumping has stopped SDPFW: - loops over one or multiple ai @@ -338,23 +334,6 @@ The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW y - takes care that only one global pn is selected at a time to avoid 10GbE link overload - loops global pn, ai via FPGA_tbuf_dump_enable_RW - . FPGA_tbuf_memory_remaining_nof_packets_R[pn][ai] - Number of packets that still need to be dumped, - starting at reg_dump_nof_pages_RW and then decrementing down to 0, when - FPGA_tbuf_dump_enable_RW is True. If it does not reach 0 then there is something wrong in SDPFW. - - FPGA_tbuf_memory_read_nof_packets_R[pn][ai] - Number of packets that have been read so far - FPGA_tbuf_memory_dropped_nof_packets_R[pn][ai] - Number of packets that have been read and had a CRC error, - count per ai, because each ai has own CRC - - FPGA_tbuf_memory_dumped_nof_packets_R[pn][ai] - Number of packets that have been dumped, packets with a read error are not passed on for dump - = FPGA_tbuf_memory_read_nof_packets_R - FPGA_tbuf_memory_dropped_nof_packets_R, - - Not: FPGA_tbuf_clear_total_counts_RW[pn] --> clear all TBuf total counts in pn, - total count not needed if MP is read after every dump Not: FPGA_tbuf_dump_enable_RW[pn] -- antenna index Not: Fixed buffer sizes, so no need to allocate pages per ai. @@ -369,12 +348,9 @@ The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW y Enable output on the pn at end of ring that has interface to CEP. * 10GbE output monitor: - . FPGA_tbuf_output_clear_counts_RW[pn] - Use dp_strobe_total_count with MM clear register rather than in_clr = revt(reg_output_enable_RW) and - rather then dp_bsn_monitor with sync = revt(reg_output_enable_RW). - . FPGA_tbuf_output_nof_packets_R[pn] - Number of dump packets that have been output while FPGA_tbuf_output_enable_RW = True, - reset to 0 when FPGA_tbuf_output_clear_counts_RW = True event + . FPGA_tbuf_output_nof_packets_R[pn] + Number of dump packets that have been output while FPGA_tbuf_output_enable_RW = True, + reset to 0 when FPGA_tbuf_clear_counts_RW = True event * Memory DDR4: With streaming use of io_ddr then the dvr_wr_flush_en = '0' (so ctlr_wr_flush_en = 0 in MP), because @@ -392,14 +368,14 @@ The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW y ctlr_rst_out_i & ctlr_wr_flush_en & ctlr_tech_miso.waitrequest_n & ctlr_tech_miso.done, c_mem_reg_dat_w); Use ddr_ prefix for DDR4 in slot I, in case in future DDR4 slot II is also used, then use e.g. ddr_ii for that module. - . FPGA_ddr_calibrated_R = ctlr_tech_miso.done = cal_ok && !cal_fail --> + . FPGA_tbuf_ddr_calibrated_R = ctlr_tech_miso.done = cal_ok && !cal_fail --> True is ddr4 memory is available, False is ddr4 calibration failed or ddr4 not present. - . FPGA_ddr_wr_fifo_full_R = wr_fifo_full_reg, True if write FIFO to ddr4 memory got full since last MP read, else False. Should remain False. - . FPGA_ddr_rd_fifo_full_R = rd_fifo_full_reg, True if read FIFO from ddr4 memory got full since last MP read, else False. Should remain False. - . FPGA_ddr_wr_fifo_usedw_R = ctlr_wr_fifo_usedw, current fill level of write FIFO to ddr4 memory in number of 512b words, should be 0 when not recording - . FPGA_ddr_rd_fifo_usedw_R = ctlr_rd_fifo_usedw, current fill level of read FIFO from ddr4 memory in number of 512b words, should be 0 when not dumping + . FPGA_tbuf_ddr_wr_fifo_full_R = wr_fifo_full_reg, True if write FIFO to ddr4 memory got full since last MP read, else False. Should remain False. + . FPGA_tbuf_ddr_rd_fifo_full_R = rd_fifo_full_reg, True if read FIFO from ddr4 memory got full since last MP read, else False. Should remain False. + . FPGA_tbuf_ddr_wr_fifo_usedw_R = ctlr_wr_fifo_usedw, current fill level of write FIFO to ddr4 memory in number of 512b words, should be 0 when not recording + . FPGA_tbuf_ddr_rd_fifo_usedw_R = ctlr_rd_fifo_usedw, current fill level of read FIFO from ddr4 memory in number of 512b words, should be 0 when not dumping There is no MP for drv_miso.done, could show hanging io_ddr_driver. All other fields in drv_miso are unused ('X'). @@ -411,7 +387,7 @@ The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW y - ctlr_tech_miso.waitrequest_n ? could show hanging io_ddr_driver or refresh cycles * Memory buffer: - REG_TBUF_RAW new in node_sdp_transient_buffer_raw.vhd + REG_TBUF new in node_sdp_transient_buffer_raw.vhd . reg_record_all_RW = FPGA_tbuf_reg_record_all_RW True = record all antenna inputs, False = record only half of the antenna inputs, the once that have even index. @@ -420,7 +396,7 @@ The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW y False = pn is frozen Constants, signals: - . c_tbuf_nof_samples_per_page = c_tbuf_nof_samples_per_packet = 2000 + . c_sdp_tbuf_nof_samples_per_block = 2000 . nof_ddr_words_per_page = 657 or 329, depends on reg_record_enable_RW SDPTR: ddr_nof_pages = floor(ddr_gigabytes * 1024**3 / (nof_ddr_words_per_page * ctrl_nof_bytes_per_ddr_word) . Moet SDPTR dit weten, heeft SDPTR dit nodig? --> Nee want SPDTR hoeft alleen pages/blocks/packets @@ -443,7 +419,6 @@ The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW y RSN of block at reg_recorded_last_page_R . reg_dump_start_page_RW --> index of start page to dump, same for all ai in pn - reg_dump_start_rsn_RW --> RSN of reg_dump_start_page_RW reg_dump_nof_pages_RW Number of pages = packets to dump, starting at reg_dump_start_page_RW, Use nof_pages instead of index of last_page, to be able to set 0 pages, same for all ai in pn @@ -454,49 +429,49 @@ The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW y dump_end_rsn = FPGA_tbuf_dump_end_timestamp_RW / T_adc # Determine start page that includes start RSN - offset_page = floor((dump_start_rsn - reg_recorded_first_rsn_R) / c_tbuf_nof_samples_per_page) + offset_page = floor((dump_start_rsn - reg_recorded_first_rsn_R) / c_sdp_tbuf_nof_samples_per_block) if offset_page < 0: offset_page = 0 dump_start_page = reg_recorded_first_page_R + offset_page # Determine end page that includes end RSN - offset_page = ceil((dump_end_rsn - reg_recorded_last_rsn_R) / c_tbuf_nof_samples_per_page) + offset_page = ceil((dump_end_rsn - reg_recorded_last_rsn_R) / c_sdp_tbuf_nof_samples_per_block) if offset_page > 0: offset_page = 0 dump_end_page = reg_recorded_last_page_R + offset_page dump_nof_pages = dump_end_page - dump_start_page + 1 - if dump_nof_pages > 0: + if dump_nof_pages < 0: dump_nof_pages = 0 # Set packets that will be dumped by SDPTR - reg_dump_start_page_RW = dump_start_page + reg_dump_start_page_RW = dump_start_page % (page_max + 1) reg_dump_nof_pages_RW = dump_nof_pages - . reg_memory_read_nof_packets_R[ai] + . reg_memory_read_nof_packets_R Number of packets that have been read so far - reg_memory_read_nof_crc_errors_R[ai] + reg_memory_read_nof_crc_errors_R Number of packets that have been read and were dropped because they had a CRC error, count per ai, because each ai has own CRC - reg_memory_read_nof_rsn_errors_R[ai] + reg_memory_read_nof_rsn_errors_R Number of packets that have been read and were dropped because they had a RSN error, count per ai, because RSN is reread for each ai - #reg_memory_remaining_nof_packets_R[ai] - # Number of packets that still need to be read, - # = reg_dump_nof_pages_RW - reg_memory_read_nof_packets_R --> Daarom niet nodig - reg_memory_dumped_nof_packets_R[ai] + reg_memory_dumped_nof_packets_R Number of packets that have been read correctly and were passed on to ring - = reg_memory_read_nof_packets_R - reg_memory_read_nof_crc_errors_R - reg_memory_read_nof_rsn_errors_R - FPGA_tbuf_memory_clear_counts_RW[pn] - Use dp_strobe_total_count with MM clear register, rather than in_clr = revt(reg_dump_enable_RW) - to clear all reg_memory_* counters. - . FPGA_tbuf_memory_read_nof_packets_R[pn][ai] = reg_memory_read_nof_packets_R - . FPGA_tbuf_memory_dropped_nof_packets_R[pn][ai] = reg_memory_read_nof_crc_errors_R + reg_memory_read_nof_rsn_errors_R - . FPGA_tbuf_memory_remaining_nof_packets_R[pn][ai] = reg_memory_remaining_nof_packets_R - . FPGA_tbuf_memory_dumped_nof_packets_R[pn][ai] = reg_memory_dumped_nof_packets_R + FPGA_tbuf_clear_counts_RW[pn] + Use dp_strobe_total_count with MM clear register, rather than in_clr = revt(reg_dump_enables_RW) + to clear all counters. + Typically clear counters before enabling a new dump. + Use central counters in dp_strobe_total_count. Use one counter for all ai, + because counting per ai is more complex to use and not needed. + . FPGA_tbuf_memory_read_nof_packets_R[pn] = reg_memory_read_nof_packets_R + . FPGA_tbuf_memory_read_nof_crc_errors_R[pn] = reg_memory_read_nof_crc_errors_R + . FPGA_tbuf_memory_read_nof_rsn_errors_R[pn] = reg_memory_read_nof_rsn_errors_R + . FPGA_tbuf_memory_dumped_nof_packets_R[pn] = reg_memory_dumped_nof_packets_R + . FPGA_tbuf_output_nof_packets_R[pn] = reg_output_nof_packets_R - . reg_dump_enable_RW[ai] + . reg_dump_enables_RW[ai] True = start / keep dumping packets for the requested ai, False = stop dumping packets immediately When True maintain reg_memory counters, when False clear reg_memory counters to 0. @@ -553,8 +528,8 @@ The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW y read FPGA_tbuf_recorded_first_timestamp_R[pn] to know recorded time interval. * Clear counts: - . write FPGA_tbuf_output_clear_counts_RW = True to clear FPGA_tbuf_output_nof_packets_R - . write FPGA_tbuf_memory_clear_counts_RW = True to clear FPGA_tbuf_memory_*_R counts + . write FPGA_tbuf_clear_counts_RW = True to clear FPGA_tbuf_output_nof_packets_R + and FPGA_tbuf_memory_*_R counts * Prepare dump: . write FPGA_tbuf_dump_start_timestamp_RW to request start time of dump interval @@ -563,37 +538,40 @@ The CP FPGA_beamlet_output_nof_beamlets_RW is not supported in SDPTR and SDPFW y . read FPGA_tbuf_dump_end_timestamp_R to check actual end time of dump interval . read FPGA_tbuf_dump_nof_packets_R to know how many packets will be dumped per ai - * Do dump: + * Start dump: for pn in range(N_pn): . write FPGA_tbuf_dump_enable_RW = True to start dumping from pn . read FPGA_tbuf_dump_enable_R to wait until dumping has started in pn, - for enabled ai in range(A_pn): - . read FPGA_tbuf_memory_read_nof_packets_R[ai] == FPGA_tbuf_dump_nof_packets_R and - read FPGA_tbuf_memory_remaining_nof_packets_R[ai] == 0 to wait until dumping has - finished for all ai. If it does not finish, then timeout break because there is - something wrong in SDPFW. - for enabled ai in range(A_pn): - . read FPGA_tbuf_memory_dropped_nof_packets_R[ai] to monitor nof dropped packets due to memory errors - . read FPGA_tbuf_memory_dumped_nof_packets_R[ai] to monitor nof packets that were put on ring + . read FPGA_tbuf_memory_read_nof_packets_R == A_en * FPGA_tbuf_dump_nof_packets_R + to wait until dumping has finished for all enabled ai. If it does not finish, + then timeout break because there is something wrong in SDPFW. + . read FPGA_tbuf_memory_dumped_nof_packets_R to monitor nof packets that were + put on to the ring. In case of dump timeout: . write FPGA_tbuf_dump_enable_RW = False to force stop dumping from pn * Monitor dump results: - . read packet counts (or use for ai loop instead of sum): - N_remaining = sum(FPGA_tbuf_memory_remaining_nof_packets_R[pn][ai]) - N_read = sum(FPGA_tbuf_memory_read_nof_packets_R[pn][ai]) - N_dropped = sum(FPGA_tbuf_memory_dropped_nof_packets_R[pn][ai]) - N_dumped = sum(FPGA_tbuf_memory_dumped_nof_packets_R[pn][ai]) - N_lost = N_read - N_dropped - N_dumped - N_output = FPGA_tbuf_output_nof_packets_R - . Expected results: - . N_remaining == 0, else something went wrong in SDPFW - . N_dumped = N_read - N_dropped, else something went wrong in SDPFW - . N_dropped == 0, else packets got lost due to DDR4 memory access errors - . N_lost == 0, else packets got lost on the ring --> check ring MP for details - . N_read == FPGA_tbuf_dump_nof_packets_R * nof active ai, else something went wrong in SDPFW - . N_output == N_read - N_dropped - N_lost, else something went wrong in SDPFW + for pn in range(N_pn): + . read packet counts: + N_read[pn] = FPGA_tbuf_memory_read_nof_packets_R[pn] + N_crc_err[pn] = FPGA_tbuf_memory_read_nof_crc_errors_R[pn] + N_rsn_err[pn] = FPGA_tbuf_memory_read_nof_rsn_errors_R[pn] + N_dumped[pn] = FPGA_tbuf_memory_dumped_nof_packets_R[pn] + N_dropped[pn] = N_read[pn] - N_dumped[pn] + N_lost[last pn] = N_output - sum(N_dumped[pn]) + N_output[pn] = FPGA_tbuf_output_nof_packets_R[pn] + . Expected results: + . N_crc_err == 0, else something went wrong in SDPFW at DDR4 interface + . N_rsn_err == 0, else something went wrong in SDPFW at DDR4 interface or with RSN + . N_dropped == 0, else packets were dropped due to DDR4 interface errors or RSN timing errors + . N_lost == 0, else packets got lost on the ring --> check ring MP for details + . N_read == FPGA_tbuf_dump_nof_packets_R * A_en, else something went wrong in SDPFW + . N_output == N_read - N_dropped - N_lost, else something went wrong in SDPFW + + * Disable dump: + for pn in range(N_pn): + . write FPGA_tbuf_dump_enable_RW = False to stop dumping 7) TBuf ICD STAT/SDP-CEP @@ -701,9 +679,19 @@ Het is gewoon een json-bestandje dat je naast een databestand met alleen complex - Verify TBuf within SDP on HW - Integration effort with SC + +The selection between recording all or half of the antennas per FPGA has the following impact: +- shows up as if-then-else selector function at various places in the code, also in the read out because the memory page size depends on the selection +- requires conditioning, to ensure that the selection does not cause the implementation to hang if it changes at any moment +- requires verification of both selections +- requires decision making at higher software, configuration or user control levels, to decide which selection to use + + 11) Transient detection (TDet) Design - no self triggering yet for MVP +- Pulse detection messages contain event info and timestamp, which is still + useful for ligthning science even without dumping buffer. - will use Hilbert transform of real input and > 30MHz BPF https://nl.mathworks.com/help/signal/ug/single-sideband-modulation-via-the-hilbert-transform.html @@ -726,8 +714,6 @@ Het is gewoon een json-bestandje dat je naast een databestand met alleen complex radio triggered message once per hour -Hoi Reinier, kun je deze voor L2SDP-18 voor mij reviewen: https://git.astron.nl/desp/hdl/-/merge_requests/324 - -en ook dit doc al een keer doorlezen. Het is nog niet af dat doe ik in L2SDP-942, maar misschien heb je al een opmerking: - -https://support.astron.nl/confluence/pages/viewpage.action?spaceKey=L2M&title=L5+SDPFW+Design+Document%3A+Transient+buffer+raw+data +12) Cosmic ray +* Katie Mulrey (RU, Cosimc Ray group) +* Stijn Buitink (VUB, Cosmic Ray group) -- GitLab