From 2ad48b6066e6ed70958b9a701a100d7bd2342ef8 Mon Sep 17 00:00:00 2001 From: Timo Millenaar <tmillenaar@gmail.com> Date: Thu, 27 Jun 2024 11:18:14 +0200 Subject: [PATCH] Mention applied fixes in README.md --- README.md | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 4a53dbb..21e8a96 100644 --- a/README.md +++ b/README.md @@ -11,15 +11,38 @@ This workflow is used by the LDV framework.  +The data in this pipeline is first fetched, then the 'compress' step is ran. +The compress step runs 'compress_pipeline.cwl', which does more than only compression. The steps take are mentioned in the section below. + ### compress_pipeline.cwl  -- identify known issues and apply them if necessary. It uses in place updates to prevent copying the measurement set - - in place update does not work well with CWL conditional steps, so a small bash script is used which either calls the fixing script or does nothing depending if the specific issue for the step was found -- Compress the measurement set with Dysco -- Produces inspection plots and collects metrics to quantify the amount of missing/flagged data +Part of this pipeline identifies known issues and applies corrections if possible. It uses in place updates to prevent copying the measurement set. + +> **_NOTE:_** In-place update does not work well with CWL conditional steps, so a small bash script is used which either calls the fixing script or does nothing depending if the specific issue for the step was found + +The checks that get performed are: +- abort if the target is the Sun +- run aoflagger if the input is raw data +- apply corrections based on the time of observation. These can be: + - FIX_ANTENNA_TABLE + - FIX_WEIGHT_SPECTRUM + - FIX_BROKEN_TILES + - FIX_STATION_ADDER + +There are also some time ranges that cannot be solved. These issues include: +- INACCURATE_FLAGGING_LBA +- FAULTY_LBA_CALIBRATION_TABLES +- STATION_SENSITIVITY_ISSUE +- DELAY_COMPENSATION_ISSUE + + +The measurement set is then compressed the with Dysco if it was not already compressed. After all measure ment sets have been compressed, inspection plots are produced and metrics to quantify the amount of missing/flagged data are collected. + +> **_NOTE:_** Some plots may be skipped if the required metadata to make the plot is missing. This is often the case for "unspecified" datasets. +When a plot is skipped, this is printed to stderr as a warning and thus is shown in the ATDB log file. ## Requirements -- GitLab