diff --git a/README.md b/README.md index 4a53dbbef2076cefc04266b1f845417a154fc624..21e8a96147ff275d9afbd28af56c9687445c0b62 100644 --- a/README.md +++ b/README.md @@ -11,15 +11,38 @@ This workflow is used by the LDV framework.  +The data in this pipeline is first fetched, then the 'compress' step is ran. +The compress step runs 'compress_pipeline.cwl', which does more than only compression. The steps take are mentioned in the section below. + ### compress_pipeline.cwl  -- identify known issues and apply them if necessary. It uses in place updates to prevent copying the measurement set - - in place update does not work well with CWL conditional steps, so a small bash script is used which either calls the fixing script or does nothing depending if the specific issue for the step was found -- Compress the measurement set with Dysco -- Produces inspection plots and collects metrics to quantify the amount of missing/flagged data +Part of this pipeline identifies known issues and applies corrections if possible. It uses in place updates to prevent copying the measurement set. + +> **_NOTE:_** In-place update does not work well with CWL conditional steps, so a small bash script is used which either calls the fixing script or does nothing depending if the specific issue for the step was found + +The checks that get performed are: +- abort if the target is the Sun +- run aoflagger if the input is raw data +- apply corrections based on the time of observation. These can be: + - FIX_ANTENNA_TABLE + - FIX_WEIGHT_SPECTRUM + - FIX_BROKEN_TILES + - FIX_STATION_ADDER + +There are also some time ranges that cannot be solved. These issues include: +- INACCURATE_FLAGGING_LBA +- FAULTY_LBA_CALIBRATION_TABLES +- STATION_SENSITIVITY_ISSUE +- DELAY_COMPENSATION_ISSUE + + +The measurement set is then compressed the with Dysco if it was not already compressed. After all measure ment sets have been compressed, inspection plots are produced and metrics to quantify the amount of missing/flagged data are collected. + +> **_NOTE:_** Some plots may be skipped if the required metadata to make the plot is missing. This is often the case for "unspecified" datasets. +When a plot is skipped, this is printed to stderr as a warning and thus is shown in the ATDB log file. ## Requirements