diff --git a/README.md b/README.md index 21e8a96147ff275d9afbd28af56c9687445c0b62..43fb0f2b2d0fa4b05e9d48763fe5def37e737177 100644 --- a/README.md +++ b/README.md @@ -9,15 +9,14 @@ There are two workflows defined. An graphical overview is given below. ### download_and_compress_pipeline.cwl This workflow is used by the LDV framework. - +<img src="docs/images/download_and_compress_pipeline.png" alt="Download pipeline Overview" width="400"/> The data in this pipeline is first fetched, then the 'compress' step is ran. The compress step runs 'compress_pipeline.cwl', which does more than only compression. The steps take are mentioned in the section below. ### compress_pipeline.cwl - - +<img src="docs/images/compress_pipeline.png" alt="Compress pipeline Overview" width="400"/> Part of this pipeline identifies known issues and applies corrections if possible. It uses in place updates to prevent copying the measurement set. @@ -38,12 +37,27 @@ There are also some time ranges that cannot be solved. These issues include: - STATION_SENSITIVITY_ISSUE - DELAY_COMPENSATION_ISSUE - The measurement set is then compressed the with Dysco if it was not already compressed. After all measure ment sets have been compressed, inspection plots are produced and metrics to quantify the amount of missing/flagged data are collected. > **_NOTE:_** Some plots may be skipped if the required metadata to make the plot is missing. This is often the case for "unspecified" datasets. When a plot is skipped, this is printed to stderr as a warning and thus is shown in the ATDB log file. +# Quality metrics +The output json conatining a summary of the processing contains a set of quality flags, usually ranging between "poor", "moderate" and "good". The qua;ity is "unknown" if the (meta)data to calculate the metric is missing. +The quality can also be "N/A" if the quality metric is not relevant for the dataset, for example a quality metric indicating dataloss of critical international stations will be "N/A" if only Dutch stations are present. +A detailed outline of the quality metrics and their meaning can be found on the following confluence page: https://support.astron.nl/confluence/pages/viewpage.action?spaceKey=LDV&title=Imaging+compression+workflow+metrics+v.2 + +Four plots are created, examples of these are: + +<img src="docs/images/3C295_800576_pointing.png" alt="Pointing image" width="400"/> + +<img src="docs/images/3C295_800576_noise_antennae.png" alt="Noise image" width="400"/> + +<img src="docs/images/3C295_800576_percflagged_antennae.png" alt="Percflagged image" width="400"/> + +<img src="docs/images/3C295_800576_dataloss_antennae.png" alt="Dataloss image" width="400"/> + +The examples are taken from the following run: https://sdc-dev.astron.nl:5554/atdb/task_details/260179/1 ## Requirements diff --git a/docs/images/3C295_800576_dataloss_antennae.png b/docs/images/3C295_800576_dataloss_antennae.png new file mode 100644 index 0000000000000000000000000000000000000000..b77dcee8fa5308bb27d1ea0cd6916f7ca27298d5 Binary files /dev/null and b/docs/images/3C295_800576_dataloss_antennae.png differ diff --git a/docs/images/3C295_800576_noise_antennae.png b/docs/images/3C295_800576_noise_antennae.png new file mode 100644 index 0000000000000000000000000000000000000000..3ea3ed0370bb493e6cadddb2d60ca2654bfd604f Binary files /dev/null and b/docs/images/3C295_800576_noise_antennae.png differ diff --git a/docs/images/3C295_800576_percflagged_antennae.png b/docs/images/3C295_800576_percflagged_antennae.png new file mode 100644 index 0000000000000000000000000000000000000000..c716317e0f80577e9d6b8c2e28dc293a4e7e4967 Binary files /dev/null and b/docs/images/3C295_800576_percflagged_antennae.png differ diff --git a/docs/images/3C295_800576_pointing.png b/docs/images/3C295_800576_pointing.png new file mode 100644 index 0000000000000000000000000000000000000000..d0d16d3c38ce92271118799ea09d2e8ad6ec64a0 Binary files /dev/null and b/docs/images/3C295_800576_pointing.png differ diff --git a/compress_pipeline.png b/docs/images/compress_pipeline.png similarity index 100% rename from compress_pipeline.png rename to docs/images/compress_pipeline.png diff --git a/download_and_compress_pipeline.png b/docs/images/download_and_compress_pipeline.png similarity index 100% rename from download_and_compress_pipeline.png rename to docs/images/download_and_compress_pipeline.png