Skip to content
Snippets Groups Projects
Commit 8190b355 authored by Timo Millenaar's avatar Timo Millenaar
Browse files

Add example images to readme

parent af6ff238
No related branches found
No related tags found
1 merge request!7Add example images to readme
...@@ -9,14 +9,14 @@ There are two workflows defined. An graphical overview is given below. ...@@ -9,14 +9,14 @@ There are two workflows defined. An graphical overview is given below.
### download_and_compress_pipeline.cwl ### download_and_compress_pipeline.cwl
This workflow is used by the LDV framework. This workflow is used by the LDV framework.
![Download pipeline Overview](download_and_compress_pipeline.png) ![Download pipeline Overview](docs/images/download_and_compress_pipeline.png)
The data in this pipeline is first fetched, then the 'compress' step is ran. The data in this pipeline is first fetched, then the 'compress' step is ran.
The compress step runs 'compress_pipeline.cwl', which does more than only compression. The steps take are mentioned in the section below. The compress step runs 'compress_pipeline.cwl', which does more than only compression. The steps take are mentioned in the section below.
### compress_pipeline.cwl ### compress_pipeline.cwl
![Compress pipeline Overview](compress_pipeline.png) ![Compress pipeline Overview](docs/images/compress_pipeline.png)
Part of this pipeline identifies known issues and applies corrections if possible. It uses in place updates to prevent copying the measurement set. Part of this pipeline identifies known issues and applies corrections if possible. It uses in place updates to prevent copying the measurement set.
...@@ -38,12 +38,22 @@ There are also some time ranges that cannot be solved. These issues include: ...@@ -38,12 +38,22 @@ There are also some time ranges that cannot be solved. These issues include:
- STATION_SENSITIVITY_ISSUE - STATION_SENSITIVITY_ISSUE
- DELAY_COMPENSATION_ISSUE - DELAY_COMPENSATION_ISSUE
The measurement set is then compressed the with Dysco if it was not already compressed. After all measure ment sets have been compressed, inspection plots are produced and metrics to quantify the amount of missing/flagged data are collected. The measurement set is then compressed the with Dysco if it was not already compressed. After all measure ment sets have been compressed, inspection plots are produced and metrics to quantify the amount of missing/flagged data are collected.
> **_NOTE:_** Some plots may be skipped if the required metadata to make the plot is missing. This is often the case for "unspecified" datasets. > **_NOTE:_** Some plots may be skipped if the required metadata to make the plot is missing. This is often the case for "unspecified" datasets.
When a plot is skipped, this is printed to stderr as a warning and thus is shown in the ATDB log file. When a plot is skipped, this is printed to stderr as a warning and thus is shown in the ATDB log file.
# Quality metrics
The output json conatining a summary of the processing contains a set of quality flags, usually ranging between "poor", "moderate" and "good". The qua;ity is "unknown" if the (meta)data to calculate the metric is missing.
The quality can also be "N/A" if the quality metric is not relevant for the dataset, for example a quality metric indicating dataloss of critical international stations will be "N/A" if only Dutch stations are present.
A detailed outline of the quality metrics and their meaning can be found on the following confluence page: https://support.astron.nl/confluence/pages/viewpage.action?spaceKey=LDV&title=Imaging+compression+workflow+metrics+v.2
Four plots are created, examples of these are:
![Pointing image](docs/images/3C295_800576_pointing.png)
![Dataloss image](docs/images/3C295_800576_dataloss_antennae.png)
![Percflagged image](docs/images/3C295_800576_percflagged_antennae.png)
![Noise image](docs/images/3C295_800576_noise_antennae.png)
The examples are taken from the following run: https://sdc-dev.astron.nl:5554/atdb/task_details/260179/1
## Requirements ## Requirements
......
docs/images/3C295_800576_dataloss_antennae.png

27.5 KiB

docs/images/3C295_800576_noise_antennae.png

27.8 KiB

docs/images/3C295_800576_percflagged_antennae.png

46 KiB

docs/images/3C295_800576_pointing.png

58.4 KiB

docs/images/compress_pipeline.png

65.4 KiB

docs/images/download_and_compress_pipeline.png

58.1 KiB

0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment