Lofar imaging compression pipeline
This is a CWL workflow used to compress Lofar imaging data.
Workflow steps
There are two workflows defined. An graphical overview is given below.
download_and_compress_pipeline.cwl
This workflow is used by the LDV framework.
compress_pipeline.cwl
- identify known issues and apply them if necessary. It uses in place updates to prevent copying the measurement set
- in place update does not work well with CWL conditional steps, so a small bash script is used which either calls the fixing script or does nothing depending if the specific issue for the step was found
- Compress the measurement set with Dysco
- Produces inspection plots and collects metrics to quantify the amount of missing/flagged data
Requirements
- CWL v1.2 compatible runner (e.g. cwltool/toil)
- Docker
Docker images
- git.astron.nl:5000/ldv/ldv-images/lofar-legacy:latest
- git.astron.nl:5000/ldv/ldv-images/lofar-ms-software:latest Docker images will be fetched the first time the workflow is used and be converted into a singularity image (sif file)
Running the workflow
In the repositories there are two workflows. One is capable to process the data as described in section workflow steps and can be executed as following
# Run the workflow
cwltool compress_pipeline.cwl [--flag_autocorrelation] --msin MEASUREMENT_SET
Another workflow, that is meant to be executed by the LDV infrastructure, takes as an input instead of a measurement set a SURL link of the data. Such a workflow can be execute with the command
# Run the workflow
cwltool download_and_compress_pipeline.cwl [--flag_autocorrelation] --surls [list of surl to process]
License
See LICENSE