Lofar imaging compression pipeline

This is a CWL workflow used to compress Lofar imaging data.

Workflow steps

• identify known issues and apply them if necessary. It uses in place updates to prevent copying the measurement set
  • in place update does not work well with CWL conditional steps, so a small bash script is used which either calls the fixing script or does nothing depending if the specific issue for the step was found
• Compress the measurement set with Dysco
• Produces inspection plots and collects metrics to quantify the amount of missing/flagged data

Requirements

• CWL v1.2 compatible runner (e.g. cwltool/toil)
• Docker

Docker images

• astronsdc/lofar-legacy
• astronsdc/lofar-ms-software

Running the workflow

In the repositories there are two workflows. One is capable to process the data as described in section workflow steps and can be executed as following

# Run the workflow
cwltool compress_pipeline.cwl [--flag_autocorrelation] --msin MEASUREMENT_SET

Another workflow, that is meant to be executed by the LDV infrastructure, takes as an input instead of a measurement set a SURL link of the data. Such a workflow can be execute with the command

# Run the workflow
cwltool download_and_compress_pipeline.cwl [--flag_autocorrelation] --surls [list of surl to process] 

License

See LICENSE