Skip to content
Snippets Groups Projects
Select Git revision
  • 3480f10053d2b182049d1c6e86175061fb4e56e0
  • master default protected
  • fix_sas_id_in_filename
  • releases/v0.5.17.tim_survey protected
  • compress_tim_survey_no_metadata_compression
  • v0.5.x
  • juelich_0_5_18
  • releases/v0.6.0.tim_survey protected
  • compress_tim_survey
  • releases/v0.5.18 protected
  • expose_elevation_for_parset
  • releases/v0.5.17 protected
  • releases/v0.6.0 protected
  • releases/v0.5.19 protected
  • releases/v0.5.16 protected
  • releases/v0.5.15 protected
  • nico_testing_juelich
  • nightly_build_test
  • releases/v0.5.14 protected
  • releases/v0.5.13 protected
  • releases/v0.5.12 protected
  • v0.5.17.tim_survey
  • v0.6.0.tim_survey
  • v0.5.18
  • v0.5.17
  • v0.6.0
  • v0.5.16
  • v0.5.15
  • v0.5.14
  • v0.5.13
  • v0.5.12
  • v0.5.11
  • v0.5.10
  • v0.5.9
  • v0.5.8
  • v0.5.7
  • v0.5.6
  • v0.5.5
  • v0.5.4
  • v0.5.3
  • v0.5.2
41 results

imaging_compress_pipeline

  • Open with
  • Download source code
  • Your workspaces

      A workspace is a virtual sandbox environment for your code in GitLab.

      No agents available to create workspaces. Please consult Workspaces documentation for troubleshooting.

  • Roy de Goei's avatar
    SDC-679: remove quote in docker command
    Roy de Goei authored
    3480f100
    History

    Lofar imaging compression pipeline

    This is a CWL workflow used to compress Lofar imaging data.

    Workflow steps

    There are two workflows defined. An graphical overview is given below.

    download_and_compress_pipeline.cwl

    This workflow is used by the LDV framework.

    Download pipeline Overview

    compress_pipeline.cwl

    Compress pipeline Overview

    • identify known issues and apply them if necessary. It uses in place updates to prevent copying the measurement set
      • in place update does not work well with CWL conditional steps, so a small bash script is used which either calls the fixing script or does nothing depending if the specific issue for the step was found
    • Compress the measurement set with Dysco
    • Produces inspection plots and collects metrics to quantify the amount of missing/flagged data

    Requirements

    • CWL v1.2 compatible runner (e.g. cwltool/toil)
    • Docker

    Docker images

    • git.astron.nl:5000/ldv/ldv-images/lofar-legacy:latest
    • git.astron.nl:5000/ldv/ldv-images/lofar-ms-software:latest Docker images will be fetched the first time the workflow is used and be converted into a singularity image (sif file)

    Running the workflow

    In the repositories there are two workflows. One is capable to process the data as described in section workflow steps and can be executed as following

    # Run the workflow
    cwltool compress_pipeline.cwl [--flag_autocorrelation] --msin MEASUREMENT_SET

    Another workflow, that is meant to be executed by the LDV infrastructure, takes as an input instead of a measurement set a SURL link of the data. Such a workflow can be execute with the command

    # Run the workflow
    cwltool download_and_compress_pipeline.cwl [--flag_autocorrelation] --surls [list of surl to process] 

    License

    See LICENSE