Skip to content
Snippets Groups Projects

Home

Last edited by Matthijs van der Wild

The Low-Frequency Array Very Long Baseline Interferometry (LOFAR-VLBI) pipeline focusses on calibration and imaging with all the international stations of the International LOFAR Telescope. It looks and solves for delays using an in-field calibrator within the field of view of the pointing. The pipeline primarily provides:

  • solution tables written in a file in the hierarchical data format (HDF, version 5), referred to as an h5parm,
  • a self-calibrated, corrected dataset for the in-field calibrator.
  • concatenated bands of target data that have been corrected for phase and instrumentation errors.

From there the pipeline can split out smaller datasets in the desired target direction(s) within the field of view.

❗ NOTE: Before processing it is important to consider the instructions on the lofar-vlbi Read the Docs page.

TL;DR

A quick way to start processing is described in the Quickstart!

Installation

The following absolute paths are assumed:

  • <installation directory> will contain all the necessary LOFAR software,
  • <input directory> will contain the data to be processed, including
    • the observation data in MeasurementSet (MS) format,
    • solution tables generated by the LOFAR Initial Calibration (LINC) pipeline,
    • a catalogue containing a suitable delay calibrator in CSV format,
  • <work directory> will contain the intermediate files produced during pipeline operation,
  • <output directory> will contain the data products produced by the pipeline.
export INSTALL_DIR=<installation directory>
export INPUT_DIR=<input directory>
export WORK_DIR=<work directory>
export OUTPUT_DIR=<output directory>

The pipeline workflow files and scripts can be The LOFAR-VLBI pipeline and necessary LOFAR software can then be installed as follows:

git clone https://git.astron.nl/RD/VLBI-cwl.git ${INSTALL_DIR}/vlbi
git clone https://github.com/jurjen93/lofar_helpers.git ${INSTALL_DIR}/lofar_helpers
git clone https://github.com/rvweeren/lofar_facet_selfcal.git ${INSTALL_DIR}/lofar_facet_selfcal

The pipeline is implemented in the Common Workflow Language (CWL). A CWL implementation must be installed on the system before the pipeline can be run. A list of implementations is available on the CWL website, although LOFAR-VLBI pipeline officially supports only cwltool or toil-cwl-runner.

python3 -m pip install cwltool toil[cwl]

The software packages above depend on various software products. These can either be installed manually, or used through a software container. Both methods are described below, but it is recommended to run the LOFAR-VLBI pipeline within a container.

Manual installation

The pipeline depends on the following external software components and their dependencies:

These have to be installed either through your package manager or from their respective repositories.

Containerisation

In case Singularity/Apptainer is used, it is recommended to use Frits' LOFAR Containers (FLoC):

wget https://lofar-webdav.grid.sara.nl/software/shub_mirror/tikk3r/lofar-grid-hpccloud/intel/lofar_sksp_v4.2.2_sandybridge_sandybridge_ddf_cuda.sif
❗ NOTE: The container version displayed above may be out of date. Always check the latest version on the FLoC website.

The pipeline can be run inside the singularity container by running

singularity <singularity options> cwltool --no-container <workflow> <configuration file>

Refer to the documentation of your CWL implementation for details.

It is important that the VLBI installation directory has been properly mounted. In addition, the VLBI installation directory should be added to the image's $PATH and $PYTHONPATH variables. See the Singularity/Apptainer documentation for details.

Configuration

Input parameters for workflows are stored in configuration files, and should be be written in the YAML or JSON formats. Introductions to these formats are available on at Learn YAML in Y minutes and Learn JSON in Y minutes, respectively.

A configuration file should, at a minimum, contain

  • the observation data in MeasurementSet (MS) format,
  • solution tables generated by the LOFAR Initial Calibration (LINC) pipeline in an HDF5 format,
  • a catalogue containing a suitable delay calibrator at a 6″ resolution, in CSV format,

A suitable pipeline input file can be generated using the VLBI pipeline's generate_input.sh:

bash ${INSTALL_DIR}/vlbi/scripts/generate_input.sh ${INPUT_DIR} ${INSTALL_DIR}
❗ NOTE: generate_input.sh assumes that the required input data is accessible in ${INPUT_DIR} and that lofar_facet_selfcal and lofar_helpers are accessible in ${INSTALL_DIR}, as described in Installation.

This will generate input.yaml in the current working directory. Optional input parameters, such as flagged baselines or solutions generated by the Direction-Dependent Faceting (DDF) pipeline should be added to this file by hand.

Pipeline overview

The LOFAR-VLBI pipeline builds on the output of the LINC and (optionally) DDFacet pipelines. In particular, it is assumed that LINC has produced sane solution files. See the LINC documentation for details on how to run and validate the initial calibration.

Delay Calibration

The delay calibration workflow comprises the following steps and subroutines:

  • Setup:
    • Applies the LINC calibrator and target solutions,
    • Predicts skymodel data and clips A-team contributions.
  • Concatenate-flag:
    • Reduces the data set by concatenation of subbands into groups,
    • Performs flagging on the data of the resulting measurement sets.
  • Phaseup-concatenate:
    • Phase shifts data to the in-field calibrator,
    • Performs direction independent calibration of the international array.
  • Store-logs: Collects all the logfiles generated by the previous subroutines.

More details can be found on the pages of each subroutine.

Split Directions

The split directions workflow comprises the following steps and subroutines:

  • Split parset
    • Generates a paramaterset file for DP3 to apply solutions and generate MeasurementSets phase-shifted to imaging directions.
  • DP3 target phaseup
    • Applies delay calibrator solutions and creates MeasurementSets of phase-shifted data.
  • Order by direction
    • Collects MeasurementSets by imaging direction.
  • Sort concatmap
    • Sorts subbands into a given number of regularly spaced frequency groups.
  • Flatten array
    • Flattens a nested array of Any type into a regular array.
  • Concatenation
    • Selects MeasurementSets from the input data based on predetermined frequency grouping.
  • Target selfcal
    • Performs self-calibration on the target source.

More details can be found on the pages of each subroutine.

Support

The LOFAR-VLBI pipeline is a continuously maintained and developed software package. Bugs or other issues can be reported at the pipeline's Git repository.

Acknowledgements

The LOFAR-VLBI pipeline and its scripts were originally developed by:

  • Leah Morabito
  • Neal Jackson
  • Sean Mooney
  • Alexander Drabent
  • Frits Sweijen
  • Etienne Bonnassieux

The pipeline makes use of the Long Baseline Calibrator Survey (LBCS):

LBCS: The LOFAR Long-Baseline Calibrator Survey
Jackson, N., et al.,
A&A 595, A86 (2016),
DOI: 10.1051/0004-6361/201629016,
arXiv:1608.02133 [astro-ph.IM]

The development of the pipeline has been supported by SURF and EGI-ACE. EGI-ACE has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 101017567.

Citation

The LOFAR-VLBI pipeline is described in Morabito et al. (2019). If your work uses the pipeline, please cite the paper above as:

Sub-arcsecond imaging with the International LOFAR Telescope - I. Foundational calibration strategy and pipeline
L. K.  Morabito, et al.,
A&A 658 A1 (2022),
DOI: 10.1051/0004-6361/202140649,
arXiv:2108.07283 [astro-ph.IM]

License

The LOFAR-VLBI pipeline is licensed under the GNU General Public License v3.0. A copy of the license is available in the pipeline's Git repository.

Comments

    Please register or sign in to add a comment.