|
|
The _Low-Frequency Array Very Long Baseline Interferometry_ (LOFAR-VLBI) pipeline is a calibration and imaging pipeline that includes all of LOFAR’s international stations.
|
|
|
It looks and solves for delays using an in-field calibrator within the field of view of the pointing.
|
|
|
The pipeline provides:
|
|
|
|
|
|
* solution tables written in a file in the [hierarchical data format](https://www.hdfgroup.org/) (version 5), referred to as an h5parm,
|
|
|
* a self-calibrated, corrected dataset for the in-field calibrator.
|
|
|
|
|
|
The delay solutions are applied back to the original data.
|
|
|
From there the pipeline can split out smaller datasets in the desired target direction(s) within the field of view.
|
|
|
|
|
|
* Unsure what data is suitable to be processed?
|
|
|
Have a look at the [Data Requirements](data-requirements).
|
|
|
* Already made sure that your data is suitable to be processed and just want the bare minimum required to get started?
|
|
|
Skip to the [Quickstart](quickstart)!
|
|
|
|
|
|
# Installation
|
|
|
|
|
|
The following absolute paths are assumed:
|
|
|
* `<installation directory>` will contain all the necessary LOFAR software,
|
|
|
* `<input directory>` will contain the data to be processed, including
|
|
|
* the observation data in _MeasurementSet_ (MS) format,
|
|
|
* solution tables generated by the _LOFAR Initial Calibration_ (LINC) pipeline,
|
|
|
* a catalogue containing a suitable delay calibrator in CSV format,
|
|
|
* `<work directory>` will contain the intermediate files produced during pipeline operation,
|
|
|
* `<output directory>` will contain the data products produced by the pipeline.
|
|
|
```
|
|
|
export INSTALL_DIR=<installation directory>
|
|
|
export INPUT_DIR=<input directory>
|
|
|
export WORK_DIR=<work directory>
|
|
|
export OUTPUT_DIR=<output directory>
|
|
|
```
|
|
|
The pipeline workflow files and scripts can be
|
|
|
The LOFAR-VLBI pipeline and necessary LOFAR software can then be installed as follows:
|
|
|
```
|
|
|
git clone https://git.astron.nl/RD/VLBI-cwl.git ${INSTALL_DIR}/vlbi
|
|
|
git clone https://github.com/jurjen93/lofar_helpers.git ${INSTALL_DIR}/lofar_helpers
|
|
|
git clone https://github.com/rvweeren/lofar_facet_selfcal.git ${INSTALL_DIR}/lofar_facet_selfcal
|
|
|
```
|
|
|
|
|
|
The pipeline is implemented in the [Common Workflow Language](https://www.commonwl.org/) (CWL).
|
|
|
A CWL implementation must be installed on the system before the pipeline can be run.
|
|
|
A list of implementations is available on the [CWL website](https://www.commonwl.org/implementations/), although LOFAR-VLBI pipeline officially supports only `cwltool` or `toil-cwl-runner`.
|
|
|
```
|
|
|
python3 -m pip install cwltool toil[cwl]
|
|
|
```
|
|
|
The software packages above depend on various software products.
|
|
|
These can either be installed manually, or used through a software container.
|
|
|
Both methods are described below, but it is recommended to run the LOFAR-VLBI pipeline within a container.
|
|
|
|
|
|
## Manual installation
|
|
|
|
|
|
The pipeline depends on the following external software components and their dependencies:
|
|
|
|
|
|
* [DP3](https://git.astron.nl/RD/DP3)
|
|
|
* [python](https://www.python.org/) (≥3.8)
|
|
|
* [PyBDSF](https://github.com/lofar-astron/PyBDSF)
|
|
|
* [scipy](https://scipy.org/)
|
|
|
* [numpy](https://numpy.org/)
|
|
|
* [scipy](https://scipy.org/)
|
|
|
* [matplotlib](https://matplotlib.org/)
|
|
|
* [pyrap](https://github.com/tslight/pyrap)
|
|
|
* [losoto](https://github.com/revoltek/losoto)
|
|
|
* [pyvo](https://github.com/astropy/pyvo)
|
|
|
* [astropy](https://www.astropy.org/about.html)
|
|
|
* [python3-regex](https://github.com/mrabarnett/mrab-regex)
|
|
|
* [statsmodels](https://www.statsmodels.org/stable/index.html)
|
|
|
* [astroquery](https://astroquery.readthedocs.io/en/latest/)
|
|
|
* [python-magic](https://github.com/ahupp/python-magic)
|
|
|
|
|
|
These have to be installed either through your package manager or from their respective repositories.
|
|
|
|
|
|
## Containerisation
|
|
|
|
|
|
In case Singularity/Apptainer is used, it is recommended to use [_Frits' LOFAR Containers_](https://tikk3r.github.io/flocs/) (FLoC):
|
|
|
```
|
|
|
wget https://lofar-webdav.grid.sara.nl/software/shub_mirror/tikk3r/lofar-grid-hpccloud/intel/lofar_sksp_v4.2.2_sandybridge_sandybridge_ddf_cuda.sif
|
|
|
```
|
|
|
|
|
|
The pipeline can be run inside the singularity container by running
|
|
|
```
|
|
|
singularity <singularity options> cwltool --no-container <workflow> <configuration file>
|
|
|
```
|
|
|
Refer to the documentation of your CWL implementation for details.
|
|
|
|
|
|
It is important that the VLBI installation directory has been properly mounted.
|
|
|
In addition, the VLBI installation directory should be added to the image's `$PATH` and `$PYTHONPATH` variables.
|
|
|
See the Singularity/Apptainer [documentation](https://apptainer.org/docs/user/main/environment_and_metadata.html) for details.
|
|
|
|
|
|
# Configuration
|
|
|
|
|
|
Input parameters for workflows are stored in configuration files, and should be be written in the [YAML](https://yaml.org/) or [JSON](https://www.json.org/) formats.
|
|
|
Introductions to these formats are available on at [Learn YAML in Y minutes](https://learnxinyminutes.com/docs/yaml/) and [Learn JSON in Y minutes](https://learnxinyminutes.com/docs/json/), respectively.
|
|
|
|
|
|
A configuration file should, at a minimum, contain
|
|
|
* the observation data in _MeasurementSet_ (MS) format,
|
|
|
* solution tables generated by the [_LOFAR Initial Calibration_](https://git.astron.nl/RD/LINC) (LINC) pipeline,
|
|
|
* a catalogue containing a suitable delay calibrator at a 6″ resolution, in CSV format,
|
|
|
|
|
|
Assuming the setup listed in [Installation](#Installation), a suitable pipeline input file can be generated using the VLBI pipeline's `generate_input.sh`:
|
|
|
```
|
|
|
bash ${INSTALL_DIR}/vlbi/scripts/generate_input.sh ${INPUT_DIR} ${INSTALL_DIR}
|
|
|
```
|
|
|
This will generate `input.yaml` in the current working directory.
|
|
|
Optional input parameters, such as flagged baselines or solutions generated by the [_Direction-Dependent Faceting_](https://github.com/saopicc/DDFacet) (DDF) pipeline should be added to this file by hand.
|
|
|
|
|
|
# Pipeline overview
|
|
|
|
|
|
The LOFAR-VLBI pipeline builds on the output of the LINC and (optionally) DDFacet pipelines.
|
|
|
In particular, it is assumed that LINC has produced sane solution files.
|
|
|
See the LINC [documentation](https://linc.readthedocs.io/en/latest/) for details on how to run and validate the initial calibration.
|
|
|
|
|
|
### [Delay Calibration](/delay-calibration/README)
|
|
|
|
|
|
The delay calibration workflow comprises the following steps and subroutines:
|
|
|
|
|
|
* [Setup](/delay-calibration/setup):
|
|
|
* Applies the LINC calibrator and target solutions,
|
|
|
* Predicts skymodel data and clips A-team contributions.
|
|
|
* [Concatenate-flag](/delay-calibration/concatenate-flag):
|
|
|
* Reduces the data set by concatenation of subbands into groups,
|
|
|
* Performs flagging on the data of the resulting measurement sets.
|
|
|
* [Phaseup-concatenate](/delay-calibration/phaseup-concatenate):
|
|
|
* Phase shifts data to the in-field calibrator,
|
|
|
* Performs direction independent calibration of the international array.
|
|
|
* [Store-logs](/steps/collect-files): Collects all the logfiles generated by the previous subroutines.
|
|
|
|
|
|
More details can be found on the pages of each subroutine.
|
|
|
|
|
|
### [Split Directions](/split-directions/README)
|
|
|
Currently the split directions pipeline is still under development.
|
|
|
|
|
|
## Support
|
|
|
|
|
|
The LOFAR-VLBI pipeline is a continuously maintained and developed software package.
|
|
|
Bugs or other issues can be reported at the pipeline's [Git repository](https://git.astron.nl/RD/VLBI-cwl/-/issues).
|
|
|
|
|
|
## Acknowledgements
|
|
|
|
|
|
The LOFAR-VLBI pipeline and its scripts were originally developed by:
|
|
|
|
|
|
* Leah Morabito
|
|
|
* Neal Jackson
|
|
|
* Sean Mooney
|
|
|
* Alexander Drabent
|
|
|
* Frits Sweijen
|
|
|
* Etienne Bonnassieux
|
|
|
|
|
|
The pipeline makes use of the Long Baseline Calibrator Survey (LBCS):
|
|
|
|
|
|
```
|
|
|
LBCS: The LOFAR Long-Baseline Calibrator Survey
|
|
|
Jackson, N., et al.,
|
|
|
A&A 595, A86 (2016),
|
|
|
DOI: 10.1051/0004-6361/201629016,
|
|
|
arXiv:1608.02133 [astro-ph.IM]
|
|
|
```
|
|
|
|
|
|
The development of the pipeline has been supported by SURF and EGI-ACE.
|
|
|
EGI-ACE has received funding from the European Union's _Horizon 2020_ research and innovation programme under grant agreement No 101017567.
|
|
|
|
|
|
## Citation
|
|
|
|
|
|
The LOFAR-VLBI pipeline is described in [Morabito et al. (2019)](https://www.aanda.org/articles/aa/full_html/2022/02/aa40649-21/aa40649-21.html).
|
|
|
If your work uses the pipeline, please cite the paper above as:
|
|
|
|
|
|
```
|
|
|
Sub-arcsecond imaging with the International LOFAR Telescope - I. Foundational calibration strategy and pipeline
|
|
|
L. K. Morabito, et al.,
|
|
|
A&A 658 A1 (2022),
|
|
|
DOI: 10.1051/0004-6361/202140649,
|
|
|
arXiv:2108.07283 [astro-ph.IM]
|
|
|
```
|
|
|
|
|
|
## License
|
|
|
|
|
|
The LOFAR-VLBI pipeline is licensed under the GNU General Public License v3.0.
|
|
|
A copy of the license is available in the pipeline's [Git repository](https://git.astron.nl/RD/VLBI-cwl/-/blob/master/LICENSE). |