|
|
This is the homepage of the LOFAR VLBI pipeline wiki. |
|
|
\ No newline at end of file |
|
|
The _Low-Frequency Array Very Long Baseline Interferometry_ (LOFAR-VLBI) pipeline is a calibration and imaging pipeline that includes all of LOFAR’s international stations.
|
|
|
It looks and solves for delays using an in-field calibrator within the field of view of the pointing.
|
|
|
The pipeline provides:
|
|
|
|
|
|
* solution tables written in a file in the [hierarchical data format](https://www.hdfgroup.org/) (version 5), referred to as an h5parm,
|
|
|
* a self-calibrated, corrected dataset for the in-field calibrator.
|
|
|
|
|
|
The delay solutions are applied back to the original data.
|
|
|
From there the pipeline can split out smaller datasets in the desired target direction(s) within the field of view.
|
|
|
|
|
|
## Installation
|
|
|
|
|
|
The LOFAR-VLBI pipeline can be downloaded from its git repository:
|
|
|
```
|
|
|
$ git clone https://git.astron.nl/RD/VLBI-cwl.git <install_dir>
|
|
|
```
|
|
|
The pipeline is implemented in the [Common Workflow Language](https://www.commonwl.org/) (CWL).
|
|
|
A CWL implementation must be installed on the system before the pipeline can be run.
|
|
|
A list of implementations is available on [the CWL website](https://www.commonwl.org/implementations/), though it is recommended to run the LOFAR-VLBI pipeline only with `cwltool` or `toil`.
|
|
|
|
|
|
The pipeline depends on the following software components:
|
|
|
|
|
|
* [DP3](https://git.astron.nl/RD/DP3)
|
|
|
* [python](https://www.python.org/) (≥3.8)
|
|
|
* [PyBDSF](https://github.com/lofar-astron/PyBDSF)
|
|
|
* [scipy](https://scipy.org/)
|
|
|
* [numpy](https://numpy.org/)
|
|
|
* [matplotlib](https://numpy.org/)
|
|
|
* [pyrap](https://github.com/tslight/pyrap)
|
|
|
* [losoto](https://github.com/revoltek/losoto)
|
|
|
* [pyvo](https://github.com/astropy/pyvo)
|
|
|
* [astropy](https://www.astropy.org/about.html)
|
|
|
* [python3-regex](https://github.com/mrabarnett/mrab-regex)
|
|
|
* [statsmodels](https://www.statsmodels.org/stable/index.html)
|
|
|
* [astroquery](https://astroquery.readthedocs.io/en/latest/)
|
|
|
* [python-magic](https://github.com/ahupp/python-magic)
|
|
|
* [lofar_helpers](https://github.com/jurjen93/lofar_helpers)
|
|
|
* [lofar_facet_selfcal](https://github.com/rvweeren/lofar_facet_selfcal)
|
|
|
|
|
|
It is recommended to run the pipeline in a container as described below.
|
|
|
|
|
|
## Containerisation
|
|
|
|
|
|
The LOFAR-VLBI pipeline currently ships with a Dockerfile.
|
|
|
It is assumed that this file has been used to construct a [Docker](https://www.docker.com/) or [Singularity/Apptainer](https://sylabs.io/) image.
|
|
|
|
|
|
**NOTE**: the resulting Docker container does not contain the packages *lofar_helpers* and *lofar_facet_selfcal*.
|
|
|
It is currently recommended to download local copies of these packages.
|
|
|
|
|
|
The CWL specification supports Docker natively.
|
|
|
That is, if using `cwltool` one simply runs
|
|
|
```
|
|
|
$ cwltool <workflow> <configuration file>
|
|
|
```
|
|
|
If using singularity, one has to add a `--singularity` flag:
|
|
|
```
|
|
|
$ cwltool --singularity <workflow> <configuration file>
|
|
|
```
|
|
|
|
|
|
It is possible to run CWL implementations within a container.
|
|
|
In this case it is important that your runner does not attempt to access external container images.
|
|
|
This can be done by passing the `--no-container` flag to your runner:
|
|
|
```
|
|
|
$ singularity <singularity options> cwltool --no-container <workflow> <configuration file>
|
|
|
```
|
|
|
Refer to the documentation of your CWL implementation for details.
|
|
|
|
|
|
If your singularity image does not contain the LOFAR-VLBI pipeline it is important that the VLBI installation directory has been properly mounted.
|
|
|
In addition, the VLBI installation directory should be added to the image's `$PATH` and `$PYTHONPATH` variables.
|
|
|
See [the Singularity/Apptainer documentation](https://apptainer.org/docs/user/main/environment_and_metadata.html) for details.
|
|
|
|
|
|
### Using cached Singularity images
|
|
|
|
|
|
CWL implementations are able to use cached Singularity images.
|
|
|
This requires resetting the `$CWL_SINGULARITY_CACHE` variable to the directory path that contains the cached images:
|
|
|
```
|
|
|
$ export CWL_SINGULARITY_CACHE=<cachedir>
|
|
|
```
|
|
|
where `<cachedir>` is the preferred location of the Singularity image.
|
|
|
|
|
|
When running the LOFAR-VLBI pipeline, the location of `<cachedir>` should be visible from all active compute nodes.
|
|
|
Once set simply restart your pipeline with `--singularity` and the image should be put into the right directory and be used for all steps.
|
|
|
|
|
|
**NOTE**: One may want more fine-grained control over image caching when building the Singularity images directly.
|
|
|
This control can be achieved by setting additional environment variables.
|
|
|
See the [Singularity build documentation](https://docs.sylabs.io/guides/latest/user-guide/build_env.html) for more details.
|
|
|
|
|
|
## Configuration
|
|
|
|
|
|
Input parameters for workflows are stored in configuration files, and should be be written in the [YAML](https://yaml.org/) or [JSON](https://www.json.org/) formats.
|
|
|
Introductions to these formats are available on at [Learn YAML in Y minutes](https://learnxinyminutes.com/docs/yaml/) and [Learn JSON in Y minutes](https://learnxinyminutes.com/docs/json/), respectively.
|
|
|
|
|
|
The configuration files should at a minimum contain the relevant measurement sets and calibrator solutions, as described in [Execution](#Execution).
|
|
|
|
|
|
## Execution
|
|
|
|
|
|
### Pipeline overview
|
|
|
|
|
|
The LOFAR-VLBI pipeline builds on the output of the [LOFAR Initial Calibration](https://linc.readthedocs.io/en/latest/) (LINC) pipeline.
|
|
|
In particular, it is assumed that LINC has produced sane solution files.
|
|
|
See the LINC documentation for details on how to run and validate the initial calibration.
|
|
|
|
|
|
### [Delay Calibration](delay-calibration/README.md)
|
|
|
The delay calibration workflow comprises the following steps and subroutines:
|
|
|
|
|
|
* [Setup](delay-calibration/setup.md):
|
|
|
* Applies the LINC calibrator and target solutions,
|
|
|
* predicts skymodel data and clips A-team contributions.
|
|
|
* [Concatenate-flag](delay-calibration/concatenate-flag.md):
|
|
|
* Reduces the data set by concatenation of subbands into groups,
|
|
|
* performs flagging on the data of the resulting measurement sets.
|
|
|
* [Phaseup-concatenate](delay-calibration/phaseup-concatenate.md):
|
|
|
* Phase shifts data to the in-field calibrator,
|
|
|
* Solves for total electron content (TEC).
|
|
|
* [Store-logs](steps/store-logs.md): Collects all the logfiles generated by the previous subroutines.
|
|
|
|
|
|
More details can be found on the pages of each subrouting.
|
|
|
|
|
|
### [Split Directions](split-directions/README.md)
|
|
|
Currently the split directions pipeline is still under development.
|
|
|
|
|
|
## Support
|
|
|
|
|
|
The LOFAR-VLBI pipeline is a continuously maintained and developed software package.
|
|
|
Bugs or other issues can be reported at the pipeline's [Git repository](https://git.astron.nl/RD/VLBI-cwl/-/issues).
|
|
|
|
|
|
## Acknowledgements
|
|
|
|
|
|
The LOFAR-VLBI pipeline and its scripts were originally developed by:
|
|
|
|
|
|
* Leah Morabito
|
|
|
* Neal Jackson
|
|
|
* Sean Mooney
|
|
|
* Alexander Drabent
|
|
|
* Frits Sweijen
|
|
|
* Etienne Bonnassieux
|
|
|
|
|
|
The pipeline makes use of the Long Baseline Calibrator Survey (LBCS):
|
|
|
|
|
|
```
|
|
|
LBCS: The LOFAR Long-Baseline Calibrator Survey
|
|
|
Jackson, N., et al.,
|
|
|
A&A 595, A86 (2016),
|
|
|
DOI: 10.1051/0004-6361/201629016,
|
|
|
arXiv:1608.02133 [astro-ph.IM]
|
|
|
```
|
|
|
|
|
|
## Citation
|
|
|
|
|
|
The LOFAR-VLBI pipeline is described in [Morabito et al. (2019)](https://www.aanda.org/articles/aa/full_html/2022/02/aa40649-21/aa40649-21.html).
|
|
|
If your work uses the pipeline, please cite the paper above as:
|
|
|
|
|
|
```
|
|
|
Sub-arcsecond imaging with the International LOFAR Telescope - I. Foundational calibration strategy and pipeline
|
|
|
L. K. Morabito, et al.,
|
|
|
A&A 658 A1 (2022),
|
|
|
DOI: 10.1051/0004-6361/202140649,
|
|
|
arXiv:2108.07283 [astro-ph.IM]
|
|
|
```
|
|
|
|
|
|
## License
|
|
|
|
|
|
The LOFAR-VLBI pipeline is licensed under the GNU General Public License v3.0.
|
|
|
A copy of the license is available in the pipeline's [Git repository](https://git.astron.nl/RD/VLBI-cwl/License). |