... | ... | @@ -8,11 +8,13 @@ The pipeline provides: |
|
|
The delay solutions are applied back to the original data.
|
|
|
From there the pipeline can split out smaller datasets in the desired target direction(s) within the field of view.
|
|
|
|
|
|
[[_TOC_]]
|
|
|
* Unsure what data is suitable to be processed?
|
|
|
Have a look at the [Data Requirements](data-requirements.md).
|
|
|
* Already made sure that your data is suitable to be processed and just want the bare minimum required to get started?
|
|
|
Skip to the [Quickstart](quickstart.md)!
|
|
|
|
|
|
## Quickstart
|
|
|
# Installation
|
|
|
|
|
|
A minimal installation necessary to run the VLBI pipeline in a Singularity container can be set up as follows.
|
|
|
The following absolute paths are assumed:
|
|
|
* `<installation directory>` will contain all the necessary LOFAR software,
|
|
|
* `<input directory>` will contain the data to be processed, including
|
... | ... | @@ -27,58 +29,35 @@ export INPUT_DIR=<input directory> |
|
|
export WORK_DIR=<work directory>
|
|
|
export OUTPUT_DIR=<output directory>
|
|
|
```
|
|
|
The LOFAR software and singularity containers can then be installed as follows:
|
|
|
The pipeline workflow files and scripts can be
|
|
|
The LOFAR-VLBI pipeline and necessary LOFAR software can then be installed as follows:
|
|
|
```
|
|
|
git clone https://git.astron.nl/RD/VLBI-cwl.git ${INSTALL_DIR}/vlbi
|
|
|
git clone https://github.com/jurjen93/lofar_helpers.git ${INSTALL_DIR}/lofar_helpers
|
|
|
git clone https://github.com/rvweeren/lofar_facet_selfcal.git ${INSTALL_DIR}/lofar_facet_selfcal
|
|
|
wget -O ${INSTALL_DIR}/vlbi.sif https://lofar-webdav.grid.sara.nl/software/shub_mirror/tikk3r/lofar-grid-hpccloud/intel/lofar_sksp_v4.2.2_sandybridge_sandybridge_ddf_cuda.sif
|
|
|
```
|
|
|
A suitable pipeline input file can be generated using the VLBI pipeline's `generate_input.sh`:
|
|
|
```
|
|
|
bash ${INSTALL_DIR}/vlbi/scripts/generate_input.sh ${INPUT_DIR} ${INSTALL_DIR}
|
|
|
```
|
|
|
This will generate `input.yaml` in the current working directory.
|
|
|
The delay calibration pipeline can then be run by executing
|
|
|
```
|
|
|
singularity exec \
|
|
|
--bind ${INSTALL_DIR},${INPUT_DIR},${OUTPUT_DIR},${WORK_DIR} \
|
|
|
--env PATH="${INSTALL_DIR}/vlbi/scripts:\$PATH" \
|
|
|
--env PYTHONPATH="${INSTALL_DIR}/vlbi/scripts:\$PYTHONPATH" \
|
|
|
--env VLBI_DATA_ROOT="${INSTALL_DIR}/vlbi" \
|
|
|
${INSTALL_DIR}/vlbi.sif \
|
|
|
cwltool \
|
|
|
--no-container \
|
|
|
--preserve-entire-environment \
|
|
|
--parallel \
|
|
|
--timestamps \
|
|
|
--outdir ${OUTPUT_DIR} \
|
|
|
--tmpdir-prefix ${WORK_DIR}/ \
|
|
|
${INSTALL_DIR}/vlbi/workflows/delay-calibration.cwl \
|
|
|
input.yaml
|
|
|
```
|
|
|
More detailed instructions can be found in the sections below.
|
|
|
|
|
|
**NOTE**: Users may want to refer to the [_Frits’ LoFAR Container_](https://tikk3r.github.io/flocs/) (FLoC) repository to ensure that their image supports the proper architecture.
|
|
|
|
|
|
## Installation
|
|
|
|
|
|
The LOFAR-VLBI pipeline can be downloaded from its git repository:
|
|
|
```
|
|
|
git clone https://git.astron.nl/RD/VLBI-cwl.git <install_dir>
|
|
|
```
|
|
|
The pipeline is implemented in the [Common Workflow Language](https://www.commonwl.org/) (CWL).
|
|
|
A CWL implementation must be installed on the system before the pipeline can be run.
|
|
|
A list of implementations is available on the [CWL website](https://www.commonwl.org/implementations/), though it is recommended to run the LOFAR-VLBI pipeline only with `cwltool` or `toil-cwl-runner`.
|
|
|
A list of implementations is available on the [CWL website](https://www.commonwl.org/implementations/), although LOFAR-VLBI pipeline officially supports only `cwltool` or `toil-cwl-runner`.
|
|
|
```
|
|
|
python3 -m pip install cwltool toil[cwl]
|
|
|
```
|
|
|
The software packages above depend on various software products.
|
|
|
These can either be installed manually, or used through a software container.
|
|
|
Both methods are described below, but it is recommended to run the LOFAR-VLBI pipeline within a container.
|
|
|
|
|
|
The pipeline depends on the following software components and their dependencies:
|
|
|
## Manual installation
|
|
|
|
|
|
The pipeline depends on the following external software components and their dependencies:
|
|
|
|
|
|
* [DP3](https://git.astron.nl/RD/DP3)
|
|
|
* [python](https://www.python.org/) (≥3.8)
|
|
|
* [PyBDSF](https://github.com/lofar-astron/PyBDSF)
|
|
|
* [scipy](https://scipy.org/)
|
|
|
* [numpy](https://numpy.org/)
|
|
|
* [matplotlib](https://numpy.org/)
|
|
|
* [scipy](https://scipy.org/)
|
|
|
* [matplotlib](https://matplotlib.org/)
|
|
|
* [pyrap](https://github.com/tslight/pyrap)
|
|
|
* [losoto](https://github.com/revoltek/losoto)
|
|
|
* [pyvo](https://github.com/astropy/pyvo)
|
... | ... | @@ -87,20 +66,18 @@ The pipeline depends on the following software components and their dependencies |
|
|
* [statsmodels](https://www.statsmodels.org/stable/index.html)
|
|
|
* [astroquery](https://astroquery.readthedocs.io/en/latest/)
|
|
|
* [python-magic](https://github.com/ahupp/python-magic)
|
|
|
* [lofar_helpers](https://github.com/jurjen93/lofar_helpers)
|
|
|
* [lofar_facet_selfcal](https://github.com/rvweeren/lofar_facet_selfcal)
|
|
|
|
|
|
It is recommended to run the pipeline in a container as described in [Containerisation](##Containerisation).
|
|
|
These have to be installed either through your package manager or from their respective repositories.
|
|
|
|
|
|
## Containerisation
|
|
|
|
|
|
It is recommended to run the LOFAR-VLBI pipeline within a container.
|
|
|
In case Singularity/Apptainer is used, it is recommended to use a [FLoC](https://tikk3r.github.io/flocs/):
|
|
|
In case Singularity/Apptainer is used, it is recommended to use [_Frits' LOFAR Containers_](https://tikk3r.github.io/flocs/) (FLoC):
|
|
|
```
|
|
|
wget https://lofar-webdav.grid.sara.nl/software/shub_mirror/tikk3r/lofar-grid-hpccloud/intel/lofar_sksp_v4.2.2_sandybridge_sandybridge_ddf_cuda.sif
|
|
|
```
|
|
|
**NOTE**: this container does not contain the packages *lofar_helpers* and *lofar_facet_selfcal*.
|
|
|
It is currently recommended to download local copies of these packages.
|
|
|
{: .alert .alert-warning}
|
|
|
|
|
|
The pipeline can be run inside the singularity container by running
|
|
|
```
|
... | ... | @@ -110,27 +87,30 @@ Refer to the documentation of your CWL implementation for details. |
|
|
|
|
|
It is important that the VLBI installation directory has been properly mounted.
|
|
|
In addition, the VLBI installation directory should be added to the image's `$PATH` and `$PYTHONPATH` variables.
|
|
|
See [the Singularity/Apptainer documentation](https://apptainer.org/docs/user/main/environment_and_metadata.html) for details.
|
|
|
See the Singularity/Apptainer [documentation](https://apptainer.org/docs/user/main/environment_and_metadata.html) for details.
|
|
|
|
|
|
## Configuration
|
|
|
# Configuration
|
|
|
|
|
|
Input parameters for workflows are stored in configuration files, and should be be written in the [YAML](https://yaml.org/) or [JSON](https://www.json.org/) formats.
|
|
|
Introductions to these formats are available on at [Learn YAML in Y minutes](https://learnxinyminutes.com/docs/yaml/) and [Learn JSON in Y minutes](https://learnxinyminutes.com/docs/json/), respectively.
|
|
|
|
|
|
A configuration file should, at minimum, contain
|
|
|
A configuration file should, at a minimum, contain
|
|
|
* the observation data in _MeasurementSet_ (MS) format,
|
|
|
* solution tables generated by the _LOFAR Initial Calibration_ (LINC) pipeline,
|
|
|
* a catalogue containing a suitable delay calibrator in CSV format,
|
|
|
A minimal configuration file can be generated by running `$VLBI_INSTALL_DIR/scripts/generate_input.sh`.
|
|
|
Optional input parameters, such as flagged baselines or solutions generated by the _Direction-Dependent Faceting_ (DDF) pipeline should be added by hand.
|
|
|
* solution tables generated by the [_LOFAR Initial Calibration_](https://git.astron.nl/RD/LINC) (LINC) pipeline,
|
|
|
* a catalogue containing a suitable delay calibrator at a 6″ resolution, in CSV format,
|
|
|
|
|
|
## Execution
|
|
|
Assuming the setup listed in [Installation](#Installation), a suitable pipeline input file can be generated using the VLBI pipeline's `generate_input.sh`:
|
|
|
```
|
|
|
bash ${INSTALL_DIR}/vlbi/scripts/generate_input.sh ${INPUT_DIR} ${INSTALL_DIR}
|
|
|
```
|
|
|
This will generate `input.yaml` in the current working directory.
|
|
|
Optional input parameters, such as flagged baselines or solutions generated by the [_Direction-Dependent Faceting_](https://github.com/saopicc/DDFacet) (DDF) pipeline should be added to this file by hand.
|
|
|
|
|
|
### Pipeline overview
|
|
|
# Pipeline overview
|
|
|
|
|
|
The LOFAR-VLBI pipeline builds on the output of the [LOFAR Initial Calibration](https://linc.readthedocs.io/en/latest/) (LINC) pipeline.
|
|
|
The LOFAR-VLBI pipeline builds on the output of the LINC pipeline.
|
|
|
In particular, it is assumed that LINC has produced sane solution files.
|
|
|
See the LINC documentation for details on how to run and validate the initial calibration.
|
|
|
See the LINC [documentation](https://linc.readthedocs.io/en/latest/) for details on how to run and validate the initial calibration.
|
|
|
|
|
|
```mermaid
|
|
|
flowchart TD
|
... | ... | |