... | @@ -10,6 +10,57 @@ From there the pipeline can split out smaller datasets in the desired target dir |
... | @@ -10,6 +10,57 @@ From there the pipeline can split out smaller datasets in the desired target dir |
|
|
|
|
|
[[_TOC_]]
|
|
[[_TOC_]]
|
|
|
|
|
|
|
|
## Quickstart
|
|
|
|
|
|
|
|
A minimal installation necessary to run the VLBI pipeline in a Singularity container can be set up as follows.
|
|
|
|
It is assumed that:
|
|
|
|
* `<installation directory>` will contain all the necessary LOFAR software,
|
|
|
|
* `<input directory>` will contain the data to be processed, including
|
|
|
|
* the observation data in _MeasurementSet_ (MS) format,
|
|
|
|
* solution tables generated by the _LOFAR Initial Calibration_ (LINC) pipeline,
|
|
|
|
* a catalogue containing a suitable delay calibrator in CSV format,
|
|
|
|
* `<work directory>` will contain the intermediate files produced during pipeline operation,
|
|
|
|
* `<output directory>` will contain the data products produced by the pipeline.
|
|
|
|
```
|
|
|
|
$ export INSTALL_DIR=<installation directory>
|
|
|
|
$ export INPUT_DIR=<input directory>
|
|
|
|
$ export WORK_DIR=<work directory>
|
|
|
|
$ export OUTPUT_DIR=<output directory>
|
|
|
|
```
|
|
|
|
The LOFAR software and singularity containers can then be installed as follows:
|
|
|
|
```
|
|
|
|
$ git clone https://git.astron.nl/RD/VLBI-cwl.git ${INSTALL_DIR}/vlbi
|
|
|
|
$ git clone https://github.com/jurjen93/lofar_helpers.git ${INSTALL_DIR}/lofar_helpers
|
|
|
|
$ git clone https://github.com/rvweeren/lofar_facet_selfcal.git ${INSTALL_DIR}/lofar_facet_selfcal
|
|
|
|
$ wget -O ${INSTALL_DIR}/vlbi.sif https://lofar-webdav.grid.sara.nl/software/shub_mirror/tikk3r/lofar-grid-hpccloud/intel/lofar_sksp_v4.2.2_sandybridge_sandybridge_ddf_cuda.sif
|
|
|
|
```
|
|
|
|
A suitable pipeline input file can be generated using the VLBI pipeline's `generate_input.sh`:
|
|
|
|
```
|
|
|
|
$ bash ${INSTALL_DIR}/vlbi/scripts/generate_input.sh ${INPUT_DIR} ${INSTALL_DIR}
|
|
|
|
```
|
|
|
|
This will generate `input.yaml` in the current working directory.
|
|
|
|
The delay calibration pipeline can then be run by executing
|
|
|
|
```
|
|
|
|
$ singularity exec \
|
|
|
|
--bind ${INSTALL_DIR},${INPUT_DIR},${OUTPUT_DIR},${WORK_DIR} \
|
|
|
|
--env PATH="${INSTALL_DIR}/vlbi/scripts:\$PATH" \
|
|
|
|
--env PYTHONPATH="${INSTALL_DIR}/vlbi/scripts:\$PATH" \
|
|
|
|
--env VLBI_DATA_ROOT="${INSTALL_DIR>" \
|
|
|
|
${INSTALL_DIR}/vlbi.sif \
|
|
|
|
cwltool \
|
|
|
|
--no-container \
|
|
|
|
--preserve-entire-environment \
|
|
|
|
--parallel \
|
|
|
|
--timestamps \
|
|
|
|
--outdir ${OUTPUT_DIR} \
|
|
|
|
--tmpdir-prefix ${WORK_DIR}/ \
|
|
|
|
${INSTALL_DIR}/vlbi/workflows/delay_calibration.cwl \
|
|
|
|
input.yaml
|
|
|
|
```
|
|
|
|
More detailed instructions can be found in the sections below.
|
|
|
|
|
|
|
|
**NOTE**: Users may want to refer to the [_Frits’ LoFAR Container_](https://tikk3r.github.io/flocs/) (FLoC) repository to ensure that their image supports the proper architecture.
|
|
|
|
|
|
## Installation
|
|
## Installation
|
|
|
|
|
|
The LOFAR-VLBI pipeline can be downloaded from its git repository:
|
|
The LOFAR-VLBI pipeline can be downloaded from its git repository:
|
... | @@ -18,9 +69,9 @@ $ git clone https://git.astron.nl/RD/VLBI-cwl.git <install_dir> |
... | @@ -18,9 +69,9 @@ $ git clone https://git.astron.nl/RD/VLBI-cwl.git <install_dir> |
|
```
|
|
```
|
|
The pipeline is implemented in the [Common Workflow Language](https://www.commonwl.org/) (CWL).
|
|
The pipeline is implemented in the [Common Workflow Language](https://www.commonwl.org/) (CWL).
|
|
A CWL implementation must be installed on the system before the pipeline can be run.
|
|
A CWL implementation must be installed on the system before the pipeline can be run.
|
|
A list of implementations is available on [the CWL website](https://www.commonwl.org/implementations/), though it is recommended to run the LOFAR-VLBI pipeline only with `cwltool` or `toil`.
|
|
A list of implementations is available on the [CWL website](https://www.commonwl.org/implementations/), though it is recommended to run the LOFAR-VLBI pipeline only with `cwltool` or `toil-cwl-runner`.
|
|
|
|
|
|
The pipeline depends on the following software components:
|
|
The pipeline depends on the following software components and their dependencies:
|
|
|
|
|
|
* [DP3](https://git.astron.nl/RD/DP3)
|
|
* [DP3](https://git.astron.nl/RD/DP3)
|
|
* [python](https://www.python.org/) (≥3.8)
|
|
* [python](https://www.python.org/) (≥3.8)
|
... | @@ -39,65 +90,39 @@ The pipeline depends on the following software components: |
... | @@ -39,65 +90,39 @@ The pipeline depends on the following software components: |
|
* [lofar_helpers](https://github.com/jurjen93/lofar_helpers)
|
|
* [lofar_helpers](https://github.com/jurjen93/lofar_helpers)
|
|
* [lofar_facet_selfcal](https://github.com/rvweeren/lofar_facet_selfcal)
|
|
* [lofar_facet_selfcal](https://github.com/rvweeren/lofar_facet_selfcal)
|
|
|
|
|
|
It is recommended to run the pipeline in a container as described below.
|
|
It is recommended to run the pipeline in a container as described in [Containerisation](##Containerisation).
|
|
|
|
|
|
## Containerisation
|
|
## Containerisation
|
|
|
|
|
|
The LOFAR-VLBI pipeline currently ships with a Dockerfile.
|
|
It is recommended to run the LOFAR-VLBI pipeline within a container.
|
|
It is assumed that this file has been used to construct a [Docker](https://www.docker.com/) or [Singularity/Apptainer](https://sylabs.io/) image.
|
|
In case Singularity/Apptainer is used, it is recommended to use a [FLoC](https://tikk3r.github.io/flocs/):
|
|
Alternatively, one can download a pre-constructed image by running
|
|
|
|
```
|
|
```
|
|
$ wget https://lofar-webdav.grid.sara.nl/software/shub_mirror/tikk3r/lofar-grid-hpccloud/lofar_sksp_v4.0.1_x86-64_generic_noavx512_ddf.sif?action=show
|
|
$ wget https://lofar-webdav.grid.sara.nl/software/shub_mirror/tikk3r/lofar-grid-hpccloud/intel/lofar_sksp_v4.2.2_sandybridge_sandybridge_ddf_cuda.sif
|
|
```
|
|
```
|
|
A repository of LOFAR images is maintained at https://tikk3r.github.io/flocs/.
|
|
**NOTE**: this container does not contain the packages *lofar_helpers* and *lofar_facet_selfcal*.
|
|
|
|
|
|
**NOTE**: the resulting Docker container does not contain the packages *lofar_helpers* and *lofar_facet_selfcal*.
|
|
|
|
It is currently recommended to download local copies of these packages.
|
|
It is currently recommended to download local copies of these packages.
|
|
|
|
|
|
The CWL specification supports Docker natively.
|
|
The pipeline can be run inside the singularity container by running
|
|
That is, if using `cwltool` one simply runs
|
|
|
|
```
|
|
|
|
$ cwltool <workflow> <configuration file>
|
|
|
|
```
|
|
|
|
If using singularity, one has to add a `--singularity` flag:
|
|
|
|
```
|
|
|
|
$ cwltool --singularity <workflow> <configuration file>
|
|
|
|
```
|
|
|
|
|
|
|
|
It is possible to run CWL implementations within a container.
|
|
|
|
In this case it is important that your runner does not attempt to access external container images.
|
|
|
|
This can be done by passing the `--no-container` flag to your runner:
|
|
|
|
```
|
|
```
|
|
$ singularity <singularity options> cwltool --no-container <workflow> <configuration file>
|
|
$ singularity <singularity options> cwltool --no-container <workflow> <configuration file>
|
|
```
|
|
```
|
|
Refer to the documentation of your CWL implementation for details.
|
|
Refer to the documentation of your CWL implementation for details.
|
|
|
|
|
|
If your singularity image does not contain the LOFAR-VLBI pipeline it is important that the VLBI installation directory has been properly mounted.
|
|
It is important that the VLBI installation directory has been properly mounted.
|
|
In addition, the VLBI installation directory should be added to the image's `$PATH` and `$PYTHONPATH` variables.
|
|
In addition, the VLBI installation directory should be added to the image's `$PATH` and `$PYTHONPATH` variables.
|
|
See [the Singularity/Apptainer documentation](https://apptainer.org/docs/user/main/environment_and_metadata.html) for details.
|
|
See [the Singularity/Apptainer documentation](https://apptainer.org/docs/user/main/environment_and_metadata.html) for details.
|
|
|
|
|
|
### Using cached Singularity images
|
|
|
|
|
|
|
|
CWL implementations are able to use cached Singularity images.
|
|
|
|
This requires resetting the `$CWL_SINGULARITY_CACHE` variable to the directory path that contains the cached images:
|
|
|
|
```
|
|
|
|
$ export CWL_SINGULARITY_CACHE=<cachedir>
|
|
|
|
```
|
|
|
|
where `<cachedir>` is the preferred location of the Singularity image.
|
|
|
|
|
|
|
|
When running the LOFAR-VLBI pipeline, the location of `<cachedir>` should be visible from all active compute nodes.
|
|
|
|
Once set simply restart your pipeline with `--singularity` and the image should be put into the right directory and be used for all steps.
|
|
|
|
|
|
|
|
**NOTE**: One may want more fine-grained control over image caching when building the Singularity images directly.
|
|
|
|
This control can be achieved by setting additional environment variables.
|
|
|
|
See the [Singularity build documentation](https://docs.sylabs.io/guides/latest/user-guide/build_env.html) for more details.
|
|
|
|
|
|
|
|
## Configuration
|
|
## Configuration
|
|
|
|
|
|
Input parameters for workflows are stored in configuration files, and should be be written in the [YAML](https://yaml.org/) or [JSON](https://www.json.org/) formats.
|
|
Input parameters for workflows are stored in configuration files, and should be be written in the [YAML](https://yaml.org/) or [JSON](https://www.json.org/) formats.
|
|
Introductions to these formats are available on at [Learn YAML in Y minutes](https://learnxinyminutes.com/docs/yaml/) and [Learn JSON in Y minutes](https://learnxinyminutes.com/docs/json/), respectively.
|
|
Introductions to these formats are available on at [Learn YAML in Y minutes](https://learnxinyminutes.com/docs/yaml/) and [Learn JSON in Y minutes](https://learnxinyminutes.com/docs/json/), respectively.
|
|
|
|
|
|
The configuration files should at a minimum contain the relevant measurement sets and calibrator solutions, as described in [Execution](#Execution).
|
|
A configuration file should, at minimum, contain
|
|
|
|
* the observation data in _MeasurementSet_ (MS) format,
|
|
|
|
* solution tables generated by the _LOFAR Initial Calibration_ (LINC) pipeline,
|
|
|
|
* a catalogue containing a suitable delay calibrator in CSV format,
|
|
|
|
A minimal configuration file can be generated by running `$VLBI_INSTALL_DIR/scripts/generate_input.sh`.
|
|
|
|
Optional input parameters, such as flagged baselines or solutions generated by the _Direction-Dependent Faceting_ (DDF) pipeline should be added by hand.
|
|
|
|
|
|
## Execution
|
|
## Execution
|
|
|
|
|
... | @@ -129,7 +154,6 @@ flowchart TD |
... | @@ -129,7 +154,6 @@ flowchart TD |
|
|
|
|
|
```
|
|
```
|
|
|
|
|
|
|
|
|
|
### [Delay Calibration](/delay-calibration/README.md)
|
|
### [Delay Calibration](/delay-calibration/README.md)
|
|
The delay calibration workflow comprises the following steps and subroutines:
|
|
The delay calibration workflow comprises the following steps and subroutines:
|
|
|
|
|
... | @@ -175,6 +199,9 @@ DOI: 10.1051/0004-6361/201629016, |
... | @@ -175,6 +199,9 @@ DOI: 10.1051/0004-6361/201629016, |
|
arXiv:1608.02133 [astro-ph.IM]
|
|
arXiv:1608.02133 [astro-ph.IM]
|
|
```
|
|
```
|
|
|
|
|
|
|
|
The development of the pipeline has been supported by SURF and EGI-ACE.
|
|
|
|
EGI-ACE has received funding from the European Union's _Horizon 2020_ research and innovation programme under grant agreement No 101017567.
|
|
|
|
|
|
## Citation
|
|
## Citation
|
|
|
|
|
|
The LOFAR-VLBI pipeline is described in [Morabito et al. (2019)](https://www.aanda.org/articles/aa/full_html/2022/02/aa40649-21/aa40649-21.html).
|
|
The LOFAR-VLBI pipeline is described in [Morabito et al. (2019)](https://www.aanda.org/articles/aa/full_html/2022/02/aa40649-21/aa40649-21.html).
|
... | | ... | |