... | ... | @@ -22,40 +22,40 @@ The following absolute paths are assumed: |
|
|
* `<work directory>` will contain the intermediate files produced during pipeline operation,
|
|
|
* `<output directory>` will contain the data products produced by the pipeline.
|
|
|
```
|
|
|
$ export INSTALL_DIR=<installation directory>
|
|
|
$ export INPUT_DIR=<input directory>
|
|
|
$ export WORK_DIR=<work directory>
|
|
|
$ export OUTPUT_DIR=<output directory>
|
|
|
export INSTALL_DIR=<installation directory>
|
|
|
export INPUT_DIR=<input directory>
|
|
|
export WORK_DIR=<work directory>
|
|
|
export OUTPUT_DIR=<output directory>
|
|
|
```
|
|
|
The LOFAR software and singularity containers can then be installed as follows:
|
|
|
```
|
|
|
$ git clone https://git.astron.nl/RD/VLBI-cwl.git ${INSTALL_DIR}/vlbi
|
|
|
$ git clone https://github.com/jurjen93/lofar_helpers.git ${INSTALL_DIR}/lofar_helpers
|
|
|
$ git clone https://github.com/rvweeren/lofar_facet_selfcal.git ${INSTALL_DIR}/lofar_facet_selfcal
|
|
|
$ wget -O ${INSTALL_DIR}/vlbi.sif https://lofar-webdav.grid.sara.nl/software/shub_mirror/tikk3r/lofar-grid-hpccloud/intel/lofar_sksp_v4.2.2_sandybridge_sandybridge_ddf_cuda.sif
|
|
|
git clone https://git.astron.nl/RD/VLBI-cwl.git ${INSTALL_DIR}/vlbi
|
|
|
git clone https://github.com/jurjen93/lofar_helpers.git ${INSTALL_DIR}/lofar_helpers
|
|
|
git clone https://github.com/rvweeren/lofar_facet_selfcal.git ${INSTALL_DIR}/lofar_facet_selfcal
|
|
|
wget -O ${INSTALL_DIR}/vlbi.sif https://lofar-webdav.grid.sara.nl/software/shub_mirror/tikk3r/lofar-grid-hpccloud/intel/lofar_sksp_v4.2.2_sandybridge_sandybridge_ddf_cuda.sif
|
|
|
```
|
|
|
A suitable pipeline input file can be generated using the VLBI pipeline's `generate_input.sh`:
|
|
|
```
|
|
|
$ bash ${INSTALL_DIR}/vlbi/scripts/generate_input.sh ${INPUT_DIR} ${INSTALL_DIR}
|
|
|
bash ${INSTALL_DIR}/vlbi/scripts/generate_input.sh ${INPUT_DIR} ${INSTALL_DIR}
|
|
|
```
|
|
|
This will generate `input.yaml` in the current working directory.
|
|
|
The delay calibration pipeline can then be run by executing
|
|
|
```
|
|
|
$ singularity exec \
|
|
|
--bind ${INSTALL_DIR},${INPUT_DIR},${OUTPUT_DIR},${WORK_DIR} \
|
|
|
--env PATH="${INSTALL_DIR}/vlbi/scripts:\$PATH" \
|
|
|
--env PYTHONPATH="${INSTALL_DIR}/vlbi/scripts:\$PYTHONPATH" \
|
|
|
--env VLBI_DATA_ROOT="${INSTALL_DIR}/vlbi" \
|
|
|
${INSTALL_DIR}/vlbi.sif \
|
|
|
cwltool \
|
|
|
--no-container \
|
|
|
--preserve-entire-environment \
|
|
|
--parallel \
|
|
|
--timestamps \
|
|
|
--outdir ${OUTPUT_DIR} \
|
|
|
--tmpdir-prefix ${WORK_DIR}/ \
|
|
|
${INSTALL_DIR}/vlbi/workflows/delay-calibration.cwl \
|
|
|
input.yaml
|
|
|
singularity exec \
|
|
|
--bind ${INSTALL_DIR},${INPUT_DIR},${OUTPUT_DIR},${WORK_DIR} \
|
|
|
--env PATH="${INSTALL_DIR}/vlbi/scripts:\$PATH" \
|
|
|
--env PYTHONPATH="${INSTALL_DIR}/vlbi/scripts:\$PYTHONPATH" \
|
|
|
--env VLBI_DATA_ROOT="${INSTALL_DIR}/vlbi" \
|
|
|
${INSTALL_DIR}/vlbi.sif \
|
|
|
cwltool \
|
|
|
--no-container \
|
|
|
--preserve-entire-environment \
|
|
|
--parallel \
|
|
|
--timestamps \
|
|
|
--outdir ${OUTPUT_DIR} \
|
|
|
--tmpdir-prefix ${WORK_DIR}/ \
|
|
|
${INSTALL_DIR}/vlbi/workflows/delay-calibration.cwl \
|
|
|
input.yaml
|
|
|
```
|
|
|
More detailed instructions can be found in the sections below.
|
|
|
|
... | ... | @@ -65,7 +65,7 @@ More detailed instructions can be found in the sections below. |
|
|
|
|
|
The LOFAR-VLBI pipeline can be downloaded from its git repository:
|
|
|
```
|
|
|
$ git clone https://git.astron.nl/RD/VLBI-cwl.git <install_dir>
|
|
|
git clone https://git.astron.nl/RD/VLBI-cwl.git <install_dir>
|
|
|
```
|
|
|
The pipeline is implemented in the [Common Workflow Language](https://www.commonwl.org/) (CWL).
|
|
|
A CWL implementation must be installed on the system before the pipeline can be run.
|
... | ... | @@ -97,14 +97,14 @@ It is recommended to run the pipeline in a container as described in [Containeri |
|
|
It is recommended to run the LOFAR-VLBI pipeline within a container.
|
|
|
In case Singularity/Apptainer is used, it is recommended to use a [FLoC](https://tikk3r.github.io/flocs/):
|
|
|
```
|
|
|
$ wget https://lofar-webdav.grid.sara.nl/software/shub_mirror/tikk3r/lofar-grid-hpccloud/intel/lofar_sksp_v4.2.2_sandybridge_sandybridge_ddf_cuda.sif
|
|
|
wget https://lofar-webdav.grid.sara.nl/software/shub_mirror/tikk3r/lofar-grid-hpccloud/intel/lofar_sksp_v4.2.2_sandybridge_sandybridge_ddf_cuda.sif
|
|
|
```
|
|
|
**NOTE**: this container does not contain the packages *lofar_helpers* and *lofar_facet_selfcal*.
|
|
|
It is currently recommended to download local copies of these packages.
|
|
|
|
|
|
The pipeline can be run inside the singularity container by running
|
|
|
```
|
|
|
$ singularity <singularity options> cwltool --no-container <workflow> <configuration file>
|
|
|
singularity <singularity options> cwltool --no-container <workflow> <configuration file>
|
|
|
```
|
|
|
Refer to the documentation of your CWL implementation for details.
|
|
|
|
... | ... | |