Skip to content
Snippets Groups Projects

Fix concat_logfiles error

Merged Mick Veldhuis requested to merge fix-concatenate-files into main
1 file
+ 0
51
Compare changes
  • Side-by-side
  • Inline
+ 0
51
Downloading and Installation
============================
Instructions for downloading and installing the Pre-Processing Pipeline are summarised below and also available on its `GitLab page <https://git.astron.nl/RD/preprocessing-cwl>`_.
Obtaining the Pre-Processing Pipeline
-------------------------------------
The pipeline is written in the `Common Workflow Language (CWL) <https://www.commonwl.org/>`_ and consists of CWL workflows, which can be obtained with:
.. code:: console
$ git clone https://git.astron.nl/RD/preprocessing-cwl
Manual Installation
-------------------
To run the pipeline, you need a CWL runner, such as ``cwltool`` or ``toil``. It's recommended to install these in a virtual environment:
.. code:: console
$ python3 -m venv venv
$ source venv/bin/active
$ pip install cwltool toil[cwl]
Note that it's also necessary to install NodeJS, which can either be installed via your favourite package manager or using the ``nodejs-wheel`` Python package.
Additionally, the Pre-Processing Pipeline depends on the following software for processing:
* `LofarStMan <https://github.com/lofar-astron/LofarStMan>`_
* `DP3 <https://git.astron.nl/RD/DP3>`_
* `AOFlagger <https://gitlab.com/aroffringa/aoflagger>`_
* `SAGECal <https://github.com/nlesc-dirac/sagecal>`_
* `Casacore <https://github.com/casacore/casacore>`_
* `EveryBeam <https://git.astron.nl/RD/EveryBeam>`_
Please follow their respective installation instructions.
Docker Installation
-------------------
To run the pipeline within a Docker container, please build the image located in the ``docker/pipeline`` directory of the repository:
.. code:: console
$ docker build ../.. -f Dockerfile -t preprocess:latest
Please modify and give the image an appropriate tag with the ``-t`` option. This requires one to install Docker on their system, either `Docker Engine <https://docs.docker.com/engine/>`_ (only available for Linux) or `Docker Desktop <https://docs.docker.com/desktop/>`_ (available for Windows, MacOS, and Linux). Instead of Docker, one could also use `Apptainer <https://apptainer.org/>`_ (previously called Singularity) on Linux.
Note: if the dependencies have not been installed manually and the pipeline is not ran inside a container, workflow components will attempt to run inside a Docker or Docker-compatible container environment. In this case, a Docker image is pulled from the repository's container registry.
Loading