Skip to content
Snippets Groups Projects

Fix some user feedback on the documentation

Merged Andre Offringa requested to merge user-feedback-on-docs into master
+ 28
13
@@ -16,16 +16,19 @@ Once you have the data and the input JSON file ready, you can run the pipeline,
$ toil-cwl-runner <cwl_options> <install_dir>/workflows/HBA_calibrator.cwl LINC.json
where ``LINC.json`` is the input JSON file as described in the chapter :doc:`parset` and ``<install_dir>`` the location of the **LINC** CWL description files.
.. note::
Do **not** run your ``cwltool`` or ``toil`` calls inside the Docker or Singularity container unless this is exactly what you intend to do (see next section)
Instead of specifying all options in ``LINC.json`` the user can also use command line options to override the defaults.
By default, **LINC** will execute the processing steps (like Dp3, etc.) inside a Docker container. If you prefer to use Singularity instead, the option
``--singularity`` can be added to the cwltool command line (see options below).
.. note::
Instead of specifying all options in ``LINC.json`` the user can also use command line options to override the defaults.
Do **not** run your ``cwltool`` or ``toil`` calls inside the Docker or Singularity container unless this is exactly what you intend to do (see next section)
For standard LOFAR observations these are the workflows to call in the command above, which provide the proper pipeline with pre-defined parameters (defaults) for **HBA** and **LBA** observations:
The following list provides the workflows to call in the command above for standard LOFAR observations. These provide the proper pipeline with pre-defined parameters (defaults) for **HBA** and **LBA** observations:
============================ ====================== =======================
**LINC workflow** **HBA** **LBA**
@@ -58,11 +61,13 @@ For Docker::
To run **LINC** you only need to add the container-specific execution command and **make sure that all necessary volumes are mounted read-write** (``<mount_points>``) and are thus accessible from inside the container, e.g.,::
$ singularity exec --bind <mount_points>:<mount_points> <linc.sif> cwltool <cwl_options> /usr/local/share/linc/workflows/HBA_calibrator.cwl LINC.json
$ singularity exec --bind <mount_points>:<mount_points> <linc.sif> cwltool --no-container <cwl_options> /usr/local/share/linc/workflows/HBA_calibrator.cwl LINC.json
where ``<linc.sif>`` is the location of the Singularity image, or ::
$ docker run --rm <docker_options> -v <mount_points>:<mount_points> -w $PWD astronrd/linc cwltool <cwl_options> /usr/local/share/linc/workflows/HBA_calibrator.cwl LINC.json
$ docker run --rm <docker_options> -v <mount_points>:<mount_points> -w $PWD astronrd/linc cwltool --no-container <cwl_options> /usr/local/share/linc/workflows/HBA_calibrator.cwl LINC.json
Since you are running **LINC** inside a container, do not forget to add the ``--no-container`` flag to your call, no matter whether you use Singularity or Docker. Do **not** use the ``--singularity``flag.
Pipeline options for ``cwltool``
--------------------------------
@@ -80,17 +85,27 @@ Please check carefully which options to choose depending on the way how you run
* **---preserve-entire-environment**: use system environment variables (only for manual installation and in case you are running from within a Docker/Singularity image)
* **---debug**: more verbose output (use only for debugging the pipeline)
.. note::
``cwltool`` has no option to resume a failed/crashed run. If you need this option have a look at ``toil``.
While the pipeline runs, the terminal will output the current state of the pipeline. For **debugging** it is highly recommended to pipe this output into a runtime logfile::
While the pipeline runs, the terminal will output the current state of the pipeline. For **debugging** it is recommended to
run cwltool inside a ``screen`` or to pipe the output into a runtime logfile::
$ cwltool <cwl_options> <install_dir>/workflows/HBA_calibrator.cwl LINC.json > logfile 2>&1
A fairly typical run that uses Singularity can look similar to this: ::
$ cwltool \
--singularity \
--outdir "/data/myproject/Linc-L628614" \
--log-dir "/data/myproject/Log-L628614" \
--tmpdir-prefix "/data/myproject/Tmp-L628614/" \
~/.local/share/linc/workflows/HBA_target.cwl \
linc-L628614.json
In the specified ``--tmpdir_prefix`` all temporary folders and files are generated. At the end of the run those files can be deleted.
.. note::
``cwltool`` has no option to resume a failed/crashed run. If you need this option have a look at ``toil``.
Pipeline options for ``toil``
--------------------------------
The following ``<cwl_options>`` are recommended to use for running **LINC** with ``toil``:
@@ -184,4 +199,4 @@ In ``toil`` the main logfile is written to ``--logFile`` and logfiles from singl
If there is no error message reported or no corresponding logfile available, check for all lines leading with ``ERROR`` or ``error`` to get additional information about the possible cause of the crash or diagnostic messages that tell you what exactly went wrong.
To get help on new or already known issues, please check :ref:`help` for further support and information.
\ No newline at end of file
To get help on new or already known issues, please check :ref:`help` for further support and information.
Loading