merge with master

Docker/fetch_latest_commits.sh

@@ -9,3 +9,4 @@ git ls-remote https://github.com/lofar-astron/LOFARBeam.git HEAD    | awk '{ pri
 git ls-remote https://git.astron.nl/RD/EveryBeam.git HEAD           | awk '{ print "EVERYBEAM_COMMIT="$1 }'
 git ls-remote https://git.astron.nl/RD/DP3.git HEAD                 | awk '{ print "DP3_COMMIT="$1 }'
 git ls-remote https://gitlab.com/aroffringa/wsclean.git HEAD        | awk '{ print "WSCLEAN_COMMIT="$1 }'
+git ls-remote https://git.astron.nl/RD/LSMTool.git HEAD             | awk '{ print "LSMTOOL_COMMIT="$1 }'

docs/source/calibrator.rst

+.. toctree::
+   :hidden:
+ 
+   calibrator_old
+      
 .. _calibrator_pipeline:
 
 Calibrator pipeline

docs/source/parset.rst

+.. toctree::
+   :hidden:
+ 
+   parset_old
+   
 .. _parset:
 
 Configuring **LINC**

docs/source/pipelineoverview.rst

+.. toctree::
+   :hidden:
+ 
+   pipelineoverview_old
+   
 .. _pipeline_overview:
 
 Pipeline overview

docs/source/preparation.rst

@@ -25,7 +25,7 @@ typically obtained from the LOFAR `Long-Term Archive`_.
     
     Older versions of **LINC**: All input measurement-sets for one pipeline run need to be in the same directory.
 
-.. _CODE: ftp://ftp.aiub.unibe.ch/CODE/
+.. _CODE: http://ftp.aiub.unibe.ch/CODE/
 .. _TGSS: http://tgssadr.strw.leidenuniv.nl/doku.php
 .. _GSM:  http://172.104.228.177/
 .. _Long-Term Archive: https://lta.lofar.eu

docs/source/running.rst

+.. toctree::
+   :hidden:
+ 
+   running_old
+   
 .. _runlinc:
 
 .. role:: blue
@@ -16,16 +21,19 @@ Once you have the data and the input JSON file ready, you can run the pipeline,
     $ toil-cwl-runner <cwl_options> <install_dir>/workflows/HBA_calibrator.cwl LINC.json
     
 where ``LINC.json`` is the input JSON file as described in the chapter :doc:`parset` and ``<install_dir>`` the location of the **LINC** CWL description files.
+
 .. note::
     
-    Do **not** run your ``cwltool`` or ``toil`` calls inside the Docker or Singularity container unless this is exactly what you intend to do (see next section)
+   Instead of specifying all options in ``LINC.json`` the user can also use command line options to override the defaults.
+
+By default, **LINC** will execute the processing steps (like Dp3, etc.) inside a Docker container. If you prefer to use Singularity instead, the option
+``--singularity`` can be added to the cwltool command line (see options below).
 
 .. note::
     
-    Instead of specifying all options in ``LINC.json`` the user can also use command line options to override the defaults.
-    
+   Do **not** run your ``cwltool`` or ``toil`` calls inside the Docker or Singularity container unless this is exactly what you intend to do (see next section)
 
-For standard LOFAR observations these are the workflows to call in the command above, which provide the proper pipeline with pre-defined parameters (defaults) for **HBA** and **LBA** observations:
+The following list provides the workflows to call in the command above for standard LOFAR observations. These provide the proper pipeline with pre-defined parameters (defaults) for **HBA** and **LBA** observations:
 
 ============================ ====================== =======================
 **LINC workflow**              **HBA**                   **LBA**
@@ -58,11 +66,13 @@ For Docker::
 
 To run **LINC** you only need to add the container-specific execution command and **make sure that all necessary volumes are mounted read-write** (``<mount_points>``) and are thus accessible from inside the container, e.g.,::
 
-    $ singularity exec --bind <mount_points>:<mount_points> <linc.sif> cwltool <cwl_options> /usr/local/share/linc/workflows/HBA_calibrator.cwl LINC.json
+    $ singularity exec --bind <mount_points>:<mount_points> <linc.sif> cwltool --no-container <cwl_options> /usr/local/share/linc/workflows/HBA_calibrator.cwl LINC.json
 
 where ``<linc.sif>`` is the location of the Singularity image, or ::
 
-    $ docker run --rm <docker_options> -v <mount_points>:<mount_points> -w $PWD astronrd/linc cwltool <cwl_options> /usr/local/share/linc/workflows/HBA_calibrator.cwl LINC.json
+    $ docker run --rm <docker_options> -v <mount_points>:<mount_points> -w $PWD astronrd/linc cwltool --no-container <cwl_options> /usr/local/share/linc/workflows/HBA_calibrator.cwl LINC.json
+
+Since you are running **LINC** inside a container, do not forget to add the ``--no-container`` flag to your call, no matter whether you use Singularity or Docker. Do **not** use the ``--singularity`` flag.
 
 Pipeline options for ``cwltool``
 --------------------------------
@@ -80,17 +90,27 @@ Please check carefully which options to choose depending on the way how you run
     * **---preserve-entire-environment**: use system environment variables (only for manual installation and in case you are running from within a Docker/Singularity image)
     * **---debug**: more verbose output (use only for debugging the pipeline)
     
-.. note::
-    
-    ``cwltool`` has no option to resume a failed/crashed run. If you need this option have a look at ``toil``.
-    
-    
-While the pipeline runs, the terminal will output the current state of the pipeline. For **debugging** it is highly recommended to pipe this output into a runtime logfile::
+While the pipeline runs, the terminal will output the current state of the pipeline. For **debugging** it is recommended to
+run cwltool inside a ``screen`` or to pipe the output into a runtime logfile::
 
     $ cwltool <cwl_options> <install_dir>/workflows/HBA_calibrator.cwl LINC.json > logfile 2>&1
     
+A fairly typical run that uses Singularity can look similar to this: ::
+
+    $ cwltool \
+      --singularity \
+      --outdir "/data/myproject/Linc-L628614" \
+      --log-dir "/data/myproject/Log-L628614" \
+      --tmpdir-prefix "/data/myproject/Tmp-L628614/" \
+      ~/.local/share/linc/workflows/HBA_target.cwl \
+      linc-L628614.json
+    
 In the specified ``--tmpdir_prefix`` all temporary folders and files are generated. At the end of the run those files can be deleted.
 
+.. note::
+    
+    ``cwltool`` has no option to resume a failed/crashed run. If you need this option have a look at ``toil``.
+
 Pipeline options for ``toil``
 --------------------------------
 The following ``<cwl_options>`` are recommended to use for running **LINC** with ``toil``:
@@ -184,4 +204,4 @@ In ``toil`` the main logfile is written to ``--logFile`` and logfiles from singl
 
 If there is no error message reported or no corresponding logfile available, check for all lines leading with ``ERROR`` or ``error`` to get additional information about the possible cause of the crash or diagnostic messages that tell you what exactly went wrong.
 
-To get help on new or already known issues, please check :ref:`help` for further support and information.
\ No newline at end of file
+To get help on new or already known issues, please check :ref:`help` for further support and information.

docs/source/target.rst

+.. toctree::
+   :hidden:
+ 
+   target_old
+   
 .. _target_pipeline:
 
 Target pipeline
@@ -321,7 +326,7 @@ A comprehensive explanation of the baseline selection syntax can be found `here`
 
 *RMextract settings*
 
-- ``ionex_server``: URL of the *IONEX* server (default: ``"ftp://ftp.aiub.unibe.ch/CODE/"``)
+- ``ionex_server``: URL of the *IONEX* server (default: ``"http://ftp.aiub.unibe.ch/CODE/"``)
 - ``ionex_prefix``: the prefix of the *IONEX* files (default: ``CODG``)
 - ``proxy_server``: specify URL or IP of proxy server if needed
 - ``proxy_port``: port of proxy server if needed

scripts/createRMh5parm.py

@@ -44,7 +44,7 @@ def makesolset(MS, data, solset_name):
 
 
 def main(MSfiles, h5parmdb, solset_name = "sol000",all_stations=True,timestepRM=300,
-                ionex_server="ftp://ftp.aiub.unibe.ch/CODE/",
+                ionex_server="http://ftp.aiub.unibe.ch/CODE/",
                 ionex_prefix='CODG',ionexPath="./",earth_rot=0,proxyServer=None,proxyPort=None,proxyType=None,proxyUser=None,proxyPass=None):
     '''Add rotation measure to existing h5parmdb
     
@@ -214,8 +214,8 @@ if __name__ == '__main__':
                         help='MS for which the parmdb should be created.')
     parser.add_argument('h5parm', type=str,
                         help='H5parm to which the results of the CommonRotationAngle is added.')
-    parser.add_argument('--server', type=str, default='ftp://ftp.aiub.unibe.ch/CODE/',
-                        help='URL of the server to use. (default: ftp://ftp.aiub.unibe.ch/CODE/)')
+    parser.add_argument('--server', type=str, default='http://ftp.aiub.unibe.ch/CODE/',
+                        help='URL of the server to use. (default: http://ftp.aiub.unibe.ch/CODE/)')
     parser.add_argument('--prefix', type=str, default='CODG',
                         help='Prefix of the IONEX files. (default: \"CODG\")')
     parser.add_argument('--ionexpath', '--path', type=str, default='./',

steps/createRMh5parm.cwl

@@ -21,7 +21,7 @@ inputs:
   - id: ionex_server
     type: string?
     doc: IONEX Server path
-    default: 'ftp://ftp.aiub.unibe.ch/CODE/'
+    default: 'http://ftp.aiub.unibe.ch/CODE/'
     inputBinding:
       prefix: '--server'
       position: 0

workflows/HBA_target.cwl

@@ -113,7 +113,7 @@ inputs:
     default: null
   - id: ionex_server
     type: string?
-    default: 'ftp://ftp.aiub.unibe.ch/CODE/'
+    default: 'http://ftp.aiub.unibe.ch/CODE/'
   - id: ionex_prefix
     type: string?
     default: 'CODG'

workflows/linc_target.cwl

@@ -113,7 +113,7 @@ inputs:
     default: null
   - id: ionex_server
     type: string?
-    default: 'ftp://ftp.aiub.unibe.ch/CODE/'
+    default: 'http://ftp.aiub.unibe.ch/CODE/'
   - id: ionex_prefix
     type: string?
     default: CODG

workflows/linc_target/prep.cwl

@@ -78,7 +78,7 @@ inputs:
     default: 48.82kHz
   - id: ionex_server
     type: string?
-    default: 'ftp://ftp.aiub.unibe.ch/CODE/'
+    default: 'http://ftp.aiub.unibe.ch/CODE/'
   - id: ionex_prefix
     type: string?
     default: CODG