Add reordering for aoflagger (RAP-129)

docs/source/calibrator.rst

@@ -278,8 +278,7 @@ A comprehensive explanation of the baseline selection syntax can be found `here`
 
 - ``max_dppp_threads``: number of threads per process for DP3 (default: 10)
 - ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20)
-- ``min_length``: minimum amount of subbands to concatenate in frequency necessary to perform the wide-band flagging in the RAM. It data is too big aoflag will use indirect-read (default: 50)
-- ``overhead``: only use this fraction of the available memory for deriving the amount of data to be concatenated (default: 0.8)
+- ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (see the `AOFlagger manual`_`)
 
 **Parameters you may want to adjust**
 
@@ -297,3 +296,4 @@ A comprehensive explanation of the baseline selection syntax can be found `here`
 
 .. _here: https://www.astron.nl/lofarwiki/doku.php?id=public:user_software:documentation:ndppp#description_of_baseline_selection_parameters
 .. _AOFlagger: https://gitlab.com/aroffringa/aoflagger.git
+.. _AOFlagger manual: https://aoflagger.readthedocs.io/en/latest/using_aoflagger.html
\ No newline at end of file

docs/source/parset.rst

@@ -74,8 +74,6 @@ There are more parameters you may want to adjust that can be added to this input
         "tables2export": "clock",
         "max_dppp_threads": 10,
         "memoryperc": 20,
-        "min_length": 50,
-        "overhead": 0.8,
         "min_separation": 30,
         "max_separation_arcmin": 1.0,
         "calibrator_path_skymodel": null,
@@ -84,7 +82,8 @@ There are more parameters you may want to adjust that can be added to this input
         "avg_freqresolution": "48.82kHz",
         "bandpass_freqresolution": "195.3125kHz",
         "lbfgs_historysize" : 10,
-        "lbfgs_robustdof" : 200
+        "lbfgs_robustdof" : 200,
+        "reorder_aoflag" : false
     }
 
 If you just want to alter one of the defaults it is sufficient to override it by specifing its new value the JSON input file::

docs/source/target.rst

@@ -303,8 +303,7 @@ A comprehensive explanation of the baseline selection syntax can be found `here`
 
 - ``max_dppp_threads``: number of threads per process for DP3 (default: 10)
 - ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20)
-- ``min_length``: minimum amount of subbands to concatenate in frequency necessary to perform the wide-band flagging in the RAM. It data is too big aoflag will use indirect-read (default: 50)
-- ``overhead``: only use this fraction of the available memory for deriving the amount of data to be concatenated (default: 0.8)
+- ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (see the `AOFlagger manual`_`)
 
 *Skymodel directory*
 
@@ -346,3 +345,4 @@ If your **LBA** data has **not** been demixed before you may still want to keep
 .. _Dysco: https://github.com/aroffringa/dysco/
 .. _AOFlagger: https://gitlab.com/aroffringa/aoflagger.git
 .. _here: https://www.astron.nl/lofarwiki/doku.php?id=public:user_software:documentation:ndppp#description_of_baseline_selection_parameters
+.. _AOFlagger manual: https://aoflagger.readthedocs.io/en/latest/using_aoflagger.html
\ No newline at end of file

steps/aoflag.cwl

@@ -51,13 +51,13 @@ inputs:
       prefix: -direct-read
       position: 0
       shellQuote: false
-  - id: indirect_read
+  - id: reorder
     type: boolean?
     doc: |
-      Will reorder the measurement set before starting, which is normally
-      faster but requires free disk space to reorder the data to.
+      Make aoflagger reorder the measurement set before running the detection.
+      This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory.
     inputBinding:
-      prefix: -indirect-read
+      prefix: -reorder
       position: 0
       shellQuote: false
   - id: memory_read

workflows/HBA_calibrator.cwl

@@ -117,6 +117,9 @@ inputs:
   - id: lbfgs_robustdof
     type: float?
     default: 200
+  - id: aoflag_reorder
+    type: boolean?
+    default: false
 outputs:
   - id: log_files
     outputSource:
@@ -214,6 +217,8 @@ steps:
         source: lbfgs_historysize
       - id: lbfgs_robustdof
         source: lbfgs_robustdof
+      - id: aoflag_reorder
+        source: aoflag_reorder
     out:
       - id: logfiles
       - id: solutions

workflows/HBA_target.cwl

@@ -148,6 +148,9 @@ inputs:
   - id: lbfgs_robustdof
     type: float?
     default: 200
+  - id: aoflag_reorder
+    type: boolean?
+    default: false
 outputs:
   - id: calibrated_data
     outputSource:
@@ -268,6 +271,8 @@ steps:
         source: lbfgs_historysize
       - id: lbfgs_robustdof
         source: lbfgs_robustdof
+      - id: aoflag_reorder
+        source: aoflag_reorder
     out:
       - id: logfiles
       - id: msout

workflows/LBA_calibrator.cwl

@@ -117,6 +117,9 @@ inputs:
   - id: lbfgs_robustdof
     type: float?
     default: 200
+  - id: aoflag_reorder
+    type: boolean?
+    default: false
 outputs:
   - id: log_files
     outputSource:
@@ -214,6 +217,8 @@ steps:
         source: lbfgs_historysize
       - id: lbfgs_robustdof
         source: lbfgs_robustdof
+      - id: aoflag_reorder
+        source: aoflag_reorder
     out:
       - id: logfiles
       - id: solutions

workflows/linc_calibrator.cwl

@@ -116,6 +116,9 @@ inputs:
   - id: lbfgs_robustdof
     type: float?
     default: 200
+  - id: aoflag_reorder
+    type: boolean?
+    default: false
 outputs:
   - id: inspection
     linkMerge: merge_flattened
@@ -209,6 +212,8 @@ steps:
         source: lbfgs_historysize
       - id: lbfgs_robustdof
         source: lbfgs_robustdof
+      - id: aoflag_reorder
+        source: aoflag_reorder
     out:
       - id: outh5parm
       - id: logfiles

workflows/linc_calibrator/prep.cwl

@@ -75,6 +75,9 @@ inputs:
   - id: lbfgs_robustdof
     type: float?
     default: 200
+  - id: aoflag_reorder
+    type: boolean?
+    default: false
 outputs:
   - id: outh5parm
     outputSource:
@@ -226,6 +229,8 @@ steps:
         default: true
       - id: strategy
         source: rfistrategy
+      - id: reorder
+        source: aoflag_reorder
     out:
       - id: output_ms
       - id: logfile

workflows/linc_target.cwl

@@ -148,6 +148,9 @@ inputs:
   - id: lbfgs_robustdof
     type: float?
     default: 200
+  - id: aoflag_reorder
+    type: boolean?
+    default: false
 outputs:
   - id: inspection
     outputSource:
@@ -303,6 +306,8 @@ steps:
         source: refant
       - id: rfistrategy
         source: rfistrategy
+      - id: aoflag_reorder
+        source: aoflag_reorder
     out:
       - id: msout
       - id: outh5parm

workflows/linc_target/gsmcal.cwl

@@ -40,6 +40,9 @@ inputs:
     type:
       - File?
       - string?
+  - id: aoflag_reorder
+    type: boolean?
+    default: false
 outputs:
   - id: msout
     outputSource:
@@ -285,6 +288,8 @@ steps:
         default: true
       - id: strategy
         source: rfistrategy
+      - id: reorder
+        source: aoflag_reorder
     out:
       - id: output_ms
       - id: logfile