Use time chunking in aoflagger/LINC (RAP-29)

docs/source/calibrator.rst

@@ -278,7 +278,8 @@ A comprehensive explanation of the baseline selection syntax can be found `here`
 
 - ``max_dp3_threads``: number of threads per process for DP3 (default: 10)
 - ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20)
-- ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (see the `AOFlagger manual`_`)
+- ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (default: false, see the `AOFlagger manual`_`)
+- ``aoflag_chunksize``: this will split the set into intervals with the given maximum size, and flag each interval independently. This lowers the amount of memory required (default: 2000)
 
 **Parameters you may want to adjust**
 

docs/source/parset.rst

@@ -83,7 +83,8 @@ There are more parameters you may want to adjust that can be added to this input
         "bandpass_freqresolution": "195.3125kHz",
         "lbfgs_historysize" : 10,
         "lbfgs_robustdof" : 200,
-        "reorder_aoflag" : false
+        "aoflag_reorder" : false,
+        "aoflag_chunksize" : 2000
     }
 
 If you just want to alter one of the defaults it is sufficient to override it by specifing its new value the JSON input file::

docs/source/target.rst

@@ -304,7 +304,8 @@ A comprehensive explanation of the baseline selection syntax can be found `here`
 
 - ``max_dp3_threads``: number of threads per process for DP3 (default: 10)
 - ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20)
-- ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (see the `AOFlagger manual`_)
+- ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (default: false, see the `AOFlagger manual`_`)
+- ``aoflag_chunksize``: this will split the set into intervals with the given maximum size, and flag each interval independently. This lowers the amount of memory required (default: 2000)
 
 *Skymodel directory*
 

steps/aoflag.cwl

@@ -133,6 +133,13 @@ inputs:
       prefix: -concatenate-frequency
       position: 0
       shellQuote: false
+  - id: chunk-size
+    type: int?
+    doc:  This will split the set into intervals with the given maximum size, and flag each interval independently
+    inputBinding:
+      prefix: -chunk-size
+      position: 0
+      shellQuote: false
 
 outputs:
   - id: output_ms

workflows/HBA_calibrator.cwl

@@ -122,6 +122,9 @@ inputs:
   - id: aoflag_reorder
     type: boolean?
     default: false
+  - id: aoflag_chunksize
+    type: int?
+    default: 2000
 outputs:
   - id: log_files
     outputSource:
@@ -221,6 +224,8 @@ steps:
         source: lbfgs_robustdof
       - id: aoflag_reorder
         source: aoflag_reorder
+      - id: aoflag_chunksize
+        source: aoflag_chunksize
     out:
       - id: logfiles
       - id: solutions

workflows/HBA_target.cwl

@@ -151,6 +151,9 @@ inputs:
   - id: aoflag_reorder
     type: boolean?
     default: false
+  - id: aoflag_chunksize
+    type: int?
+    default: 2000
 outputs:
   - id: calibrated_data
     outputSource:
@@ -273,6 +276,8 @@ steps:
         source: lbfgs_robustdof
       - id: aoflag_reorder
         source: aoflag_reorder
+      - id: aoflag_chunksize
+        source: aoflag_chunksize
     out:
       - id: logfiles
       - id: msout

workflows/LBA_calibrator.cwl

@@ -120,6 +120,9 @@ inputs:
   - id: aoflag_reorder
     type: boolean?
     default: false
+  - id: aoflag_chunksize
+    type: int?
+    default: 2000
 outputs:
   - id: log_files
     outputSource:
@@ -219,6 +222,8 @@ steps:
         source: lbfgs_robustdof
       - id: aoflag_reorder
         source: aoflag_reorder
+      - id: aoflag_chunksize
+        source: aoflag_chunksize
     out:
       - id: logfiles
       - id: solutions

workflows/linc_calibrator.cwl

@@ -121,6 +121,9 @@ inputs:
   - id: aoflag_reorder
     type: boolean?
     default: false
+  - id: aoflag_chunksize
+    type: int?
+    default: 2000
 outputs:
   - id: inspection
     linkMerge: merge_flattened
@@ -216,6 +219,8 @@ steps:
         source: lbfgs_robustdof
       - id: aoflag_reorder
         source: aoflag_reorder
+      - id: aoflag_chunksize
+        source: aoflag_chunksize
     out:
       - id: outh5parm
       - id: logfiles

workflows/linc_calibrator/prep.cwl

@@ -80,6 +80,9 @@ inputs:
   - id: aoflag_reorder
     type: boolean?
     default: false
+  - id: aoflag_chunksize
+    type: int?
+    default: 2000
 outputs:
   - id: outh5parm
     outputSource:
@@ -233,6 +236,8 @@ steps:
         source: rfistrategy
       - id: reorder
         source: aoflag_reorder
+      - id: chunk-size
+        source: aoflag_chunksize
     out:
       - id: output_ms
       - id: logfile

workflows/linc_target.cwl

@@ -151,6 +151,9 @@ inputs:
   - id: aoflag_reorder
     type: boolean?
     default: false
+  - id: aoflag_chunksize
+    type: int?
+    default: 2000
 outputs:
   - id: inspection
     outputSource:
@@ -310,6 +313,8 @@ steps:
         source: rfistrategy
       - id: aoflag_reorder
         source: aoflag_reorder
+      - id: aoflag_chunksize
+        source: aoflag_chunksize
     out:
       - id: msout
       - id: outh5parm

workflows/linc_target/gsmcal.cwl

@@ -43,6 +43,9 @@ inputs:
   - id: aoflag_reorder
     type: boolean?
     default: false
+  - id: aoflag_chunksize
+    type: int?
+    default: 2000
 outputs:
   - id: msout
     outputSource:
@@ -290,6 +293,8 @@ steps:
         source: rfistrategy
       - id: reorder
         source: aoflag_reorder
+      - id: chunk-size
+        source: aoflag_chunksize
     out:
       - id: output_ms
       - id: logfile