Skip to content
Snippets Groups Projects
Commit 3dcea906 authored by alex's avatar alex
Browse files

Use time chunking in aoflagger/LINC (RAP-29)

parent 60a46bcf
No related branches found
No related tags found
1 merge request!125Use time chunking in aoflagger/LINC (RAP-29)
Pipeline #34710 passed
...@@ -278,7 +278,8 @@ A comprehensive explanation of the baseline selection syntax can be found `here` ...@@ -278,7 +278,8 @@ A comprehensive explanation of the baseline selection syntax can be found `here`
- ``max_dp3_threads``: number of threads per process for DP3 (default: 10) - ``max_dp3_threads``: number of threads per process for DP3 (default: 10)
- ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20) - ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20)
- ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (see the `AOFlagger manual`_`) - ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (default: false, see the `AOFlagger manual`_`)
- ``aoflag_chunksize``: this will split the set into intervals with the given maximum size, and flag each interval independently. This lowers the amount of memory required (default: 2000)
**Parameters you may want to adjust** **Parameters you may want to adjust**
......
...@@ -83,7 +83,8 @@ There are more parameters you may want to adjust that can be added to this input ...@@ -83,7 +83,8 @@ There are more parameters you may want to adjust that can be added to this input
"bandpass_freqresolution": "195.3125kHz", "bandpass_freqresolution": "195.3125kHz",
"lbfgs_historysize" : 10, "lbfgs_historysize" : 10,
"lbfgs_robustdof" : 200, "lbfgs_robustdof" : 200,
"reorder_aoflag" : false "aoflag_reorder" : false,
"aoflag_chunksize" : 2000
} }
If you just want to alter one of the defaults it is sufficient to override it by specifing its new value the JSON input file:: If you just want to alter one of the defaults it is sufficient to override it by specifing its new value the JSON input file::
......
...@@ -304,7 +304,8 @@ A comprehensive explanation of the baseline selection syntax can be found `here` ...@@ -304,7 +304,8 @@ A comprehensive explanation of the baseline selection syntax can be found `here`
- ``max_dp3_threads``: number of threads per process for DP3 (default: 10) - ``max_dp3_threads``: number of threads per process for DP3 (default: 10)
- ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20) - ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20)
- ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (see the `AOFlagger manual`_) - ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (default: false, see the `AOFlagger manual`_`)
- ``aoflag_chunksize``: this will split the set into intervals with the given maximum size, and flag each interval independently. This lowers the amount of memory required (default: 2000)
*Skymodel directory* *Skymodel directory*
......
...@@ -133,6 +133,13 @@ inputs: ...@@ -133,6 +133,13 @@ inputs:
prefix: -concatenate-frequency prefix: -concatenate-frequency
position: 0 position: 0
shellQuote: false shellQuote: false
- id: chunk-size
type: int?
doc: This will split the set into intervals with the given maximum size, and flag each interval independently
inputBinding:
prefix: -chunk-size
position: 0
shellQuote: false
outputs: outputs:
- id: output_ms - id: output_ms
......
...@@ -122,6 +122,9 @@ inputs: ...@@ -122,6 +122,9 @@ inputs:
- id: aoflag_reorder - id: aoflag_reorder
type: boolean? type: boolean?
default: false default: false
- id: aoflag_chunksize
type: int?
default: 2000
outputs: outputs:
- id: log_files - id: log_files
outputSource: outputSource:
...@@ -221,6 +224,8 @@ steps: ...@@ -221,6 +224,8 @@ steps:
source: lbfgs_robustdof source: lbfgs_robustdof
- id: aoflag_reorder - id: aoflag_reorder
source: aoflag_reorder source: aoflag_reorder
- id: aoflag_chunksize
source: aoflag_chunksize
out: out:
- id: logfiles - id: logfiles
- id: solutions - id: solutions
......
...@@ -151,6 +151,9 @@ inputs: ...@@ -151,6 +151,9 @@ inputs:
- id: aoflag_reorder - id: aoflag_reorder
type: boolean? type: boolean?
default: false default: false
- id: aoflag_chunksize
type: int?
default: 2000
outputs: outputs:
- id: calibrated_data - id: calibrated_data
outputSource: outputSource:
...@@ -273,6 +276,8 @@ steps: ...@@ -273,6 +276,8 @@ steps:
source: lbfgs_robustdof source: lbfgs_robustdof
- id: aoflag_reorder - id: aoflag_reorder
source: aoflag_reorder source: aoflag_reorder
- id: aoflag_chunksize
source: aoflag_chunksize
out: out:
- id: logfiles - id: logfiles
- id: msout - id: msout
......
...@@ -120,6 +120,9 @@ inputs: ...@@ -120,6 +120,9 @@ inputs:
- id: aoflag_reorder - id: aoflag_reorder
type: boolean? type: boolean?
default: false default: false
- id: aoflag_chunksize
type: int?
default: 2000
outputs: outputs:
- id: log_files - id: log_files
outputSource: outputSource:
...@@ -219,6 +222,8 @@ steps: ...@@ -219,6 +222,8 @@ steps:
source: lbfgs_robustdof source: lbfgs_robustdof
- id: aoflag_reorder - id: aoflag_reorder
source: aoflag_reorder source: aoflag_reorder
- id: aoflag_chunksize
source: aoflag_chunksize
out: out:
- id: logfiles - id: logfiles
- id: solutions - id: solutions
......
...@@ -121,6 +121,9 @@ inputs: ...@@ -121,6 +121,9 @@ inputs:
- id: aoflag_reorder - id: aoflag_reorder
type: boolean? type: boolean?
default: false default: false
- id: aoflag_chunksize
type: int?
default: 2000
outputs: outputs:
- id: inspection - id: inspection
linkMerge: merge_flattened linkMerge: merge_flattened
...@@ -216,6 +219,8 @@ steps: ...@@ -216,6 +219,8 @@ steps:
source: lbfgs_robustdof source: lbfgs_robustdof
- id: aoflag_reorder - id: aoflag_reorder
source: aoflag_reorder source: aoflag_reorder
- id: aoflag_chunksize
source: aoflag_chunksize
out: out:
- id: outh5parm - id: outh5parm
- id: logfiles - id: logfiles
......
...@@ -80,6 +80,9 @@ inputs: ...@@ -80,6 +80,9 @@ inputs:
- id: aoflag_reorder - id: aoflag_reorder
type: boolean? type: boolean?
default: false default: false
- id: aoflag_chunksize
type: int?
default: 2000
outputs: outputs:
- id: outh5parm - id: outh5parm
outputSource: outputSource:
...@@ -233,6 +236,8 @@ steps: ...@@ -233,6 +236,8 @@ steps:
source: rfistrategy source: rfistrategy
- id: reorder - id: reorder
source: aoflag_reorder source: aoflag_reorder
- id: chunk-size
source: aoflag_chunksize
out: out:
- id: output_ms - id: output_ms
- id: logfile - id: logfile
......
...@@ -151,6 +151,9 @@ inputs: ...@@ -151,6 +151,9 @@ inputs:
- id: aoflag_reorder - id: aoflag_reorder
type: boolean? type: boolean?
default: false default: false
- id: aoflag_chunksize
type: int?
default: 2000
outputs: outputs:
- id: inspection - id: inspection
outputSource: outputSource:
...@@ -310,6 +313,8 @@ steps: ...@@ -310,6 +313,8 @@ steps:
source: rfistrategy source: rfistrategy
- id: aoflag_reorder - id: aoflag_reorder
source: aoflag_reorder source: aoflag_reorder
- id: aoflag_chunksize
source: aoflag_chunksize
out: out:
- id: msout - id: msout
- id: outh5parm - id: outh5parm
......
...@@ -43,6 +43,9 @@ inputs: ...@@ -43,6 +43,9 @@ inputs:
- id: aoflag_reorder - id: aoflag_reorder
type: boolean? type: boolean?
default: false default: false
- id: aoflag_chunksize
type: int?
default: 2000
outputs: outputs:
- id: msout - id: msout
outputSource: outputSource:
...@@ -290,6 +293,8 @@ steps: ...@@ -290,6 +293,8 @@ steps:
source: rfistrategy source: rfistrategy
- id: reorder - id: reorder
source: aoflag_reorder source: aoflag_reorder
- id: chunk-size
source: aoflag_chunksize
out: out:
- id: output_ms - id: output_ms
- id: logfile - id: logfile
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment