Skip to content
Snippets Groups Projects
Commit 6cab2d64 authored by Mattia Mancini's avatar Mattia Mancini
Browse files

Refactor naming and clean up old files

parent 6f8ffe53
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 325 additions and 188 deletions
steps/EmptyParsetGenerator.cwl → steps/DP3.EmptyParsetGenerator.cwl
+ 0
− 0
View file @ 6cab2d64
File moved
steps/parsetselector.cwl → steps/DP3.ParsetSelector.cwl 100644 → 100755
+ 2
− 0
View file @ 6cab2d64
#!/usr/bin/env cwl-runner
class: CommandLineTool
cwlVersion: v1.0
$namespaces:
......
steps/PreflaggingStepGenerator.cwl → steps/DP3.PreflaggerStepGenerator.cwl
+ 306
− 0
View file @ 6cab2d64
......@@ -2,235 +2,305 @@
class: CommandLineTool
cwlVersion: v1.0
id: average_step_generator
baseCommand: [parset_concat.py]
requirements:
DockerRequirement:
dockerPull: prefactor-utils:latest
arguments: ['preflagger']
$namespaces:
sbg: 'https://www.sevenbridges.com/'
id: preflagging_step_generator
baseCommand:
- parset_concat.py
inputs:
- id: input_parset
type: File
inputBinding:
position: -1
prefix: --input_parset
- id: step_name
position: -1
prefix: '--input_parset'
- default: average
id: step_name
type: string
default: average
doc: unique name for the step
inputBinding:
position: -1
position: -1
doc: unique name for the step
- id: input_secondary_files
type: File[]
type: 'File[]'
doc: Files other than the input used to process the data
###############################################################################
- id: count.save
- default: false
id: count.save
type: boolean
default: false
doc: If true, the flag percentages per frequency are saved to a table with extension .flagfreq and percentages per station to a table with extension .flagstat. The basename of the table is the MS name (without extension) followed by the stepname and extension.
inputBinding:
prefix: <step>.count.save=
position: 0
prefix: count.save=
separate: false
- id: count.path
doc: >-
If true, the flag percentages per frequency are saved to a table with
extension .flagfreq and percentages per station to a table with extension
.flagstat. The basename of the table is the MS name (without extension)
followed by the stepname and extension.
- default: ''
id: count.path
type: string
default: ''
doc: The directory where to create the flag percentages table. If empty, the path of the input MS is used.
inputBinding:
position: 0
prefix: count.path=
separate: false
- id: mode
doc: >-
The directory where to create the flag percentages table. If empty, the
path of the input MS is used.
- default: set
id: mode
type: string
default: set
doc: Case-insensitive string telling what to do with the flags of the data matching (or not matching) the selection criteria given in the other parameters.
inputBinding:
position: 0
prefix: mode=
separate: false
- id: expr
type: string
default: []
doc: 'Expression of preflagger keyword sets (see above). Operators AND, OR, and NOT are possible (or their equivalents &&,&, ||, |, and !). Parentheses can be used to change precedence order. For example:: c1 and (c2 or c3)'
doc: >-
Case-insensitive string telling what to do with the flags of the data
matching (or not matching) the selection criteria given in the other
parameters.
- default: ''
id: expr
type: string?
inputBinding:
position: 0
prefix: expr=
separate: false
- id: timeofday
doc: >-
Expression of preflagger keyword sets (see above). Operators AND, OR, and
NOT are possible (or their equivalents &&,&, ||, |, and !). Parentheses
can be used to change precedence order. For example:: c1 and (c2 or c3)
- default:
id: timeofday
type: string
default: []
doc: Ranges of UTC time-of-day given as st..end or val+-delta. Each value must be given as 12:34:56.789, 12h34m56.789, or as a value followed by a unit like h, min, or s.
inputBinding:
position: 0
prefix: timeofday=
separate: false
- id: abstime
doc: >-
Ranges of UTC time-of-day given as st..end or val+-delta. Each value must
be given as 12:34:56.789, 12h34m56.789, or as a value followed by a unit
like h, min, or s.
- default: '[]'
id: abstime
type: string
default: '[]'
doc: Ranges of absolute UTC date/time given as st..end or val+-delta. Each value (except delta) must be given as a date/time in casacore MVTime format, for instance 12-Mar-2010/11:31:00.000. A delta value must be given as a time (for instance 1:30:0 or 20s).
inputBinding:
position: 0
prefix: abstime=
separate: false
- id: reltime
doc: >-
Ranges of absolute UTC date/time given as st..end or val+-delta. Each
value (except delta) must be given as a date/time in casacore MVTime
format, for instance 12-Mar-2010/11:31:00.000. A delta value must be given
as a time (for instance 1:30:0 or 20s).
- default: '[]'
id: reltime
type: string
default: '[]'
doc: Ranges of times (using .. or +-) since the start of the observation. A time can be given like 1:30:0 or 20s.
inputBinding:
position: 0
prefix: reltime=
separate: false
- id: timeslot
doc: >-
Ranges of times (using .. or +-) since the start of the observation. A
time can be given like 1:30:0 or 20s.
- default: '[]'
id: timeslot
type: string
default: '[]'
doc: Time slot sequence numbers. First time slot is 0. st..end means end inclusive.
inputBinding:
position: 0
prefix: timeslot=
separate: false
- id: lst
type: string[]
default: []
doc: Ranges of Local Apparent Sidereal Times like 1:30:0 +- 20min. The LST of a time slot is calculated for the array position, thus not per antenna.
doc: >-
Time slot sequence numbers. First time slot is 0. st..end means end
inclusive.
- default: []
id: lst
type: 'string[]'
inputBinding:
position: 0
prefix: lst=
separate: false
- id: azimuth
type: float[]
default: []
doc: Ranges of azimuth angles given as st..end or val+-delta. Each value has to be given as a casacore direction like 12:34:56.789 or 12h34m56.789, 12.34.56.789 or 12d34m56.789, or a value followed by a unit like rad or deg.
doc: >-
Ranges of Local Apparent Sidereal Times like 1:30:0 +- 20min. The LST of a
time slot is calculated for the array position, thus not per antenna.
- default: []
id: azimuth
type: 'float[]'
inputBinding:
position: 0
prefix: azimuth=
separate: false
doc: >-
Ranges of azimuth angles given as st..end or val+-delta. Each value has to
be given as a casacore direction like 12:34:56.789 or 12h34m56.789,
12.34.56.789 or 12d34m56.789, or a value followed by a unit like rad or
deg.
- id: elevation
type: float[]
default: []
doc: 'Ranges of elevation angles (similar to azimuth). For example:: 0deg..10deg'
type: string?
inputBinding:
position: 0
prefix: elevation=
separate: false
- id: baseline
doc: 'Ranges of elevation angles (similar to azimuth). For example:: 0deg..10deg'
- default: []
id: baseline
type: string[]
default: ''
doc: See Description of baseline selection parameters.
inputBinding:
position: 0
prefix: baseline=
separate: false
- id: corrtype
itemSeparator: ','
doc: See Description of baseline selection parameters.
- default: ''
id: corrtype
type: string
default: ''
doc: Correlation type to match? Must be auto, cross, or an empty string.
inputBinding:
position: 0
prefix: corrtype=
separate: false
- id: blmin
type: double
default: -1
doc: If blmin > 0, baselines with length < blmin meter will match.
doc: 'Correlation type to match? Must be auto, cross, or an empty string.'
- default: -1
id: blmin
type: float
inputBinding:
position: 0
prefix: blmin=
separate: false
- id: blmax
type: double
default: -1
doc: If blmax > 0, baselines with length > blmax meter will match.
doc: 'If blmin > 0, baselines with length < blmin meter will match.'
- default: -1
id: blmax
type: float
inputBinding:
position: 0
prefix: blmax=
separate: false
- id: uvmmin
type: double
default: -1
doc: If uvmmin > 0, baselines with UV-distance < uvmmin meter will match. Note that the UV-distance is the projected baseline length.
doc: 'If blmax > 0, baselines with length > blmax meter will match.'
- default: -1
id: uvmmin
type: float
inputBinding:
position: 0
prefix: uvmmin=
separate: false
- id: uvmmax
type: double
default: -1
doc: If uvmmax > 0, baselines with UV-distance > uvmmax meter will match.
doc: >-
If uvmmin > 0, baselines with UV-distance < uvmmin meter will match. Note
that the UV-distance is the projected baseline length.
- default: -1
id: uvmmax
type: float
inputBinding:
position: 0
prefix: uvmmax=
separate: false
- id: freqrange
doc: 'If uvmmax > 0, baselines with UV-distance > uvmmax meter will match.'
- default: ''
id: freqrange
type: string
default: []
doc: 'Channels in the given frequency ranges will match. Each value in the vector is a range which can be given as start..end or start+-delta. A value can be followed by a unit like KHz. If only one value in a range has a unit, the unit is also applied to the other value. If a range has no unit, it defaults to MHz. For example: freqrange=[1.2 .. 1.4 MHz, 1.8MHz+-50KHz] flags channels between 1.2MHz and 1.4MHz and between 1.75MHz and 1.85MHz. The example shows that blanks can be used at will.'
inputBinding:
position: 0
prefix: freqrange=
separate: false
- id: chan
doc: >-
Channels in the given frequency ranges will match. Each value in the
vector is a range which can be given as start..end or start+-delta. A
value can be followed by a unit like KHz. If only one value in a range has
a unit, the unit is also applied to the other value. If a range has no
unit, it defaults to MHz. For example: freqrange=[1.2 .. 1.4 MHz,
1.8MHz+-50KHz] flags channels between 1.2MHz and 1.4MHz and between
1.75MHz and 1.85MHz. The example shows that blanks can be used at will.
- default: ''
id: chan
type: string
default: []
doc: 'The given channels will match (start counting at 0). Channels exceeding the number of channels are ignored. Similar to msin, it is possible to specify the channels as an expression of nchan. Furthermore, .. can be used to specify ranges. For example: chan=[0..nchan/32-1, 31*nchan/32..nchan-1] to flag the first and last 2 or 8 channels (depending on 64 or 256 channels in the observation).'
inputBinding:
position: 0
prefix: chan=
separate: false
- id: amplmin
doc: >-
The given channels will match (start counting at 0). Channels exceeding
the number of channels are ignored. Similar to msin, it is possible to
specify the channels as an expression of nchan. Furthermore, .. can be
used to specify ranges. For example: chan=[0..nchan/32-1,
31*nchan/32..nchan-1] to flag the first and last 2 or 8 channels
(depending on 64 or 256 channels in the observation).
- default: -1e+30
id: amplmin
type: float
default: -1.e30
doc: "Correlation data with amplitude < amplmin will match. It can be given per correlation. For example, amplmin=[100,,,100] matches data points with XX or YY amplitude < 100. The non-specified amplitudes get the default value."
inputBinding:
position: 0
prefix: amplmin=
separate: false
- id: amplmax
doc: >-
Correlation data with amplitude < amplmin will match. It can be given per
correlation. For example, amplmin=[100,,,100] matches data points with XX
or YY amplitude < 100. The non-specified amplitudes get the default value.
- default: 1e+30
id: amplmax
type: float
default: 1e+030
doc: Correlation data with amplitude > amplmax will match.
inputBinding:
position: 0
prefix: amplmax=
separate: false
- id: phasemin
doc: Correlation data with amplitude > amplmax will match.
- default: -1e+30
id: phasemin
type: float
default: -1.e30
doc: Correlation data with phase < phasemin (in radians) will match.
inputBinding:
position: 0
prefix: phasemin=
separate: false
- id: phasemax
doc: Correlation data with phase < phasemin (in radians) will match.
- default: 1e+30
id: phasemax
type: float
default: +1.e30
doc: Correlation data with phase > phasemax (in radians) will match.
inputBinding:
position: 0
prefix: phasemax=
separate: false
- id: realmin
doc: Correlation data with phase > phasemax (in radians) will match.
- default: -1e+30
id: realmin
type: float
default: -1.e30
doc: Correlation data with real complex part < realmin will match.
inputBinding:
position: 0
prefix: realmin=
separate: false
- id: realmax
doc: Correlation data with real complex part < realmin will match.
- default: 1e+30
id: realmax
type: float
default: 1.e30
doc: Correlation data with real complex part > realmax will match.
inputBinding:
position: 0
prefix: realmax=
separate: false
- id: imagmin
doc: Correlation data with real complex part > realmax will match.
- default: -1e+30
id: imagmin
type: float
default: -1.e+30
doc: Correlation data with imaginary complex part < imagmin will match.
inputBinding:
position: 0
prefix: imagmin=
separate: false
- id: imagmax
doc: Correlation data with imaginary complex part < imagmin will match.
- default: 1e+30
id: imagmax
type: float
default: 1.e+30
doc: Correlation data with imaginary complex part > imagmax will match.
inputBinding:
position: 0
prefix: imagmax=
separate: false
stdout: output_parset
doc: Correlation data with imaginary complex part > imagmax will match.
outputs:
- id: output_parset
doc: Parset output file
streamable: True
type: File
outputBinding:
glob: output_parset
glob: output_parset
streamable: true
- id: output_secondary_files
doc: files needed to execute the step
type: File[]
type: 'File[]'
outputBinding:
outputEval: $(inputs.input_secondary_files)
arguments:
- preflagger
requirements:
- class: DockerRequirement
dockerPull: 'prefactor-utils:latest'
- class: InlineJavascriptRequirement
stdout: output_parset
steps/DP3_flag_averaging.cwl deleted 100644 → 0
+ 0
− 123
View file @ 6f8ffe53
cwlVersion: 1.1
class: CommandLineTool
baseCommand: [DPPP]
label: "Flagging and Averaging"
hints:
DockerRequirements:
dockerPull: lofaruser/imaging-pipeline
inputs:
msin:
type: Directory
msin_data_column:
type: string
default: "DATA"
msin_writefullresflag:
type: boolean
default: False
msin_baselines:
type: string
default: ""
msin_autoweight:
type: boolean
default: False
msout_data_column:
type: string
default: "DATA"
msout_writefullresflag:
type: boolean
default: False
msout_overwrite:
type: boolean
default: True
msout_storagemanager:
type: string
default: "Dysco"
msout_storagemanager_databitrate:
type: int
default: 0
flagger_memory_used:
type: int
default: 0
flagger_keep_statistics:
type: boolean
default: False
flag_baseline:
type: string
default: ""
flag_elevation:
type: string
default: "0deg..20deg"
flag_amplitude:
type: double
default: 1.e-30
interpolate_windowsize:
type: int
avg_timeresolution:
type: int
avg_frequenciesresolution:
type: int
demixer_baseline:
type: string
default: ""
demixer_frequency_steps:
type: int
default: 4
demixer_time_steps:
type: int
default: 4
demixer_ignore_target:
type: boolean
default: False
demixer_target_source:
type: string
demixer_subtract_sources:
type: string
outputs:
ndppp_prep_cal.control.type = dppp
ndppp_prep_cal.control.max_per_node = {{ num_proc_per_node_limit }}
ndppp_prep_cal.control.error_tolerance = {{ error_tolerance }}
ndppp_prep_cal.argument.numthreads = {{ max_dppp_threads }}
ndppp_prep_cal.argument.steps = [{{ initial_flagging }},{{ demix_step }}interp,avg]
ndppp_prep_cal.argument.flagedge.type = preflagger
ndppp_prep_cal.argument.flagedge.chan = [0..nchan/32-1,31*nchan/32..nchan-1] # we are running on a single subband
ndppp_prep_cal.argument.flagbaseline.type = preflagger
ndppp_prep_cal.argument.flagamp.type = preflagger
ndppp_prep_cal.argument.interp.type = interpolate
ndppp_prep_cal.argument.avg.type = average
ndppp_prep_cal.argument.flagamp.type = preflagger
ndppp_prep_cal.argument.demix.type = demixer
steps/DPPPStep.cwl → steps/DPPP.cwl
+ 0
− 0
View file @ 6cab2d64
File moved
steps/GenericParsetGeneratorStep.cwl deleted 100755 → 0
+ 0
− 42
View file @ 6f8ffe53
#!/usr/bin/env cwl-runner
class: CommandLineTool
cwlVersion: v1.0
id: _generic_parset_generator_step
baseCommand: [parset_concat.py]
requirements:
DockerRequirement:
dockerPull: prefactor-utils:latest
inputs:
- id: step_name
type: string
inputBinding:
position: 1
doc: name of the DPPP step
- id: step_type
type: string
inputBinding:
position: 2
doc: type of the DPPP step
- id: step_arguments
type: 'string[]'
inputBinding:
position: 3
doc: List of key=value parameters to pass
outputs:
- id: output_parset
doc: Parset output file
type: File
streamable: true
outputBinding: a_stdout_file
- id: input_secondary_files
doc: files needed to execute the step
type: File[]
outputBinding:
outputEval: $(inputs.input_secondary_files)
stdout: a_stdout_file
steps/check_ateam_separation.cwl 100644 → 100755
+ 2
− 0
View file @ 6cab2d64
#!/usr/bin/env cwl-runner
class: CommandLineTool
cwlVersion: v1.0
id: check_ateam_separation
......
steps/echo.cwl deleted 100644 → 0
+ 0
− 12
View file @ 6f8ffe53
class: CommandLineTool
cwlVersion: v1.0
id: echo
baseCommand:
- cat
inputs:
file:
type: File
inputBinding:
position: 1
outputs: []
\ No newline at end of file
steps/make_sourcedb_ateam.cwl → steps/makesourcedb.cwl 100644 → 100755
+ 2
− 0
View file @ 6cab2d64
#!/usr/bin/env cwl-runner
class: CommandLineTool
cwlVersion: v1.0
$namespaces:
......
subworkflow/ndppp_prep_cal.cwl
+ 13
− 11
View file @ 6cab2d64
#!/usr/bin/env cwl-runner
class: Workflow
cwlVersion: v1.0
id: ndppp_prep_cal
......@@ -10,7 +12,7 @@ inputs:
'sbg:x': 612.3626098632812
'sbg:y': -442.8657531738281
- id: elevation_to_flag
type: float
type: string
'sbg:x': 847
'sbg:y': -501.7469482421875
- id: min_amplitude_to_flag
......@@ -52,7 +54,7 @@ steps:
out:
- id: output_parset
- id: output_secondary_files
run: ../steps/PreflaggingStepGenerator.cwl
run: ../steps/DP3.PreflaggerStepGenerator.cwl
'sbg:x': 1160.802978515625
'sbg:y': -95.15213775634766
- id: flagelev
......@@ -70,7 +72,7 @@ steps:
out:
- id: output_parset
- id: output_secondary_files
run: ../steps/PreflaggingStepGenerator.cwl
run: ../steps/DP3.PreflaggerStepGenerator.cwl
'sbg:x': 1459.2421875
'sbg:y': -83.970458984375
- id: flagamp
......@@ -87,7 +89,7 @@ steps:
out:
- id: output_parset
- id: output_secondary_files
run: ../steps/PreflaggingStepGenerator.cwl
run: ../steps/DP3.PreflaggerStepGenerator.cwl
'sbg:x': 1659.302001953125
'sbg:y': -23
- id: empty_parset_generator
......@@ -95,7 +97,7 @@ steps:
out:
- id: output_parset
- id: output_secondary_files
run: ../steps/EmptyParsetGenerator.cwl
run: ../steps/DP3.EmptyParsetGenerator.cwl
'sbg:x': -187.92431640625
'sbg:y': 35.5892333984375
- id: flagedge
......@@ -112,7 +114,7 @@ steps:
out:
- id: output_parset
- id: output_secondary_files
run: ../steps/PreflaggingStepGenerator.cwl
run: ../steps/DP3.PreflaggerStepGenerator.cwl
'sbg:x': 53.59375762939453
'sbg:y': -119.96077728271484
- id: aoflag
......@@ -131,7 +133,7 @@ steps:
out:
- id: output_parset
- id: output_secondary_files
run: ../steps/AOFlaggerStepGenerator.cwl
run: ../steps/DP3.AOFlaggerStepGenerator.cwl
'sbg:x': 357.934326171875
'sbg:y': -129.337890625
- id: demix_select
......@@ -151,7 +153,7 @@ steps:
out:
- id: output_secondary_files
- id: output_parset
run: ../steps/parsetselector.cwl
run: ../steps/DP3.ParsetSelector.cwl
label: demix?
'sbg:x': 2144.327392578125
'sbg:y': -22.676969528198242
......@@ -165,7 +167,7 @@ steps:
out:
- id: output_parset
- id: output_secondary_files
run: ../steps/DemixingStepGenerator.cwl
run: ../steps/DP3.DemixerStepGenerator.cwl
'sbg:x': 1921.2374267578125
'sbg:y': 62.99578094482422
- id: process_raw_data_select
......@@ -185,7 +187,7 @@ steps:
out:
- id: output_secondary_files
- id: output_parset
run: ../steps/parsetselector.cwl
run: ../steps/DP3.ParsetSelector.cwl
label: ParsetSelector
'sbg:x': 572.44970703125
'sbg:y': 38.326629638671875
......@@ -200,7 +202,7 @@ steps:
- demix_select/output_secondary_files
out:
- id: msout
run: ../steps/DPPPStep.cwl
run: ../steps/DPPP.cwl
'sbg:x': 2516.561767578125
'sbg:y': -25.858686447143555
requirements:
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment