Skip to content
Snippets Groups Projects
Commit 60a46bcf authored by alex's avatar alex
Browse files

Rename all uses of NDPPP and DPPP to DP3 (RAP-31)

parent 59093c7c
Branches
Tags
1 merge request!126Rename all uses of NDPPP and DPPP to DP3 (RAP-31)
Pipeline #34655 failed
Stage: versioning
Stage: build
Stage: tests
Stage: docs
Stage: deploy
Show whitespace changes
Inline Side-by-side
Showing
with 117 additions and 117 deletions
docs/source/calibrator.rst
+ 3
− 3
View file @ 60a46bcf
...@@ -39,7 +39,7 @@ The workflow ``prep`` consists of: ...@@ -39,7 +39,7 @@ The workflow ``prep`` consists of:
- determining suitable calibrator skymodel (steps ``find_skymodel_cal``, ``make_sourcedb``) - determining suitable calibrator skymodel (steps ``find_skymodel_cal``, ``make_sourcedb``)
- checking for nearby A-Team sources (step ``check_Ateam_separation``) - checking for nearby A-Team sources (step ``check_Ateam_separation``)
- creating a model of A-Team sources to be subtracted (step ``make_sourcedb_ateam``) - creating a model of A-Team sources to be subtracted (step ``make_sourcedb_ateam``)
- basic flagging and averaging (subworkflow ``ndppp_prep_cal``) - basic flagging and averaging (subworkflow ``dp3_prep_cal``)
- edges of the band (``flagedge``) -- only used if ``raw_data : true`` - edges of the band (``flagedge``) -- only used if ``raw_data : true``
- statistical flagging (``aoflag``) -- only used in ``raw_data : true`` - statistical flagging (``aoflag``) -- only used in ``raw_data : true``
- baseline flagging (``flagbaseline``) - baseline flagging (``flagbaseline``)
...@@ -276,7 +276,7 @@ A comprehensive explanation of the baseline selection syntax can be found `here` ...@@ -276,7 +276,7 @@ A comprehensive explanation of the baseline selection syntax can be found `here`
**Parameters for pipeline performance** **Parameters for pipeline performance**
- ``max_dppp_threads``: number of threads per process for DP3 (default: 10) - ``max_dp3_threads``: number of threads per process for DP3 (default: 10)
- ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20) - ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20)
- ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (see the `AOFlagger manual`_`) - ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (see the `AOFlagger manual`_`)
...@@ -294,6 +294,6 @@ A comprehensive explanation of the baseline selection syntax can be found `here` ...@@ -294,6 +294,6 @@ A comprehensive explanation of the baseline selection syntax can be found `here`
- ``avg_freqresolution`` : final frequency resolution of the data after averaging (default: 48.82kHz, which translates to 4 channels per subband) - ``avg_freqresolution`` : final frequency resolution of the data after averaging (default: 48.82kHz, which translates to 4 channels per subband)
- ``bandpass_freqresolution``: frequency resolution of the bandpass solution table (default: 195.3125kHz, which translates to 1 channel per subband) - ``bandpass_freqresolution``: frequency resolution of the bandpass solution table (default: 195.3125kHz, which translates to 1 channel per subband)
.. _here: https://www.astron.nl/lofarwiki/doku.php?id=public:user_software:documentation:ndppp#description_of_baseline_selection_parameters .. _here: https://dp3.readthedocs.io/en/latest/steps/Description%20of%20baseline%20selection%20parameters.html
.. _AOFlagger: https://gitlab.com/aroffringa/aoflagger.git .. _AOFlagger: https://gitlab.com/aroffringa/aoflagger.git
.. _AOFlagger manual: https://aoflagger.readthedocs.io/en/latest/using_aoflagger.html .. _AOFlagger manual: https://aoflagger.readthedocs.io/en/latest/using_aoflagger.html
\ No newline at end of file
docs/source/changelog.rst
+ 1
− 1
View file @ 60a46bcf
...@@ -36,7 +36,7 @@ Version 2.0 ...@@ -36,7 +36,7 @@ Version 2.0
* applying Ionospheric RM corrections * applying Ionospheric RM corrections
* grouping of subbands by actual frequency * grouping of subbands by actual frequency
* speed and disk usage improvements by optimized usage of NDPPP * speed and disk usage improvements by optimized usage of DP3
* (optional) wide-band cleaning in Initial-Subtract * (optional) wide-band cleaning in Initial-Subtract
* more diagnostic plots * more diagnostic plots
* documentation moved to GitHib wiki: https://github.com/lofar-astron/prefactor/wiki * documentation moved to GitHib wiki: https://github.com/lofar-astron/prefactor/wiki
......
docs/source/parset.rst
+ 41
− 41
View file @ 60a46bcf
...@@ -19,16 +19,16 @@ A minimum input file may look like this:: ...@@ -19,16 +19,16 @@ A minimum input file may look like this::
{ {
"msin": [ "msin": [
{"class": "Directory", "path": "3C286/L228161_SB000_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB000_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB001_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB001_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB002_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB002_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB003_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB003_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB004_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB004_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB005_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB005_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB006_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB006_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB007_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB007_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB008_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB008_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB009_uv.dppp.MS"} {"class": "Directory", "path": "3C286/L228161_SB009_uv.MS"}
], ],
} }
...@@ -36,16 +36,16 @@ There are more parameters you may want to adjust that can be added to this input ...@@ -36,16 +36,16 @@ There are more parameters you may want to adjust that can be added to this input
{ {
"msin": [ "msin": [
{"class": "Directory", "path": "3C286/L228161_SB000_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB000_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB001_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB001_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB002_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB002_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB003_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB003_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB004_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB004_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB005_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB005_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB006_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB006_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB007_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB007_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB008_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB008_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB009_uv.dppp.MS"} {"class": "Directory", "path": "3C286/L228161_SB009_uv.MS"}
], ],
"refant": "CS00.*", "refant": "CS00.*",
"flag_baselines": [], "flag_baselines": [],
...@@ -72,7 +72,7 @@ There are more parameters you may want to adjust that can be added to this input ...@@ -72,7 +72,7 @@ There are more parameters you may want to adjust that can be added to this input
"ion_3rd": false, "ion_3rd": false,
"clock_smooth": true, "clock_smooth": true,
"tables2export": "clock", "tables2export": "clock",
"max_dppp_threads": 10, "max_dp3_threads": 10,
"memoryperc": 20, "memoryperc": 20,
"min_separation": 30, "min_separation": 30,
"max_separation_arcmin": 1.0, "max_separation_arcmin": 1.0,
...@@ -90,16 +90,16 @@ If you just want to alter one of the defaults it is sufficient to override it by ...@@ -90,16 +90,16 @@ If you just want to alter one of the defaults it is sufficient to override it by
{ {
"msin": [ "msin": [
{"class": "Directory", "path": "3C286/L228161_SB000_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB000_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB001_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB001_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB002_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB002_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB003_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB003_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB004_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB004_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB005_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB005_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB006_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB006_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB007_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB007_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB008_uv.dppp.MS"}, {"class": "Directory", "path": "3C286/L228161_SB008_uv.MS"},
{"class": "Directory", "path": "3C286/L228161_SB009_uv.dppp.MS"} {"class": "Directory", "path": "3C286/L228161_SB009_uv.MS"}
], ],
"demix": true, "demix": true,
} }
...@@ -108,16 +108,16 @@ If you run the target pipeline you also need to provide the calibrator solution ...@@ -108,16 +108,16 @@ If you run the target pipeline you also need to provide the calibrator solution
{ {
"msin": [ "msin": [
{"class": "Directory", "path": "?????/L228163_SB000_uv.dppp.MS"}, {"class": "Directory", "path": "?????/L228163_SB000_uv.MS"},
{"class": "Directory", "path": "?????/L228163_SB001_uv.dppp.MS"}, {"class": "Directory", "path": "?????/L228163_SB001_uv.MS"},
{"class": "Directory", "path": "?????/L228163_SB002_uv.dppp.MS"}, {"class": "Directory", "path": "?????/L228163_SB002_uv.MS"},
{"class": "Directory", "path": "?????/L228163_SB003_uv.dppp.MS"}, {"class": "Directory", "path": "?????/L228163_SB003_uv.MS"},
{"class": "Directory", "path": "?????/L228163_SB004_uv.dppp.MS"}, {"class": "Directory", "path": "?????/L228163_SB004_uv.MS"},
{"class": "Directory", "path": "?????/L228163_SB005_uv.dppp.MS"}, {"class": "Directory", "path": "?????/L228163_SB005_uv.MS"},
{"class": "Directory", "path": "?????/L228163_SB006_uv.dppp.MS"}, {"class": "Directory", "path": "?????/L228163_SB006_uv.MS"},
{"class": "Directory", "path": "?????/L228163_SB007_uv.dppp.MS"}, {"class": "Directory", "path": "?????/L228163_SB007_uv.MS"},
{"class": "Directory", "path": "?????/L228163_SB008_uv.dppp.MS"}, {"class": "Directory", "path": "?????/L228163_SB008_uv.MS"},
{"class": "Directory", "path": "?????/L228163_SB009_uv.dppp.MS"} {"class": "Directory", "path": "?????/L228163_SB009_uv.MS"}
], ],
"cal_solutions": {"class": "File", "path": "cal_solutions.h5"} "cal_solutions": {"class": "File", "path": "cal_solutions.h5"}
} }
......
docs/source/target.rst
+ 6
− 6
View file @ 60a46bcf
...@@ -42,7 +42,7 @@ The workflow ``prep`` consists of: ...@@ -42,7 +42,7 @@ The workflow ``prep`` consists of:
.. image:: RMextract.png .. image:: RMextract.png
- basic flagging, applying solutions, and averaging (subworkflow ``ndppp_prep_target``) - basic flagging, applying solutions, and averaging (subworkflow ``dp3_prep_target``)
- edges of the band (``flagedge``) -- only used if ``raw_data : true`` - edges of the band (``flagedge``) -- only used if ``raw_data : true``
- statistical flagging (``aoflag``) -- only used in ``raw_data : true`` - statistical flagging (``aoflag``) -- only used in ``raw_data : true``
- baseline flagging (``flagbaseline``) - baseline flagging (``flagbaseline``)
...@@ -302,9 +302,9 @@ A comprehensive explanation of the baseline selection syntax can be found `here` ...@@ -302,9 +302,9 @@ A comprehensive explanation of the baseline selection syntax can be found `here`
**Parameters for pipeline performance** **Parameters for pipeline performance**
- ``max_dppp_threads``: number of threads per process for DP3 (default: 10) - ``max_dp3_threads``: number of threads per process for DP3 (default: 10)
- ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20) - ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20)
- ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (see the `AOFlagger manual`_`) - ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (see the `AOFlagger manual`_)
*Skymodel directory* *Skymodel directory*
...@@ -342,8 +342,8 @@ If your **LBA** data has **not** been demixed before you may still want to keep ...@@ -342,8 +342,8 @@ If your **LBA** data has **not** been demixed before you may still want to keep
.. _RMextract: https://github.com/lofar-astron/RMextract/ .. _RMextract: https://github.com/lofar-astron/RMextract/
.. _factor: https://github.com/lofar-astron/factor/ .. _factor: https://github.com/lofar-astron/factor/
.. _killMS: https://github.com/saopicc/killMS/ .. _killMS: https://github.com/saopicc/killMS/
.. _TGSS ADR: https://http://tgssadr.strw.leidenuniv.nl/ .. _TGSS ADR: https://tgssadr.strw.leidenuniv.nl/
.. _Dysco: https://github.com/aroffringa/dysco/ .. _Dysco: https://github.com/aroffringa/dysco/
.. _AOFlagger: https://gitlab.com/aroffringa/aoflagger.git .. _AOFlagger: https://gitlab.com/aroffringa/aoflagger.git
.. _here: https://www.astron.nl/lofarwiki/doku.php?id=public:user_software:documentation:ndppp#description_of_baseline_selection_parameters .. _here: https://dp3.readthedocs.io/en/latest/steps/Description%20of%20baseline%20selection%20parameters.html
.. _AOFlagger manual: https://aoflagger.readthedocs.io/en/latest/using_aoflagger.html .. _AOFlagger manual: https://aoflagger.readthedocs.io/en/latest/using_aoflagger.html
scripts/sort_times_into_freqGroups.py
+ 13
− 13
View file @ 60a46bcf
...@@ -59,7 +59,7 @@ def input2int(invar): ...@@ -59,7 +59,7 @@ def input2int(invar):
######################################################################## ########################################################################
def main(MSfile, numSB=10, NDPPPfill=True, stepname=None, mergeLastGroup=False, truncateLastSBs=True, firstSB=None): def main(MSfile, numSB=10, DP3fill=True, stepname=None, mergeLastGroup=False, truncateLastSBs=True, firstSB=None):
""" """
Check a list of MS files for missing frequencies Check a list of MS files for missing frequencies
...@@ -71,13 +71,13 @@ def main(MSfile, numSB=10, NDPPPfill=True, stepname=None, mergeLastGroup=False, ...@@ -71,13 +71,13 @@ def main(MSfile, numSB=10, NDPPPfill=True, stepname=None, mergeLastGroup=False,
How many files should go into one frequency group. Values <= 0 mean put How many files should go into one frequency group. Values <= 0 mean put
all files of the same time-step into one group. all files of the same time-step into one group.
default = -1 default = -1
NDPPPfill : bool, optional DP3fill : bool, optional
Add dummy file-names for missing frequencies, so that NDPPP can Add dummy file-names for missing frequencies, so that DP3 can
fill the data with flagged dummy data. fill the data with flagged dummy data.
default = True default = True
target_path : str, optional target_path : str, optional
Change the path of the "groups" files to this. (I.e. write output files Change the path of the "groups" files to this. (I.e. write output files
into this directory with the subsequent NDPPP call.) into this directory with the subsequent DP3 call.)
default = keep path of input files default = keep path of input files
stepname : str, optional stepname : str, optional
Add this step-name into the file-names of the output files. Add this step-name into the file-names of the output files.
...@@ -90,7 +90,7 @@ def main(MSfile, numSB=10, NDPPPfill=True, stepname=None, mergeLastGroup=False, ...@@ -90,7 +90,7 @@ def main(MSfile, numSB=10, NDPPPfill=True, stepname=None, mergeLastGroup=False,
mergeLastGroup = False, truncateLastSBs = True: mergeLastGroup = False, truncateLastSBs = True:
ignore last files, that don't make for a full group (not all files are used). ignore last files, that don't make for a full group (not all files are used).
mergeLastGroup = False, truncateLastSBs = False: mergeLastGroup = False, truncateLastSBs = False:
keep inclomplete last group, or - with NDPPPfill=True - fill keep inclomplete last group, or - with DP3fill=True - fill
last group with dummies. last group with dummies.
firstSB : int, optional firstSB : int, optional
If set, then reference the grouping of files to this station-subband. As if a file If set, then reference the grouping of files to this station-subband. As if a file
...@@ -104,7 +104,7 @@ def main(MSfile, numSB=10, NDPPPfill=True, stepname=None, mergeLastGroup=False, ...@@ -104,7 +104,7 @@ def main(MSfile, numSB=10, NDPPPfill=True, stepname=None, mergeLastGroup=False,
""" """
NDPPPfill = input2bool(NDPPPfill) DP3fill = input2bool(DP3fill)
mergeLastGroup = input2bool(mergeLastGroup) mergeLastGroup = input2bool(mergeLastGroup)
truncateLastSBs = input2bool(truncateLastSBs) truncateLastSBs = input2bool(truncateLastSBs)
firstSB = input2int(firstSB) firstSB = input2int(firstSB)
...@@ -197,15 +197,15 @@ def main(MSfile, numSB=10, NDPPPfill=True, stepname=None, mergeLastGroup=False, ...@@ -197,15 +197,15 @@ def main(MSfile, numSB=10, NDPPPfill=True, stepname=None, mergeLastGroup=False,
# this is the case for concatenating all subbands # this is the case for concatenating all subbands
groupBW = maxfreq-minfreq groupBW = maxfreq-minfreq
truncateLastSBs = input2bool(False) truncateLastSBs = input2bool(False)
NDPPPfill = input2bool(True) DP3fill = input2bool(True)
freqborders = np.arange(minfreq,maxfreq,groupBW) freqborders = np.arange(minfreq,maxfreq,groupBW)
if mergeLastGroup: if mergeLastGroup:
freqborders[-1] = maxfreq freqborders[-1] = maxfreq
elif truncateLastSBs: elif truncateLastSBs:
pass #nothing to do! # left to make the logic more clear! pass #nothing to do! # left to make the logic more clear!
elif not truncateLastSBs and NDPPPfill: elif not truncateLastSBs and DP3fill:
freqborders = np.append(freqborders,(freqborders[-1]+groupBW)) freqborders = np.append(freqborders,(freqborders[-1]+groupBW))
elif not truncateLastSBs and not NDPPPfill: elif not truncateLastSBs and not DP3fill:
freqborders = np.append(freqborders,maxfreq) freqborders = np.append(freqborders,maxfreq)
freqborders = freqborders[freqborders>(np.min(freqliste)-groupBW)] freqborders = freqborders[freqborders>(np.min(freqliste)-groupBW)]
...@@ -233,7 +233,7 @@ def main(MSfile, numSB=10, NDPPPfill=True, stepname=None, mergeLastGroup=False, ...@@ -233,7 +233,7 @@ def main(MSfile, numSB=10, NDPPPfill=True, stepname=None, mergeLastGroup=False,
else: else:
(freq,fname) = (1e12,'This_shouldn\'t_show_up') (freq,fname) = (1e12,'This_shouldn\'t_show_up')
skip_this = False skip_this = False
elif NDPPPfill: elif DP3fill:
files.append('dummy.ms') files.append('dummy.ms')
if not skip_this: if not skip_this:
freqID = int((freqborders[groupIdx]+freqborders[groupIdx+1])/2e6) freqID = int((freqborders[groupIdx]+freqborders[groupIdx+1])/2e6)
...@@ -265,8 +265,8 @@ if __name__ == '__main__': ...@@ -265,8 +265,8 @@ if __name__ == '__main__':
help='One (or more MSs) for which you want to get the antenna list.') help='One (or more MSs) for which you want to get the antenna list.')
parser.add_argument('--numbands', type=int, default=10, parser.add_argument('--numbands', type=int, default=10,
help='Number of how many files should be grouped together in frequency. (default=10)') help='Number of how many files should be grouped together in frequency. (default=10)')
parser.add_argument('--NDPPPfill', type=bool, default=True, parser.add_argument('--DP3fill', type=bool, default=True,
help='Add dummy file-names for missing frequencies, so that NDPPP can fill the data with flagged dummy data.') help='Add dummy file-names for missing frequencies, so that DP3 can fill the data with flagged dummy data.')
parser.add_argument('--stepname', type=str, default=None, parser.add_argument('--stepname', type=str, default=None,
help='Add this step-name into the file-names of the output files.') help='Add this step-name into the file-names of the output files.')
parser.add_argument('--mergeLastGroup', type=bool, default=False, parser.add_argument('--mergeLastGroup', type=bool, default=False,
...@@ -289,4 +289,4 @@ if __name__ == '__main__': ...@@ -289,4 +289,4 @@ if __name__ == '__main__':
logging.root.addHandler(log) logging.root.addHandler(log)
# first argument: pattern for measurement-sets # first argument: pattern for measurement-sets
main(args.MSfiles, numSB=args.numbands, NDPPPfill=args.NDPPPfill, stepname=args.stepname, mergeLastGroup=args.mergeLastGroup, truncateLastSBs=args.truncateLastSBs, firstSB=args.firstSB) main(args.MSfiles, numSB=args.numbands, DP3fill=args.DP3fill, stepname=args.stepname, mergeLastGroup=args.mergeLastGroup, truncateLastSBs=args.truncateLastSBs, firstSB=args.firstSB)
skymodels/3C295-sweijen.skymodel
+ 1
− 1
View file @ 60a46bcf
...@@ -2,7 +2,7 @@ ...@@ -2,7 +2,7 @@
# This model of 3C 295 was created by Frits Sweijen. # This model of 3C 295 was created by Frits Sweijen.
# A pointed 8 hour synthesis observation was taken early 2019 (surveys setup in terms of subbands) and all available international (DE, FR, IE, UK, PL, SE) stations were present. # A pointed 8 hour synthesis observation was taken early 2019 (surveys setup in terms of subbands) and all available international (DE, FR, IE, UK, PL, SE) stations were present.
# Selfcal was done using the full bandwidth from about 121 MHz up to 168 MHz, using DPPP and WSClean. # Selfcal was done using the full bandwidth from about 121 MHz up to 168 MHz, using DP3 and WSClean.
# WSClean's -spectral-correction option was used to try and keep the model at least somewhat near the Scaife&Heald scale, but no guarantees are given. # WSClean's -spectral-correction option was used to try and keep the model at least somewhat near the Scaife&Heald scale, but no guarantees are given.
# The bandpasses are not perfect yet, but are a lot better compared to the current two-component model. # The bandpasses are not perfect yet, but are a lot better compared to the current two-component model.
# If you want to share this model, please ask me first as it is planned to be used in a publication at some point. # If you want to share this model, please ask me first as it is planned to be used in a publication at some point.
......
steps/LoSoTo.Flag.cwl
+ 1
− 1
View file @ 60a46bcf
...@@ -82,7 +82,7 @@ inputs: ...@@ -82,7 +82,7 @@ inputs:
- id: preflagzeros - id: preflagzeros
type: boolean? type: boolean?
doc: doc:
Flag zeros/ones (bad solutions in BBS/DPPP). Flag zeros/ones (bad solutions in BBS/DP3).
They should be flagged at import time. By default False. They should be flagged at import time. By default False.
- id: mode - id: mode
type: string? type: string?
......
steps/dppp_make_parset_cal.cwl → steps/dp3_make_parset_cal.cwl
+ 2
− 2
View file @ 60a46bcf
...@@ -60,7 +60,7 @@ outputs: ...@@ -60,7 +60,7 @@ outputs:
- id: output - id: output
type: File type: File
outputBinding: outputBinding:
glob: DPPP.parset glob: DP3.parset
baseCommand: baseCommand:
- cp - cp
arguments: arguments:
...@@ -71,7 +71,7 @@ arguments: ...@@ -71,7 +71,7 @@ arguments:
- prefix: '' - prefix: ''
shellQuote: false shellQuote: false
position: 0 position: 0
valueFrom: DPPP.parset valueFrom: DP3.parset
requirements: requirements:
- class: InlineJavascriptRequirement - class: InlineJavascriptRequirement
- class: ShellCommandRequirement - class: ShellCommandRequirement
......
steps/dppp_make_parset_target.cwl → steps/dp3_make_parset_target.cwl
+ 2
− 2
View file @ 60a46bcf
...@@ -78,7 +78,7 @@ outputs: ...@@ -78,7 +78,7 @@ outputs:
- id: output - id: output
type: File type: File
outputBinding: outputBinding:
glob: DPPP.parset glob: DP3.parset
baseCommand: baseCommand:
- cp - cp
arguments: arguments:
...@@ -89,7 +89,7 @@ arguments: ...@@ -89,7 +89,7 @@ arguments:
- prefix: '' - prefix: ''
shellQuote: false shellQuote: false
position: 0 position: 0
valueFrom: DPPP.parset valueFrom: DP3.parset
requirements: requirements:
- class: InlineJavascriptRequirement - class: InlineJavascriptRequirement
- class: ShellCommandRequirement - class: ShellCommandRequirement
......
steps/dppp_prep_cal.cwl → steps/dp3_prep_cal.cwl
+ 4
− 4
View file @ 60a46bcf
class: CommandLineTool class: CommandLineTool
cwlVersion: v1.2 cwlVersion: v1.2
id: dppp id: dp3
baseCommand: baseCommand:
- DP3 - DP3
inputs: inputs:
...@@ -107,12 +107,12 @@ outputs: ...@@ -107,12 +107,12 @@ outputs:
- id: logfile - id: logfile
type: File[] type: File[]
outputBinding: outputBinding:
glob: 'DPPP*.log' glob: 'DP3*.log'
hints: hints:
- class: DockerRequirement - class: DockerRequirement
dockerPull: astronrd/linc dockerPull: astronrd/linc
stdout: DPPP.log stdout: DP3.log
stderr: DPPP_err.log stderr: DP3_err.log
requirements: requirements:
- class: InlineJavascriptRequirement - class: InlineJavascriptRequirement
- class: ResourceRequirement - class: ResourceRequirement
......
steps/dppp_prep_target.cwl → steps/dp3_prep_target.cwl
+ 4
− 4
View file @ 60a46bcf
class: CommandLineTool class: CommandLineTool
cwlVersion: v1.2 cwlVersion: v1.2
id: dppp id: dp3
baseCommand: baseCommand:
- DP3 - DP3
inputs: inputs:
...@@ -163,12 +163,12 @@ outputs: ...@@ -163,12 +163,12 @@ outputs:
- id: logfile - id: logfile
type: File[] type: File[]
outputBinding: outputBinding:
glob: 'DPPP*.log' glob: 'DP3*.log'
hints: hints:
- class: DockerRequirement - class: DockerRequirement
dockerPull: astronrd/linc dockerPull: astronrd/linc
stdout: DPPP.log stdout: DP3.log
stderr: DPPP_err.log stderr: DP3_err.log
requirements: requirements:
- class: InlineJavascriptRequirement - class: InlineJavascriptRequirement
- class: ResourceRequirement - class: ResourceRequirement
......
steps/dpppconcat.cwl → steps/dp3concat.cwl
+ 1
− 1
View file @ 60a46bcf
class: CommandLineTool class: CommandLineTool
cwlVersion: v1.2 cwlVersion: v1.2
id: dpppconcat id: dp3concat
baseCommand: baseCommand:
- DP3 - DP3
inputs: inputs:
......
steps/sort_times_into_freqGroups.cwl
+ 6
− 6
View file @ 60a46bcf
...@@ -14,10 +14,10 @@ inputs: ...@@ -14,10 +14,10 @@ inputs:
type: int? type: int?
default: 10 default: 10
doc: Number of how many files should be grouped together in frequency. doc: Number of how many files should be grouped together in frequency.
- id: NDPPPfill - id: DP3fill
type: boolean? type: boolean?
default: True default: True
doc: Add dummy file-names for missing frequencies, so that NDPPP can fill the data with flagged dummy data. doc: Add dummy file-names for missing frequencies, so that DP3 can fill the data with flagged dummy data.
- id: stepname - id: stepname
type: string? type: string?
default: '.ms' default: '.ms'
...@@ -29,11 +29,11 @@ inputs: ...@@ -29,11 +29,11 @@ inputs:
- id: mergeLastGroup - id: mergeLastGroup
type: boolean? type: boolean?
default: False default: False
doc: Add dummy file-names for missing frequencies, so that NDPPP can fill the data with flagged dummy data. doc: Add dummy file-names for missing frequencies, so that DP3 can fill the data with flagged dummy data.
- id: truncateLastSBs - id: truncateLastSBs
type: boolean? type: boolean?
default: True default: True
doc: Add dummy file-names for missing frequencies, so that NDPPP can fill the data with flagged dummy data. doc: Add dummy file-names for missing frequencies, so that DP3 can fill the data with flagged dummy data.
requirements: requirements:
- class: InlineJavascriptRequirement - class: InlineJavascriptRequirement
- class: InitialWorkDirRequirement - class: InitialWorkDirRequirement
...@@ -49,12 +49,12 @@ requirements: ...@@ -49,12 +49,12 @@ requirements:
numbands = inputs['numbands'] numbands = inputs['numbands']
stepname = inputs['stepname'] stepname = inputs['stepname']
NDPPPfill = inputs['NDPPPfill'] DP3fill = inputs['DP3fill']
mergeLastGroup = inputs['mergeLastGroup'] mergeLastGroup = inputs['mergeLastGroup']
truncateLastSBs = inputs['truncateLastSBs'] truncateLastSBs = inputs['truncateLastSBs']
firstSB = inputs['firstSB'] firstSB = inputs['firstSB']
output = sort_times_into_freqGroups(mss, numbands, NDPPPfill, stepname, mergeLastGroup, truncateLastSBs, firstSB) output = sort_times_into_freqGroups(mss, numbands, DP3fill, stepname, mergeLastGroup, truncateLastSBs, firstSB)
filenames = output['filenames'] filenames = output['filenames']
groupnames = output['groupnames'] groupnames = output['groupnames']
......
workflows/HBA_calibrator.cwl
+ 3
− 3
View file @ 60a46bcf
...@@ -86,7 +86,7 @@ inputs: ...@@ -86,7 +86,7 @@ inputs:
- id: tables2export - id: tables2export
type: string? type: string?
default: clock default: clock
- id: max_dppp_threads - id: max_dp3_threads
type: int? type: int?
default: 10 default: 10
- id: memoryperc - id: memoryperc
...@@ -197,8 +197,8 @@ steps: ...@@ -197,8 +197,8 @@ steps:
source: clock_smooth source: clock_smooth
- id: tables2export - id: tables2export
source: tables2export source: tables2export
- id: max_dppp_threads - id: max_dp3_threads
source: max_dppp_threads source: max_dp3_threads
- id: memoryperc - id: memoryperc
source: memoryperc source: memoryperc
- id: min_separation - id: min_separation
......
workflows/HBA_target.cwl
+ 3
− 3
View file @ 60a46bcf
...@@ -74,7 +74,7 @@ inputs: ...@@ -74,7 +74,7 @@ inputs:
- id: updateweights - id: updateweights
type: boolean? type: boolean?
default: true default: true
- id: max_dppp_threads - id: max_dp3_threads
type: int? type: int?
default: 10 default: 10
- id: memoryperc - id: memoryperc
...@@ -221,8 +221,8 @@ steps: ...@@ -221,8 +221,8 @@ steps:
source: demix source: demix
- id: updateweights - id: updateweights
source: updateweights source: updateweights
- id: max_dppp_threads - id: max_dp3_threads
source: max_dppp_threads source: max_dp3_threads
- id: memoryperc - id: memoryperc
source: memoryperc source: memoryperc
- id: min_separation - id: min_separation
......
workflows/LBA_calibrator.cwl
+ 3
− 3
View file @ 60a46bcf
...@@ -86,7 +86,7 @@ inputs: ...@@ -86,7 +86,7 @@ inputs:
- id: tables2export - id: tables2export
type: string? type: string?
default: phaseOrig default: phaseOrig
- id: max_dppp_threads - id: max_dp3_threads
type: int? type: int?
default: 10 default: 10
- id: memoryperc - id: memoryperc
...@@ -195,8 +195,8 @@ steps: ...@@ -195,8 +195,8 @@ steps:
source: clock_smooth source: clock_smooth
- id: tables2export - id: tables2export
source: tables2export source: tables2export
- id: max_dppp_threads - id: max_dp3_threads
source: max_dppp_threads source: max_dp3_threads
- id: memoryperc - id: memoryperc
source: memoryperc source: memoryperc
- id: min_separation - id: min_separation
......
workflows/linc_calibrator.cwl
+ 3
− 3
View file @ 60a46bcf
...@@ -85,7 +85,7 @@ inputs: ...@@ -85,7 +85,7 @@ inputs:
- id: tables2export - id: tables2export
type: string? type: string?
default: clock default: clock
- id: max_dppp_threads - id: max_dp3_threads
type: int? type: int?
default: 10 default: 10
- id: memoryperc - id: memoryperc
...@@ -178,8 +178,8 @@ steps: ...@@ -178,8 +178,8 @@ steps:
source: flagunconverged source: flagunconverged
- id: demix - id: demix
source: demix source: demix
- id: max_dppp_threads - id: max_dp3_threads
source: max_dppp_threads source: max_dp3_threads
- id: memoryperc - id: memoryperc
source: memoryperc source: memoryperc
- id: flag_baselines - id: flag_baselines
......
workflows/linc_calibrator/ndppp_prep_cal.cwl → workflows/linc_calibrator/dp3_prep_cal.cwl
+ 8
− 8
View file @ 60a46bcf
class: Workflow class: Workflow
cwlVersion: v1.2 cwlVersion: v1.2
id: ndppp_prep_cal id: dp3_prep_cal
label: ndppp_prep_cal label: dp3_prep_cal
inputs: inputs:
- id: baselines_to_flag - id: baselines_to_flag
type: string[]? type: string[]?
...@@ -67,7 +67,7 @@ outputs: ...@@ -67,7 +67,7 @@ outputs:
type: Directory type: Directory
- id: logfile - id: logfile
outputSource: outputSource:
- concat_logfiles_dppp/output - concat_logfiles_dp3/output
type: File type: File
- id: flagged_fraction_dict - id: flagged_fraction_dict
outputSource: outputSource:
...@@ -114,7 +114,7 @@ steps: ...@@ -114,7 +114,7 @@ steps:
source: lbfgs_robustdof source: lbfgs_robustdof
out: out:
- id: output - id: output
run: ../../steps/dppp_make_parset_cal.cwl run: ../../steps/dp3_make_parset_cal.cwl
label: make_parset label: make_parset
- id: dp3_execute - id: dp3_execute
in: in:
...@@ -143,19 +143,19 @@ steps: ...@@ -143,19 +143,19 @@ steps:
- id: msout - id: msout
- id: flagged_fraction_dict - id: flagged_fraction_dict
- id: logfile - id: logfile
run: ../../steps/dppp_prep_cal.cwl run: ../../steps/dp3_prep_cal.cwl
label: DP3.Execute label: DP3.Execute
- id: concat_logfiles_dppp - id: concat_logfiles_dp3
in: in:
- id: file_list - id: file_list
source: source:
- dp3_execute/logfile - dp3_execute/logfile
- id: file_prefix - id: file_prefix
default: dppp default: dp3
out: out:
- id: output - id: output
run: ../../steps/concatenate_files.cwl run: ../../steps/concatenate_files.cwl
label: concat_logfiles_dppp label: concat_logfiles_dp3
requirements: requirements:
- class: SubworkflowFeatureRequirement - class: SubworkflowFeatureRequirement
- class: StepInputExpressionRequirement - class: StepInputExpressionRequirement
......
workflows/linc_calibrator/prep.cwl
+ 9
− 9
View file @ 60a46bcf
...@@ -19,7 +19,7 @@ inputs: ...@@ -19,7 +19,7 @@ inputs:
default: false default: false
- id: demix - id: demix
type: boolean? type: boolean?
- id: max_dppp_threads - id: max_dp3_threads
type: int? type: int?
default: 10 default: 10
- id: memoryperc - id: memoryperc
...@@ -111,7 +111,7 @@ outputs: ...@@ -111,7 +111,7 @@ outputs:
type: string[] type: string[]
- id: flagged_fraction_dict - id: flagged_fraction_dict
outputSource: outputSource:
- ndppp_prep_cal/flagged_fraction_dict - dp3_prep_cal/flagged_fraction_dict
type: string[] type: string[]
- id: logfiles - id: logfiles
outputSource: outputSource:
...@@ -136,7 +136,7 @@ steps: ...@@ -136,7 +136,7 @@ steps:
- id: output - id: output
run: ../../steps/selectfirstdirectory.cwl run: ../../steps/selectfirstdirectory.cwl
label: select_only_first label: select_only_first
- id: ndppp_prep_cal - id: dp3_prep_cal
in: in:
- id: baselines_to_flag - id: baselines_to_flag
default: [] default: []
...@@ -180,7 +180,7 @@ steps: ...@@ -180,7 +180,7 @@ steps:
source: find_skymodel_cal/model_name source: find_skymodel_cal/model_name
- id: ntimechunk - id: ntimechunk
default: 10 default: 10
source: max_dppp_threads source: max_dp3_threads
- id: subtract_sources - id: subtract_sources
source: check_demix/out_demix_sources source: check_demix/out_demix_sources
- id: lbfgs_historysize - id: lbfgs_historysize
...@@ -191,8 +191,8 @@ steps: ...@@ -191,8 +191,8 @@ steps:
- id: msout - id: msout
- id: flagged_fraction_dict - id: flagged_fraction_dict
- id: logfile - id: logfile
run: ./ndppp_prep_cal.cwl run: ./dp3_prep_cal.cwl
label: ndppp_prep_cal label: dp3_prep_cal
scatter: scatter:
- msin - msin
- id: predict_calibrate - id: predict_calibrate
...@@ -224,7 +224,7 @@ steps: ...@@ -224,7 +224,7 @@ steps:
in: in:
- id: msin - id: msin
linkMerge: merge_flattened linkMerge: merge_flattened
source: ndppp_prep_cal/msout source: dp3_prep_cal/msout
- id: verbose - id: verbose
default: true default: true
- id: concatenate-frequency - id: concatenate-frequency
...@@ -300,9 +300,9 @@ steps: ...@@ -300,9 +300,9 @@ steps:
- id: file_list - id: file_list
linkMerge: merge_flattened linkMerge: merge_flattened
source: source:
- ndppp_prep_cal/logfile - dp3_prep_cal/logfile
- id: file_prefix - id: file_prefix
default: ndppp_prep_cal default: dp3_prep_cal
out: out:
- id: output - id: output
run: ../../steps/concatenate_files.cwl run: ../../steps/concatenate_files.cwl
......
workflows/linc_target.cwl
+ 3
− 3
View file @ 60a46bcf
...@@ -74,7 +74,7 @@ inputs: ...@@ -74,7 +74,7 @@ inputs:
- id: updateweights - id: updateweights
type: boolean? type: boolean?
default: true default: true
- id: max_dppp_threads - id: max_dp3_threads
type: int? type: int?
default: 10 default: 10
- id: memoryperc - id: memoryperc
...@@ -229,8 +229,8 @@ steps: ...@@ -229,8 +229,8 @@ steps:
source: apply_beam source: apply_beam
- id: updateweights - id: updateweights
source: updateweights source: updateweights
- id: max_dppp_threads - id: max_dp3_threads
source: max_dppp_threads source: max_dp3_threads
- id: memoryperc - id: memoryperc
source: memoryperc source: memoryperc
- id: min_separation - id: min_separation
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment