VLBI-cwl merge requestshttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests2024-03-25T16:36:33Zhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/51Find dis2 solutions2024-03-25T16:36:33ZFrits SweijenFind dis2 solutionsImplements RAP-605. Given the path to ddf-pipeline's SOLSDIR folder, this adds steps that will:
1. Automatically find the DIS2 solutions in npz format given the SOLSDIR directory.
2. Convert them to H5parm format
3. Merge them into a si...Implements RAP-605. Given the path to ddf-pipeline's SOLSDIR folder, this adds steps that will:
1. Automatically find the DIS2 solutions in npz format given the SOLSDIR directory.
2. Convert them to H5parm format
3. Merge them into a single H5parm
This eliminates the need for a user to do this manually and replaces the `ddf_solset` parameter that previously took the H5parm from step 3.Frits SweijenFrits Sweijenhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/52Reinstated dysco compression for data weights2024-03-14T13:19:30ZMatthijs van der WildReinstated dysco compression for data weightsRD/VLBI-cwl!50 removed dysco compression in the A-team clipping step. As a result the weights weren't compressed, causing a substantial increase in the produced MS. This MR reinstates the dysco compression for the weights.RD/VLBI-cwl!50 removed dysco compression in the A-team clipping step. As a result the weights weren't compressed, causing a substantial increase in the produced MS. This MR reinstates the dysco compression for the weights.Matthijs van der WildMatthijs van der Wildhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/50Implemented new workflow for A-team clipping2024-03-14T11:58:17ZMatthijs van der WildImplemented new workflow for A-team clippingThis MR changes the A-team clipping workflow. Visibility prediction and A-team clipping are now done entirely by DP3, using the new [clipper](https://dp3.readthedocs.io/en/latest/steps/Clipper.html) step.This MR changes the A-team clipping workflow. Visibility prediction and A-team clipping are now done entirely by DP3, using the new [clipper](https://dp3.readthedocs.io/en/latest/steps/Clipper.html) step.Matthijs van der WildMatthijs van der Wildhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/47Added config to target solve2024-02-27T15:03:14ZJames PetleyAdded config to target solveTarget solve was not actually using the config file that could be controlled and instead just using auto settings. Have added this option to the `steps/target_solve.cwl` workflow.Target solve was not actually using the config file that could be controlled and instead just using auto settings. Have added this option to the `steps/target_solve.cwl` workflow.Matthijs van der WildMatthijs van der Wildhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/46Bugfix prep delay2024-02-27T11:43:50ZMatthijs van der WildBugfix prep delayCatalogues of sources that are not covered by LoTSS are not processed correctly by `TargetListToCoords.py`. For example, a Source_id of `"example"` will be output as `'e,x,a,m,p,l,e". This causes issues later in the pipeline, as the late...Catalogues of sources that are not covered by LoTSS are not processed correctly by `TargetListToCoords.py`. For example, a Source_id of `"example"` will be output as `'e,x,a,m,p,l,e". This causes issues later in the pipeline, as the later steps will search this string in the MS data, which does not exist.
Additionally, I noticed that if multiple sources are imaged in split_directions the resulting source_id may be incorrectly formatted, resulting in a similar issue.
Both issues are probably due to a bug introduced in RD/VLBI-cwl!41. This MR corrects the output of `TargetListToCoords.py`, such that both delay calibration and split directions both result in correctly formatted strings.Matthijs van der WildMatthijs van der Wildhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/45Merged delay_solve and target_solve.2024-02-27T10:26:24ZMatthijs van der WildMerged delay_solve and target_solve.Both the delay calibration and split directions workflows
use facetselfcal.py, but do so through different step files.
The steps are similar enough that, in principle, they can
be replaced by a single step file. This commit creates a
fac...Both the delay calibration and split directions workflows
use facetselfcal.py, but do so through different step files.
The steps are similar enough that, in principle, they can
be replaced by a single step file. This commit creates a
facet_selfcal step file that can be used by both pipeline
steps.Matthijs van der WildMatthijs van der Wildhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/48Flag baselines input2024-02-23T16:17:43ZMatthijs van der WildFlag baselines inputThis MR changes the type of the input `flag_baselines` in the delay calibration workflow and all of its relevant subparts from `string` to `string[]`. In doing so the input matches its analogue in LINC.This MR changes the type of the input `flag_baselines` in the delay calibration workflow and all of its relevant subparts from `string` to `string[]`. In doing so the input matches its analogue in LINC.Matthijs van der WildMatthijs van der Wildhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/41Restructured TargetListToCoords.py2024-02-15T13:22:21ZMatthijs van der WildRestructured TargetListToCoords.pyThe script now distinguishes between
the delay calibration and the split
directions pipelines, and in the
later case checks if there is a single
direction.
Closes RAP-548The script now distinguishes between
the delay calibration and the split
directions pipelines, and in the
later case checks if there is a single
direction.
Closes RAP-548https://git.astron.nl/RD/VLBI-cwl/-/merge_requests/42Added solution files to output2024-02-01T15:56:40ZJames PetleyAdded solution files to outputThis branch means that h5 solsets will be saved during the target solve step and output from the split-directions.cwl workflowThis branch means that h5 solsets will be saved during the target solve step and output from the split-directions.cwl workflowJames PetleyJames Petleyhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/43Added missing input parameter prep_delay2024-02-01T15:51:41ZMatthijs van der WildAdded missing input parameter prep_delayThis MR adds a missing input parameter to the call to prep_delay in get_delay_cal_beam_dir. This issue was discovered by @jwpetley during !42.This MR adds a missing input parameter to the call to prep_delay in get_delay_cal_beam_dir. This issue was discovered by @jwpetley during !42.Matthijs van der WildMatthijs van der Wildhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/38Update order of beam corrections in Split-Directions2024-02-01T15:13:57ZFrits SweijenUpdate order of beam corrections in Split-DirectionsThis updates the beam corrections done in the split directions workflow for proper commutation between the delay calibrator solutions and the beam. Instead of
1. phaseshift
2. apply target beam
3. apply delay solutions
The new order is...This updates the beam corrections done in the split directions workflow for proper commutation between the delay calibrator solutions and the beam. Instead of
1. phaseshift
2. apply target beam
3. apply delay solutions
The new order is now
1. phaseshift
2. apply delay cal beam
3. apply delay solutions
4. apply target beam
The H5parm from delay calibration was already passed, so instead of adding an additional parameter this is used to generate a new "delaycal.csv" on the fly from which the beam direction is calculated.
Closes #6Frits SweijenFrits Sweijenhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/39Change aoflagger memory parameter to accept both float and int2024-01-31T13:40:16ZFrits SweijenChange aoflagger memory parameter to accept both float and intAOFlagger's `aoflagger.memorymax` argument accepts floating point values, but the `aoflagger.cwl` workflow enforces `int`. This changes the input to accept both.AOFlagger's `aoflagger.memorymax` argument accepts floating point values, but the `aoflagger.cwl` workflow enforces `int`. This changes the input to accept both.Frits SweijenFrits Sweijenhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/29Removed Dockerfiles2024-01-31T11:14:51ZMatthijs van der WildRemoved DockerfilesThe pipeline is currently expected to use a singularity image
(FLoC). In this case, Docker support isn't necessary, and the
dockerfiles have been removed.
The `DockerRequirements` have been kept in order to allow for
containerisation wi...The pipeline is currently expected to use a singularity image
(FLoC). In this case, Docker support isn't necessary, and the
dockerfiles have been removed.
The `DockerRequirements` have been kept in order to allow for
containerisation with the FLoC images.Matthijs van der WildMatthijs van der Wildhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/26Split target_phaseup.2024-01-25T11:00:13ZMatthijs van der WildSplit target_phaseup.This MR replaces `steps/target_phaseup.cwl` with `workflows/subworkflows/split_parset.cwl`. The actions of `steps/target_phaseup.cwl` and its underlying scripts have been split over separate steps. This allows the split directions workfl...This MR replaces `steps/target_phaseup.cwl` with `workflows/subworkflows/split_parset.cwl`. The actions of `steps/target_phaseup.cwl` and its underlying scripts have been split over separate steps. This allows the split directions workflow to use step files that were implemented for the delay calibration pipeline.Matthijs van der WildMatthijs van der Wildhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/37Added AOflagged data as an output of the pipeline.2024-01-23T15:31:05ZMatthijs van der WildAdded AOflagged data as an output of the pipeline.The output data in the `concatenation` workflow only contains flagged data if the data is additionally corrected for direction-dependent effects. That is, the pipeline either captures the output of `dp3_applycal` (which contains the flag...The output data in the `concatenation` workflow only contains flagged data if the data is additionally corrected for direction-dependent effects. That is, the pipeline either captures the output of `dp3_applycal` (which contains the flagging information from `AOflagging`) or `dp3_concat`. If `dp3_applycal` has not been executed the flagging information is lost.
The change in this MR allows the pipeline to collect the output of `AOflagging` if it has been executed without subsequently running `dp3_applycal`.https://git.astron.nl/RD/VLBI-cwl/-/merge_requests/40Run DP3 filter step before counts2024-01-16T10:46:52ZFrits SweijenRun DP3 filter step before countsThis puts the `filter` step in the DP3 call first, such that subsequent `count` steps are consistent when they are ultimate used in the summary step.
Reported by Nadia Biava:
>The pipeline, however, crashed at the job summary step. Thi...This puts the `filter` step in the DP3 call first, such that subsequent `count` steps are consistent when they are ultimate used in the summary step.
Reported by Nadia Biava:
>The pipeline, however, crashed at the job summary step. This is the summary_err.log:
>```
>Error. nthreads cannot be larger than environment variable "NUMEXPR_MAX_THREADS" (64)Traceback (most recent call last):
> File "/home/nbiava/software/VLBI-cwl/vlbi/scripts/make_summary.py", line 242, in <module>
> main(flagFiles=args.flagFiles, pipeline=args.pipeline, run_type=args.run_type,
> File "/home/nbiava/software/VLBI-cwl/vlbi/scripts/make_summary.py", line 151, in main
> index = [i for (i, item) in enumerate(station_statistics) if item['station'] == antenna][0]
>IndexError: list index out of range
>Closing remaining open files:/data/nbiava/MS0735/vlbi-cwl/work/nmri_91v/stg60fe3013-0986-49ad-97c6-8c0346d4eb83/merged_addCS_selfcalcyle009_linearfulljones_ILTJ117.3+74.34_142MHz_uv.dp3-concat.copy.phaseup.h5....done
>```Frits SweijenFrits Sweijenhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/36Made LINC a pipeline dependency.2023-11-03T09:16:19ZMatthijs van der WildMade LINC a pipeline dependency.The files and scripts that originate from LINC have now been
removed from the VLBI-cwl pipeline. The pipeline now instead
requires a local copy of LINC as an input, and will then use
LINC's files and scripts instead.The files and scripts that originate from LINC have now been
removed from the VLBI-cwl pipeline. The pipeline now instead
requires a local copy of LINC as an input, and will then use
LINC's files and scripts instead.Matthijs van der WildMatthijs van der Wildhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/35Fix sort_times_into_freqGroups.py like in LINC2023-10-19T15:27:30ZFrits SweijenFix sort_times_into_freqGroups.py like in LINCImplements the fix in https://git.astron.nl/RD/LINC/-/merge_requests/169 in this repository as a stop-gap until LINC is made a dependency of the pipeline.Implements the fix in https://git.astron.nl/RD/LINC/-/merge_requests/169 in this repository as a stop-gap until LINC is made a dependency of the pipeline.https://git.astron.nl/RD/VLBI-cwl/-/merge_requests/34Cleaned up doc field, removed unused parameter.2023-10-10T09:37:01ZMatthijs van der WildCleaned up doc field, removed unused parameter.This MR cleans up a few things missed in [!30](https://git.astron.nl/RD/VLBI-cwl/-/merge_requests/30).This MR cleans up a few things missed in [!30](https://git.astron.nl/RD/VLBI-cwl/-/merge_requests/30).Matthijs van der WildMatthijs van der Wildhttps://git.astron.nl/RD/VLBI-cwl/-/merge_requests/30Added memory allocation to AOFlagger.2023-10-10T09:36:53ZMatthijs van der WildAdded memory allocation to AOFlagger.The aoflagger.memoryperc option has been removed in favour of
a CWL step that explicitly computes the required amount of
memory. This amount is then passed to the AOFlagger step, where
it is used by both aoflagger.memorymax and ResourceR...The aoflagger.memoryperc option has been removed in favour of
a CWL step that explicitly computes the required amount of
memory. This amount is then passed to the AOFlagger step, where
it is used by both aoflagger.memorymax and ResourceRequirement.Matthijs van der WildMatthijs van der Wild