Toil issues with optional parameters and when-statements
When running the concatenation.cwl
step, as part of split-directions and delay-calibration, toil gives an error about missing required input parameters (linc_libraries
) from aoflagger.cwl
when line 106 is False (https://git.astron.nl/RD/VLBI-cwl/-/blob/master/workflows/subworkflows/concatenation.cwl?ref_type=heads#L106). However, as line 106 is False, it should not validate aoflagger.cwl (and completely skip this step). I can fix this by making linc_libraries
and memory
optional parameters in aoflagger.cwl
, but it is unclear why this error is raised.
This is the complete error message:
[2024-01-24T12:16:43+0000] [MainThread] [I] [toil.leader] Issued job 'CWLJob' concatenation.dp3_concatenate_logfiles.concatfiles kind-CWLJob/instance-9gag3z5a v1 with job batch system
ID: 23 and disk: 1.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2024-01-24T12:16:45+0000] [Thread-26 (daddy)] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker CWLJobWrapper file:/home/tjlv53/softwar
e/LOFAR/VLBI-cwl/jobstore kind-CWLJobWrapper/instance-0d8x3516.
[2024-01-24T12:16:45+0000] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'CWLJobWrapper' concatenation.AOflagging.aoflagging._wrapper kind-CWLJobWrapper/instance-0d8x35
16 v1
Exit reason: None
[2024-01-24T12:16:45+0000] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'CWLJobWrapper' concatenation.AOflagging.aoflagging._wrapper kind-
CWLJobWrapper/instance-0d8x3516 v2
[2024-01-24T12:16:45+0000] [MainThread] [W] [toil.leader] Log from job "kind-CWLJobWrapper/instance-0d8x3516" follows:
=========>
[2024-01-24T12:16:44+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
[2024-01-24T12:16:44+0000] [MainThread] [I] [toil] Running Toil version 6.0.0-0e2a07a20818e593bfdfde3cc51ca4ad809fde96 on host PHY-TJLV53.
[2024-01-24T12:16:44+0000] [MainThread] [I] [toil.worker] Working on job 'CWLJobWrapper' concatenation.AOflagging.aoflagging._wrapper kind-CWLJobWrapper/instance-0d8x3516 v1
[2024-01-24T12:16:44+0000] [MainThread] [I] [toil.worker] Loaded body Job('CWLJobWrapper' concatenation.AOflagging.aoflagging._wrapper kind-CWLJobWrapper/instance-0d8x3516 v1)
from description 'CWLJobWrapper' concatenation.AOflagging.aoflagging._wrapper kind-CWLJobWrapper/instance-0d8x3516 v1
Traceback (most recent call last):
File "/home/tjlv53/.local/lib/python3.10/site-packages/cwltool/process.py", line 434, in fill_in_defaults
raise WorkflowException(
cwltool.errors.WorkflowException: Missing required input parameter 'linc_libraries'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/tjlv53/.local/lib/python3.10/site-packages/toil/worker.py", line 407, in workerScript
job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
File "/home/tjlv53/.local/lib/python3.10/site-packages/toil/job.py", line 2829, in _runner
returnValues = self._run(jobGraph=None, fileStore=fileStore)
File "/home/tjlv53/.local/lib/python3.10/site-packages/toil/job.py", line 2746, in _run
return self.run(fileStore)
File "/home/tjlv53/.local/lib/python3.10/site-packages/toil/cwl/cwltoil.py", line 2333, in run
fill_in_defaults(
File "/home/tjlv53/.local/lib/python3.10/site-packages/cwltool/process.py", line 425, in fill_in_defaults
with SourceLine(inputs, e, WorkflowException, debug):
File "/home/tjlv53/.local/lib/python3.10/site-packages/schema_salad/sourceline.py", line 249, in __exit__
raise self.makeError(str(exc_value)) from exc_value
cwltool.errors.WorkflowException: steps/aoflagger.cwl:37:7: Missing required input parameter 'linc_libraries'
[2024-01-24T12:16:45+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host PHY-TJLV53
<=========
Edited by Jurjen de Jong