process_slow_gains can't handle inf solutions
My dataset results in solution tables that contain infs, which seems to crash process_slow_gains during calibrate_2 with a ValueError: array must not contain infs or NaNs. The error I get is
[2023-01-14T00:46:34+0100] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
[2023-01-14T00:46:34+0100] [MainThread] [I] [toil] Running Toil version 5.7.1-b5cae9634820d76cb6c13b2a6312895122017d54 on host rijn5.strw.leidenuniv.nl.
[2023-01-14T00:46:34+0100] [MainThread] [I] [toil.worker] Working on job 'CWLJob' pipeline_parset.cwl.process_slow_gains2.process_slow_gains.cwl kind-CWLJob/instance-9xoy1z_3 v5
[2023-01-14T00:46:34+0100] [MainThread] [I] [toil.worker] Loaded body Job('CWLJob' pipeline_parset.cwl.process_slow_gains2.process_slow_gains.cwl kind-CWLJob/instance-9xoy1z_3 v5) from description 'CWLJob' pipeline_parset.cwl.process_slow_gains2.process_slow_gains.cwl kind-CWLJob/instance-9xoy1z_3 v5
[2023-01-14T00:46:34+0100] [MainThread] [I] [cwltool] [job pipeline_parset.cwl.process_slow_gains2.process_slow_gains.cwl] /data2/sweijen/Quasar_Anniek/rundir_P240+30_rapthor/tmpdir/tmpmonb7bvv$ process_slow_gains.py \
--normalize=True \
--smooth=True \
/data2/sweijen/Quasar_Anniek/rundir_P240+30_rapthor/tmpdir/tmpmonb7bvv/slow_gains2.h5parm
Error. nthreads cannot be larger than environment variable "NUMEXPR_MAX_THREADS" (64)/net/rijn5/data2/sweijen/Quasar_Anniek/rundir_P240+30_rapthor/localrapthor/bin/process_slow_gains.py:158: RuntimeWarning: divide by zero encountered in log10
parms = np.log10(parms)
/net/rijn5/data2/sweijen/Quasar_Anniek/rundir_P240+30_rapthor/localrapthor/bin/process_slow_gains.py:72: RuntimeWarning: divide by zero encountered in log10
parms = np.log10(parms)
Traceback (most recent call last):
File "/net/rijn5/data2/sweijen/Quasar_Anniek/rundir_P240+30_rapthor/localrapthor/bin/process_slow_gains.py", line 492, in <module>
main(args.h5parmfile, solsetname=args.solsetname, ampsoltabname=args.ampsoltabname,
File "/net/rijn5/data2/sweijen/Quasar_Anniek/rundir_P240+30_rapthor/localrapthor/bin/process_slow_gains.py", line 468, in main
amp, damp = normalize_direction(ampsoltab, remove_core_gradient=True,
File "/net/rijn5/data2/sweijen/Quasar_Anniek/rundir_P240+30_rapthor/localrapthor/bin/process_slow_gains.py", line 104, in normalize_direction
popt, pcov = curve_fit(func, x, y, sigma=w, p0=[0.0, 1.0])
File "/net/rijn5/data2/sweijen/Quasar_Anniek/rundir_P240+30_rapthor/localrapthor/scipy/optimize/_minpack_py.py", line 782, in curve_fit
ydata = np.asarray_chkfinite(ydata, float)
File "/net/rijn5/data2/sweijen/Quasar_Anniek/rundir_P240+30_rapthor/localrapthor/numpy/lib/function_base.py", line 627, in asarray_chkfinite
raise ValueError(
ValueError: array must not contain infs or NaNs
Closing remaining open files:/data2/sweijen/Quasar_Anniek/rundir_P240+30_rapthor/tmpdir/tmpmonb7bvv/slow_gains2.h5parm...done
[2023-01-14T00:46:35+0100] [MainThread] [W] [cwltool] [job pipeline_parset.cwl.process_slow_gains2.process_slow_gains.cwl] exited with status: 1
[2023-01-14T00:46:35+0100] [MainThread] [W] [cwltool] [job pipeline_parset.cwl.process_slow_gains2.process_slow_gains.cwl] completed permanentFail
Traceback (most recent call last):
File "/opt/lofar/pyenv-py3/lib/python3.9/site-packages/toil/worker.py", line 407, in workerScript
job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
File "/opt/lofar/pyenv-py3/lib/python3.9/site-packages/toil/job.py", line 2406, in _runner
returnValues = self._run(jobGraph=None, fileStore=fileStore)
File "/opt/lofar/pyenv-py3/lib/python3.9/site-packages/toil/job.py", line 2324, in _run
return self.run(fileStore)
File "/opt/lofar/pyenv-py3/lib/python3.9/site-packages/toil/cwl/cwltoil.py", line 2217, in run
raise cwltool.errors.WorkflowException(status)
cwltool.errors.WorkflowException: permanentFail
[2023-01-14T00:46:35+0100] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host rijn5.strw.leidenuniv.nl
Edited by Frits Sweijen