prep.losoto_plot_RM.losot_plot - file existing error
Hi,
Running the latest master of linc (together with latest singularity) on a completely new run I can the following error:
[lotss-tshimwell@ui-01 L2005609]$ cat failed_CWLJob_prep.losoto_plot_RM.losoto_plot_kind--CWLJob-instance--kjg0k_48_v1000.log
[2024-02-05T12:26:37+0100] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
[2024-02-05T12:26:37+0100] [MainThread] [I] [toil] Running Toil version 5.9.2-54bfe0b146b76ecc6221de384c255e1be89547c6 on host wn-dc-03.spider.surfsara.nl.
[2024-02-05T12:26:37+0100] [MainThread] [I] [toil.worker] Working on job 'CWLJob' prep.losoto_plot_RM.losoto_plot kind-CWLJob/instance-kjg0k_48 v1
[2024-02-05T12:26:37+0100] [MainThread] [I] [toil.worker] Loaded body Job('CWLJob' prep.losoto_plot_RM.losoto_plot kind-CWLJob/instance-kjg0k_48 v1) from description 'CWLJob' prep.losoto_plot_RM.losoto_plot kind-CWLJob/instance-kjg0k_48 v1
[2024-02-05T12:26:37+0100] [MainThread] [I] [cwltool] Using local copy of Singularity image found in /project/lotss/Software/linc-master//pull
[2024-02-05T12:26:38+0100] [MainThread] [I] [cwltool] [job prep.losoto_plot_RM.losoto_plot] /project/lotss/Software/linc-master/detailed-logs/L2005609/L2005609ns69cg87$ singularity \
--quiet \
exec \
--contain \
--ipc \
--cleanenv \
--userns \
--home \
/project/lotss/Software/linc-master/detailed-logs/L2005609/L2005609ns69cg87:/DpUURT \
--bind \
/project/lotss/Software/linc-master/tmpdir/L2005609/7zlb0jy9:/tmp \
--bind \
/project/lotss/Software/linc-master/detailed-logs/L2005609/L2005609w0xoi_cc/cal_solutions.h5:/DpUURT/cal_solutions.h5:ro \
--pwd \
/DpUURT \
--net \
--network \
none \
/project/lotss/Software/linc-master/pull/astronrd_linc.sif \
bash \
run_losoto_plot.sh > /project/lotss/Software/linc-master/detailed-logs/L2005609/L2005609ns69cg87/cal_solutions.h5-losoto.log 2> /project/lotss/Software/linc-master/detailed-logs/L2005609/L2005609ns69cg87/cal_solutions.h5-losoto_err.log
[2024-02-05T12:26:41+0100] [MainThread] [W] [cwltool] [job prep.losoto_plot_RM.losoto_plot] exited with status: 1
[2024-02-05T12:26:41+0100] [MainThread] [W] [cwltool] [job prep.losoto_plot_RM.losoto_plot] completed permanentFail
Traceback (most recent call last):
File "/home/lotss-tshimwell/.local/lib/python3.9/site-packages/toil/worker.py", line 403, in workerScript
job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
File "/home/lotss-tshimwell/.local/lib/python3.9/site-packages/toil/job.py", line 2743, in _runner
returnValues = self._run(jobGraph=None, fileStore=fileStore)
File "/home/lotss-tshimwell/.local/lib/python3.9/site-packages/toil/job.py", line 2660, in _run
return self.run(fileStore)
File "/home/lotss-tshimwell/.local/lib/python3.9/site-packages/toil/cwl/cwltoil.py", line 2301, in run
raise cwl_utils.errors.WorkflowException(status)
cwl_utils.errors.WorkflowException: permanentFail
[2024-02-05T12:26:41+0100] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host wn-dc-03.spider.surfsara.nl
Looking at the error log that it points to I see:
[lotss-tshimwell@ui-01 L2005609]$ cat /project/lotss/Software/linc-master/detailed-logs/L2005609/L2005609ns69cg87/cal_solutions.h5-losoto_err.log
Traceback (most recent call last):
File "/usr/local/bin/losoto", line 141, in <module>
H = h5parm(args.h5parm, readonly=False)
File "/usr/local/lib/python3.8/dist-packages/losoto/h5parm.py", line 80, in __init__
self.H = tables.open_file(h5parmFile, 'r+', IO_BUFFER_SIZE=1024*1024*10, BUFFER_TIMES=500)
File "/usr/local/lib/python3.8/dist-packages/tables/file.py", line 300, in open_file
return File(filename, mode, title, root_uep, filters, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/tables/file.py", line 750, in __init__
self._g_new(filename, mode, **params)
File "tables/hdf5extension.pyx", line 366, in tables.hdf5extension.File._g_new
File "/usr/local/lib/python3.8/dist-packages/tables/utils.py", line 173, in check_file_access
raise PermissionError(f"file ``{path}`` exists but it can not be written")
PermissionError: file ``/DpUURT/cal_solutions.h5`` exists but it can not be written
[lotss-tshimwell@ui-01 L2005609]$
Edited by Matthijs van der Wild