LINC_Target step fail at DP3
Hi, I'm trying to run the LINC pipeline from a singularity with cwltool on a local computing cluster. The calibrator pipeline ran successfully with 3C295, however it is failing after ~2 hours on the target pipeline with 8 hours + 30 subbands of HBA core and remote station observations. I've attached a text file with the end of the log file:
LINC_Target_pipeline_error.txt
I ran the pipeline in verbose mode and in a DP3 log file (within the first dp3_execute folder in the log folder) the error:
terminate called after throwing an instance of 'dyscostman::DyscoStManError' terminate called recursively what(): Table DataManager error: I/O error: error while writing file '/tmp/l2rwcywm/L793560_SAP000_SB001_uv.dppp.MS/table.f6' -- Error occured inside the Dysco Storage Manager
was displayed.
The same error was then propagated down through the remaining measurement set error log files. Is this a file accessibility issue or some kind of memory error?
The command I'm running within the singularity is:
cwltool --no-container --preserve-entire-environment --parallel --log-dir /.../HBA_target.cwl /.../LINC_target.json > logfiletar 2>&1
(with the full directories to the .cwl and .json files)
Thanks, Jeremy