LINC Docker permanentFail at find_skymodel_cal step
Hi,
I have set up the LINC docker on my local machine, and have started the processing within it via cwltool on a subset of calibrator measurement set data from a LOFAR core station observation.
The command run to set up the docker volume with a bind mount between the host machine for file access/modification:
docker run -it --name lofar -v $(pwd):/opt/Data:z -u (my_uid) astronrd/linc
then within docker:
cwltool --no-container /usr/local/share/linc/workflows/HBA_calibrator.cwl /opt/Data/LINC.json > logfile 2>&1
The pipeline currently fails at the find skymodel cal step with the following error (logfile below):
Click to expand
INFO /usr/local/bin/cwltool 3.1.20220406080846
INFO Resolved '/usr/local/share/linc/workflows/HBA_calibrator.cwl' to 'file:///usr/local/share/linc/workflows/HBA_calibrator.cwl'
INFO [workflow ] start
INFO [workflow ] starting step linc
INFO [step linc] start
INFO [workflow linc] start
INFO [workflow linc] starting step prep
INFO [step prep] start
INFO [workflow prep] start
INFO [workflow prep] starting step select
INFO [step select] start
INFO [step select] completed success
INFO [workflow prep] starting step check_ateam_separation
INFO [step check_ateam_separation] start
INFO [job check_ateam_separation] /tmp/9vtpd9x_$ check_Ateam_separation.py \
/tmp/njfnu3th/stgdc6b2fb6-63e3-429e-9d46-fea2344c0735/L793556_SAP000_SB000_uv.dppp.MS \
--min_separation \
30 \
--outputimage \
Ateam_separation.png > /tmp/9vtpd9x_/Ateam_separation.log
INFO [job check_ateam_separation] Max memory used: 45MiB
INFO [job check_ateam_separation] completed success
INFO [step check_ateam_separation] completed success
INFO [workflow prep] starting step find_skymodel_cal
INFO [step find_skymodel_cal] start
INFO [job find_skymodel_cal] /tmp/ljk701c7$ python3 \
find_sky.py \
/tmp/3kbmujdl/stg5e8c8e71-b3df-43b5-a737-11d0b94d7b89/L793556_SAP000_SB000_uv.dppp.MS > /tmp/ljk701c7/find_skymodel_cal.log 2> /tmp/ljk701c7/find_skymodel_cal_err.log
WARNING [job find_skymodel_cal] exited with status: 1
ERROR Expecting value: line 1 column 1 (char 0)
script was:
01 "use strict";
02 var inputs = {
03 "msin": {
04 "class": "Directory",
05 "location": "file:///opt/Data/3C295/25_08_20/L793556_SAP000_SB000_uv.dppp.MS",
06 "basename": "L793556_SAP000_SB000_uv.dppp.MS",
07 "path": "/tmp/3kbmujdl/stg5e8c8e71-b3df-43b5-a737-11d0b94d7b89/L793556_SAP000_SB000_uv.dppp.MS",
08 "dirname": "/tmp/3kbmujdl/stg5e8c8e71-b3df-43b5-a737-11d0b94d7b89"
09 },
10 "skymodels": null,
11 "max_separation_arcmin": 1.0,
12 "skymodels_extension": null
13 };
14 var self = [];
15 var runtime = {
16 "cores": 1,
17 "ram": 256,
18 "tmpdirSize": 1024,
19 "outdirSize": 1024,
20 "exitCode": 1,
21 "tmpdir": "/tmp/inkb2qe3",
22 "outdir": "/tmp/ljk701c7"
23 };
24 (function(){return ((JSON.parse(self[0].contents).file));})()
stdout was: ''
stderr was: 'evalmachine.<anonymous>:24
(function(){return ((JSON.parse(self[0].contents).file));})()
^
TypeError: Cannot read property 'contents' of undefined
at evalmachine.<anonymous>:24:41
at evalmachine.<anonymous>:24:60
at Script.runInContext (vm.js:133:20)
at Script.runInNewContext (vm.js:139:17)
at Object.runInNewContext (vm.js:322:38)
at Socket.<anonymous> ([eval]:11:57)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:265:13)
at Socket.Readable.push (_stream_readable.js:224:10)'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/cwltool/sandboxjs.py", line 384, in execjs
return cast(CWLOutputType, json.loads(stdout))
File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/cwltool/expression.py", line 393, in do_eval
return interpolate(
File "/usr/local/lib/python3.8/dist-packages/cwltool/expression.py", line 297, in interpolate
e = evaluator(
File "/usr/local/lib/python3.8/dist-packages/cwltool/expression.py", line 232, in evaluator
return execjs(
File "/usr/local/lib/python3.8/dist-packages/cwltool/sandboxjs.py", line 386, in execjs
raise JavascriptException(
cwltool.sandboxjs.JavascriptException: Expecting value: line 1 column 1 (char 0)
script was:
01 "use strict";
02 var inputs = {
03 "msin": {
04 "class": "Directory",
05 "location": "file:///opt/Data/3C295/25_08_20/L793556_SAP000_SB000_uv.dppp.MS",
06 "basename": "L793556_SAP000_SB000_uv.dppp.MS",
07 "path": "/tmp/3kbmujdl/stg5e8c8e71-b3df-43b5-a737-11d0b94d7b89/L793556_SAP000_SB000_uv.dppp.MS",
08 "dirname": "/tmp/3kbmujdl/stg5e8c8e71-b3df-43b5-a737-11d0b94d7b89"
09 },
10 "skymodels": null,
11 "max_separation_arcmin": 1.0,
12 "skymodels_extension": null
13 };
14 var self = [];
15 var runtime = {
16 "cores": 1,
17 "ram": 256,
18 "tmpdirSize": 1024,
19 "outdirSize": 1024,
20 "exitCode": 1,
21 "tmpdir": "/tmp/inkb2qe3",
22 "outdir": "/tmp/ljk701c7"
23 };
24 (function(){return ((JSON.parse(self[0].contents).file));})()
stdout was: ''
stderr was: 'evalmachine.<anonymous>:24
(function(){return ((JSON.parse(self[0].contents).file));})()
^
TypeError: Cannot read property 'contents' of undefined
at evalmachine.<anonymous>:24:41
at evalmachine.<anonymous>:24:60
at Script.runInContext (vm.js:133:20)
at Script.runInNewContext (vm.js:139:17)
at Object.runInNewContext (vm.js:322:38)
at Socket.<anonymous> ([eval]:11:57)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:265:13)
at Socket.Readable.push (_stream_readable.js:224:10)'
[job find_skymodel_cal] Job error:
('Error collecting output for parameter \'output_models\': ../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: Expression evaluation error:\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: Expecting value: line 1 column 1 (char 0)\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: script was:\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 01 "use strict";\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 02 var inputs = {\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 03 "msin": {\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 04 "class": "Directory",\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 05 "location": "file:///opt/Data/3C295/25_08_20/L793556_SAP000_SB000_uv.dppp.MS",\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 06 "basename": "L793556_SAP000_SB000_uv.dppp.MS",\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 07 "path": "/tmp/3kbmujdl/stg5e8c8e71-b3df-43b5-a737-11d0b94d7b89/L793556_SAP000_SB000_uv.dppp.MS",\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 08 "dirname": "/tmp/3kbmujdl/stg5e8c8e71-b3df-43b5-a737-11d0b94d7b89"\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 09 },\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 10 "skymodels": null,\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 11 "max_separation_arcmin": 1.0,\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 12 "skymodels_extension": null\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 13 };\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 14 var self = [];\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 15 var runtime = {\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 16 "cores": 1,\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 17 "ram": 256,\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 18 "tmpdirSize": 1024,\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 19 "outdirSize": 1024,\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 20 "exitCode": 1,\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 21 "tmpdir": "/tmp/inkb2qe3",\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 22 "outdir": "/tmp/ljk701c7"\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 23 };\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: 24 (function(){return ((JSON.parse(self[0].contents).file));})()\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: stdout was: \'\'\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: stderr was: \'evalmachine.<anonymous>:24\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: (function(){return ((JSON.parse(self[0].contents).file));})()\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: ^\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: \n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: TypeError: Cannot read property \'contents\' of undefined\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: at evalmachine.<anonymous>:24:41\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: at evalmachine.<anonymous>:24:60\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: at Script.runInContext (vm.js:133:20)\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: at Script.runInNewContext (vm.js:139:17)\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: at Object.runInNewContext (vm.js:322:38)\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: at Socket.<anonymous> ([eval]:11:57)\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: at Socket.emit (events.js:198:13)\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: at addChunk (_stream_readable.js:288:12)\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: at readableAddChunk (_stream_readable.js:265:13)\n../../usr/local/share/linc/steps/find_skymodel_cal.cwl:35:9: at Socket.Readable.push (_stream_readable.js:224:10)\'', {})
WARNING [job find_skymodel_cal] completed permanentFail
ERROR [step find_skymodel_cal] Output is missing expected field file:///usr/local/share/linc/workflows/linc_calibrator/prep.cwl#prep/find_skymodel_cal/output_models
ERROR [step find_skymodel_cal] Output is missing expected field file:///usr/local/share/linc/workflows/linc_calibrator/prep.cwl#prep/find_skymodel_cal/model_name
ERROR [step find_skymodel_cal] Output is missing expected field file:///usr/local/share/linc/workflows/linc_calibrator/prep.cwl#prep/find_skymodel_cal/logfile
WARNING [step find_skymodel_cal] completed permanentFail
INFO [workflow prep] completed permanentFail
WARNING [step prep] completed permanentFail
INFO [workflow linc] completed permanentFail
WARNING [step linc] completed permanentFail
INFO [workflow ] completed permanentFail
{
"log_files": null,
"inspection_plots": null,
"summary": null,
"solutions": null
}
WARNING Final process status is permanentFail
Any help with what may be the issue would be greatly appreciated. I'm unsure if it is a file access issue or if perhaps the file that the pipeline is searching for can't be created for some reason.
Thanks,
Jeremy