LINC target is currently incompatible with compute clusters that restrict internet access
Hi,
This may be a relatively uncommon scenario, but I am currently mainly running LINC target on a compute cluster that (for security reasons?) does not allow any internet access on the compute nodes. The login nodes have internet access, obviously, but batch jobs are run on nodes without internet access.
Currently, this presents two problems as far as I've encountered: the TGSS sky model and the IONEX data can not be downloaded.
I have managed to hack my way around it by downloading these on the login nodes and then injecting them into the pipeline. For the TGSS sky model, this is relatively straight forward, as the pipeline simply tries to find a target.skymodel file and only attempts to download it if the skymodel is not present. However, for the IONEX data, this is more challenging as this required modifying the createRMh5parm.py file:
if ionexf == -1:
logging.error("IONEX data not available, even not from fast product server. Trying to find existing file.")
from glob import glob
os.system('mv ../../IONEX/* .')
IONEX_files = glob(f'./*')
for f in IONEX_files:
if f is not "./cal_solutions.h5":
ionexf = f
if ionexf == -1:
logging.error("IONEX data not available")
return(-1)
The above solution requires the aid of an additional script to download the IONEX data and place them in the temporary work directory, so it can be found by the pipeline. So this is probably not a robust implementation that would work for everyone. I've only added it as a demonstration of how I've managed to work around it.
To make it easier for myself, and potentially others, to keep our software up-to-date without needing to implement custom edits, it would be great if the official LINC version could be updated to accommodate users on compute infrastructure where internet access is not available.
Thanks,
Roland