Task #8437: Add support for global fs in node scripts to prevent rsyncing files within the same filesystem

Task #8991 Change for the generic pipeline -> executable_casa.py: Merge with the multi-epoch version.

Task #8679 Fixes to run wsclean with multiple input files. Exact matching during inputkeys replacement instead of substrings. Loop break condition change as requested by David.

Task #6559 Bring the mapfile argument for subpipelines in line with the syntax for regular recipe steps (mapfile -> mapfile_in)

Task #6559 busy week update. string replacement: parameters to change can be easier be put on top of the parset. syntax corresponds to jinja2 in factor. fixed nested loops and break condition from python steps. started api for creating pipelines from python console.

Task #6559 Added a calibrate stand alone node script that directly calls bbs-reducer and not the calibrate-stand-alone shell script. This way bbs-reducer gets called by the pipeline infrastructure and not as a subprocess called by a different mechanism. Also the order of the arguments should not matter anymore when they are given as parset arguments (long options).

Task #6559 python plugins as node script. they can return a dict that gets saved by the master script as mapfile and can be used as input for later steps.

Task #6559 small update changing the automatic naming of results. names were too long and now only contain the last stepname the data went through.

Task #6559 Update from march busy week. Functionality for extreme peeling preparation and bug fixes.

Task #7491: Small rename of variables. Add output of correct files per image (including skip set), not used in the pipeline but for validity

Task #6559 Next iteration: executable-args recipe handles multiple input mapfiles with inputkeys for the including files. parameter for individual processes per node. the master recipe handles the executable_args and executable_casa noderecipes. casa now has individual tmpdirs as working dir which enables multiple instances on one node. changes in genericpipeline so that more than one occurence of outputreference can be replaced.

Task #7336: Give all temporary directories a suffix of the python file that created them, and clean up dangling temporary directories from tests

Task #6559 First experiments with a casapy recipe. Works in basic form for now but with workarounds for proper return values to the pipeline. casapy does not seem to return in a usual way. when there is an error the pipeline does not recognize the process has finished. without the encapsuleting in a shell script even a successful run does not return. Needs more checking and a cleaner way of calling. Maybe integrate casa python classes to the Lofar install? Loops: Nested loops work. Breaking a loop only works for one full cycle. Outputdictionaries you reference with step.output.stepval are now always for the last call of that step.

Task #6559 overhaul of the executable_* recipes. The arg one now covers all argument based calls and calls with saved parsets on the node side.

Task #6559 This is the third iteration. Introducing loops and additional recipes for executables in general. Both not 100% finisehd. Loops work but more work has to be done concerning in/output of the steps within the loop. Loops are pipeline steps themself which hold a list of steps they add to the pipeline run. If not using a fixed number of iterations one step in the loop should be a break condition step. This step specifies in its output if the loop should break and the pipeline continue its work. The parsetonly recipe now handles dppp and awimager steps. For patching the parset on node level in/output keywords have to given in the step control section.

Task #6559 adding recipes for smart demixing for backup purpose and there might be a need for the busy week to play with this. Warning those a are copy pastes of the old ndppp recipes and again special purpose only. A proper dppp recipe has to developed that covers all functionality of the program.

It is not part of the current design. Tested in locus102 in integration test. 

Task #6559 second iteration of generic pipeline stuff. special case for vdsmaker atm. all inputs should be reduced to dicts in next phase. more cleanup and structure needed as well. for now mss_calibration pipeline is looking good and gets to step5.

Task #6559 An attempt for generic processing a pipeline defined by parsets. Worked backwards from the dppp scripts. The node script still has patch parsets routines.
For the moment this can only handle some basic stuff. Chain recipes that as only input get a mapfile and a parset and throw some "plugins" inbetween. The plugins have to be given some thought. Are they useful/needed? Need templates/interfaces? etc
Should all recipes be required to handle mapfiles/MS?
The go() method in the pipeline scripts are only required to set a name for the pipeline. Moved this functionality to the baserecipe to avoid redundancies. Now only the pipeline_logic has to be reimplemented.
We need standard/templates for recipes so they can be used in a unified way by pipeline scripts.