Skip to content
Snippets Groups Projects

Add initial top-level workflow

Merged Mick Veldhuis requested to merge add-pipeline-workflow into main
All threads resolved!
Files
3
tests/README.md 0 → 100644
+ 27
0
# Testing the Pre-Processing Pipeline
The `/tests` directory contains a number of sample inputs that can be used to test and verify the pipeline.
## Sample MeasurementSets
The sample inputs, defined in the JSON format, reference a number of relatively small raw LOFAR MeasurementSets (MSs), 3 GB each:
```
L888536_SAP000_SB026_uv.MS
L888536_SAP000_SB027_uv.MS
L888536_SAP000_SB028_uv.MS
```
These MSs are stored on the [DAS-6](https://www.cs.vu.nl/das6/) cluster in the following directory:
```
/var/scratch/veldhuis/data/raw
```
This folder also contains a tarball with all three MSs: `raw_lofar_data.tar`. If you don't have access to DAS-6 and would like to test the pipeline, but do not have access to raw LOFAR data, please contact Team Rapthor.
## Running the tests
To run these tests you need to have a CWL runner, such as `cwltool` or `toil`, installed on your system.
For example, with `cwltool` you could run the top-level `pipeline.cwl` workflow as follows:
```
cwltool --debug --preserve-entire-environment --outdir=pipeline-out $PREPROCESS_ROOT/workflows/pipeline.cwl $PREPROCESS_ROOT/tests/pipeline_input.json
```
where `PREPROCESS_ROOT` refers to the location of the repository on your system. Please modify the command as desired, for an overview of possible arguments run `cwltool --help` or refer to their [Read the Docs pages](https://cwltool.readthedocs.io/).
\ No newline at end of file
Loading