diff --git a/doc/Doxyfile.in b/doc/Doxyfile.in index 3347f7f825cfddf904c1c61a61fd7a09c3853d8c..4e4a309c46ddf5f1d9fa46926fbe8dc44e74c460 100644 --- a/doc/Doxyfile.in +++ b/doc/Doxyfile.in @@ -842,7 +842,7 @@ EXAMPLE_RECURSIVE = NO # that contain images that are to be included in the documentation (see the # \image command). -IMAGE_PATH = +IMAGE_PATH = @CMAKE_CURRENT_LIST_DIR@/images # The INPUT_FILTER tag can be used to specify a program that doxygen should # invoke to filter for each input file. Doxygen will invoke the filter program diff --git a/doc/design.md b/doc/design.md new file mode 100644 index 0000000000000000000000000000000000000000..4a944945ab9abaa9c7fc90804b8540607542b7ff --- /dev/null +++ b/doc/design.md @@ -0,0 +1,95 @@ +# Design + +## DPPP: a framework for streaming radio interferometric pipelines +DPPP (DP3, short for Default PreProcessing Pipeline) is a framework +for efficiently processing time-ordered visibilities, +e.g. freshly correlated data. It was developed for the LOFAR +telescope, and is used for preprocessing its imaging pipelines. +DPPP user documentation is found at the [LOFAR Wiki](https://www.astron.nl/lofarwiki/doku.php?id=public:user_software:documentation:ndppp). + +For avoiding high memory usage, DPPP reads the input data in chunks instead of reading all input data at once. +For each chunk, it calls all the steps to do the processing on this data. The last step will then +write out the processed data. + + + +DPPP runs a pipeline in clearly defined stages, to make spotting +errors as easy as possible. +* **Initialise** This connects all the processing steps together. This step is data independent. +* **Tune At** This stage reads metadata, such as the number of channels, and adjusts the steps accordingly. +* **Process** This stage pipes the data through the steps, one buffer at a time. The steps define the buffer size. + +## Measurement Set standard +DPPP adheres to the Measurement Set (MS) standard, which is the common format for radio telescope data. DPPP can +also create a new MS. When an appropriate input step is in +place (e.g. a streaming connection to a correlator), MSs can +be written from scratch. + +DPPP supports only regularly shaped MSs with the following restrictions: +- Every time slot has the same number of antennas. +- Time slots have the same duration and are equally spaced (missing time slots are allowed). +- Every integration has the same number of correlations and channels. +- Only one spectral window can be handled. + +## Built-in steps +DPPP contains built-in steps that are linked together and process one time slot at a time. This way, DPPP is therefore suitable for large measurement sets. +All the built-in steps derive from the [DPStep](@ref DP3::DPPP::DPStep) class. +Its inheritance diagram shows all step types, including the built-in steps. + +[DPRun](@ref DP3::DPPP::DPRun) initialises and orchestrates the configured steps. +It begins with setting the required info for the first step and its next step(s). +Next, each buffer (containing a single time slot) is processed by the first step. +When processed, it invokes the process function of the next step. +Once all buffers are processed, DPPP calls the `finish` function of the first step and subsequent steps. +This function flushes all remaining data from the steps to their output channel(s). +Finally, DPPP calls the `addToMS` function on each step for updating the metadata of the MS. +The figure below gives a graphical overview of the [DPStep](@ref DP3::DPPP::DPStep) flow. + + + +### Calibration +The DPPP step [GainCal](@ref DP3::DPPP::GainCal) implements many variants of direction +independent calibration. It uses [StefCal](https://ieeexplore.ieee.org/abstract/document/6930038), with many +extra features, such as fitting a function over frequency. +Also full-Jones calibration is supported. Results are written +to [ParmDB](https://www.astron.nl/lofarwiki/doku.php?id=public:user_software:documentation:makesourcedb); [H5Parm](https://github.com/revoltek/losoto/wiki/H5parm-specifications) support is in development. + +### AOFlagger +DPPP contains a DPStep class which is a shallow wrapper that calls AOFlagger. +This makes it possible to flag with AOFlagger and immediately +afterwards average the data, without writing the +data to disk in between. + +### User-defined steps +Apart from the built-in steps, also custom steps can +be defined, following the existing interface description. +Recompiling DPPP is not necessary: steps are found as shared +libraries. + +A special custom step is the [Python step](@ref DP3::DPPP::PythonStep), which uses a step defined in a Python module. +This makes it very easy to implement custom steps even for non C++-wizards. An +example Python step is bundled with the LOFAR software. + +## Example reduction +Arguments are given to DPPP as a 'parameter set', a simple +key-value format. Optionally, parameters can be overridden +on the command line. The example also shows that you can +use the out step to store intermediate results. + + msin = [bla.MS, bla2.MS] + steps = [aoflag, avg1, out1, cal, out2, avg2 ] + avg1.type = average + avg1.timestep = 4 # Average down every 4 timeslots + out1.type = out + out1.name = myfile_averaged.MS + cal.type = calibrate + cal.skymodel = 3C196.skymodel + cal.applysolution = true + cal.parmdb = instrument.parmdb + out2.type = out + out2.name = . + out2.datacolumn = CORRECTED_DATA + avg2.type = average + avg2.timestep = 2 # average down some more + avg2.freqstep = 16 + msout = bla_combined.MS diff --git a/doc/images/diagram.png b/doc/images/diagram.png new file mode 100644 index 0000000000000000000000000000000000000000..3a94abe45eedd3bf8d54bd39c4ac35c684f9bdad Binary files /dev/null and b/doc/images/diagram.png differ diff --git a/doc/images/flow.png b/doc/images/flow.png new file mode 100644 index 0000000000000000000000000000000000000000..754ce883a263a6a6257e4fd25a3dffdc8f929946 Binary files /dev/null and b/doc/images/flow.png differ