Resolve RAP-202

Former-commit-id: df518679 [formerly 3105c505]
Former-commit-id: 7619b97b
Former-commit-id: 33052033

docs/Makefile

+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SPHINXPROJ    = Prefactor
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
\ No newline at end of file

docs/source/MFS-I-image-pb.plot_im_high_i.png.REMOVED.git-id

+4e67d4cbcae86474b49270b3b8f0994336edc3fd
\ No newline at end of file

docs/source/MFS-V-image-pb.plot_im_high_v.png.REMOVED.git-id

+171e3e127ccac94e51586d8dd00d867e0c577073
\ No newline at end of file

docs/source/acknowledgements.rst

+.. _acknowledgements:
+
+Acknowledgements
+================
+
+The **prefactor** pipeline and its scripts were developed by:
+
+    * Alexander Drabent
+    * David Rafferty
+    * Andreas Horneffer
+    * Francesco de Gasperin
+    * Marco Iacobelli
+    * Emanuela Orru
+    * Björn Adebahr
+    * Martin Hardcastle
+    * George Heald
+    * Soumyajit Mandal
+    * Carole Roskowinski
+    * Jose Sabater Montes
+    * Timothy Shimwell
+    * Sarrvesh Sridhar
+    * Reinout van Weeren
+    * Wendy Williams
+
+With special thanks to Stefan Fröhlich for developing the genericpipeline.
+
+The procedure is also mostly described in these papers:
+
+    * de Gasperin, F.; Dijkema, T. J.; Drabent, A.; Mevius, M.; Rafferty, van Weeren, R., et al. 2018, arXiv:1811.07954
+    * van Weeren, R. J., Williams, W. L., Hardcastle, M. J., et al. 2016, ApJS, 223, 2
+    * Williams, W. L., van Weeren, R. J., Röttgering, H. J. A., et al. 2016, MNRAS, 460, 2385

docs/source/ampAFlag_polXX.png.REMOVED.git-id

+ec6c1d1a16fcb48c7a8a1bd4d67ca4120747fbc1
\ No newline at end of file

docs/source/ampBFlag_polXX.png.REMOVED.git-id

+73483c9d0c198f7c47f60082baae6995a9572700
\ No newline at end of file

docs/source/calibrator.rst

+.. _calibrator_pipeline:
+
+Calibrator pipeline
+===================
+
+.. note::
+
+   If you are running the deprecated genericpipeline version of the pipeline (**prefactor** 3.2 or older), please check the :doc:`old instrunctions page<calibrator_old>`.
+
+This pipeline processes the calibrator data in order to derive direction-independent corrections.
+It will take into account the correct order of distortions to be calibrated for.
+This chapter will present the specific steps of the calibrator pipeline in more detail.
+
+All results (diagnostic plots and calibration solutions) will be stored usually in the ``--outdir`` directory specified with your ``cwltool`` or ``toil`` command.
+
+    .. image:: calibscheme.png
+
+Prepare calibrator, incl. "demixing" (``prep``)
+-----------------------------------------------
+
+This part of the pipeline prepares the calibrator data in order to be calibration-ready.
+This mainly includes mitigation of bad data (RFI, bad antennas, contaminations from A-Team sources), selection of the data to be calibrated, and some averaging to reduce data size and enhance the signal-to-noise ratio.
+The user can specify whether to do raw data or pre-processed data flagging and whether demixing should be performed.
+
+The basic workflows are:
+
+- preparation of data (``prep``)
+- correcting for polarization alignment (``PA``)
+- correcting for Faraday Rotation (``FR``)
+- correcting for bandpass (``BP``)
+- correcting for ionospheric disturbances (``ion``)
+
+The workflow ``prep`` consists of: 
+    - determining suitable calibrator skymodel (steps ``find_skymodel_cal``, ``make_sourcedb``)
+    - checking for nearby A-Team sources (step ``check_Ateam_separation``)
+    - creating a model of A-Team sources to be subtracted (step ``make_sourcedb_ateam``)
+    - basic flagging and averaging (subworkflow ``ndppp_prep_cal``)
+        - edges of the band (``flagedge``) -- only used if ``raw_data : true``
+        - statistical flagging (``aoflag``) -- only used in ``raw_data : true``
+        - baseline flagging (``flagbaseline``)
+        - low elevation flagging (below 15 degress elevation) (``flagelev``)
+        - low amplitude flagging (below 1e-30) (``flagamp``)
+        - demix A-Team sources (``demix``) -- only used if specified ``demix : true``
+        - averaging of the data in time and frequency
+    - wide-band statistical flagging (steps ``ms_concat`` and ``aoflag``)
+    - write the calibrator skymodel into the ``MODEL_DATA`` column (step ``predict``) and perform direction-independent phase-only calibration (diagonal terms + common rotation angle, step ``calib_cal``) (subworkflow ``predict_cal``, baseline-dependent smoothing (step ``BLsmooth``) if specified ``do_smooth : true``)
+
+Calibration of the polarization alignment (``PA``)
+--------------------------------------------------
+The phase solutions derived from the preparation step are now collected and loaded into **LoSoTo**.
+**LoSoTo** will derive the polarizion alignment and provide diagnostic plots under ``results/inspection/``:
+
+- ``polalign_ph_pol??``: matrix plot of the phase solutions for the XX and YY polarization
+    .. image:: polalign_ph_polXX.png
+- ``polalign_ph_poldif``: matrix plot of the phase solutions from XX-YY
+    .. image:: polalign_ph_poldif.png
+- ``polalign_rotange``: matrix plot of the common rotation angle solutions
+    .. image:: polalign_rotangle.png
+- ``polalign_amp_pol??``: matrix plot of the amplitude solutions for the XX and YY polarization
+    .. image:: polalign_amp_polXX.png
+- ``polalign``: matrix plot of the derived polarization alignment between XX and YY
+    .. image:: polalign.png
+- ``polalign_ph-res_pol??``: matrix plot of the residual phase solutions for the XX and YY polarization after subtraction the derived polarization alignment
+- ``polalign_ph-res_poldif``: matrix plot of the residual phase solutions for XX-YY after subtraction of the derived polarization alignment
+
+The workflow ``PA`` consists of:
+    - deriving the polarization alignment from the calibration solutions (subworkflow ``PolAlign``)
+    - creating diagnostic plots (steps ``losoto_plot``)
+    - applying polarization alignment solutions (step ``applyPA``) and the element beam correction (step ``applybeam``) to the original data and re-calibrate (diagonal terms + common rotation angle, step ``calib_cal``) (subworkflow ``apply_calibrate_pa``, baseline-dependend smoothing (step ``BLsmooth``) if specified ``do_smooth : true``)
+
+Calibration of the Faraday Rotation (``FR``)
+--------------------------------------------
+The outcome of the re-calibration **after** correcting for the polarization alignment is again loaded into **LoSoTo** in order to derive corrections for Faraday Rotation.
+The following diagnostic plots are created:
+
+- ``fr_ph_pol??``: matrix plot of the phase solutions for the XX and YY polarization
+- ``fr_ph_poldif``: matrix plot of the phase solutions from XX-YY
+- ``fr_rotange``: matrix plot of the common rotation angle solutions
+- ``fr_amp_pol??``: matrix plot of the amplitude solutions for the XX and YY polarization
+- ``fr``: matrix plot of the derived differential Rotation Measure from Faraday Rotation
+    .. image:: fr.png
+
+The workflow ``FR`` consists of:
+    - deriving the Faraday Rotation from the calibration solutions (subworkflow ``FaradayRot``)
+    - creating diagnostic plots (steps ``losoto_plot``)
+    - applying Faraday Rotation solutions (step ``applyFR``) and re-calibrate (diagonal terms + common rotation angle, step ``calib_cal``) (subworkflow ``apply_calibrate_fr``, baseline-dependend smoothing (step ``BLsmooth``) if specified ``do_smooth : true``)
+    
+Calibration of the Bandpass (``BP``)
+------------------------------------
+The outcome of the re-calibration **after** correcting for the polarization alignment and Faraday Rotation is loaded into **LoSoTo** in order to derive corrections for the bandpass. A robust flagging on the amplitude solutions as well as a Savitzky-Golay filter is applied in order to reject bad solutions and smooth the outcome. Frequency regimes up to a certain maximum width (parameter ``max2interpolate``) will be interpolated if flagged.
+The following diagnostic plots are created:
+
+- ``ampBFlag__??``: matrix plot of the amplitude solutions for the XX and YY polarization ``before`` flagging
+    .. image:: ampBFlag_polXX.png
+- ``ampAFlag__??``: matrix plot of the amplitude solutions for the XX and YY polarization ``after`` flagging
+    .. image:: ampAFlag_polXX.png
+- ``bandpass_pol??``: the derived bandpass of all stations in the XX and YY polarization
+- ``bandpass_time??``: matrix plot of the derived bandpass, where both polarizations are colorcoded
+    .. image:: bandpass.png
+- ``bandpass_time??_pol??``: plot of the derived bandpass of the XX and YY polarization, where all stations are colorcoded
+    .. image:: bandpass_polXX.png
+
+The workflow ``BP`` consists of:
+    - deriving the bandpass from the calibration solutions (subworkflow ``bandpass``)
+    - creating diagnostic plots (steps ``losoto_plot``)
+    - transfer solutions for international stations for non-trusted calibrator sources (step ``transfer_solutions``), see parameters ``do_transfer`` and ``trusted_sources``
+    - applying polarization alignment solutions (step ``applyPA``), the bandpass (step ``applyBP``), the element beam correction (step ``applybeam``) and the Faraday Rotation solutions (``applyFR``) to the original data and calibrate (only single scalar phase, step ``calib_cal``) (subworkflow ``apply_calibrate_bp``, baseline-dependend smoothing (step ``BLsmooth``) if specified ``do_smooth : true``)
+    - deriving final amount of flags applied to the data (step ``final_flags``)
+
+Calibration of the instrumental and ionospheric delays (``ion``)
+----------------------------------------------------------------
+The outcome of the re-calibration **after** correcting for the polarization alignment, the bandpass and the Faraday Rotation is loaded into **LoSoTo** in order to derive corrections for the instrumental and ionospheric delays. A robust flagging on the amplitude solutions is applied in order to reject bad solutions. These flags are applied to the phase solutions. These phase solutions should be mainly affected by instrumental (clock) and ionospheric (TEC) delays. This **LoSoTo** step will aim to separate both effects (clock-TEC separation).
+The following diagnostic plots are created:
+
+- ``ion_ph``: matrix plot of the phase solutions
+- ``clock``: matrix plot of the derived (instrumental) clock offsets in seconds
+    .. image:: clock.png
+- ``tec``: matrix plot of the derived differential TEC in TECU
+    .. image:: tec.png
+- ``ion_ph-res``: matrix plot of the residual phase solutions after the subtraction of the derived instrumental and ionospheric delays
+
+The workflow ``ion`` consists of:
+    - deriving the bandpass from the calibration solutions (subworkflow ``clocktec``)
+    - creating diagnostic plots (steps ``losoto_plot``)
+    - create a summary file (step ``summary``)
+
+.. note::
+    All solutions are written in the h5parm file format via the steps ``H5parm_collector`` and called during all the workflows.
+    
+    The solutions are stored in the final calibrator solution set ``results/cal_values/cal_solutions.h5``.
+
+Further diagnostics
+-------------------
+
+The ``results`` directory will contain all relevant outputs of the current **prefactor** run, once the pipeline has finished:
+    - logfiles in ``results/logs``
+    - summary file (JSON format) in ``results/summary``
+    - calibration solutions in ``results/cal_values/cal_solutions.h5``
+    - inspection plots in ``results/inspection``
+
+``Ateam_separation.png`` shows the distance and the elevation of A-Team sources with respect to the analyzed observation.
+
+You can also check the calibration solutions for more details::
+
+    $ losoto -i results/cal_values/cal_solutions.h5
+    
+    Summary of results/cal_values/cal_solutions.h5
+
+    Solution set 'calibrator':
+    ==========================
+
+    Directions: 3c286
+
+    Stations: CS001HBA0     CS001HBA1       CS002HBA0       CS002HBA1
+              CS003HBA0     CS003HBA1       CS004HBA0       CS004HBA1
+              CS005HBA0     CS005HBA1       CS006HBA0       CS006HBA1
+              CS007HBA0     CS007HBA1       CS011HBA0       CS011HBA1
+              CS017HBA0     CS017HBA1       CS021HBA0       CS021HBA1
+              CS024HBA0     CS024HBA1       CS026HBA0       CS026HBA1
+              CS028HBA0     CS028HBA1       CS030HBA0       CS030HBA1
+              CS031HBA0     CS031HBA1       CS032HBA0       CS032HBA1
+              CS101HBA0     CS101HBA1       CS103HBA0       CS103HBA1
+              CS201HBA0     CS201HBA1       CS301HBA0       CS301HBA1
+              CS302HBA0     CS302HBA1       CS401HBA0       CS401HBA1
+              CS501HBA0     CS501HBA1       RS106HBA        RS205HBA
+              RS208HBA      RS210HBA        RS305HBA        RS306HBA
+              RS307HBA      RS310HBA        RS406HBA        RS407HBA
+              RS409HBA      RS503HBA        RS508HBA        RS509HBA
+
+    Solution table 'bandpass' (type: amplitude): 120 times, 11 freqs, 60 ants, 2 pols
+        Flagged data: 0.000%
+
+    Solution table 'clock' (type: clock): 120 times, 60 ants
+        Flagged data: 0.000%
+
+    Solution table 'faraday' (type: rotationmeasure): 60 ants, 120 times
+        Flagged data: 0.222%
+
+    Solution table 'polalign' (type: phase): 120 times, 60 ants, 40 freqs, 2 pols
+        Flagged data: 0.000%
+
+For an overall summary it is advised to check the summary logfile::
+
+    $ cat results/logs/3c286_summary.log
+    
+    *********************************************
+    *** prefactor calibrator pipeline summary ***
+    *********************************************
+
+    Field name: 3c286
+
+    User-specified baseline filter: *&
+    Additional antennas removed from the data: NONE
+    A-Team sources close to the phase reference center: VirA
+        Of which were demixed: NONE
+        Of which were clipped: NONE
+
+    Amount of flagged solutions per station and solution table:
+    Station   bandpass  clock   faraday  polalign
+    CS001HBA0   9.08%  100.00%    0.00%    0.00%
+    CS001HBA1   9.08%  100.00%    0.00%    0.00%
+    CS002HBA0   9.08%  100.00%    0.00%    0.00%
+    CS002HBA1   9.08%  100.00%    0.00%    0.00%
+    CS003HBA0   9.08%  100.00%    0.00%    0.00%
+    CS003HBA1   9.08%  100.00%    0.00%    0.00%
+
+    Amount of flagged data per station at a given state:
+    Station    initial  final
+    CS001HBA0   3.18%   3.96%
+    CS001HBA1   2.98%   3.97%
+    CS002HBA0   3.18%   4.42%
+    CS002HBA1   2.95%   3.67%
+    CS003HBA0   2.96%   3.94%
+    CS003HBA1   3.10%   4.21%
+
+    **********
+    Summary file is written to: 3c286_prefactor_calibrator_summary.json
+    Summary has been created.
+
+User-defined parameter configuration
+------------------------------------
+
+**Parameters you will need to adjust**
+
+*Location of the calibrator solutions*
+
+- ``msin``: location of the input calibrator data, for instructions look at the :doc:`configuration instructions<parset>` page
+
+**Parameters you may need to adjust**
+
+*Data selection and calibration options*
+
+- ``refant``: regular expression of the stations that are allowed to be selected as a reference antenna by the pipeline (default: ``CS00.*``)
+- ``flag_baselines``: DP3-compatible pattern for baselines or stations to be flagged (may be an empty list, i.e.: ``[]`` )
+- ``process_baselines_cal``: performs A-Team-clipping/demixing and direction-independent phase-only self-calibration only on these baselines. Choose ``[CR]S*&`` if you want to process only cross-correlations and remove international stations (default: ``*&``)
+- ``filter_baselines``: selects only this set of baselines to be processed. Choose ``[CR]S*&`` if you want to process only cross-correlations and remove international stations (default: ``*&``)
+- ``do_smooth``: enable or disable baseline-based smoothing (default: ``false``)
+- ``rfistrategy``: strategy to be applied with the statistical flagger (`AOFlagger`_, default: ``HBAdefault.rfis``)
+- ``max2interpolate``: amount of channels in which interpolation should be performed for deriving the bandpass (default: 30)
+- ``fit_offset_PA``: assume that together with a delay each station has also a differential phase offset (important for old LBA observations, default: ``false``)
+- ``interp_windowsize``: size of the window over which a value is interpolated. Should be odd. (default: 15)
+- ``ampRange``: range of median amplitudes accepted per station (default: ``[0,0]``)
+- ``skip_international``: skip fitting the bandpass for international stations (this avoids flagging them in many cases, default: ``true``)
+- ``raw_data``: use autoweight, set to True in case you are using raw data (default: ``false``)
+- ``propagatesolutions``: use already derived solutions as initial guess for the upcoming time slot (default: ``true``)
+- ``flagunconverged`` : flag solutions for solves that did not converge (if they were also detected to diverge, default: ``false``)
+- ``maxStddev``: maximum allowable standard deviation when outlier clipping is done. For phases, this should value should be in radians, for amplitudes in log(amp). If None (or negative), a value of 0.1 rad is used for phases and 0.01 for amplitudes (default: ``-1.0``)
+- ``solutions2transfer``: provide own solutions from a reference calibrator observation in case calibrator source is not trusted
+- ``antennas2transfer``: DP3-compatible baseline pattern for those stations who should get calibration solutions from a reference solution set in case calibrator source is not trusted (default: ``[FUSPID].*``)
+- ``do_transfer``: enable solutions transfer for non-trusted calibrator sources (default: ``false``)
+- ``trusted_sources``: comma-separated list of trusted calibrator sources. Solutions are only transferred from a reference solution set in case the observed calibrator is not among them (default: ``3C48,3C147,3C196,3C295,3C380``)
+- ``ion_3rd``: take into account also 3rd-order effects for the clock-TEC separation (ionospheric calibration, default: ``false``)
+- ``clock_smooth``: only take the median of the derived clock solutions (enable this in case of non-joint observations, default: ``true``)
+
+A comprehensive explanation of the baseline selection syntax can be found `here`_.
+
+
+*Demixing options* (only used if demix step is added to the ``prep_cal_strategy`` variable)
+
+- ``demix_sources``: choose sources to demix (provided as list), e.g., ``[CasA,CygA]``
+- ``demix_target``: if given, the target source model (its patch in the SourceDB) is taken into account when solving (default: ``""``)
+- ``demix_freqstep``: number of channels to average when demixing (default: 16)
+- ``demix_timestep`` : number of time slots to average when demixing (default: 10)
+- ``demix``: enable demixing (default: ``false``)
+
+*Further pipeline options*
+
+- ``min_separation``: minimum accepted distance to an A-team source on the sky in degrees (will raise a WARNING, default: ``30``)
+- ``tables2export``: choose which tables to export to the solutions file after the ionospheric calibration (default: ``clock``)
+
+**Parameters for pipeline performance**
+
+- ``max_dppp_threads``: number of threads per process for DP3 (default: 10)
+- ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20)
+- ``min_length``: minimum amount of subbands to concatenate in frequency necessary to perform the wide-band flagging in the RAM. It data is too big aoflag will use indirect-read (default: 50)
+- ``overhead``: only use this fraction of the available memory for deriving the amount of data to be concatenated (default: 0.8)
+
+**Parameters you may want to adjust**
+
+*Skymodel directory*
+
+- ``calibrator_path_skymodel``: location of the prefactor calibrator skymodels
+- ``max_separation_arcmin``: maximum separation between phase center of the observation and the patch of a calibrator skymodel which is accepted to be chosen as a skymodel (default: 1.0)
+- ``A-Team_skymodel``: location of the prefactor A-Team skymodels
+
+*Averaging for the calibrator data*
+
+- ``avg_timeresolution``: final time resolution of the data in seconds after averaging (default: 4)
+- ``avg_freqresolution`` : final frequency resolution of the data after averaging (default: 48.82kHz, which translates to 4 channels per subband)
+- ``bandpass_freqresolution``: frequency resolution of the bandpass solution table (default: 195.3125kHz, which translates to 1 channel per subband)
+
+.. _here: https://www.astron.nl/lofarwiki/doku.php?id=public:user_software:documentation:ndppp#description_of_baseline_selection_parameters
+.. _AOFlagger: https://gitlab.com/aroffringa/aoflagger.git

docs/source/calibrator_old.rst

+.. _calibrator_pipeline_old:
+
+Calibrator pipeline
+===================
+
+.. note::
+
+   These instructions are outdated and only valid for **prefactor** 3.2 or older. Please check the :doc:`recent instrunctions page<calibrator>`.
+
+
+This pipeline processes the calibrator data in order to derive direction-independent corrections.
+It will take into account the correct order of distortions to be calibrated for.
+This chapter will present the specific steps of the calibrator pipeline in more detail.
+You will find the single steps in the parameter ``pipeline.steps`` in line 98.
+All results (diagnostic plots and calibration solutions) are usually stored in a subfolder of the results directory, see ``inspection_directory`` (line 79) and ``cal_values_directory`` (line 80), respectively.
+
+    .. image:: calibscheme.png
+
+Prepare calibrator (incl. "demixing")
+-------------------------------------
+
+This part of the pipeline prepares the calibrator data in order to be calibration-ready.
+This mainly includes mitigation of bad data (RFI, bad antennas, contaminations from A-Team sources), selection of the data to be calibrated (usually Dutch stations only), and some averaging to reduce data size and enhance the signal-to-noise ratio.
+The user can specify whether to do raw data or pre-processed data flagging and whether demixing should be performed.
+
+The basic steps are:
+
+- mapping of data to be used (``createmap_cal``)
+- creating a model of A-Team sources to be subtracted (``make_sourcedb_ateam``)
+- basic flagging and averaging (``ndppp_prep_cal``)
+    - edges of the band (``flagedge``) -- only used in ``raw_flagging`` mode
+    - statistical flagging (``aoflag``) -- only used in ``raw_flagging`` mode
+    - baseline flagging (``flag``)
+    - low elevation flagging (below 20 degress elevation) (``elev``)
+    - demix A-Team sources (``demix``) -- only used if specified
+    - interpolation of flagged data (``interp``)
+    - averaging of the data to 4 sec and 4 channels per subband (``avg``)
+- wide-band statistical flagging (``aoflag``)
+- find needed skymodel of calibrator automatically (``sky_cal``)
+- write the calibrator skymodel into the MODEL_DATA column (``predict_cal``)
+- interpolate flagged data from the wide-band statistical flagging step (``interp_cal``)
+- baseline-dependent smoothing of the data (``smooth_data``)
+- perform direction-independent phase-only calibration (diagonal terms + common rotation angle) (``calib_cal``)
+
+The solutions are stored in the h5parm file format.
+
+Calibration of the polarization alignment (PA)
+----------------------------------------------
+The phase solutions derived from the preparation step are now collected and loaded into **LoSoTo**.
+**LoSoTo** will derive the polarizion alignment and provide diagnostic plots:
+
+- ``polalign_ph_pol??``: matrix plot of the phase solutions for the XX and YY polarization
+    .. image:: polalign_ph_polXX.png
+- ``polalign_ph_poldif``: matrix plot of the phase solutions from XX-YY
+    .. image:: polalign_ph_poldif.png
+- ``polalign_rotange``: matrix plot of the common rotation angle solutions
+    .. image:: polalign_rotangle.png
+- ``polalign_amp_pol??``: matrix plot of the amplitude solutions for the XX and YY polarization
+    .. image:: polalign_amp_polXX.png
+- ``polalign``: matrix plot of the derived polarization alignment between XX and YY
+    .. image:: polalign.png
+- ``polalign_ph-res_pol??``: matrix plot of the residual phase solutions for the XX and YY polarization after subtraction the derived polarization alignment
+- ``polalign_ph-res_poldif``: matrix plot of the residual phase solutions for XX-YY after subtraction of the derived polarization alignment
+
+The solutions are then stored in the final calibrator solution set ``cal_solutions`` (line 83) and applied to the interpolated data (``apply_PA``), together with the LOFAR beam correction (``apply_beam``)
+The calibration (``calib_cal``) is then repeated on the corrected and re-smoothed data (``smooth_corrected``).
+
+Calibration of the Faraday Rotation (FR)
+----------------------------------------
+The outcome of the re-calibration **after** correcting for the polarization alignment is again loaded into **LoSoTo** in order to derive corrections for Faraday Rotation.
+The following diagnostic plots are created:
+
+- ``fr_ph_pol??``: matrix plot of the phase solutions for the XX and YY polarization
+- ``fr_ph_poldif``: matrix plot of the phase solutions from XX-YY
+- ``fr_rotange``: matrix plot of the common rotation angle solutions
+- ``fr_amp_pol??``: matrix plot of the amplitude solutions for the XX and YY polarization
+- ``fr``: matrix plot of the derived differential Rotation Measure from Faraday Rotation
+    .. image:: fr.png
+- ``fr_ph-res_pol??``: matrix plot of the residual phase solutions for the XX and YY polarization after subtraction the derived Rotation Measure
+- ``fr_ph-res_poldif``: matrix plot of the residual phase solutions for XX-YY after subtraction of the derived Rotation Measure
+
+The solutions are then stored in the final calibrator solution set ``cal_solutions`` (line 83) and applied, together with the polarization alignment and the LOFAR beam correction, to the interpolated data (``apply_PA`` + ``apply_beam`` + ``apply_FR``).
+The calibration (``calib_cal``) is then repeated on the corrected and re-smoothed data (``smooth_corrected``).
+
+Calibration of the Bandpass (bandpass)
+----------------------------------------
+The outcome of the re-calibration **after** correcting for the polarization alignment and Faraday Rotation is loaded into **LoSoTo** in order to derive corrections for the bandpass. A robust flagging on the amplitude solutions as well as a Savitzky-Golay filter is applied in order to reject bad solutions and smooth the outcome. Frequency regimes up to a certain maximum width (``maxFlaggedWidth``) will be interpolated if flagged.
+The following diagnostic plots are created:
+
+- ``ampBFlag__??``: matrix plot of the amplitude solutions for the XX and YY polarization ``before`` flagging
+    .. image:: ampBFlag_polXX.png
+- ``ampAFlag__??``: matrix plot of the amplitude solutions for the XX and YY polarization ``after`` flagging
+    .. image:: ampAFlag_polXX.png
+- ``bandpass_pol??``: the derived bandpass of all stations in the XX and YY polarization
+- ``bandpass_time??``: matrix plot of the derived bandpass, where both polarizations are colorcoded
+    .. image:: bandpass.png
+- ``bandpass_time??_pol??``: plot of the derived bandpass of the XX and YY polarization, where all stations are colorcoded
+    .. image:: bandpass_polXX.png
+
+The solutions are then stored in the final calibrator solution set ``cal_solutions`` (line 83) and applied, together with the polarization alignment, the LOFAR beam correction and the Faraday Rotation corrections to the interpolated data in the correct order (``apply_PA`` + ``apply_bandpass`` + ``apply_beam`` + ``apply_FR`` ).
+The calibration (``calib_cal``) is then repeated on the corrected and re-smoothed data (``smooth_corrected``).
+
+Calibration of the instrumental and ionospheric delays (ion)
+------------------------------------------------------------
+The outcome of the re-calibration **after** correcting for the polarization alignment, the bandpass and the Faraday Rotation is loaded into **LoSoTo** in order to derive corrections for the instrumental and ionospheric delays. A robust flagging on the amplitude solutions is applied in order to reject bad solutions. These flags are applied to the phase solutions. These phase solutions should be mainly affected by instrumental (clock) and ionospheric (TEC) delays. This **LoSoTo** step will aim for seperating both effects (clock-TEC separation).
+The following diagnostic plots are created:
+
+- ``ion_ampBFlag__??``: matrix plot of the amplitude solutions for the XX and YY polarization **before** flagging
+- ``ion_ampAFlag__??``: matrix plot of the amplitude solutions for the XX and YY polarization **after** flagging
+- ``ion_ph_pol??``: matrix plot of the phase solutions for the XX and YY polarization
+- ``ion_ph_poldif``: matrix plot of the phase solutions from XX-YY
+- ``clock``: matrix plot of the derived (instrumental) clock offsets in seconds
+    .. image:: clock.png
+- ``tec``: matrix plot of the derived differential TEC in TECU
+    .. image:: tec.png
+- ``ion_ph-res_pol??``: matrix plot of the residual phase solutions for the XX and YY polarization after subtraction the derived instrumental and ionospheric delays
+- ``ion_ph-res_poldif``: matrix plot of the residual phase solutions for XX-YY after subtraction of the derived instrumental and ionospheric delays
+    .. image:: ion_ph-res_poldif.png
+
+The solutions are then stored in the final calibrator solution set ``cal_solutions`` (line 83).
+
+User-defined parameter configuration
+------------------------------------
+**Parameters you will need to adjust**
+
+*Information about the input data*
+
+- ``cal_input_path``: specify the directory where your calibrator data is stored (a full UNIX-compatible directory is required)
+- ``cal_input_pattern``: regular expression pattern of all your calibrator files (e.g. ``L72318*.MS``)
+
+*Location of the software*
+
+- ``prefactor_directory``: full path to your prefactor copy
+- ``losoto_directory``: full path to your local LoSoTo installation
+- ``aoflagger``: full path to your aoflagger executable
+
+
+
+**Parameters you may need to adjust**
+
+*Data selection and calibration options*
+
+- ``refant``:name of the station that will be used as a reference for the phase-plots
+- ``flag_baselines``: NDPPP-compatible pattern for baselines or stations to be flagged (may be an empty list, i.e.: ``[]`` )
+- ``process_baselines_cal``: performs A-Team-clipping/demixing and direction-independent phase-only self-calibration only on these baselines. Choose [CR]S*& if you want to process only cross-correlations and remove international stations.
+- ``filter_baselines``: selects only this set of baselines to be processed. Choose [CR]S*& if you want to process only cross-correlations and remove international stations.
+- ``do_smooth``: enable or disable baseline-based smoothing
+- ``rfistrategy``: strategy to be applied with the statistical flagger (AOFlagger), default: ``HBAdefault.rfis``
+- ``max_length``: amount of subbands to concatenate for full-bandwidth flagging (for an HBA calibrator, you can take all SBs if memory allows)
+- ``max2interpolate``: amount of channels in which interpolation should be performed for deriving the bandpass (default: 30)
+- ``interp_windowsize``: size of the window over which a value is interpolated. Should be odd. (default: 15)
+- ``ampRange``: range of median amplitudes accepted per station
+- ``skip_international``: skip fitting the bandpass for international stations (this avoids flagging them in many cases)
+- ``raw_data``: use autoweight, set to True in case you are using raw data (default: False)
+- ``propagatesolutions``: use already derived solutions as initial guess for the upcoming time slot
+- ``maxStddev``: maximum allowable standard deviation when outlier clipping is done. For phases, this should value should be in radians, for amplitudes in log(amp). If None (or negative), a value of 0.1 rad is used for phases and 0.01 for amplitudes
+
+A comprehensive explanation of the baseline selection syntax can be found `here`_.
+
+
+*Demixing options* (only used if demix step is added to the ``prep_cal_strategy`` variable)
+
+- ``demix_sources``: choose sources to demix (provided as list), e.g., ``[CasA,CygA]``
+- ``demix_target``: if given, the target source model (its patch in the SourceDB) is taken into account when solving (default: ``""``)
+- ``demix_freqstep``: number of channels to average when demixing (default: 16)
+- ``demix_timestep`` : number of time slots to average when demixing (default: 10)
+
+*Definitions for pipeline options*
+
+- ``default_flagging``: regular flagging steps after pre-processing by the observatory pipelines (default: ``flag,elev,flagamp``)
+- ``raw_flagging``: full set flagging steps (usually only necessary for raw data, default: ``flagedge,aoflag,{{ default_flagging }}``)
+- ``1st_order``: steps for first order clock-TEC separation (Do not change! Only ``cal_ion`` should be edited if needed, default: ``ct,plotTEC,residuals``)
+- ``3rd_order``: steps for third order clock-TEC separation (Do not change! Only ``cal_ion`` should be edited if needed, default: ``ct3,plotTEC3,residuals3``)
+- ``prep_cal_strategy``: steps to be performed for the  preparation of the calibrator data. Add ``,demix`` if you want to enable demixing. (default: ``{{ default_flagging }}``)
+- ``cal_ion``: choose whether you want to perform 1st or 3rd order ionospheric effects during clock-TEC separation (default: ``{{ 1st_order }}``)
+
+
+**Parameters for pipeline performance**
+
+- ``num_proc_per_node``: number of processes to use per step per node (default: ``input.output.max_per_node``, reads the parameter ``max_per_node`` from the ``pipeline.cfg``)
+- ``num_proc_per_node_limit``: number of processes to use per step per node for tasks with high I/O (DPPP or cp) or memory (e.g. calibration) (default: 4)
+- ``max_dppp_threads``: number of threads per process for NDPPP (default: 10)
+- ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent
+- ``min_length``: minimum amount of subbands to concatenate in frequency necessary to perform the wide-band flagging in the RAM. It data is too big aoflag will use indirect-read.
+- ``overhead``: Only use this fraction of the available memory for deriving the amount of data to be concatenated.
+- ``min_separation``: minimal accepted distance to an A-team source on the sky in degrees (will raise a WARNING)
+- ``error_tolerance``: defines whether pipeline run will continue if single bands fail (default: False)
+
+**Parameters you may want to adjust**
+
+*Main directories*
+
+- ``lofar_directory``: base directory of your **LOFAR** installation (default: $LOFARROOT)
+- ``job_directory``: directory of the prefactor outputs (usually the ``job_directory`` as defined in the ``pipeline.cfg``, default: ``input.output.job_directory``)
+
+*Script and plugin directories*
+
+- ``scripts``: location of the prefactor scripts (default: ``{{ prefactor_directory }}/scripts``)
+- ``pipeline.pluginpath``: location of the prefactor plugins: (default: ``{{ prefactor_directory }}/plugins``)
+
+*Skymodel directory*
+
+- ``calibrator_path_skymodel``: location of the prefactor skymodels (default: ``{{ prefactor_directory }}/skymodels``)
+
+*Result directories*
+
+- ``results_directory``: location of the prefactor results (default: ``{{ job_directory }}/results``)
+- ``inspection_directory``: location of the inspection plots (default: ``{{ results_directory }}/inspection``)
+- ``cal_values_directory``: directory of the calibration solutions (h5parm file, default: ``{{ results_directory }}/cal_values``)
+
+*Location of calibrator solutions*
+
+- ``cal_solutions``: location of the calibration solutions (h5parm file, default: ``{{ cal_values_directory }}/cal_solutions.h5``)
+
+*Averaging for the calibrator data*
+
+- ``avg_timeresolution``: final time resolution of the data in seconds after averaging (default: 4)
+- ``avg_freqresolution`` : final frequency resolution of the data after averaging (default: 48.82kHz, which translates to 4 channels per subband)
+- ``bandpass_freqresolution``: frequency resolution of the bandpass solution table (default: 195.3125kHz, which translates to 1 channel per subband)
+
+Parameters for **HBA** and **LBA** observations
+-----------------------------------------------
+====================== =============== =======================
+**parameter**          **HBA**         **LBA**
+---------------------- --------------- -----------------------
+``do_smooth``          False           True
+``rfistrategy``        HBAdefault.rifs LBAdefaultwideband.rfis
+``cal_ion``            {{ 1st_order }} {{ 3rd_order }}
+``tables2export``      clock           phaseOrig
+====================== =============== =======================
+
+In case of **LBA** observation you might also want to enable demixing in the ``prep_cal_strategy`` variable.
+
+.. _here: https://www.astron.nl/lofarwiki/doku.php?id=public:user_software:documentation:ndppp#description_of_baseline_selection_parameters

docs/source/changelog.rst

+.. _changelog:
+
+Changelog
+=========
+
+
+Version 4.0
+-----------
+
+    * First re-written version of the pipeline based on the `Common Workflow Language`_ (CWL)
+    * Additional diagnostics (UV-plots and summary file) and improved logging
+    * Full software is based on python3
+    * Easy-to-use with integrated Docker support
+
+Version 3.2 (last version based on the `genericpipeline`_ framework)
+-----------
+
+    * Several bugfixes and minor improvements
+    * Added high-resolution models for increased compatibility if using data including long baselines
+    * Option to add back missing stations for the use with long-baseline data
+    * Automated selection of the reference antenna
+
+Version 3.0
+-----------
+
+    * Major overhaul of calibrator and target pipelines to support LBA and HBA data
+    * Addition of production versions of calibrator, target, and image pipelines
+    * Documentation moved to http://www.astron.nl/citt/prefactor
+
+Version 2.0
+-----------
+
+    * applying Ionospheric RM corrections
+    * grouping of subbands by actual frequency
+    * speed and disk usage improvements by optimized usage of NDPPP
+    * (optional) wide-band cleaning in Initial-Subtract
+    * more diagnostic plots
+    * documentation moved to GitHib wiki: https://github.com/lofar-astron/prefactor/wiki
+    * automatic selection of calibrator sky model
+    * automatic generation of TGSS sky model
+
+Version 1.0
+-----------
+
+    * First release of the Pre-Facet-Calibration pipeline
+
+.. _Common Workflow Language: https://www.commonwl.org/
+.. _genericpipeline:  https://www.astron.nl/citt/genericpipeline/

docs/source/concatenate.rst

+.. _concatenate_pipeline:
+
+Concatenate pipeline
+=========================
+
+This pipeline concatenates single-subband target data produced by production
+runs and retrieved through the LTA. The resulting concatenated files are
+required for further processing with the initial-subtract pipeline.
+
+.. note::
+
+    If you processed the target data yourself instead of retrieving them from the
+    LTA, this pipeline is not required as the concatenated datasets are already
+    produced by the user version of the target pipeline.
+
+
+Prepare data
+------------
+This part of the pipeline prepares the target data in order to be concatenated. The steps are
+as follows:
+
+``createmap_target``
+    Generate a mapfile of all the target data (the single-subband datasets retrieved
+    from the LTA, with the direction-independent phase-only calibration applied).
+``combine_target_map``
+    Generate a mapfile with all files in a single entry. This mapfile is used as
+    input to the next step.
+``sortmap_target``
+    Compute frequency groupings
+``do_magic_maps``, ``do_sortmap_maps``
+    Convert the output of do_magic into usable mapfiles.
+
+
+Concatenation
+-------------
+Subbands are concatenated into "bands".
+
+``dpppconcat``
+    Concatenate the data, averaging to the specified frequency and time resolution.
+``make_results_mapfile``, ``move_results``
+    Move the concatenated files to the results directory.
+
+
+
+User-defined parameter configuration
+------------------------------------
+
+**Parameters you will need to adjust**
+
+*Information about the input data*
+
+``! target_input_path``
+    Directory where your single-subband target data are stored.
+``! target_input_pattern``
+    Regular expression pattern of all your target files.
+    .. note::
+
+        These files should have the direction-independent calibration applied to the DATA
+        column.
+
+*Location of the software*
+
+``! prefactor_directory``
+    Path to your prefactor copy
+
+**Parameters you may need to adjust**
+
+*Interpolation options*
+
+- ``interp_windowsize``: size of the window over which a value is interpolated. Should be odd. (default: 15)
+
+*Averaging options*
+
+- ``avg_timeresolution_concat``: final time resolution of the data in seconds after averaging and concatenation (default: 8)
+- ``avg_freqresolution_concat``: final frequency resolution of the data after averaging and concatenation (default: 97.64kHz, which translates to 2 channels per subband)
+
+*Concatenation options*
+
+- ``num_SBs_per_group``: make concatenated measurement-sets with that many subbands (default: 10)
+- ``reference_stationSB``: station-subband number to use as reference for grouping, (default: ``None`` -> use lowest frequency input data as reference)

docs/source/conf.py

+# -*- coding: utf-8 -*-
+#
+# Configuration file for the Sphinx documentation builder.
+#
+# This file does only contain a selection of the most common options. For a
+# full list see the documentation:
+# http://www.sphinx-doc.org/en/master/config
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'Prefactor'
+copyright = '2021, Alexander Drabent and David Rafferty'
+author = 'Alexander Drabent and David Rafferty'
+
+# The short X.Y version
+version = '4.0'
+# The full version, including alpha/beta/rc tags
+release = '4.0'
+
+
+# -- General configuration ---------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#
+# needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.autodoc',
+    'sphinx.ext.mathjax'
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+# source_suffix = ['.rst', '.md']
+source_suffix = '.rst'
+
+# The master toctree document.
+master_doc = 'index'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path .
+exclude_patterns = []
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'classic'
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#
+# html_theme_options = {}
+html_logo = 'bandpass.png'
+html_favicon = 'favicon.ico'
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# Custom sidebar templates, must be a dictionary that maps document names
+# to template names.
+#
+# The default sidebars (for documents that don't match any pattern) are
+# defined by theme itself.  Builtin themes are using these templates by
+# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
+# 'searchbox.html']``.
+#
+# html_sidebars = {}
+
+
+# -- Options for HTMLHelp output ---------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'Prefactordoc'
+
+
+# -- Options for LaTeX output ------------------------------------------------
+
+latex_elements = {
+    # The paper size ('letterpaper' or 'a4paper').
+    #
+    # 'papersize': 'letterpaper',
+
+    # The font size ('10pt', '11pt' or '12pt').
+    #
+    # 'pointsize': '10pt',
+
+    # Additional stuff for the LaTeX preamble.
+    #
+    # 'preamble': '',
+
+    # Latex figure (float) alignment
+    #
+    # 'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (master_doc, 'Prefactor.tex', 'Prefactor Documentation',
+     'Alexander Drabent and David Rafferty', 'manual'),
+]
+
+
+# -- Options for manual page output ------------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, 'prefactor', 'Prefactor Documentation',
+     [author], 1)
+]
+
+
+# -- Options for Texinfo output ----------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (master_doc, 'Prefactor', 'Prefactor Documentation',
+     author, 'Prefactor', 'One line description of project.',
+     'Miscellaneous'),
+]
+