initial commit corresponds to version 22.0 release

.gitignore

+build/*

Makefile

+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SPHINXPROJ    = LOFARImagingCookbook
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
\ No newline at end of file

addTracking.sh

+#!/bin/bash
+for file in ./build/html/*.html; 
+do
+   x="$(tail -n 2 $file | wc -c)"
+   truncate --size=-$x $file
+   cat ./statcounter >> $file
+   echo -e '\n</body>\n</html>' >> $file
+done

source/acknowledgements.rst

+Acknowledgements
+================
+
+Many commissioners and software developers have contributed to this cookbook. As the routines, the hardware, and the software needed for LOFAR develop very quickly, what is reported in this manual might be sometimes incorrect. We try to keep it as up to date as possible, but we surely need your feedback to improve its quality. Please send comments and suggestions of improvements to `Sarrvesh Sridhar <mailto:sarrvesh@astron.nl>`_.
+
+The contact points for the various versions of the LOFAR Imaging Cookbook are listed below.
+
++ **Version 1.0**: Louise Ker
+
++ **Version 1.1**: Louise Ker
+
++ **Version 1.2**: Annalisa Bonafede
+
++ **Version 2.0**: Emanuela Orru' & Fabien Batejat
+
++ **Version 2.1**: Roberto Francesco Pizzo
+
++ **Version 2.2**: Roberto Francesco Pizzo
+
++ **Version 2.3**: Roberto Francesco Pizzo
+
++ **Version 3.0**: Roberto Francesco Pizzo 
+
++ **Version 4.0**: Roberto Francesco Pizzo
+
++ **Version 5.0**: Roberto Francesco Pizzo 
+
++ **Version 5.1**: Roberto Francesco Pizzo 
+
++ **Version 6.0**: Roberto Francesco Pizzo, Laura Birzan, Ger van Diepen, Sven Duscha,  John McKean, Andr\'e Offringa, David Rafferty, Cyril Tasse,  Bas van der Tol,  Reinout van Weeren, and Joris van Zwieten
+
++ **Version 7.0**: Roberto Francesco Pizzo, Laura Birzan, Ger van Diepen, Sven Duscha,  John McKean, Andr\'e Offringa, David Rafferty, Cyril Tasse,  Bas van der Tol,  Reinout van Weeren, Sarod Yatawatta, and Joris van Zwieten 
+
++ **Version 8.0**: Roberto Francesco Pizzo, Laura Birzan, Ger van Diepen, Sven Duscha,  John McKean, Andr\'e Offringa, David Rafferty, Aleksandar Shulevski, Cyril Tasse,  Bas van der Tol,  Reinout van Weeren, Sarod Yatawatta , and Joris van Zwieten
+
++ **Version 9.0**: Roberto Francesco Pizzo, Laura Birzan, Ger van Diepen, Sven Duscha,  Geaorge Heald, John McKean, Andr\'e Offringa, David Rafferty, Cyril Tasse,  Bas van der Tol,  Reinout van Weeren, Sarod Yatawatta , and Joris van Zwieten
+
++ **Version 10.0 - 12.0**: Roberto Francesco Pizzo, Laura Birzan, Ger van Diepen, Sven Duscha,  George Heald, John McKean, Andr\'e Offringa, Emanuela Orr\'u, David Rafferty, Cyril Tasse,  Bas van der Tol,  Reinout van Weeren, Sarod Yatawatta , and Joris van Zwieten
+
++ **Version 13.0**: Roberto Francesco Pizzo, Ger van Diepen, Tammo Jan Dijkema, John McKean, Andr\'e Offringa, Emanuela Orr\'u, David Rafferty, Cyril Tasse,  Bas van der Tol,  Valentina Vacca, Reinout van Weeren, Wendy Williams, Sarod Yatawatta , and Joris van Zwieten
+
++ **Version 14.0**: Roberto Francesco Pizzo, Ger van Diepen, Tammo Jan Dijkema, G. Heald, Francesco de Gasperin, John McKean, Maaijke Mevius, Andr\'e Offringa, Emanuela Orr\'u, David Rafferty, Cyril Tasse,  Bas van der Tol,  Valentina Vacca, Nicolas Vilchez, Reinout van Weeren, Wendy Williams and Sarod Yatawatta, 
+
++ **Version 15.0**: Roberto~F.~Pizzo, Ger van Diepen, Tammo Jan Dijkema, G. Heald, Francesco de Gasperin, M. Iacobelli, John McKean, Maaijke Mevius, Andr\'e Offringa, Emanuela Orr\'u, David Rafferty, Cyril Tasse,  Bas van der Tol,  Valentina Vacca, Nicolas Vilchez, Reinout van Weeren, Wendy Williams and Sarod Yatawatta
+
++ **Version 16.0**: Roberto~F.~Pizzo, Ger van Diepen, Tammo Jan Dijkema, G. Heald, Francesco de Gasperin, M. Iacobelli, John McKean, Maaijke Mevius, Andr\'e Offringa, Emanuela Orr\'u, David Rafferty, Cyril Tasse,  Bas van der Tol,  Valentina Vacca, Nicolas Vilchez, Reinout van Weeren, Wendy Williams and Sarod Yatawatta
+
++ **Version 17.0**: Roberto~F.~Pizzo, Ger van Diepen, Tammo Jan Dijkema, G. Heald, Francesco de Gasperin, M. Iacobelli, John McKean, Maaijke Mevius, Andr\'e Offringa, Emanuela Orr\'u, David Rafferty, Cyril Tasse,  Bas van der Tol,  Valentina Vacca, Nicolas Vilchez, Reinout van Weeren, Wendy Williams and Sarod Yatawatta
+
++ **Version 18.0**: Aleksandar~Shulevski, Roberto~F.~Pizzo, Ger van Diepen, Tammo Jan Dijkema, G. Heald, Francesco de Gasperin, M. Iacobelli, John McKean, Maaijke Mevius, Andr\'e Offringa, Emanuela Orr\'u, David Rafferty, Cyril Tasse,  Bas van der Tol, T. J. Dijkema,  Valentina Vacca, Nicolas Vilchez, Reinout van Weeren, Wendy Williams and Sarod Yatawatta
+
++ **Version 19.0 - 22.0**: Aleksandar~Shulevski, Roberto~F.~Pizzo, Ger van Diepen, Tammo Jan Dijkema, G. Heald, Francesco de Gasperin, M. Iacobelli, John McKean, Maaijke Mevius, Andr\'e Offringa, Emanuela Orr\'u, David Rafferty, Cyril Tasse,  Bas van der Tol, T. J. Dijkema,  Valentina Vacca, Nicolas Vilchez, Reinout van Weeren, Wendy Williams and Sarod Yatawatta

source/aoflagger.rst

+AOFlagger [#f1]_
+================
+
+The frequencies covered by LOFAR are considerably affected by RFI, both in the low and the high band. Without proper flagging of the RFI, the RFI affects the image quality or can make calibration fail. This chapter describes how to analyze and flag LOFAR observations using the AOFlagger.
+
+In most cases, one would normally use the AOFlagger step in NDPPP with the default settings to flag the data. In this chapter we look at what the flagger does and how to alter its behaviour. 
+
+--------
+Tutorial
+--------
+
+The tutorial data can be downloaded from the LOFAR LTA. Please refer to the chapter on `practical examples <./tutorial.html>`_ for details.
+
+The AOFlagger program comes with a tool called **rfigui** that is part of the **lofar** module on the CEP3 cluster. If you haven't done so, load the module::
+
+    module load lofar
+
+We will use the rfigui to display the measurement set. The rfigui will not write to the measurement set, so you do not need to copy the measurement set yet. Use the following command to open the measurement set in rfigui::
+
+    rfigui L74759_SAP000_SB001_uv.MS
+
+^^^^^^^^^^^^^^^^^^
+Browsing baselines
+^^^^^^^^^^^^^^^^^^
+
+A window should pop-up which can be used to select how to open the measurement set. Note that the visualized column can be selected here, which is useful to analyze data in a different column (e.g. CORRECTED_DATA). Since this is raw data, it will only have a DATA ("observed") column, so leave the settings as they are and press "Open".
+
+Another window will pop-up which allows you to select two antennas, a band and a field. Since LOFAR measurement sets never have multiple bands or fields, they will always have only one option. With the two antenna selection boxes, you can select which cross-correlation to visualize. Select antenna CS002HBA0 in both selection frames and press "Load". It will take some while before the visibilities are loaded.
+
+You should now see a dynamic spectrum (time on the x-axis, frequency on the y-axis) of auto-correlation CS002HBA0 x CS002HBA0. Note several things:
+
++ When you move the mouse over the dynamic spectrum, the status bar will provide some information of the visibilities at the location of the mouse.
++ At the bottom of the plot is a purple line. The rfigui uses purple to indicate visibilities that are flagged. In this case, it is flagged because in raw LOFAR data, channel 0 contains invalid data due to how the poly-phase filter works. The purple overlay can be turned off with the "Original flags" toggle button in the toolbox.
++ Note the sharp patterns that recur in a horizontal way, such as at 119.05 and 119.18 MHz. These are common, and are caused by normal transmitters. Use the mouse to find which features I mean.
++ Around 13.30 h, vertical features are visible. These can be caused by broadband RFI such as lightning, or by a malfunctioning instrument.
++ Finally, there are some big wavy features visible. These are intermodulations in the receivers, and are only present in auto-correlations (and sometimes in the cross-baseline of a dual split station, e.g. CS002HBA0 x CS002HBA1).
+
+Auto-correlations are much more sensitive to RFI. Because the auto-correlations are normally not used in imaging, in fact this RFI situation is not representative. Let's go to a cross-correlation: press the "Forward" button. This will load the correlations for the baseline of this antenna by the next antenna, in this case CS002HBA0 x CS002HBA1. Notice that the RFI situation is much better, and the dominating features are the consistent transmitters at 119.05 MHz and 119.18 MHz. Press "Forward" a few more times and notice the changes.
+
+Let's go to a remote station: go to the "Go" (newer version: "Browse") menu and select "Go to...". Load baseline CS004HBA0 x RS106HBA. Note the smooth vertical features in the background. These are fringes caused by observed sources. Therefore, a flagging strategy should not flag these.
+
+^^^^^^^^^^^^^^^^^^^
+Plotting & flagging
+^^^^^^^^^^^^^^^^^^^
+
+Power spectra and time-power plots can be helpful to analyze the magnitude of RFI. In menu "Plot", select "Power spectrum". A pop-up window appears with the power over frequency over the selected region in the main window. The RFI is clearly visible as spikes in the spectrum. Note also that the bandpass shape is curving down at 119.23 MHz. One normally does not want to include these edge channels because of these features, and they are therefore cut-off in NDPPP (see the NDPPP tutorial).
+
+Now select "Plot time scatter" in the "Plot" menu, and notice how the RFI looks like in this plot. In this plot, different colours show the different polarizations. Also try the "mean spectrum" and "power vs time" plots, and notice how they differ. 
+
+Normally, the time-frequency plot shows the Stokes I power of the visibilities. However, internally AOFlagger has loaded complex correlations for all four polarizations. Select "Data" :math:`\rightarrow` "Keep real data". Fringes are more pronounced in the real data. When you press one of the "Keep..." options, the rfigui will no longer keep the other parts in memory. This can be confusing: for example, try selecting "Data" :math:`\rightarrow` "Keep imaginary data" and understand the error message. To see the imaginary data, first reload the baseline with "Go" :math:`\rightarrow` "Go to..." (new version: refresh button). Try analyzing the different parts of this correlation by using the other "Data" :math:`\rightarrow` "Keep..." buttons and reloading the baseline when necessary.
+
+Before continuing, close the plot window and reload the baseline.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Testing a flagging strategy
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Select "Actions" :math:`\rightarrow` "Execute strategy". A progress window will pop-up, which you can close once the flagging is done. Notice that yellow flags have appeared in the time-frequency plot. The default flagging strategy has indeed flagged all visible RFI. Additionally, it has flagged a bit of the edge channels (e.g. in the top-right), because of the curvature in the pass-band at the edges. This is normally not a problem, since these channels will be cut off with NDPPP.
+
+Try toggling the yellow flags off and on with the "Alternative flags" button on the toolbar. Before continuing, make sure that the yellow flags are shown. Plot the power spectrum and time scatter plots as before. The power spectrum will now have two lines: the original one, and one that shows the spectrum after the flagged visibilities have been taken off. The time scatter plot will only show unflagged samples. Indeed, the RFI samples have been removed, and a clear difference between XX/YY and XY/YX is visible. If you hide the yellow flags and replot the scatter plot, the flagged samples will be shown again.
+
+Clear the flags by reloading the baseline.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Editing the flagging strategy
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Press "Action" :math:`\rightarrow` "Edit strategy". A window will pop-up that shows a tree that represents the flagging script. Every row is an action, and certain actions can have sub-actions [#f2]_.
+
+Notice that there are two "SumThreshold" actions: one is in the "iterate 2 times" loop, while the other is below the loop. When you press one of the SumThreshold actions, a settings box will appear. Set the "Base sensitivity" to 0.7 and press "Apply" for both SumThreshold actions. Now, rerun the strategy with "Actions" :math:`\rightarrow` "Execute strategy" and notice the difference.
+
+Try playing with the settings below, and rerun the flagger each time. Do not forget to press "Apply" after changing the settings of an action. If you make modifications you don't like, you can press the "Default" button in the edit startegy window to reset the strategy.
+
++ The strategy normally slowly converges by iterating a couple of times. Only in the last SumThreshold action, the maximum sensitivity is used. The "Iterate 2 times" action performs this slow convergence. In the "Iterate 2 times" action, try changing the number of iterations to 1 and see if this is enough for this baseline.
++ Normally, each polarization is searched for RFI. Change the "For each polarization" action so that only Stokes Q is searched for RFI. 
++ In the "Statistical flagging" action, play with the "Minimum time ratio" and "Minimum frequency ratio". For example, is a "Minimum time ratio" of 60\% a good setting?
++ Try setting the threshold of the "Time selection" action to 2.5.
+
+This subband is a rather good subbands. While for LOFAR many subbands are like this, there are also subbands which are much worse. Typically you want to try the strategy on some good and bad subband, as well as on some small on some long baselines. You can chose to flag different subbands with a different strategy, but in general it is easier to have one generic strategy. It is also generally better to flag slightly too much than too little. Finally, it is not necessary to consider the flagging speed.
+
+Once you have made a modified strategy, you can save it with the "Save" button in the "Edit strategy window". When saving a strategy, give the filename an extension of ".rfis". These strategies can be used by the AOFlagger step inside NDPPP, to flag and average the data at the same time, as well as that they can be used by the "**aoflagger**" command line program. It is normally faster to do this with NDPPP.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Running the **aoflagger** command line program
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+We are now going to flag the measurement set without running NDPPP. To do this, we need write access to the measurement set. Therefore, copy the measurement set mentioned above to your scratch directory. Measurement sets that are created by the LOFAR correlator have a special "raw" format. To make the flag table writeable by aoflagger, it is necessary to store the flags with a different format. This can be done with the following command::
+
+    > makeFLAGwritable L74759_SAP000_SB001_uv.MS
+    Successful read/write open of default-locked table L74759_SAP000_SB001_uv.MS:
+      23 columns, 5337090 rows
+    Created new FLAG column; copying old values ...
+    FLAG column now stored with SSM to make it writable
+
+The measurement set can now be flagged with the aoflagger command. To get a list of commands, run the program without parameters::
+
+    > aoflagger
+    AOFlagger 2.7.1 (2015-07-01) command line application
+    This program will execute ...
+
+Two settings need to be changed: i) for large raw sets, it is best to use the indirect reading mode; ii) since we have made a modified strategy, we want to specify our new strategy. If you are using aoflagger 2.7, you can immediately specify the .rfis file that you have created in the gui. At the point of writing, you will need to use a file which has been prepared by me, because there is an issue with different versions of the gui and the command line aoflagger.
+
+Combined all together, run aoflagger like this::
+
+    > aoflagger -indirect-read -strategy ~offringa/tutorials/aoflagger-tutorial.rfis L74759_SAP000_SB001_uv.MS
+    AOFlagger 2.7.1 (2015-07-01) command line application
+    ...
+    0% : +-+-+-+-+-+-Initializing...
+    ...
+
+Please note that the current working directory will be used as a temporary storage location, and a lot of temporary data will be written there. Once the flagger is done, you can open the measurement set in the rfigui, and the found flags will be shown with purple.
+
+-------------
+Documentation
+-------------
+
+The manual for AOFlagger can be found on the site `http://aoflagger.sourceforge.net/ <http://aoflagger.sourceforge.net/>`_. The properties and performance of the AOFlagger have been described in the following papers:
+
++ `Post-correlation radio frequency interference classification methods <http://arxiv.org/abs/1002.1957>`_.
++ `A morphological algorithm for improving radio-frequency interference detection <http://arxiv.org/abs/1201.3364>`_.
+
+
+
+
+
+
+.. rubric:: Footnotes
+
+.. [#f1] This author of this chapter is `Andre Offringa <mailto:offringa@astron.nl>`_.
+.. [#f2] A description of the steps in the pipeline is given in `this paper <http://pos.sissa.it/archive/conferences/107/036/RFI2010_036.pdf>`_.

source/awimager.rst

+The AW Imager [#f1]_
+====================
+
+------------
+Introduction
+------------
+
+In this section we will describe the necessary steps needed to perform
+successful imaging of LOFAR data using the AWimager [#f2]_. 
+
+Note that the AWimager is still in the development phase, therefore this documentation is very dynamic and it is currently meant to provide the basic instructions on how to use the code.
+
+----------
+Background
+----------
+
+The AWimager is specially adapted to image wide fields of view, and imaging data produced by non-coplanar arrays, where the W term in the measured visibilities is not negligible. Furthermore, AWimager corrects for direction dependent effects (LOFAR beam and ionosphere) varying in time and frequency. The used algorithm is A-projection.
+
+The algorithm is implemented using some CASA libraries.
+
+------
+Usage
+------
+
+To run AWimager, you first need to setup your environment using::
+
+   module load lofar
+
+Before running AWimager, it is necessary to calibrate the dataset and correct
+the visibilities towards the phase center of the observation. This can now be
+done by not specifying any direction in the **correct** step of BBS. ::
+
+   Step.correct.Model.Sources = []
+
+AWimager can run in a parallel fashion. The number of processing cores
+(**n**) to be used during imaging can be specified::
+
+   export OMP_NUM_THREADS=n
+
+If not specified, all cores will be used. AWimager is quite memory hungry, so the number of cores should be limited in case it fails due a **'bad alloc'** error.
+
+------------
+Output files
+------------
+
+AWimager creates several image output files. Note that in the following list **<image>** is the image name given using the **image** parameter. 
+
++ **<image>.model** is the uncorrected dirty image.
++ **<image>.model.corr** is the dirty image corrected for the average primary beam.
++ **<image>.restored** and **<image>.restored.corr** are the restored images.
++ **<image>.residual** and **<image>.residual.corr** are the residual images.
++ **<image>.psf** is the point spread function.
++ **<image>0.avgpb** is the average primary beam.
+
+Furthermore, a few other files might be created for AWimager's internal use.
+
+----------------
+Input Parameters
+----------------
+
+An extensive list of the parameters that can be used by the AWimager
+can be obtained by typing::
+
+   awimager -h
+   
+Eventually, to run the imager, you can type::
+
+   awimager ms=test.MS image=test.img weight=natural wprojplanes=64 npix=512 cellsize=60arcsec data=CORRECTED_DATA padding=1. niter=2000 timewindow=300 stokes=IQUV threshold=0.1Jy operation=csclean
+
+It is also possible to specify these parameters in a parset and run it like::
+
+   awimager parsetname
+
+Many parameters can be set for the AWimager. Several of them are currently being tested by the commissioners. The most important parameters are listed below.
+
+^^^^^^^^^^^^^^
+Data selection
+^^^^^^^^^^^^^^
+
+These parameters specify the input data.
+
++ **ms** - The name of the input MeasurementSet.
++ **data** - The name of the data column to use in the MeasurementSet. The default is DATA.
++ **antenna** - Baseline selection following the CASA baseline selection syntax
++ **wmax** - Ignore baselines whose w-value exceeds wmax (in meters).
++ **uvdist** - Ignore baselines whose length (in wavelengths) exceed uvdist.
++ **select** - Only use data matching this TaQL selection string. For example, **sumsqr(UVW[:2])<1e8** selects baselines with length <10km.
+
+^^^^^^^^^^^^^^^^
+Image properties
+^^^^^^^^^^^^^^^^
+
+These parameters define the properties of the output image.
+
++ **image** - The name of the image.
++ **npix** - The number of pixels in the RA and DEC direction
++ **cellsize** - The size of each pixel. An unit can be given at the end. E.g.  **30arcsec**
++ **padding** - The padding factor to use when imaging. It can be used to get rid of effects at the edges of the image. If, say, 1.5 is given, the number of pixels used internally is 50% more.
++ **stokes** - The Stokes parameters for which an image is made. If A-projection is used, it must be IQUV.
+
+^^^^^^^^^
+Weighting
+^^^^^^^^^
+
+These parameters select the weighting scheme to be used.
+
++ **weight** - Weighting scheme (uniform, superuniform, natural, briggs (robust), briggsabs, or radial)
++ **robust** - Robust weighting parameter.
+
+^^^^^^^^^
+Operation
+^^^^^^^^^
+
+This parameter selects the operation to be performed by awimager.
+
++ **operation** - The operation to be performed by the AWImager.
+
+   + csclean - make an image and clean it (using Cotton-Schwab).
+   + multiscale - use multiscale cleaning.
+   + image - make dirty image only.
+   + predict - fill the data column in the MeasurementSet by predicting the data from the image.
+   + empty - make an empty image. This can be useful if only image coordinate info is needed.
+   
+^^^^^^^^^^^^^
+Deconvolution
+^^^^^^^^^^^^^
+
+These parameters control the deconvolution algorithm. Only those parameters that are applicable to the selected operation will be used.
+
++ **niter** - The number of clean iterations to be done. The default is 1000.
++ **gain** - The loop gain for cleaning. The default is 0.1.
++ **threshold** - The flux level at which to stop cleaning. The default is 0Jy.
++ **uservector** - Comma separated list of scales (in pixels) to be used by multiscale deconvolution
+   
+^^^^^^^^
+Gridding
+^^^^^^^^
+
+These parameters control the AW-projection algorithm.
+
++ **wprojplanes** - The number of W projection planes to use.
++ **maxsupport** - The maximum of W convolution functions. The default is 1024.
++ **oversample** - The oversampling to use for the convolution functions. The default is 8.
++ **timewindow** - The width of the time window (in sec) where the AW-term is assumed to be constant. Default is 300 sec. The wider the window, the faster the imager will be.
++ **splitbeam** - Evaluate station beam and element beam separately? The default is true. AWimager will work much faster if the correction for the station and element beam can be applied separately. This should only be done if the element beam is the same for all stations used.
+
+For more details, the user can refer to the Busy Wednesday `commissioning reports <http://www.lofar.org/operations/doku.php?id=commissioning:busy_wednesdays>`_ (specifically those from September 28 and October 26 2011).
+
+.. rubric:: Footnotes
+
+.. [#f1] This chapter is maintained by `Bas van der Tol <mailto:tol@astron.nl>`_.
+.. [#f2] Cyril Tasse, Bas van der Tol, Joris van Zwieten, Ger van Diepen, Sanjay Bhatnagar 2013, **Applying full polarization A-Projection to very wide field of view instruments: An imager for LOFAR}. A&A, 553, A105**.

source/changelog.rst

+Changelog
+=========
+
+The latest released version of the cookbook is available `online <http://www.astron.nl/radio-observatory/lofar/lofar-imaging-cookbook>`_.
+
+This link is advertised on the LOFAR wiki. The very latest (development) version of the cookbook can also be found on the `USG repository <http://usg.lofar.org/svn/documents/trunk/Tutorials/Imaging/>`_.
+
+The LOFAR software is continuously improving and, as a consequence, several procedures (and the cookbook itself) continuously change. In the following, we report an overview of the (recent) changes applied to the manual.
+
+--------------------------
+Overview of recent changes
+--------------------------
+
+^^^^^^^^^^^^^^^^^^^^^^^^
+Version 22 (2018/01/10)
+^^^^^^^^^^^^^^^^^^^^^^^^
+
++ Converted source from LATEX to reStructuredText.
++ Removed now-obsolete appendix on "Automated self-calibration" and "GNU Screen"
++ Replaced all "use" to "module load".
++ Replaced DSA with RSA in generating SSH keys.
++ Added a new section on generating skymodels from TGSS-ADR.
++ Added a section on drawer to Data Inspection.

source/conf.py

+# -*- coding: utf-8 -*-
+#
+# LOFAR Imaging Cookbook documentation build configuration file, created by
+# sphinx-quickstart on Tue Dec 12 09:37:16 2017.
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#
+# needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = ['sphinx.ext.mathjax',
+    'sphinx.ext.githubpages', 'sphinx.ext.intersphinx']
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+# source_suffix = ['.rst', '.md']
+source_suffix = '.rst'
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = u'LOFAR Imaging Cookbook'
+copyright = u'2017, ASTRON'
+author = u'Edited by Sarrvesh Sridhar'
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The short X.Y version.
+version = u'22.0'
+# The full version, including alpha/beta/rc tags.
+release = u'22.0'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This patterns also effect to html_static_path and html_extra_path
+exclude_patterns = []
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = False
+
+# Enable numbering for figures and images
+numfig = True
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+#html_theme = 'alabaster'
+html_theme = 'sphinx_rtd_theme'
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#
+# html_theme_options = {}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# Custom sidebar templates, must be a dictionary that maps document names
+# to template names.
+#
+# This is required for the alabaster theme
+# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars
+#html_sidebars = {
+#    '**': [
+#        'relations.html',  # needs 'show_related': True theme option to display
+#        'searchbox.html',
+#    ]
+#}
+
+# Enable last updated date
+html_last_updated_fmt = '%b %d, %Y'
+
+# Remove copyright information
+html_show_copyright = False
+
+# -- Options for HTMLHelp output ------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'LOFARImagingCookbookdoc'
+
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+    # The paper size ('letterpaper' or 'a4paper').
+    #
+    # 'papersize': 'letterpaper',
+
+    # The font size ('10pt', '11pt' or '12pt').
+    #
+    # 'pointsize': '10pt',
+
+    # Additional stuff for the LaTeX preamble.
+    #
+    # 'preamble': '',
+
+    # Latex figure (float) alignment
+    #
+    # 'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (master_doc, 'LOFARImagingCookbook.tex', u'LOFAR Imaging Cookbook Documentation',
+     u'Edited by Sarrvesh Sridhar', 'manual'),
+]
+
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, 'lofarimagingcookbook', u'LOFAR Imaging Cookbook Documentation',
+     [author], 1)
+]
+
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (master_doc, 'LOFARImagingCookbook', u'LOFAR Imaging Cookbook Documentation',
+     author, 'LOFARImagingCookbook', 'One line description of project.',
+     'Miscellaneous'),
+]
+
+
+

source/dpppcalibrate.rst

+Gain calibration with DPPP [#f1]_
+=================================
+
+The most common gain calibration procedures can be `performed <http://www.lofar.org/operations/doku.php?id=public:user_software:documentation:ndppp#gaincal>`_ with DPPP. Formerly `BBS <./bbs.html>`_ was the standard calibration tool, but for the supported calibration scenarios DPPP is at least 10 times faster. For calibration scenarios that are not (yet) supported in DPPP, it may be necessary to resort to BBS, see the chapter on `BBS <./bbs.html>`_.
+
+The calibration problem that DPPP can solve is the following: find a set of Jones matrices :math:`{\mathbf{G}_p}` (one for every station :math:`p`) which corrects the measured visibilities :math:`\mathbf{V}_{pq}` to closely resemble the model visibilities :math:`\mathbf{M}_{pq}` (for all baselines :math:`pq`), i.e. minimize
+
+.. math::
+
+    \| \mathbf{V}_{\!pq}-\mathbf{G}_{p}\mathbf{M}_{pq}\mathbf{G}_{q}^H \|
+    
+The matrices :math:`\mathbf{G}` will be referred to as **gain matrices** although they correct for more than just electrical gains.
+
+--------------------
+Calibration variants
+--------------------
+
+There are various options to restrict the shape of :math:`\mathbf{G}`. The main difference is whether to solve for the amplitude of the solutions or only for the phase. Also, the number of free parameters can be restricted.
+
+.. csv-table:: 
+   :header: "Shape", "Calibration type", "Free parameters"
+   
+   ":math:`\mathbf{G}_{{p}} = \left( \begin{array}{cc} A_{{xx}}^{{(p)}}e^{\phi_{{xx}}^{{(p)}}} & A_{{xy}}^{{(p)}}e^{\phi_{{xy}}^{{(p)}}} \\ A_{{yx}}^{{(p)}}e^{\phi_{{yx}}^{{(p)}}} & A_{{yy}}^{{(p)}}e^{\phi_{{yy}}^{{(p)}}} \end{array} \right)`", "fulljones", "8"
+   ":math:`\mathbf{G}_{{p}} = \left( \begin{array}{cc} A_{{xx}}^{{(p)}}e^{\phi_{{xx}}^{{(p)}}} & 0 \\ 0 & A_{{yy}}^{{(p)}}e^{\phi_{{yy}}^{{(p)}}} \end{array} \right)`", "diagonal", "4"
+   ":math:`\mathbf{G}_{{p}} = \left( \begin{array}{cc} e^{\phi_{{xx}}^{{(p)}}} & 0 \\ 0 & e^{\phi_{{yy}}^{{(p)}}} \end{array} \right)`", "phaseonly", "2"
+   ":math:`\mathbf{G}_{{p}} = \left( \begin{array}{cc} \,e^{\phi^{{(p)}}} & 0 \\ 0 & \,e^{\phi^{{(p)}}} \end{array} \right)`", "scalarphase", "1"
+
+
+-----------------
+Making a skymodel
+-----------------
+
+To perform a calibration, you need a sky model (see section :ref:`sourcecatalog` for details). You can get one from the catalogs in **gsm.py** (see section on :ref:`gsm` for details), or make your own. To make the sky model (in text format) readable by DPPP, it needs to be converted from plain text to a **sourcedb**. That is done with the program **makesourcedb**. Usually the sourcedb is called 'sky' and copied into the data set you're reducing. If you put it elsewhere, it is customary to give it the extension **.sourcedb**. ::
+
+    makesourcedb in=my.skymodel out=L123.MS/sky format='<'
+    
+The part **format='<'** is necessary to convince makesourcedb that the format is given by the first line in the file. You can use the program **showsourcedb** to verify the output sourcedb.
+
+It is also possible to calibrate on model visibilities - in this case no sky model is necessary. See the online documentation of DPPP, the parameter to look for is **usemodelcolumn**.
+
+.. _calibrationdppp:
+
+-----------
+Calibration
+-----------
+
+To perform a phase only calibration, the following parset can be given to DPPP. ::
+
+    msin=L123.MS
+    msout=
+    steps=[gaincal]
+    gaincal.sourcedb=L123.MS/sky
+    gaincal.parmdb=L123.MS/instrument
+    gaincal.caltype=phaseonly
+    gaincal.solint=2
+    gaincal.nchan=0
+
+The part **solint=2** specifies that we only want one solution for every two time intervals. This can improve the signal to noise ratio - but one should have a physical argument that tells that the solutions do not change within the solution interval. The part **nchan=0** tells that one solution is computed that is assumed constant over the entire band. Setting it to e.g. **nchan=1** will compute a separate solution for each channel (again, at the cost of signal to noise).
+
+The parset above performs a phase only calibration, and stores the calibration result in the **parmdb** (parameter database) **L123.MS/instrument**. This file will be created if it is not there yet. If it does exist, the solution in it will be overwritten. Note that it is a convention to save the calibration tables in a file (casa table) called **instrument** in the data set being reduced. If you store it outside the data set, the convention is to give it the extension **.parmdb**.
+
+The solution table can (and should!) be inspected with **parmdbplot.py**, see Section~\ref{bbs:inspect-solutions}. Note that this calibration step does not yet change the visibilities. To perform complex operations on the solutions, like smoothing them, LoSoTo can be used, see Chapter~\ref{losoto}.
+
+------------------
+Applying solutions
+------------------
+
+To apply the calibration solutions in DPPP, the step **applycal** can be used. The following parset applies the solutions that were obtained by gaincal. Note that currently. the solution table is only written at the very end of DPPP, so that it is not possible to solve and apply the solutions in the same run of DPPP. ::
+
+    msin=L123.MS
+    msout=.
+    msout.datacolumn=CORRECTED_DATA
+    steps=[applycal]
+    applycal.parmdb=L123.MS/instrument
+
+It is a convention to write the output to the column **CORRECTED_DATA**, to avoid changing the original data column **DATA**.
+
+-----------------------------------
+Transferring solutions and the beam
+-----------------------------------
+
+When transferring solutions from a calibrator to a target, the sensitivity of the beam across the sky needs to be taken into account: the instrument does not have the same sensitivity at the position of the calibrator field as at the position of the target field. You can compensate for this by using a model for the LOFAR beam. Effectively, then instead of equation~\ref{eq:dpppcal} the following equation is solved for :math:`{\mathbf{G}_p}`:
+
+.. math::
+
+    \| \mathbf{V}_{\!pq}-\mathbf{G}_{p}\mathbf{B}_p\mathbf{M}_{pq}\mathbf{B}_q^H\mathbf{G}_{q}^H \|
+    
+In the case of transferring solutions, the calibration is usually about the amplitude and the calibration type should be either **diagonal** or **fulljones**.
+
+A parset for calibrating on the calibrator field, taking the beam into account, is given below::
+
+    msin=L123.MS
+    msout=
+    steps=[gaincal]
+    gaincal.sourcedb=L123.MS/sky
+    gaincal.parmdb=L123.MS/instrument
+    gaincal.caltype=diagonal
+    gaincal.usebeammodel=true
+
+-----------------    
+Applying the beam
+-----------------
+
+When applying the solutions of the calibrator to the target, you should probably not apply the beam, so that another round of calibration is possible afterwards. Only after you are done with all calibration, the beam should be applied (just before imaging). Applying the beam is possible with ::
+
+    msin=L123.MS
+    msin.datacolumn=CORRECTED_DATA
+    msout=.
+    msout.datacolumn=CORRECTED_DATA
+    steps=[applybeam]
+
+.. rubric:: Footnotes
+
+.. [#f1] This section is maintained by `Tammo Jan Dijkema <mailto:dijkema@astron.nl>`_.