diff --git a/source/acknowledgements.rst b/source/acknowledgements.rst index 36ed67fd853dc617c1ac9d7e810bcd1ccd5a30ff..a9b28a5d101f8ce9977d57bf35425e4e83bd5b19 100644 --- a/source/acknowledgements.rst +++ b/source/acknowledgements.rst @@ -1,7 +1,7 @@ Acknowledgements ================ -Many commissioners and software developers have contributed to this cookbook. As the routines, the hardware, and the software needed for LOFAR develop very quickly, what is reported in this manual might be sometimes incorrect. We try to keep it as up to date as possible, but we surely need your feedback to improve its quality. Please send comments and suggestions of improvements to `Sarrvesh Sridhar <mailto:sarrvesh@astron.nl>`_. +Many commissioners and software developers have contributed to this cookbook. As the routines, the hardware, and the software needed for LOFAR develop very quickly, what is reported in this manual might be sometimes incorrect. We try to keep it as up to date as possible, but we surely need your feedback to improve its quality. Please send comments and suggestions of improvements to `Marco Iacobelli <mailto:iacobelli@astron.nl>`_. The contact points for the various versions of the LOFAR Imaging Cookbook are listed below. @@ -50,3 +50,5 @@ The contact points for the various versions of the LOFAR Imaging Cookbook are li + **Version 18.0**: Aleksandar~Shulevski, Roberto~F.~Pizzo, Ger van Diepen, Tammo Jan Dijkema, G. Heald, Francesco de Gasperin, M. Iacobelli, John McKean, Maaijke Mevius, Andr\'e Offringa, Emanuela Orr\'u, David Rafferty, Cyril Tasse, Bas van der Tol, T. J. Dijkema, Valentina Vacca, Nicolas Vilchez, Reinout van Weeren, Wendy Williams and Sarod Yatawatta + **Version 19.0 - 22.0**: Aleksandar~Shulevski, Roberto~F.~Pizzo, Ger van Diepen, Tammo Jan Dijkema, G. Heald, Francesco de Gasperin, M. Iacobelli, John McKean, Maaijke Mevius, Andr\'e Offringa, Emanuela Orr\'u, David Rafferty, Cyril Tasse, Bas van der Tol, T. J. Dijkema, Valentina Vacca, Nicolas Vilchez, Reinout van Weeren, Wendy Williams and Sarod Yatawatta + ++ **Version 23.0 - 24.2**: Aleksandar~Shulevski, Roberto~F.~Pizzo, Ger van Diepen, Tammo Jan Dijkema, George Heald, Francesco de Gasperin, M. Iacobelli, John McKean, Maaijke Mevius, Andr\'e Offringa, Emanuela Orr\'u, David Rafferty, Cyril Tasse, Bas van der Tol, Tammo J. Dijkema, Valentina Vacca, Sander ter Veen, Reinout van Weeren, Wendy Williams and Sarod Yatawatta \ No newline at end of file diff --git a/source/aoflagger.rst b/source/aoflagger.rst index e273572455f96c54c77c501d304e0a1b45ca276c..a0fc67654c6e58c6ed72b9d5d13e3e34d8c43fa2 100644 --- a/source/aoflagger.rst +++ b/source/aoflagger.rst @@ -114,7 +114,7 @@ Please note that the current working directory will be used as a temporary stora Documentation ------------- -The manual for AOFlagger can be found on the site `http://aoflagger.sourceforge.net/ <http://aoflagger.sourceforge.net/>`_. The properties and performance of the AOFlagger have been described in the following papers: +The manual for AOFlagger can be found on the site `https://aoflagger.readthedocs.io/ <https://aoflagger.readthedocs.io/>`_. The properties and performance of the AOFlagger have been described in the following papers: + `Post-correlation radio frequency interference classification methods <http://arxiv.org/abs/1002.1957>`_. + `A morphological algorithm for improving radio-frequency interference detection <http://arxiv.org/abs/1201.3364>`_. diff --git a/source/buildlofar.rst b/source/buildlofar.rst index 72bbc47d0db96f112f74f3e1d45fa572493c1464..f564dcc003df06c3f9b5a5840c3357ff59036e06 100644 --- a/source/buildlofar.rst +++ b/source/buildlofar.rst @@ -10,7 +10,7 @@ With this new docker framework [#f2]_, initializing your LOFAR processing framew and frees the user (and sysadmins) from building all the required software packages. The following sections provide an overview of how to install and run LOFAR imaging pipelines using the docker. -Any questions concerning the docker image should be addressed to `science operations and support <https://support.astron.nl/rohelpdesk>`_. Note however that **this docker framework is experimental and support for it from the SOS group will be provided on a best effort basis**. +Any questions concerning the docker image should be addressed to `Science Data Center Operations (SDCO) <https://support.astron.nl/sdchelpdesk>`_. Note however that **this docker framework is experimental and support for it from the SDCO group will be provided on a best effort basis**. ---------------------------------- What is in the LOFAR docker image? @@ -221,7 +221,7 @@ Once inside the container, you should source :: source /opt/lofarsoft/lofarinit.sh -Alternately, you can also download the singularity image (containing the same packages as the docker image) from this `link <https://www.dropbox.com/s/r5w5p1qdefp64g4/lofar.simg?dl=0>`_. Contact the `Science Operations & Support <https://support.astron.nl/rohelpdesk>`_ if you have trouble running the singularity image. +Alternately, you can also download the singularity image (containing the same packages as the docker image) from this `link <https://www.dropbox.com/s/r5w5p1qdefp64g4/lofar.simg?dl=0>`_. Contact the `SDCO <https://support.astron.nl/sdchelpdesk>`_ if you have trouble running the singularity image. --- FAQ @@ -238,10 +238,10 @@ FAQ **I get an "Illegal instruction error" when I run the docker image. What does this mean?** - This probably means that you are running an older/incompatible CPU. A docker image might have to be built on your machine. Please contact Science Operations & Support for further assistance if you encounter this. + This probably means that you are running an older/incompatible CPU. A docker image might have to be built on your machine. Please contact SDCO for further assistance if you encounter this. .. rubric:: Footnotes -.. [#f1] This chapter is maintained by `S. Sridhar <mailto:sarrvesh@astron.nl>`_. +.. [#f1] This chapter is maintained by `M. Iacobelli <mailto:iacobelli@astron.nl>`_. .. [#f2] Docker is an opensource platform that makes use of `container technology <https://opensource.com/resources/what-are-linux-containers?intcmp=7016000000127cYAAQ>`_ to create, deploy, and run applications easily. Detailed information about docker can be found `here <https://www.docker.com/>`_ and elsewhere on the internet. diff --git a/source/calculator.rst b/source/calculator.rst index 001e60f875f059f4974ac738e01c41201c8ec2f8..5d5caf78e3ff55c1a591b88146b133d858c07757 100644 --- a/source/calculator.rst +++ b/source/calculator.rst @@ -3,7 +3,7 @@ LOFAR Unified Calculator for Imaging (LUCI) [#f1]_ The LOFAR Unified Calculator for Imaging, or LUCI, is a unified web interface that allows users to compute several parameters relevant for planning an interferometric observation with LOFAR. The web interface can be accessed at `https://support.astron.nl/luci <https://support.astron.nl/luci/>`_. A similar tool for planning beamformed observations is under development. -LUCI is developed and maintained by the LOFAR Science Operations & Support (SOS) group at ASTRON. The source code is publicly available on GitHub at `https://github.com/scisup/LOFAR-calculator <https://github.com/scisup/LOFAR-calculator>`_. For comments and/or feature requests, please contact the SOS group using the `JIRA Helpdesk <https://support.astron.nl/rohelpdesk>`_. +LUCI is developed and maintained by the SDC Operations group (SDCO) at ASTRON. The source code is publicly available on GitHub at `https://github.com/scisup/LOFAR-calculator <https://github.com/scisup/LOFAR-calculator>`_. For comments and/or feature requests, please contact the SDCO group using the `JIRA Helpdesk <https://support.astron.nl/rohelpdesk>`_. For a given observation and pipeline setup, the web interface allows users to @@ -134,4 +134,4 @@ Frequently Asked Questions (FAQ) .. rubric:: Footnotes -.. [#f1] This chapter was written by `Sarrvesh Sridhar <mailto:sarrvesh@astron.nl>`_. +.. [#f1] This chapter was written by `Sarrvesh Sridhar <mailto:sarrvesh@astron.nl>` and is manitained by `Sander ter Veen <mailto:veen@astron.nl>`_. diff --git a/source/changelog.rst b/source/changelog.rst index 784837022cf1ce054c7f1362ab884aa8e17d84ab..0ef419fb568fe39ac14e85c2d8eae64f9bc39fc3 100644 --- a/source/changelog.rst +++ b/source/changelog.rst @@ -3,11 +3,11 @@ Changelog The latest released version of the cookbook is available `online <http://www.astron.nl/radio-observatory/lofar/lofar-imaging-cookbook>`_. -The LOFAR software is continuously improving and, as a consequence, several procedures (and the cookbook itself) continuously change. In the following, we report an overview of the (recent) changes applied to the manual. +The LOFAR software is continuously improving and, as a consequence, several procedures (and the cookbook itself) continuously change. In the following, we report an overview of the (recent) major changes applied to the manual. --------------------------- -Overview of recent changes --------------------------- +-------------------------------- +Overview of recent major changes +-------------------------------- ^^^^^^^^^^^^^^^^^^^^^^^^^ Version 24.0 (2020/01/08) diff --git a/source/conf.py b/source/conf.py index a06d5d6457fd61105277cd050a4704d85de7e9f3..4391cb79ddc616573ff756aecb13bd47837ac55b 100644 --- a/source/conf.py +++ b/source/conf.py @@ -47,17 +47,17 @@ master_doc = 'index' # General information about the project. project = u'LOFAR Imaging Cookbook' -copyright = u'2020, ASTRON' -author = u'Edited by Sarrvesh Sridhar' +copyright = u'2021, ASTRON' +author = u'Edited by Marco Iacobelli' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. -version = u'24.1' +version = u'24.2' # The full version, including alpha/beta/rc tags. -release = u'24.1' +release = u'24.2' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. @@ -148,7 +148,7 @@ latex_elements = { # author, documentclass [howto, manual, or own class]). latex_documents = [ (master_doc, 'LOFARImagingCookbook.tex', u'LOFAR Imaging Cookbook Documentation', - u'Edited by Sarrvesh Sridhar', 'manual'), + u'Edited by Marco Iacobelli', 'manual'), ] diff --git a/source/datainspection.rst b/source/datainspection.rst index d6fcc0d3a012856a099b0617709f8798816b5fbb..36046dc22a24b857c2c152dbd39f5ede7567f8ac 100644 --- a/source/datainspection.rst +++ b/source/datainspection.rst @@ -4,7 +4,7 @@ Data inspection [#f1]_ Data inspection is essential for a proper data reduction and can be carried out using either scripts (by means of a python interface to the Measurement Set) or `CASA <http://casa.nrao.edu/>`_. In this Chapter, we summarize the various tools that can be used to inspect the LOFAR visibilities at the beginning and during the processing of your data. -+ The quality of raw measurement sets can be inspected using data quality inspection plots that are generated by the Radio Observatory. These plots are available `online <https://proxy.lofar.eu/inspect/HTML/>`_ for up to three weeks after the time of observation. After that period, they are compressed and stored offline. Users can access them by creating a ticket through the `JIRA Helpdesk <https://support.astron.nl/rohelpdesk>`_. ++ The quality of raw measurement sets can be inspected using data quality inspection plots that are generated by the ASTRON staff. These plots are available `online <https://proxy.lofar.eu/inspect/HTML/>`_ for up to three weeks after the time of observation. After that period, they are compressed and stored offline. Users can access them by creating a ticket through the `JIRA Helpdesk <https://support.astron.nl/sdchelpdesk>`_. + The program **msoverview** provides you with details on the contents of a Measurement Set, no matter if it is raw or it has been already processed. You are advised to use this script when you start working on your data. You can run it by typing :: @@ -14,7 +14,11 @@ where the **verbose** parameter allows you to have more detailed information abo + Python-casacore (formerly "pyrap") is a python interface to the casacore library, which allows the raw data tables (Measurement Sets) to be manipulated and the data plotted via python scripts. These allow you to customize what is plotted, and can be significantly faster than CASA for plotting large datasets. An extensive description of the python-casacore utilities is available `online <http://casacore.github.io/python-casacore/>`_. + Visibilities can be plotted relatively rapidly by making use of the combination of python-casacore and the plotting package PGPLOT, both of which work quickly. The script **uvplot.py** [#f2]_ can be used to plot visibility data and can be significantly faster than the PLOTMS task in CASA. +<<<<<<< HEAD ++ `CASA <http://casa.nrao.edu/>`_ is the python-based next generation replacement for AIPS/AIPS++ and can be used to display the data. Be aware that trying to inspect the raw visibilities with CASA will produce a "**segmentation fault**" error. To avoid this, you should make a copy of the dataset with DPPP (see the first example parset in Sect.~\ref{theparsetfile}). Moreover, when opening with CASA a MeasurementSet observed between May and October 2011, you will get an **Unrecognised mount type** error [#f3]_. CASA has multiple functionalities that allows one to inspect different data products. +======= + `CASA <http://casa.nrao.edu/>`_ is the python-based next generation replacement for AIPS/AIPS++ and can be used to display the data. Be aware that trying to inspect the raw visibilities with CASA will produce a "**segmentation fault**" error. To avoid this, you should make a copy of the dataset with DPPP (see the first example parset in Sect.~\ref{theparsetfile}). **Also note** that the Radio Observatory now stores the LOFAR data as Dysco compressed measurement sets and hence CASA cannot be used to inspect them. You must decompress the measurement set first if you want to use CASA for data inspection [#f6]_. Moreover, when opening with CASA a MeasurementSet observed between May and October 2011, you will get an **Unrecognized mount type** error [#f3]_. CASA has multiple functionalities that allows one to inspect different data products. +>>>>>>> 7c916f800eedb02e44dc1fa3bc0b6a9441263e1f * **casabrowser** can be used to inspect the content of a Measurement Set. * **casaplotms** can be used to visibility data against different parameters. @@ -23,13 +27,13 @@ where the **verbose** parameter allows you to have more detailed information abo + The low frequency radio sky is dominated by a few bright sources that form the so called A-team: CasA, CygA, VirA, TauA, HydA, HerA. The removal of these sources from the target visibilities is very important in order to achieve high dynamic range images. The script **plot_Ateam_elevation.py**, which is available as part of the `LOFAR repository <https://github.com/lofar-astron/LOFAR-Contributions>`_, can be used to inspect the contribution of these A-team sources to the observed visibility data. -+ For manipulating measurement sets, you can use TaQL (Table Query Language); this is an SQL-like language which works on MS, and can perform all kinds of selections (and more). A detailed documentation of TaQL can be found `here <http://www.astron.nl/casacore/trunk/casacore/doc/notes/199.html>`__ and `here <http://taql.astron.nl>`__. ++ For manipulating measurement sets, you can use TaQL (Table Query Language); this is an SQL-like language which works on MS, and can perform all kinds of selections (and more). A detailed documentation of TaQL can be found `here <http://www.astron.nl/casacore/trunk/casacore/doc/notes/199.html>`__. ------------------------------------------- Analyzing the data quality with **aoqplot** ------------------------------------------- -Once you have succesfully run DPPP on the measurement sets in your observation, it is recommended that you validate the results of the flagger and get an impression of the quality of the full observation. For this, the **aoqplot** tool that is part of the LofIm build can be used. +Once you have successfully run DPPP on the measurement sets in your observation, it is recommended that you validate the results of the flagger and get an impression of the quality of the full observation. For this, the **aoqplot** tool that is part of the LofIm build can be used. Basically, the tool allows you to plot standard deviations and RFI percentages and some other quantities, over time, frequency, baselines and the time-frequency domain of the full observation, i.e., over all sub-bands. Since an observation can be several terabytes of data, the performance of standard tools to perform such analysis is an issue, and the **aoqplot** was designed to overcome this problem. Fig.~\ref{aoqplot-window} shows an example of the **aoqplot** interface, plotting the data standard deviations of an LBA set over the entire frequency range. It also allows one to plot differential statistics. "Differential" in this context means that the standard deviation is calculated over the difference between adjacent channels. Therefore, they quantify the noise, because the difference of signal in adjacent (3 kHz) channels is tiny and can be neglected. Differential quantities are prefixed with a "D", such as DMean and DStdDev. @@ -235,5 +239,5 @@ The following command lets you make a movie of the line plots like the one shown .. [#f2] The script was written by George Heald and is available through the `LOFAR GitHub repository <https://github.com/lofar-astron/LOFAR-Contributions>`_. .. [#f6] For more information, consult `this webpage <https://old.astron.nl/radio-observatory/lofar-data-processing/dysco-compression/compressing-lofar-measurement-sets-using-d>`_. .. [#f3] This is due to the fact that the MS writer version used during those months was specifying the antenna mount as FIXED, and not as ALT-AZ, which is CASA friendly. To solve this problem, you can run the following **taql** command on your MS as *taql 'update <ms name>/ANTENNA set MOUNT="X-Y" '* -.. [#f4] The **SPLIT** task will be deprecated in favor of the **MSTRANSFORM** task beginning with CASA v.4.1.0. +.. [#f4] The **SPLIT** task will be deprecated in favour of the **MSTRANSFORM** task beginning with CASA v.4.1.0. .. [#f5] The script is available through the `LOFAR GitHub repository <https://github.com/lofar-astron/LOFAR-Contributions>`_. diff --git a/source/gettingstarted.rst b/source/gettingstarted.rst index e05cbe24f05b6466756c5d305c706184f5ce6efa..76b5a2dd417e5595f5ca2b634e3587e71c77837c 100644 --- a/source/gettingstarted.rst +++ b/source/gettingstarted.rst @@ -5,7 +5,7 @@ Introduction to LOFAR computing facilities [#f1]_ The LOFAR cluster layout ------------------------ -The correlated data coming from the Cobalt [#f2]_ correlator are stored on a cluster of machines called CEP4_ cluster. CEP4_ has been added to the LOFAR offline system at the beginning of 2017. Till December 2016, another cluster of computing machines was used to store and process the data (CEP2). Nowadays, CEP4_ is normally used by the Radio Observatory to process the data through the initial stages of the data reduction (flagging and averaging of the visibilities), while another CEP facility is currently used by both the commissioners and LOFAR users to manually play with the data and understand which strategy to use for the calibration and the imaging: the new commissioning cluster `CEP3`_. +The correlated data coming from the Cobalt [#f2]_ correlator are stored on a cluster of machines called CEP4_ cluster. CEP4_ has been added to the LOFAR offline system at the beginning of 2017. Till December 2016, another cluster of computing machines was used to store and process the data (CEP2). Nowadays, CEP4_ is normally used by ASTRON Telescope and Operations team to process the data through the initial stages of the data reduction (flagging and averaging of the visibilities), while another CEP facility is currently used by both the commissioners and LOFAR users to manually play with the data and understand which strategy to use for the calibration and the imaging: the new commissioning cluster `CEP3`_. CEP4_ facility provides technologies that were not available on CEP2, especially with respect to resource management. In the following sections we will focus on discussing the architecture of CEP facilities as well as their usage policies. @@ -17,7 +17,7 @@ The LOFAR phase 4 cluster - CEP4 -------------------------------- The LOFAR CEP4 cluster is composed by 50 compute nodes (**cpu01-50**), 4 GPU nodes (**gpu01-04**), 18 storage nodes (**data01-18**), 50 compute nodes (**cpu01-50**), 2 meta-data nodes (**meta01-02**), 2 head nodes (**head01-head02**) and 1 management node (**mgmt01**). -A detailed description of all the packages available on the new cluster and on its network interface can be found on the `wiki <https://www.astron.nl/lofarwiki/doku.php?id=cep4:system>`_. Processed data products will usually made available to the user via the Long-Term Archive, but may also be copied to the `CEP3`_ cluster upon request for further analysis by the user in the original proposal. Due to the intensive nature of the standard data pipelines and the need for these compute resources to be allocated and scheduled by Radio Observatory staff, access to the resources on CEP4 will be strictly limited, with a few exceptions, to the Radio Observatory. In the following, a short description of the computing characteristics/performances of the new cluster is given. +A detailed description of all the packages available on the new cluster and on its network interface can be found on the `wiki <https://www.astron.nl/lofarwiki/doku.php?id=cep4:system>`_. Processed data products will usually made available to the user via the Long-Term Archive, but may also be copied to the `CEP3`_ cluster upon request for further analysis by the user in the original proposal. Due to the intensive nature of the standard data pipelines and the need for these compute resources to be allocated and scheduled by the ASTRON Science Data Center Operations (SDCO) staff, access to the resources on CEP4 will be strictly limited, with a few exceptions, to the SDCO. In the following, a short description of the computing characteristics/performances of the new cluster is given. The Lofar Phase 4 cluster consists of: @@ -55,7 +55,7 @@ The other nodes are not accessible (storage, meta-data, and management nodes). **Processing:** CEP4 will use a SLURM batch scheduling system to schedule and run all observation and processing jobs on the cluster. -NOTE: It is emphasised again that CEP4 is not meant for commissioning work. For that, commissioners can use CEP3 (see access policies here). It is emphasized that the disks on CEP3 are not intended for long term storage of results. As it is impossible for the Observatory to micro-manage the disk space, commissioners should be aware that disk deletions could happen with very little warning. +NOTE: It is emphasised again that CEP4 is not meant for commissioning work. For that, commissioners can use CEP3 (see access policies here). It is emphasized that the disks on CEP3 are not intended for long term storage of results. As it is impossible for the SDCO to micro-manage the disk space, commissioners should be aware that disk deletions could happen with very little warning. ---- CEP3 @@ -74,9 +74,9 @@ In the future the servers can be fitted with up to two GPU cards (e.g. NVIDIA K2 Two head nodes (lhd001 & lhd002) are available for logging in and (limited) interactive development and processing purposes. Access to CEP3 can only be done through the **lhdhead.offline.lofar** head node, access to the twenty two worker nodes is managed through a job management system. Users are required to submit requests for processing jobs/sessions on the worker nodes. In general, data will be distributed across the local disks on the worker nodes and processing jobs are distributed accordingly. -**USAGE POLICY:** Observing, CEP4 processing time and the use of CEP3 are allocated by the LOFAR Programme Committee and the ILT director during the regular proposal evaluation stages, or under Director's Discretionary Time. Access and use of CEP3 is under the sole control of the Radio Observatory's Science Operations & Support (SOS). Access for Users will be granted only at the discretion of the Science Operations & Support. Users should conform to the access, resource allocation and data deletion policies issued by the SOS at all times. +**USAGE POLICY:** Observing, CEP4 processing time and the use of CEP3 are allocated by the LOFAR Programme Committee and the ILT director during the regular proposal evaluation stages, or under Director's Discretionary Time. Access and use of CEP3 is under the sole control of the SDCO department. Access for Users will be granted only at the discretion of the SDCO staff. Users should conform to the access, resource allocation and data deletion policies issued by SDCO at all times. -To have access to CEP3 a formal request must be submitted to sos@astron.nl or be explicitly given in a proposal for LOFAR observing time. When submitting a request users should clearly include the following information: +To have access to CEP3 a formal request must be submitted to the SDCO helpdesk `<https://support.astron.nl/sdchelpdesk>`_ or be explicitly given in a proposal for LOFAR observing time. When submitting a request users should clearly include the following information: + A brief explanation of why access to CEP3 is required (e.g., you do not have access to suitable computing resources elsewhere) + Project to be worked on (i.e. commissioning, cycle or archived data (post) processing) @@ -87,12 +87,12 @@ To have access to CEP3 a formal request must be submitted to sos@astron.nl or be Users awarded with access to CEP3 will be able to access the cluster for a limited period of time (2 months by default). The awarded period starts: -+ from the moment the user's data is copied from CEP4 (after Radio Observatory pipeline processing) to CEP3 -+ following the timeline communicated to the user via the SOS notification and available on the `wiki <http://www.lofar.org/operations/doku.php?id=cep3:earlyaccess>`__. ++ from the moment the user's data is copied from CEP4 (after SDCO pipeline processing) to CEP3 ++ following the timeline communicated to the user via the SDCO notification and available on the `wiki <http://www.lofar.org/operations/doku.php?id=cep3:earlyaccess>`__. -At the beginning of a Cycle, users requesting CEP3 processing time in their observing proposals can derive this timeline by checking the observing schedule, which is available `here <http://www.astron.nl/radio-observatory/cycles/cycles>`__. Access timelines related to observing programs involving observations spread in time will be discussed between the PI and Science Support. In general, info about CEP3 access of users are detailed on the LOFAR `wiki <http://www.lofar.org/operations/doku.php?id=cep3:earlyaccess>`__. After the granted period on CEP3 has expired, all users data products generated on the cluster will be automatically and promptly removed, to enable new users to have enough disk space to perform their data reduction. +At the beginning of a Cycle, users requesting CEP3 processing time in their observing proposals can derive this timeline by checking the observing schedule, which is available `here <http://old.astron.nl/radio-observatory/cycles/cycles>`__. Access timelines related to observing programs involving observations spread in time will be discussed between the PI and SDCO staff. In general, info about CEP3 access of users are detailed on the LOFAR `wiki <http://www.lofar.org/operations/doku.php?id=cep3:earlyaccess>`__. After the granted period on CEP3 has expired, all users data products generated on the cluster will be automatically and promptly removed, to enable new users to have enough disk space to perform their data reduction. -Extensions to the default 8-weeks period will be granted only in exceptional circumstances and only if properly justified through a formal request to be sent to sos@astron.nl no later than 1 week before the expiration of your access privileges. Monitoring of node usage during allocated time will be performed and the evaluation of extension requests will be based on such statistics. +Extensions to the default 8-weeks period will be granted only in exceptional circumstances and only if properly justified through a formal request to be sent to the SDCO helpdesk `<https://support.astron.nl/sdchelpdesk>`_ no later than 1 week before the expiration of your access privileges. Monitoring of node usage during allocated time will be performed and the evaluation of extension requests will be based on such statistics. .. _Logging on to CEP3: @@ -100,7 +100,7 @@ Extensions to the default 8-weeks period will be granted only in exceptional cir Logging on to CEP3 ------------------ -As mentioned above, normal users have access only to CEP3 head node(s), while the access to processing nodes is controlled using the Slurm cluster management software. In the head node users can experience the quality of the data and understand the best approach to use in the lof node(s) for the calibration and imaging of the visibilities. After Science Support has set up a reservation on a particular processing node(s), you should have a reservationID needed for setting up access to the working node(s). +As mentioned above, normal users have access only to CEP3 head node(s), while the access to processing nodes is controlled using the Slurm cluster management software. In the head node users can experience the quality of the data and understand the best approach to use in the lof node(s) for the calibration and imaging of the visibilities. After SDCO staff has set up a reservation on a particular processing node(s), you should have a reservationID needed for setting up access to the working node(s). To access CEP3, begin by logging on to **portal.lofar.eu** [#f3]_ :: @@ -213,7 +213,7 @@ Processing can now take place. Once you have logged onto this compute node, you mkdir /data/scratch/<username> -You can now **cd** into it and use it as your working space. You can copy in here the data provided by the Radio Observatory by e.g. typing:: +You can now **cd** into it and use it as your working space. You can copy in here the data provided by SDCO by e.g. typing:: > scp -r <user name>@lhdhead.offline.lofar:/data/<user>/<LOFAR dataset> . @@ -264,7 +264,7 @@ multiple SSH connections, that can get pretty annoying. To get around it, set Copying data from and to CEP3 cluster ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Data transfers from CEP4 to CEP3 should always be coordinated with the Radio Observatory [#f7]_. Data retrieval from LTA locations as well as from/to other computing facilities is also possible as detailed in `the LOFAR wiki <http://www.lofar.org/wiki/doku.php?id=cep3:userdata>`__. Although access to the CEP3 systems can only be done through the **lhdhead.offline.lofar** head node, data transfers to the outside world can be done directly. *Because data will be transferred via the LOFAR portal, care should be taken not to flood the available network bandwidth with the public internet. Thus we recommend to limit the bandwidth to not disturb the portal access of other users. Note that the portal capacity is 120MB/s*. +Data transfers from CEP4 to CEP3 should always be coordinated with the SDCO [#f7]_. Data retrieval from LTA locations as well as from/to other computing facilities is also possible as detailed in `the LOFAR wiki <http://www.lofar.org/wiki/doku.php?id=cep3:userdata>`__. Although access to the CEP3 systems can only be done through the **lhdhead.offline.lofar** head node, data transfers to the outside world can be done directly. *Because data will be transferred via the LOFAR portal, care should be taken not to flood the available network bandwidth with the public internet. Thus we recommend to limit the bandwidth to not disturb the portal access of other users. Note that the portal capacity is 120MB/s*. .. rubric:: Footnotes @@ -274,6 +274,6 @@ Data transfers from CEP4 to CEP3 should always be coordinated with the Radio Obs .. [#f3] The actual host name is lfw.lofar.eu (lfw=LOFAR firewall), but this alias will work fine. .. [#f4] Your default password will be communicated to you at the moment of the creation of your Lofar account by `Teun Grit <mailto:grit@astron.nl>`_. .. [#f5] You may have to log out of and log in again to the portal first. -.. [#f6] The Radio Observatory will assign you with a suitable *lof* node to work on. -.. [#f7] sos[at]astron[dot]nl +.. [#f6] The SDCO staff will assign you with a suitable *lof* node to work on. +.. [#f7] SDCO helpdesk `<https://support.astron.nl/sdchelpdesk>`_ .. [#f8] **"L"** stays for LOFAR, **XXXXX** is the ID number of the observation, which is assigned to it at the moment of the scheduling. diff --git a/source/resources.rst b/source/resources.rst index a2cc21b51c57b849a8b6193290fd4f4839e56b31..e123e950c81c3324ea9843e7b729484d61daf7b7 100644 --- a/source/resources.rst +++ b/source/resources.rst @@ -5,7 +5,7 @@ Useful resources Webpages -------- -The LOFAR wiki is a key resource, and you need an account to access the software areas. You can register for an account `here <http://www.lofar.org/operations/doku.php?id=start&do=register>`_. +The LOFAR wiki is a key resource, and you need an account to access the software areas. You can register for an account `here <https://www.astron.nl/lofarwiki>`_. Essential pages on the wiki are: @@ -51,7 +51,7 @@ Contact points Some key contact points are listed below: -+ **LOFAR Imaging Cookbook** - `Sarrvesh Sridhar <mailto:sarrvesh@astron.nl>`_ ++ **LOFAR Imaging Cookbook** - `Marco Iacobelli <mailto:iacobelli@astron.nl>`_ + **DPPP** - `Ger van Diepen <mailto:diepen@astron.nl>`_, `Tammo Jan Dijkema <mailto:dijkema@astron.nl>`_, and `David Rafferty <mailto:rafferty@strw.leidenuniv.nl>`_ + **AOFlagger** - `Andre Offringa <mailto:offringa@astron.nl>`_ + **BBS** - `Tammo Jan Dijkema <mailto:dijkema@astron.nl>`_, `Vishambhar Nath Pandey <mailto:pandey@astron.nl>`_ @@ -62,4 +62,4 @@ Some key contact points are listed below: .. rubric:: Footnotes -.. [#f1] If you have other scripts that could be useful for other commissioners, please contact `Sarrvesh Sridhar <mailto:sarrvesh@astron.nl>`_. +.. [#f1] If you have other scripts that could be useful for other commissioners, please contact `Marco Iacobelli <mailto:iacobelli@astron.nl>`_. diff --git a/source/tutorial.rst b/source/tutorial.rst index ff9732f6fd56a205df3a143c4dfbf31a6bde2ddb..578600501d26520b5bb5cd2202279c752de3bbe4 100644 --- a/source/tutorial.rst +++ b/source/tutorial.rst @@ -569,7 +569,7 @@ From this, you should see that HBA_DUAL_INNER mode was used, i.e. the core stat Flagging and data compression ----------------------------- -The data set that we are using is uncompressed and unflagged; the total size of each MS is $11$\,GB. The data flagging and compression are carried out using `DPPP <./dppp.html>`_. Typically, initial RFI flagging and averaging will be done by the averaging pipeline run by the Radio Observatory. Note that the limitation on the compression in frequency is set by the size of the field you wish to image and the amount of bandwidth smearing at the edges of the field. The time averaging is limited not only by the amount of time smearing you will allow but also by the changes in the ionosphere. +The data set that we are using is uncompressed and unflagged; the total size of each MS is $11$\,GB. The data flagging and compression are carried out using `DPPP <./dppp.html>`_. Typically, initial RFI flagging and averaging will be done by the averaging pipeline run by the SDCO staff. Note that the limitation on the compression in frequency is set by the size of the field you wish to image and the amount of bandwidth smearing at the edges of the field. The time averaging is limited not only by the amount of time smearing you will allow but also by the changes in the ionosphere. In this example we are flagging the data using the aoflagger algorithm within DPPP. Here we will compress the sub-band to 4 channels in frequency and 5 s in time. The parset file for the flagging and compression should be copied to your working directory, :: @@ -624,7 +624,7 @@ The flagged and compressed data set should now be in your working directory and Post-compression data inspection and flagging --------------------------------------------- -We will use the CASA task plotms to inspect the data. Only limited information for using plotms is given here, the User is directed to the `CASA cookbook <\href{http://casa.nrao.edu/Doc/Cookbook/casa_cookbook.pdf>`_ for full details. :: +We will use the CASA task plotms to inspect the data. Only limited information for using plotms is given here, the User is directed to the `CASA cookbook <\href{https://casa.nrao.edu/docs/cookbook/>`_ for full details. :: module load casa casaplotms @@ -1027,7 +1027,7 @@ From this, you should see that 32 stations were used for this observation, that Flagging and demixing --------------------- -As with the HBA, the data set is uncompressed and unflagged; the total size of each MS is 3.9 GB. The data flagging and compression are carried out using DPPP. We will compress the sub-band to 4 channel in frequency and 4s in time. Note that the limitation on the compression in time is set by the changes in the ionosphere. For LBA data it is almost always necessary to demix the data to remove the bright radio sources from the data. Demixing has been implemented in DPPP. Usually this will be performed by the Radio Observatory but we include it here so you can learn how to do it. +As with the HBA, the data set is uncompressed and unflagged; the total size of each MS is 3.9 GB. The data flagging and compression are carried out using DPPP. We will compress the sub-band to 4 channel in frequency and 4s in time. Note that the limitation on the compression in time is set by the changes in the ionosphere. For LBA data it is almost always necessary to demix the data to remove the bright radio sources from the data. Demixing has been implemented in DPPP. Usually this will be performed by the SDCO staff but we include it here so you can learn how to do it. To see which A-team sources need to be demixed use the **plot_Ateam_elevation.py** script :: diff --git a/source/wsclean.rst b/source/wsclean.rst index 9162e5c41b65baf992e97da4cab176b35cca8489..a83a37467e68a5b839c793606be9b6b7b4fe609c 100644 --- a/source/wsclean.rst +++ b/source/wsclean.rst @@ -14,7 +14,7 @@ A list of command line options can be acquired by running **wsclean** without pa This software package is released under the GPL version 3. ... -An extensive manual for WSClean is available on-line, at `wsclean homepage <https://sourceforge.net/p/wsclean/wiki/Home/>`_. It includes a chapter about LOFAR beam correction. WSClean has also been described in the following article: `http://arxiv.org/abs/1407.1943 <http://arxiv.org/abs/1407.1943>`_. +An extensive manual for WSClean is available on-line, at `wsclean homepage <https://wsclean.readthedocs.io/>`_. It includes a chapter about LOFAR beam correction. WSClean has also been described in the following article: `http://arxiv.org/abs/1407.1943 <http://arxiv.org/abs/1407.1943>`_. .. rubric:: Footnotes