diff --git a/source/calculator.rst b/source/calculator.rst index 5d5caf78e3ff55c1a591b88146b133d858c07757..891a3058240b9734e0988e6b21d14a46b72e031e 100644 --- a/source/calculator.rst +++ b/source/calculator.rst @@ -1,137 +1,196 @@ LOFAR Unified Calculator for Imaging (LUCI) [#f1]_ -================================================== +==================================================== -The LOFAR Unified Calculator for Imaging, or LUCI, is a unified web interface that allows users to compute several parameters relevant for planning an interferometric observation with LOFAR. The web interface can be accessed at `https://support.astron.nl/luci <https://support.astron.nl/luci/>`_. A similar tool for planning beamformed observations is under development. +The LOFAR Unified Calculator for Imaging, or LUCI, is a unified web interface that allows users to compute several parameters relevant for planning an interferometric observation with LOFAR. The web interface can be accessed at `https://support.astron.nl/luci <https://support.astron.nl/luci/>`_. LUCI is developed and maintained by the SDC Operations group (SDCO) at ASTRON. The source code is publicly available on GitHub at `https://github.com/scisup/LOFAR-calculator <https://github.com/scisup/LOFAR-calculator>`_. For comments and/or feature requests, please contact the SDCO group using the `JIRA Helpdesk <https://support.astron.nl/rohelpdesk>`_. For a given observation and pipeline setup, the web interface allows users to -+ compute the raw and processed data size, -+ compute theoretical image sensitivity, -+ estimate the pipeline processing time, -+ plot the elevation of the sources to observe on any given date, -+ plan multi-beam observations, and -+ export the observation and pipeline setup in a PDF format. ++ Compute the raw and pre-processed data size ++ Compute theoretical and effective image sensitivity ++ Estimate the pipeline processing time ++ Plot the the visibility profiles of the sky sources on any given date ++ Plot the beam layout ++ Calculate angular distances to bright sky sources ++ Export the observation and pipeline setup in a PDF format. -The landing page of the LUCI web interface is shown below in :numref:`luci_interface`. The page is divided into three columns each allowing the user to set a specific aspect of the observation. By setting different input fields, users can compute some or all of the parameters listed above. The different functionalities of LUCI are explained below. +This manual will elaborate on the basic LUCI usage, how the different parameters are calculated and how to interpret the calculated results. -.. _luci_interface: -.. figure:: figures/luci_interface.png - :align: center - - The web interface to the LOFAR Unified Calculator for Imaging (LUCI). - -------------------------- -Setting up an observation -------------------------- +------------------------------ +How to set up an observation +------------------------------ -The **Observational setup** section in LUCI allows users to specify an interferometric observing run with LOFAR. The various fields under this section are: +The *Observational setup* section in LUCI allows users to specify an interferometric observing run with LOFAR. The various fields under this section are: + **Observation time (in seconds)** -- Duration of an observing run (default: 28800 seconds). LUCI throws an error if a negative value is provided as input. + **No. of core stations (0 - 24)** (default: 24) + **No. of remote stations (0 - 14)** (default: 14) + **No. of international stations (0 - 14)** (default: 14) + **Number of channels per subband** -- Frequency resolution of an observing run. Allowed values are 64, 128, and 256 channels per subband (default: 64) -+ **Number of subbands** -- Number of subbands to record as part of the observing run (default: 488). LUCI thows an error if the number of subbands is greater than 488. ++ **Number of subbands** -- Number of subbands to record as part of the observing run (default: 488). LUCI thows an error if the number of subbands is greater than 488. If multiple targets are provided, the maximum number of subbands allowed becomes 488 divided by the number of targets. + **Integration time (in seconds)** - Correlator integration time. This input field determines the time resolution of the observing run (default: 1 second). LUCI thows an error if correlator time is smaller than 0.16 seconds. -+ **Antenna set** -- specifies the mode in which the stations are operated. Allowed values are LBA Outer, HBA Dual, and HBA Dual Inner. (default: HBA Dual Inner). ++ **Antenna set** -- specifies the mode in which the stations are operated. Allowed values are LBA Outer, LBA Sparse, HBA Dual, and HBA Dual Inner. (default: HBA Dual Inner). -Upon clicking the button **Calculate**, LUCI computes the theoretical image sensitivity and the raw data size. The results of this computation is displayed to the user under the **Results** section. Note that before doing the calculations, LUCI validates the input specified by the user. If an invalid input is detected, LUCI displays an error as shown in :numref:`luci_error`. +Upon clicking the button *Calculate*, LUCI computes the theoretical image sensitivity, effective image sensitivity, mean elevation of the selected input target(s) as well as the raw data size. The results of this computation is displayed to the user under the Results section. Note that before doing the calculations, LUCI validates the input specified by the user. If an invalid input is detected, LUCI displays an error. -.. _luci_error: - -.. figure:: figures/luci_error.png - :align: center - - Example error message displayed to the user. Before performing the calculations, LUCI validates the user input. If an invalid input is detected, LUCI shows a similar error message. +If the pipeline set-up is also selected, then the Results section will get an additional population as described below. ------------------------------------ -Setting up a preprocessing pipeline ------------------------------------ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +How to set up the preprocessing pipeline +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Pipelines can be setup in LUCI using the field **Pipeline** dropdown button under **Pipeline setup**. At the moment, only a preprocessing pipeline can be specified. By default, no pipeline is attached to the observing run and hence the field **Pipeline** is set to None. To attach a preprocessing pipeline, set the **Pipeline** field to **Preprocessing**. Toggling the **Pipeline** field reveals a few more input fields (see :numref:`luci_pipeline`) under **Pipeline setup** which can be used to tweak the behaviour of the pipeline. The new, pipeline-specific fields are +Pipelines can be setup in LUCI using the field *Pipeline* dropdown button under *Pipeline setup*. At the moment, only a preprocessing pipeline can be specified. By default, no pipeline is attached to the observing run and hence the field *Pipeline* is set to None. To attach a preprocessing pipeline, set the *Pipeline* field to *Preprocessing*. Toggling the *Pipeline* field reveals a few more input fields under *Pipeline setup* which can be used to tweak the behaviour of the pipeline. The new, pipeline-specific fields are + **Time averaging factor** How much averaging to perform along the time axis (default: 1), + **Frequency averaging factor** How much averaging to perform along the freqency axis (default: 1), and + **Enable dysco compression?** Should the output measurement sets produced by the preprocessing pipeline be compressed with Dysco? (default: Enabled). -Upon clicking the **Calculate** button, LUCI calculates **Processed data size** and **Pipeline processing time** in addition to **Theoretical image sensitivity** and **Raw data size** mentioned earlier. +Upon clicking the *Calculate* button, LUCI calculates *Processed data size* and *Pipeline processing time* in addition to the previously mentioned parameters. -.. _luci_pipeline: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Specifying targets, calibrators, and A-team sources +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. figure:: figures/luci_pipeline.png - :align: center - - LUCI web interface showing the pipeline fields that are revealed when the field **Pipeline** is set to **Preprocessing**. +Besides calculating the data sizes and processing times, LUCI also allows users to check if their target of interest is visible to LOFAR on a specific date. The target of interest can be specified in the *Target* field under the *Target setup* section. If the specified source name is known in Simbad, the Resolve button can be used to obtain the coordinates of that source. Note that LUCI can resolve valid LoTSS pointing names as well (like P17 or P205+67). If however, the user wishes to manually input the coordinates, a target name must also be specified. +In addition to plotting the target, users can also plot the elevations of standard LOFAR calibrators and A-team sources by selecting them using the *Calibrators* and the *A-team sources* dropdown boxes. -.. _target_setup: +In addition to plotting the target, users can also plot the elevations of standard LOFAR calibrators and A-team sources by selecting them using the Calibrators and the A-team sources drop-down boxes. Note that the calibrators and A-team sources will not be used in any of the calculations. ---------------------------------------------------- -Specifying targets, calibrators, and A-team sources ---------------------------------------------------- +Upon clicking the *Calculate* button, LUCI produces three plots like the once shown below: -Besides calculating the data sizes and processing times, LUCI also allows users to check if their target of interest is visible to LOFAR on a specific date. The target of interest can be specified in the **Target** field under the **Target setup** section. If the specified source is valid, the **Resolve** button can be used to obtain the coordinates of that source. Note that LUCI can resolve valid LoTSS pointing names as well (like P17 or P205+67). If however, the user wishes to manually input the coordinates, a target name must also be specified. The **Observation date** field allows the user to specify a date on which the target elevation needs to be calculated. ++ **Target visibility plot** shows the elevation of a target as seen by LOFAR on the specified date as an interactive plot. If the field No. of international stations (0-14) is set to 0 (i.e. observing with only the Dutch array), the target elevation is calculated with respect to the LOFAR core. On the other hand, if the user specifies at least 1 No. of international stations (0-14), LUCI plots the minimum apparent elevation of the target as seen by all LOFAR stations and takes the longest baseline as reference. In addition to the user-specified target, LUCI also plots the elevations of the Sun, the Moon, and Jupiter by default. If a calibrator or A-team source is selected from the Target setup section, these will also be added to the plot. The two blue regions indicate the sunrise and sunset times. A thicker line on top of the input target elevation plot is also shown to highlight the observation time around transit for each input target. ++ **Beam Layout** plots the tile and station beams for the specified observation. If multiple targets were selected, it will plot a station beam for each target. ++ **Angular distance** between specified target and other bright sources are presented in a plotly table. The table expands to calculate values for each input target. -In addition to plotting the target, users can also plot the elevations of standard LOFAR calibrators and A-team sources by selecting them using the **Calibrators** and the **A-team sources** dropdown boxes. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Setting up multi-beam observations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Upon clicking the **Calculate** button, LUCI produces three plots like the once shown below: +To observe multiple sources with LOFAR simultaeously making use of its multi-beam capability, the user can specify multiple sources in the *Target* field as a comma-separated value. LUCI can handle coordinate resolution of both single and multiple sources. While specifying multiple sources, LUCI will throw an error if -+ **Target visibility plot** shows the elevation of a target as seen by LOFAR on the specified date as an interactive plot. If the field **No. of international stations (0-14)** is set to 0 (i.e.) observing with only the Dutch array, the target elevation is calculated with respect to the LOFAR core. On the other hand, if the user specifies the full array, LUCI plots the minimum apparent elevation of the target as seen by all LOFAR stations. In addition to the user-specified target, LUCI also plots the elevations of the Sun, the Moon, and Jupiter by default. The two blue regions indicate the sunrise and sunset times. -+ **Beam Layout** plots the tile and station beams for the specified observation. -+ **Angular distance between specified target and other bright sources** table. ++ the number of sources mentioned in the *Target* field does not match the number of coordinates, ++ the number of sources times the number of subbands is greater than 488. This is because any LOFAR observation cannot have more than 488 subbands. -.. _luci_single_target: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Exporting the setup +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. figure:: figures/luci_single_target.png - :align: center - - LUCI web interface showing the target visibility and beam layout plots along with a table showing the angular distance between the specified target and known bright radio sources. Note that the target visibility plot shows the elevation of the Sun, the Moon and Jupiter in addition to the user-specified target. Also note that both the target visibility and beam layout plots are interactive plots. +Users can export their "observational setup" to a PDF file using the *Generate PDF* button. Upon clicking the *Generate PDF* button, LUCI exposes the *Download file* link below the two buttons which can be used to download the generated PDF file. ----------------------------------- -Setting up multi-beam observations ----------------------------------- +**Note** that if you click on the *Generate PDF* button before using the *Calculate* button, LUCI will throw an error. -To observe multiple sources with LOFAR simultaeously making use of its multi-beam capability, the user can specify multiple sources in the **Target** field as a comma-separated value. LUCI can handle coordinate resolution of both single and multiple sources. While specifying multiple sources, LUCI will throw an error if +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Warnings and errors +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -+ the number of sources mentioned in the **Target** field does not match the number of coordinates, -+ the number of sources times the number of subbands is greater than 488. This is because any LOFAR observation cannot have more than 488 subbands. +LUCI has warning and errors integrated so that the user knows why a setting does not work and potentially halts the calculator. Warnings are messages that pop up to warn the user about the validity of some results (mainly if a target is < 30 degrees for LBA and < 40 degrees for HBA). These warning do not crash the calculator but the user should treat the returned results with care. The current warning message pops up in orange below the Results section. +Errors are integrated hard crashes that won't allow the calculator to run and return results. These are shown in a separate pop up window that the user has to manually close. These errors include unrealistic parameter inputs or unobservable sources. When an error message pops up, the user can simply close the message and adjust the set up accordingly and retry the calculation. -.. _luci_multi_target: -.. figure:: figures/luci_multi_target.png - :align: center - - LUCI web interface showing the target visibility and beam layout plots along with a table showing the angular distance between the specified targets and known bright radio sources. +--------------------------- +Calculated parameters +--------------------------- -------------------- -Exporting the setup -------------------- +The following calculated values are presented in the Results section: Mean Elevation, Theoretical Sensitivity, Effective Sensitivity, Raw Data Size, Processed Data Size and Processing Time. These parameters are calculated using the input parameters with a certain logic. This section will elaborate on how these values are calculated and why it was done that way. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Mean elevation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The mean elevation is calculated for each input target and is calculated to be at around +/- 0.5 times the observation time, centred around transit (maximum elevation) of the specified observation date. The mean elevation can change with respect to what selection of stations is chosen (i.e. if international stations are chosen, the projection will change compared to just the superterp). This mean elevation is then used in further calculations. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Theoretical sensitivity +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Users can export their "observational setup" to a PDF file using the **Generate PDF** button. Upon clicking the **Generate PDF** button, LUCI exposes the **Download file** link (see :numref:`luci_export`) below the two buttons which can be used to download the generated PDF file. +The theoretical sensitivity is based on 2 major things: the instrumentation set up and the target's mean elevation. The calculation starts with the instrumentation setup, where the number of different stations, number of subbands, observation time and observing mode play the most important role. +The different observing modes specify different hard-coded SEFD values (based on 60 MHz for LBA and 150 MHz for HBA) from which the rms is calculated. The exact formula for the theoretical calculation is given below where we set the weight to 1 (natural). This value will then get a standard geometrical correction factor applied to it that is a function of the mean elevation to reflect how the rms changes for targets especially at low elevations. This corrective factor is coded a :math:`cos(90-elev)^{-1}` -.. _luci_export: +.. math:: -.. figure:: figures/luci_export.png + \Delta S_{theor} = \frac{W}{\sqrt{ 2(2 \delta \nu \delta t) \left ( \frac{N_{core}(N_{core} - 1)/2}{S^2_{core}} + \frac{N_{remote} (N_{remote} -1)/2}{S^2_{remote}} \frac{N_{int} (N_{int} -1)/2}{S^2_{int}} + \frac{N_{core} N_{remote}}{S_{core} S_{remote}} + \frac{N_{core} N_{int}}{S_{core} S_{int}} + \frac{N_{remote} N_{int}}{S_{remote} S_{int}} \right ) }}} + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Effective sensitivity +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The effective sensitivity calculation method is handled differently for HBA and LBA. Currently for LBA there is no specific effective sensitivity calculation, so it returns the theoretical sensitivity with an uncertainty of 0. For HBA there is an effective rms value returned that is based on the noise maps from the LoTTS survey. In the figure below, the purple dots belong to the average noise in each LoTTS pointing as a function of pointing elevation on the sky. +This is then fitted to a power cosine function to result in the effective noise with different uncertainties coming from the uncertainty from the fitting parameters. This fit was then extended to incorporate the theoretical sensitivity dependence so the values scale with number of subbands, observation time, etc. + +.. _luci_HBA_fitting: + +.. figure:: figures/output.png :align: center - LUCI web interface showing the PDF download link after a user clicks on the **Generate PDF** button. + Plot of the rms fitting based on the LoTSS-DR2, plotted together with the standard theoretical noise, theoretical noise with geometrical correction, and the final model fitting with error bars at 1, 3 and 5 times sigma. + +The final formula obtained from this fitting results in: + +.. math:: + + \Delta S_{eff} = 3.35619407 \cdot cos(90-elev)^{-1.62306591} \cdot \Delta S_{theor} + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Raw data size +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The raw data size of a LOFAR measurement is calculated with a given length of the observation, correlator integration time, number of baselines, number of channels per subband, and number of subbands. A single row in LofarStMan format contains 32-bit sequence numbers (4 bytes), number of channels times 16-bit samples for weight and sigma calculations (2 times number of channels bytes) and 8 times number of channels times float data array (each float data array being 4 bytes). This result in the following used formula: + +.. math:: + + Size_{total} = n_{rows} \times LofarStMan = n_{subbands} \times \left ( (n_{baselines} \cdot \frac{t_{obs}}{t_{integration}}) - n_{baselines} \right ) \times \frac{ 4 + 2 n_{chan} + 4 n_{chan} \cdot 2 \cdot 4 }{1024^3} + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Processed data size +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Depending on which pipeline is chosen, different processed data sized can be returned. Currently, LUCI only has the pre-processing pipeline option. The data size will then be averaged in time, frequency and/or be dysco compressed. The result will be presented in the Results section once the pipeline option is not set to None. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Processing time +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The processing time is calculated based on different hard-coded P/O factors (processing over observing time) per subband derived from commissioning. These factors are then used together with the input set-up (specifically observation time and number of subbands) to give an estimate in hours of how long the processing might take. -**Note** that if you click on the **Generate PDF** button before using the **Calculate** button, LUCI will throw an error. -------------------------------- -Frequently Asked Questions (FAQ) +How to interpret results -------------------------------- -**In what framework is LUCI implemented?** +LUCI's most notable usage is for creating proposals with a good set up and estimates for disk space required and what sensitivity levels an observation can reach. That makes it important to properly interpret the results that are returned and what the user can do to optimise the results for a given science goal. Below are a list of different tips for interpreting LUCI results. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Sensitivities +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Older versions of LUCI only returned theoretical sensitivity values which were often found to be far too optimistic. The effective sensitivity should return a more realistic value based on empirical evidence. It is therefore recommended that users consult that number. Note that this is true for HBA only, so with LBA take the calculated sensitivity as the best case scenario. +Also note that often, even though they have been used in an observation, some pipelines do not include international stations. This does result in a different resolution and therefore sensitivity. It is recommended to therefore also calculate for a set-up without international stations. + +Note that sensitivities are best when observation time and number of subbands are high, however this results in larger data sizes. Additionally targets should ideally be high in the sky to avoid atmospheric effects. At lower frequencies (especially low LBA frequencies) the sky temperature can have significant impact on the data, and that gets even worse as sources approach the horizon. +LUCI warns the user about the validity of the rms values for sources below 30 degrees for LBA and sources below 40 degrees for HBA as the ionosphere starts to severely impact the data quality. Try to keep the mean elevation in mind when planning the observation (date). For targets at lower elevations, it can be recommended to observe in several small time chunks rather than one large observation at low elevation. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Processing time +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +LUCI returns estimates of pipeline processing times (currently only for pre-processing). The pre-processing pipeline is run by the observatory, where the P/O fractions are reflective of the system. However, for other pipelines (e.g. LINC or RAPTHOR), if run by the user on a local machine, these estimates change depending on the local specs. LUCI provides only values from ASTRON's services. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Elevation plot +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The elevation plot is a visual tool to see the visibility profile of the target together with other offending sources. The user can opt for adding calibrators and/or A-team sources to the plot for quick inspection. Combined with the angular distance table the user can be made aware of what sources can influence the data. The elevation plot also shows a thicker line for the entire observation time for the input targets, highlighting the evolution of the source elevation across the observation date. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +LOFAR 2.0 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - LUCI is implemented using `Dash <https://github.com/plotly/dash>`_ which is a simple Python framework for building web applications. Interactive plotting is done using `Plotly <https://github.com/plotly/plotly.py>`_. +LUCI is currently hardwired for regular LOFAR observations in interferometric mode. For LOFAR2.0 users that want to utilise LUCI, please refer to `this page <https://support.astron.nl/confluence/display/ROSDC/Usage+of+LUCI+for+LOFAR2.0+proposers>`_. -**Can I contribute to LUCI?** - Sure! The source code of LUCI is available on `GitHub <https://github.com/scisup/LOFAR-calculator>`_. If you have an interesting feature, feel free to submit a pull request to this repository. .. rubric:: Footnotes -.. [#f1] This chapter was written by `Sarrvesh Sridhar <mailto:sarrvesh@astron.nl>` and is manitained by `Sander ter Veen <mailto:veen@astron.nl>`_. +.. [#f1] This chapter was written by `Sarrvesh Sridhar <mailto:sarrvesh@astron.nl>`_, updated by `Noa Peters <mailto:peters@astron.nl>`_ and is maintained by `Sander ter Veen <mailto:veen@astron.nl>`_.