diff --git a/tangostationcontrol/docs/source/calibration.rst b/tangostationcontrol/docs/source/calibration.rst
index 9e53efbd326a4f74d68a8ab801662da339cc18ba..c07f0d6b78f85a0e9779cf569fcc26b3a0d31ebd 100644
--- a/tangostationcontrol/docs/source/calibration.rst
+++ b/tangostationcontrol/docs/source/calibration.rst
@@ -48,7 +48,7 @@ We equalise the signals of the different antennas to compensate for the delay an
The *coarse delay compensation* is done in SDP, by delaying all inputs to line up with the latest arriving one. The FPGAs do this through a *sample shift*, in which the samples from each input is delayed a fixed number of samples. At the 200 MHz clock, samples are 5 ns. The sample shift aligns the inputs with a remaining difference of +/- 2.5 ns.
-This remainder is corrected for in the *fine delay compensation*, by shifting the phases of each input backwards. A phase shift is frequency dependent (``-2pi * frequency * delay``), and is thus applied at the higher frequency resolution after creating subbands. The ``FPGA_subband_weights_RW`` in SDP allows us to configure a complex correction factor for each subband from each input. A phase shift ``phi`` is converted into a complex factor through ``cos(phi) + i * sin(phi)``.
+his remainder is corrected for in the *fine delay compensation*, by shifting the phases of each input backwards. A phase shift is frequency dependent (``-2pi * frequency * delay``), and is thus applied at the higher frequency resolution after creating subbands. The ``FPGA_subband_weights_RW`` in SDP allows us to configure a complex correction factor for each subband from each input. A phase shift ``phi`` is converted into a complex factor through ``cos(phi) + i * sin(phi)``.
.. note:: The delay compensation shifts all antenna signals by a fixed amount: the number of samples to delay to line up with the longest cable. Yet we mark those signals as "now" in SDP. This introduces a temporal shift of the order of 200ns. This is deemed acceptable, as after the station FFT (that creates the subbands), we have 5.12ms samples, which is an order of magnitude higher time scale.
@@ -126,23 +126,11 @@ The individual calibration tables for each frequency are provided through:
:type: ``float64[N_antennas * N_pol][N_subbands * VALUES_PER_COMPLEX]``
-Managing Calibration Tables
-----------------------------
-
-The calibration tables for SDP are stored in the HDF5 file format, described at XXX, and easily read and written in Python by using the ``common.calibration_table.CalibrationTable`` class in this package, or with the more generic ``h5py`` Python package. Each file is typically named ``CalTable-CS001-HBA0-150MHz.h5``, and is thus specific for an antenna field and the frequency band used to determine it. Each file contains the subband weights, as well as metadata on how and when they were determined.
-
-The AntennaField device reads these files from disk, maintained in a dedicated Docker volume. New files can be downloaded from a central location on demand, providing the follow functionality:
-
-:Calibration_Table_Base_URL: Property which contains the root URL for the calibration tables. The remote location of a calibration table is f.e. ``{Calibration_Table_Base_URL}/CS001/CalTable-CS001-HBA0-150MHz.h5``.
-
-:download_calibration_tables(): Command to download and apply the latest calibration tables, caching them in the Docker volume.
-
-:calibrate(): Command to apply the calibration tables present in the Docker volume.
Applying Calibration Values
----------------------------
-The following commands in AntennaField upload new calibration values to the signal chain in RECV and SDP:
+The following commands in AntennaField upload new calibration values to the signal chain in RECV and SDP. They are called automatically if the frequency of the signal changes, but can be invoked manually as well:
:calibrate_recv(): Configure ``recv.RCU_attenuator_dB_RW`` for the antennas in the field.
@@ -157,19 +145,50 @@ Celestial & Geodetic Calibration
The ``TileBeam`` and ``DigitalBeam`` devices use `python-casacore <https://casacore.github.io/python-casacore/index.html>`_ to compute the direction of a given pointing with respect to our antennas and reference positions. Casacore in turn uses *measures* tables for the precise measurements of celestial positions, geodetical information, and time calibrations (f.e. leap seconds). These tables need to be installed and periodically updated to maintain the pointing accuracy:
-:measures_directory_R: Directory of the active set of measures tables. The directory name includes the timestamp denoting their age.
+:IERS_timestamp_R: Creation timestamp of the IERS tables.
+
+ :type: ``float64``
+
+Updating Calibration Tables
+=============================
+
+The calibration tables for SDP are stored in the HDF5 file format, described at XXX, and easily read and written in Python by using the ``common.calibration_table.CalibrationTable`` class in this package, or with the more generic ``h5py`` Python package. Each file is typically named ``CalTable-CS001-HBA0-150MHz.h5``, and is thus specific for an antenna field and the frequency band used to determine it. Each file contains the subband weights, as well as metadata on how and when they were determined.
+
+These files are maintained and distributed as follows:
+
+* The source of truth is at http://s3.lofar.net/browser/central-caltables/CS001/,
+* The ``object-replication`` Nomad job on the station periodically downloads them the local S3 store (http://localhost:9001/browser/caltables/CS001),
+* The Calibration device downloads them from the local S3 store every time a recalibration is triggered.
- :type: ``str``
+So, by uploading new calibration tables to the central S3 store, they will be applied automagically in the next observation.
-:measures_directories_available_R: List of installed sets of measures tables.
+Updating IERS Tables
+-----------------------------
- :type: ``str[64]``
+The IERS tables contain the celestial and geodetic models used by the beam devices. These files are maintained and distributed as follows:
-:download_measures(): Download (but do not activate) the latest measures tables from ftp://ftp.astron.nl/outgoing/Measures/WSRT_Measures.ztar. Returns the directory name in which the measures were installed.
+* The latest tables can be downloaded from ftp://ftp.astron.nl/outgoing/Measures/WSRT_Measures.ztar.
+* The source of truth for the stations is at http://s3.lofar.net/browser/central-iers/,
+* The ``object-replication`` Nomad job on the station periodically downloads them the local S3 store (http://localhost:9001/browser/iers),
+* The ``sync-IERS`` Nomad job device periodically downloads them from the local S3 store into the Beam devices.
+* The Beam devices read and apply these tables at the first use after their Device Server starts.
+* *To actually apply the new tables, the Beam device servers need to be restarted*. It is not sufficient to turn the station OFF and ON. An example of how to do this for a core station::
- :returns: ``str``
+ # create proxies to all relevant devices
+ digitalbeam_h0 = DeviceProxy("STAT/DigitalBeam/HBA0")
+ digitalbeam_h1 = DeviceProxy("STAT/DigitalBeam/HBA1")
+ digitalbeam_l = DeviceProxy("STAT/DigitalBeam/LBA")
+ tilebeam_h0 = DeviceProxy("STAT/TileBeam/HBA0")
+ tilebeam_h1 = DeviceProxy("STAT/TileBeam/HBA1")
-:use_measures(dir): Activate the measures tables in the provided directory. This necessitates turning off and restarting the TileBeam device, so the command will always appear to fail. Turn the device back and the selected measures tables will be active.
+ # reset the server on any device to reset all devices of that class
+ digitalbeam_l.ds_restart_server()
+ tilebeam_1.ds_restart_server()
- :returns: ``(does not return)``
+ # reinitialise all devices
+ digitalbeam_h0.boot()
+ digitalbeam_h1.boot()
+ digitalbeam_l.boot()
+ tilebeam_h0.boot()
+ tilebeam_h1.boot()
diff --git a/tangostationcontrol/docs/source/index.rst b/tangostationcontrol/docs/source/index.rst
index 44bb396b53c1b50305f3161a02ec5a166805a462..79f2d4f1e57052c7748def0ef90eb434e01d8d9a 100644
--- a/tangostationcontrol/docs/source/index.rst
+++ b/tangostationcontrol/docs/source/index.rst
@@ -35,6 +35,7 @@ Even without having access to any LOFAR2.0 hardware, you can install the full st
devices/temperature-manager
devices/configure
configure_station
+ protection_control
observing
signal_chain
calibration
diff --git a/tangostationcontrol/docs/source/protection_control.rst b/tangostationcontrol/docs/source/protection_control.rst
new file mode 100644
index 0000000000000000000000000000000000000000..76e4c422acf4de1080744c5e815781d9b7922505
--- /dev/null
+++ b/tangostationcontrol/docs/source/protection_control.rst
@@ -0,0 +1,73 @@
+Protection Control
+=============================
+
+The station provides mechanisms to protect itself (or its environment) from immediate damage. The ProtectionControl device ``STAT/ProtectionControl/1`` monitors key attributes of other devices, and automatically triggers a protective station shutdown if their values exceed preconfigured limits.
+
+
+Observed Attributes
+-----------------------------
+
+ProtectionControl is configured to observe the following attributes:
+
+.. literalinclude:: ../../tangostationcontrol/devices/protection_control.py
+ :dedent:
+ :start-after: START PROTECTION_CONFIG
+ :end-before: END PROTECTION_CONFIG
+
+Protection measures are triggered as follows:
+
+* All devices belonging to the described class are tracked,
+* A subscription is taken on the described attributes,
+* Values that do not fall between ``lower_limit`` and ``upper_limit`` are considered erroneous and are discarded,
+* If a value above ``maximum`` appears, protection measures are triggered,
+* The ``minimal`` value is not used for now.
+
+Protection Measures
+-----------------------------
+
+If ProtectionControl triggers, it calls its `protective_shutdown()` command, which will:
+
+* Expose the (last) attribute that went out of bounds in ``last_threshold_device_attribute_R``,
+* Enable the ``stationmanager.protection_lock_RW``, preventing state transitions that turn on more hardware,
+* Order the StationManager to transition the station to ``HIBERNATE``, turning off most hardware.
+
+ProtectionControl furthermore exposes the enum ``protection_state_R`` to expose the current state of protecting the station:
+
++---------------+---------------------------------------------------+
+| State | Description |
++===============+===================================================+
+| DEACTIVATED | All protection is disabled. |
++---------------+---------------------------------------------------+
+| OFF | No shutdown is ongoing. |
++---------------+---------------------------------------------------+
+| ARMED | Protection is activated. Station is locked. |
++---------------+---------------------------------------------------+
+| ACTIVE | Protection is ongoing. |
++---------------+---------------------------------------------------+
+| TRANSITIONING | Protection is ongoing (going to HIBERNATE). |
++---------------+---------------------------------------------------+
+| FAULT | An unrecoverable error occurred during shutdown. |
++---------------+---------------------------------------------------+
+| ABORTED | Shutdown was aborted. |
++---------------+---------------------------------------------------+
+
+Furthermore, manual intervention is possible through the following commands offered by ProtectionControl:
+
+:protective_shutdown(): Manually trigger a shutdown.
+
+:abort(): Stop an ongoing shutdown.
+
+Unlock Station
+-----------------------------
+
+As long as the ``protection_lock_RW`` is enabled, the station cannot be booted. To unlock the station, run::
+
+ stationmanager = DeviceProxy("STAT/StationManager/1")
+ stationmanager.protection_lock_RW = False
+
+Note that if the attributes monitored by ProtectionControl are still out of bounds, the station will almost instantly lock again. If you really want to disable the protection, and thus risk damaging the station, run::
+
+ protectioncontrol = DeviceProxy("STAT/ProtectionControl/1")
+ protectioncontrol.off()
+
+
diff --git a/tangostationcontrol/tangostationcontrol/devices/protection_control.py b/tangostationcontrol/tangostationcontrol/devices/protection_control.py
index f64118ce29dec0b7e32d9843fe817e9af2239cd9..1194354dd8ff1a42256b314e6e64cf73d01ac5a1 100644
--- a/tangostationcontrol/tangostationcontrol/devices/protection_control.py
+++ b/tangostationcontrol/tangostationcontrol/devices/protection_control.py
@@ -54,6 +54,7 @@ __all__ = ["ProtectionManager"]
class ProtectionControl(LOFARDevice):
"""Station protection control Device for LOFAR2.0"""
+ # START PROTECTION_CONFIG (needed for sphinx)
PROTECTION_CONFIG: protection_config_type = CaseInsensitiveDict(
{
"UNB2": CaseInsensitiveDict(
@@ -103,6 +104,7 @@ class ProtectionControl(LOFARDevice):
),
}
)
+ # END PROTECTION_CONFIG (needed for sphinx)
# Generated one time at process start
METRIC_MAPPING = generate_device_attribute_mapping(PROTECTION_CONFIG)