Tango Station Control

Station Control software related to Tango devices.
Please consult the online documentation!

Index

• Installation
  • Prerequisites
  • Bootstrap
• User documentation (ReadTheDocs (Sphinx / ReStructuredText))
• Docker compose & station services documentation
  • Jupyter startup files
  • Tango Prometheus exporter
• Developer Documentation
  • Deployments
  • Versioning
• Source code documentation
  • Attribute wrapper documentation
  • Adding a new tango device
  • HDF5 statistics
• Unit tests
• Integration tests
• Release Notes

Installation

Prerequisites

After checking out this repo, be sure to also check out the submodules for this git needs to be installed:

git submodule init
git submodule update

You will also need:

• docker
• docker-compose
• make
• bash
• dig (dnsutils package on ubuntu/debian)
• jq
• wget
• hexdump (bsdextrautils package on ubuntu/debian)
• gcc
• g++
• shellcheck
• graphviz
• pkg-config
• python3.10
• pip (python3-pip)
• tox (python3-tox)

On Ubuntu / Debian based systems these can typically be installed by executing: ./sbin/install-deps-ubuntu-debian.sh

Of these docker-compose must be at least 2.0. Alternatively, tox can be installed through pip using pip install tox, at least version 4.0 must be used.

Finally, running unit tests relies on availability of casacore data see: lofar-device-base Dockerfile for details.

Development and Testing

Setup environment variables

This should be done prior to any form of development or use and needs to be repeated everytime the environment variables are reset.

source setup.sh

Start dev environment

For local development a dev environment is needed. To setup this environment run

source ./sbin/prepare_dev_env.sh

This will install jumppad, if not present yet as well as creating the docker volume needed to simulate the station nomad cluster.

Afterwards run

jumppad up infra/dev/all.hcl

to start the dev environment including tango.

Nomad is now available at http://localhost:4646/

The development environment and its network can be stopped using

jumppad down

The entire dev environment needs to be torn down this way prior to running integration tests! You can check for the existence of this 'station' network using docker network ls.

If you think the network is lingering as an error you can use docker network rm station 9000-station to remove it.

Inspect services

Now we can start all containers, and make sure everything is up:

make start
make status

If not, you can inspect why with docker logs <container>. The containers will automatically be restarted on reboot or failure. Stop them explicitly to bring them down (make stop <container>).

Most notably, you will have web interfaces available at:

• http://localhost:8888 (Jupyter Lab)
• http://localhost:3000 (Grafana).

Running integration tests

Notably, before running integration you should NOT run jumppad up or source sbin/prepare_dev_env.sh but still need to have run source setup.sh.

Also ensure the station networks do not exist or remove them using docker network remove station 9000-station.

You can run the integration using

# source setup.sh
unset TANGO_HOST
./sbin/run_integration_test.sh

Versioning

When changing behavior a new version for Lofar Station Control should be reserved. To do this please follow semantic versioning.

Next change the version in the following places:

1. The VERSION file.
2. Add a Release note for the given version.
3. Once the merge requests is merged to master, add a tag with the version such as v0.3.2 or v0.3.3 etc
4. The tag can be deployed to the environments, manually, through https://git.astron.nl/lofar2.0/tango/-/tags

Release Notes

• 0.26.6 Replace pysnmp with pysnmplib
• 0.26.3 Fix minor deployment issues
• 0.26.2 Fix minor deployment issues
• 0.26.1 Code quality: comply with flake8 7.0.0
• 0.26.0 Expose all attributes as metrics by device server. Renamed sst.integration_interval_R -> sst.sst_integration_interval_R.
• 0.25.3 Add environment decorators to devices in integration tests
• 0.25.2 Increase prometheus retention time
• 0.25.1 Add RCU_DTH_setup command in RECV devices
• 0.25.0 Pull IERS tables from object store
• 0.24.8 Push docker images to own registry and pull from there for deployments
• 0.24.6 Bugfix: deploy tagged version instead of latest
• 0.24.5 Bugfix: install 'tango' in Jupyter Lab again
• 0.24.4 Apply calibration values in reverse order if subband frequencies are decreasing
• 0.24.3 Tune python logging format and parsing
• 0.24.2 Ensure code base is PyTango 9.5 compatible
• 0.24.1 Let all devices emit basic prometheus metrics
• 0.24.0 Allow multiple antenna fields to be used in single observation, This renames the Observation device to ObservationField.
• 0.23.0 Migrate execution environment to nomad
• 0.22.0 Split Antennafield in AFL and AFH devices in order to separate Low-Band and High-Band functionalities Removed Antenna_Type_R attribute from antennafield devices
• 0.21.4 Replace ACC-MIB.mib with SP2-MIB.mib source file in PCON device
• 0.21.3 Added DigitalBeam.Antenna_Usage_Mask_R to expose antennas used in beamforming
• 0.21.2 Removed deprecated "Boot" device (use StationManager now)
• 0.21.1 Implement multi project integration downstream pipeline
• 0.21.0 Use radians instead of degrees when interpreting pointings
• 0.20.5 Manage both polarisations in RCU_band_select_R(W), Antenna_Loss_R, and Frequency_Band_RW
• 0.20.4 Collapse AbstractHierarchyDevice and AbstractHierarchy into one class
• 0.20.3 Fix application of Field_Attenuation_R
• 0.20.2 Support only one parent in hierarchies
• 0.20.1 Create an abstract AntennaMapper class which implements behavior of both AntennaToSdpMapper and AntennaToRecvMapper
• 0.20.0 Complete implementation of station-state transitions in StationManager device. Unified power management under power_hardware_on/off(), dropping prepare_hardware(), disable_hardware(). Replaced device.warm_boot() by device.boot().
• 0.19.0 Ensure requirements.txt are installed when using pip install
• 0.18.3 Many configuration fixes in tango device configs, Fixed APS & EC device port mapping, fixed variable initialization in several devices, Fixed XST device going into fault state, prevent UDP packet loss and verify UDP buffer size for XSTs, Fixed several tests due to use of numpy.array in properties, Implement control hierarchy, Version pin PyASN, Fix code coverage for PyTango devices, Fix beam tracker not starting again after being stopped.
• 0.18.2 Fix documentation links in README
• 0.18.1 Various improvements including: better error handling for commands and resolving a configuration issue related to beamlets
• 0.18.0 Expose attribute related to SDP rings such as FPGA_bf_ring_nof_transport_hops_RW_default and FPGA_ring_use_cable_to_next_rn_RW_default
• 0.17.1 Ensure OPCUA devices reconnect automatically if the connection is lost
• 0.17.0 Add Power Hierarchy state transition
• 0.16.2 Add Power_Parent and Parent_Children properties in LOFAR devices
• 0.16.1 AntennaField: Do not put device in FAULT if an attribute cannot be read/written. AntennaField: Avoid archiving HBA-specific attributes for LBA fields.
• 0.16.0 Observation: Removed antenna mask from specification DigitalBeam: Removed beamlet and antenna selection
• 0.15.0 Split recv device into rcu2h and rcu2l and split recv-sim translator into rcu2h-sim and rcu2l-sim
• 0.14.0 Create async device base and make tilebeam and digitalbeam async device servers, allowing for cooperative multitasking and preventing issues with beamtracking.
• 0.13.1 Upgrade PyTango to 9.4.x and ensure it is installed through requirements.txt
• 0.13.0 Remove all archiver-timescale, hdbppts-cm, hdbppts-es functionalities
• 0.12.1 Add AbstractHierarchy and AbstractHierarchyDevice classes and functionality
• 0.12.0 Add Calibration_SDP_Subband_Weights_<XXX>MHz_R attributes to implement HDF5 calibration tables
• 0.11.2 Fix sleep duration in archiver test
• 0.11.1 Fix event unsubscription in TemperatureManager
• 0.11.0 Added StationManager device
• 0.10.0 Add AntennaToSdpMapper and fpga_sdp_info_* mapped attributes in Antennafield device
• 0.9.0 Statistics writer: moved the whole functionality to lofar-station-client repository
• 0.8.0 Statistics writer: HDF5 format overhaul (removed values, added and moved attributes), Statistics writer: Added --field parameter to record statistics of a specific AntennaField, AntennaField: Added RCU_DTH_on_R, RCU_DTH_freq_R(W), RCU_band_select_R, RCU_attenuator_dB_R.
• 0.7.2 Added sdp.subband_frequency_R, antennafield.Frequency_Band_RW, and support for spectral inversion
• 0.7.1 Add restore backup configuration for Configuration device
• 0.7.0 Raised required Python version to 3.10
• 0.6.0 Changed recv.ANT_mask_RW and recv.ANT_mask_R into 32x3 matrices
• 0.5.1 Add loading and updating methods for Configuration device
• 0.5.0 Add Configuration device
• 0.4.1 Fix for missing SDP attributes for spectral inversion
• 0.4.0 Have most containers report health status and add make await command
• 0.3.1 Fix for applying boot device dsconfig
• 0.3.0 Initial version of deployment scripts and functionality
• 0.2.0 Extend Beamlet device with FPGA source address attributes
• 0.1.2 Fix StatisticsClient accessing last_invalid_packet_exception parameter