L2SS-434: Added control, monitoring intro

docs/source/control.rst

-Controlling the station
-============================
+Control access
+========================
+
+The main API to control the station is through the `Tango Controls https://tango-controls.readthedocs.io/en/latest/` API we expose on port 10000, which is most easily accessed using a `PyTango https://pytango.readthedocs.io/en/stable/client_api/index.html` client.
 
 Jupyter Notebooks
 ------------------------
 
 The station offers Juypyter notebooks On http://localhost:8888, which allow one to interact with the station, for example to set control points, access monitoring points, or to graph their values.
 
-The notebooks provide some predefined variables:
+The notebooks provide some predefined variables, so you don't have to look them up:
 
 .. literalinclude:: ../../docker-compose/jupyter/ipython-profiles/stationcontrol-jupyter/startup/01-devices.py
 
+Note: the Jupyter notebooks use enhancements from the `itango` suite, which provide tab completions, but also the `Device` alias for `DeviceProxy` as was used in the Python examples in the next section.
+
+For example, you can start a new *Station Control* notebook (File->New Notebook->StationControl), and access these devices:
+
+.. image:: jupyter_basic_example.png
+
+
+Python
+------------------------
+
+To access a station from scratch using Python, we need to install some dependencies::
+
+  pip3 install tango
+
+Then, if we know what devices are available on the station, we can access them directly::
+
+  import tango
+  import os
+
+  # Tango needs to know where our Tango API is running.
+  os.environ["TANGO_HOST"] = "localhost:10000"
+
+  # Construct a remote reference to a specific device.
+  # One can also use "tango://localhost:10000/LTS/Boot/1" if TANGO_HOST is not set
+  boot_device = tango.DeviceProxy("LTS/Boot/1")
+
+  # Print the device's state.
+  print(boot_device.state())
+
+To obtain a list of all devices, we need to access the database::
+
+  import tango
+
+  # Tango needs to know where our Tango API is running.
+  import os
+  os.environ["TANGO_HOST"] = "localhost:10000"
+
+  # Connect to the database.
+  db = tango.Database()
+
+  # Retrieve the available devices, excluding any Tango-internal ones.
+  # This returns for example: ['LTS/Boot/1', 'LTS/Docker/1', ...]
+  devices = list(db.get_device_exported("LTS/*"))
+
+  # Connect to any of them.
+  any_device = tango.DeviceProxy(devices[0])
+
+  # Print the device's state.
+  print(any_device.state())

docs/source/devices/devices.rst

 Using Devices
 =============
 
-The station exposes the following devices:
+The station exposes *devices*, each of which is a remote software object that manages part of the station. Each device has the following properties:
+
+- It has a *state*,
+- Many devices manage and represent hardware in the station,
+- It exposes *read-only attributes*, that expose values from within the device or from the hardware it represents,
+- It exposes *read-write attributes*, that allow controlling the functionality of the device, or the hardware it represents,
+- It exposes *properties*, which are fixed configuration parameters (such as port numbers and timeouts).
 
 To access a device, one creates a `Device` object. For example::
 

docs/source/monitoring.rst

+Monitoring
+========================
+
+Each device exposes a list of monitoring points as attributes with the `_R` prefix. These can be accessed interactively from a controle console (such as Jupyter), but that will not scale.
+
+Grafana
+------------------------
+
+We offer `Grafana https://grafana.com/` dashboards on http://localhost:3000 that provide a quick overview of the station's status, including temperatures and settings. Several dashboards are included. An example::
+
+.. image:: grafana_dashboard_1.png
+.. image:: grafana_dashboard_2.png
+
+NOTE: These dashboards are highly subject to change. The above examples provide an impression of a possible overview of the station state.
+
+You are encouraged to inspect each panel (graph) to see the underlying database query and settings. Use the small arrow in the panel's title to get a drop-down menu of options, and select *inspect*. See the Grafana documentation for further information.
+
+The Grafana dashboards are configured with the following data sources:
+
+- *Prometheus*, the time-series database that caches the latest values of all monitoring points (see next section),
+- *Archiver DB*, the database that provides a long-term cache of attributes,
+- *Tango DB*, providing access to device properties (fixed settings),
+- *ELK*, the log output of the devices.
+
+Prometheus
+-------------------------
+
+`Prometheus https://prometheus.io/docs/introduction/overview/` is a low-level monitoring system that allows us to periodically retrieve the values of all the attributes of all our devices, and cache them to be used in Grafana:
+
+- Every several seconds, Prometheus scrapes our `TANGO-Grafana Exporter https://git.astron.nl/lofar2.0/ska-tango-grafana-exporter` (our local fork of https://gitlab.com/ska-telescope/TANGO-grafana.git), collecting all values of all the device attributes (except the large ones, for performance reasons).
+- Prometheus can be queried directly on http://localhost:9090,
+- The query language is `PromQL https://prometheus.io/docs/prometheus/latest/querying/basics/`, which is also used in Grafana to query Prometheus,
+
+Prometheus stores attributes in the following format::
+
+  device_attribute{device="lts/recv/1", dim_x="32", dim_y="0", instance="tango-prometheus-exporter:8000", job="tango", label="RCU_temperature_R", name="RCU_temperature_R", type="float", x="00", y="0"} 
+
+The above describes a single data point and its labels. Each point furthermore has a value (integer) and a timestamp. The following transformations take place:
+
+- For 1D and 2D attributes, each array element is its own monitoring point, with `x` and `y` labels describing the indices. The labels `dim_x` and `dim_y` describe the array dimensionality,
+- Attributes with string values get a `str_value` label describing their value.

docs/source/usage/remote_interfaces.rst

@@ -30,8 +30,3 @@ Futhermore, there are some low-level interfaces:
 +---------------------+--------------+----------------------+-------------------+
 |Log Database         |ElasticSearch |http://localhost:9200 |                   |
 +---------------------+--------------+----------------------+-------------------+
-
-Jupyter: Interactive Scripting
-------------------------------------
-
-The station offers Juypyter notebooks On http://localhost:8888, which allow one to interact with the station, for example to set control points, access monitoring points, or to graph their values.