diff --git a/docs/source/index.rst b/docs/source/index.rst
index b5281d49b2d0c08fcf3394e92858ae03d4c0e405..54c7e318a0bcde66e441ee335f0f10c65019f015 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -10,8 +10,8 @@ Welcome to LOFAR2.0 Station Control's documentation!
:maxdepth: 2
:caption: Contents:
- usage/installation
- usage/remote_interfaces
+ installation
+ remote_interfaces
devices/devices
devices/recv
control
diff --git a/docs/source/installation.rst b/docs/source/installation.rst
index b2d8f763a79b00042ed777a7a58ecb077db50e51..51e534f31f041694789ffc13ede8e8c4e80c38a1 100644
--- a/docs/source/installation.rst
+++ b/docs/source/installation.rst
@@ -42,7 +42,7 @@ You should see the following state:
If not, you can inspect why with `docker logs <container>`. Note that the containers will automatically be restarted on failure, and also if you reboot. Stop them explicitly to bring them down (`make stop <container>`).
Post-boot Initialisation
-----------------
+---------------------------
The following procedure describes how to initialise the system, which is required after installation and after a system reboot.
diff --git a/docs/source/logs.rst b/docs/source/logs.rst
index de6ca454325bf966240b99d49cf4694d172dcddf..f0a92a386b123c995c7f4b3baa11b590945da25b 100644
--- a/docs/source/logs.rst
+++ b/docs/source/logs.rst
@@ -15,10 +15,14 @@ To monitor the logs remotely, or to browse older logs, use the *ELK stack* that
- Logs of all devices,
- Logs of the Jupyter notebook server.
-If you browse to the ELK stack (actually, it is Kibana providing the GUI), your go-to is the *Discover* view at http://localhost:5601/app/discover. There, you can construct (and save, load) a dashboard that provides a custom view of the logs. For example, `this dashboard http://localhost:5601/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-60m,to:now))&_a=(columns:!(extra.tango_device,level,message),filters:!(),index:'1e8ca200-1be0-11ec-a85f-b97e4206c18b',interval:auto,query:(language:kuery,query:''),sort:!())` shows the logs of the last hour, with some useful columns added to the default timestamp and message columns. Expand the time range if no logs appear, to look further back. You should see something like:
+If you browse to the ELK stack (actually, it is Kibana providing the GUI), your go-to is the *Discover* view at http://localhost:5601/app/discover. There, you can construct (and save, load) a dashboard that provides a custom view of the logs, based on the *index pattern* `logstash-*`. There is a lot to take in, and there are excellent Kibana tutorials on the web.
+
+To get going, use for example `this dashboard <http://localhost:5601/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-60m,to:now))&_a=(columns:!(extra.tango_device,level,message),filters:!(),index:'1e8ca200-1be0-11ec-a85f-b97e4206c18b',interval:auto,query:(language:kuery,query:''),sort:!())>`_, which shows the logs of the last hour, with some useful columns added to the default timestamp and message columns. Expand the time range if no logs appear, to look further back. You should see something like:
.. image:: elk_last_hour.png
ELK allows you to filter, edit the columns, and a lot more. We enrich the log entries with several extra fields, for example the device that generated it, and stack traces if available. Click on the `>` before a log entry and the information expands, showing for example:
.. image:: elk_log_fields.png
+
+Furthermore, statistics from the ELK stack, such as the number of ERROR log messages, are made available as a data source in :doc:`monitoring`.
diff --git a/docs/source/monitoring.rst b/docs/source/monitoring.rst
index faf71d36427f2949992b15fd797ac51a1f103dad..5c5c30c52019ea68b3a907b8b4b858bb27d09d80 100644
--- a/docs/source/monitoring.rst
+++ b/docs/source/monitoring.rst
@@ -6,7 +6,7 @@ Each device exposes a list of monitoring points as attributes with the `_R` pref
Grafana
------------------------
-We offer `Grafana https://grafana.com/` dashboards on http://localhost:3000 that provide a quick overview of the station's status, including temperatures and settings. Several dashboards are included. An example::
+We offer `Grafana <https://grafana.com/>`_ dashboards on http://localhost:3000 that provide a quick overview of the station's status, including temperatures and settings. Several dashboards are included. An example:
.. image:: grafana_dashboard_1.png
.. image:: grafana_dashboard_2.png
@@ -25,17 +25,24 @@ The Grafana dashboards are configured with the following data sources:
Prometheus
-------------------------
-`Prometheus https://prometheus.io/docs/introduction/overview/` is a low-level monitoring system that allows us to periodically retrieve the values of all the attributes of all our devices, and cache them to be used in Grafana:
+`Prometheus <https://prometheus.io/docs/introduction/overview/>`_ is a low-level monitoring system that allows us to periodically retrieve the values of all the attributes of all our devices, and cache them to be used in Grafana:
-- Every several seconds, Prometheus scrapes our `TANGO-Grafana Exporter https://git.astron.nl/lofar2.0/ska-tango-grafana-exporter` (our local fork of https://gitlab.com/ska-telescope/TANGO-grafana.git), collecting all values of all the device attributes (except the large ones, for performance reasons).
+- Every several seconds, Prometheus scrapes our `TANGO-Grafana Exporter <https://git.astron.nl/lofar2.0/ska-tango-grafana-exporter>`_ (our fork of https://gitlab.com/ska-telescope/TANGO-grafana.git), collecting all values of all the device attributes (except the large ones, for performance reasons).
- Prometheus can be queried directly on http://localhost:9090,
-- The query language is `PromQL https://prometheus.io/docs/prometheus/latest/querying/basics/`, which is also used in Grafana to query Prometheus,
+- The query language is `PromQL <https://prometheus.io/docs/prometheus/latest/querying/basics/>`, which is also used in Grafana to query Prometheus,
Prometheus stores attributes in the following format::
- device_attribute{device="lts/recv/1", dim_x="32", dim_y="0", instance="tango-prometheus-exporter:8000", job="tango", label="RCU_temperature_R", name="RCU_temperature_R", type="float", x="00", y="0"}
+ device_attribute{device="lts/recv/1",
+ dim_x="32", dim_y="0",
+ instance="tango-prometheus-exporter:8000",
+ job="tango",
+ label="RCU_temperature_R",
+ name="RCU_temperature_R",
+ type="float",
+ x="00", y="0"}
-The above describes a single data point and its labels. Each point furthermore has a value (integer) and a timestamp. The following transformations take place:
+The above describes a single data point and its labels. The primary identifying labels are `device` and `name`. Each point furthermore has a value (integer) and a timestamp. The following transformations take place:
- For 1D and 2D attributes, each array element is its own monitoring point, with `x` and `y` labels describing the indices. The labels `dim_x` and `dim_y` describe the array dimensionality,
- Attributes with string values get a `str_value` label describing their value.
diff --git a/docs/source/remote_interfaces.rst b/docs/source/remote_interfaces.rst
index 632147db45125ff6dd17918eb0c14d10da1ac8fa..8dea68f92cb69333c9ad20d11575571f07dd87f5 100644
--- a/docs/source/remote_interfaces.rst
+++ b/docs/source/remote_interfaces.rst
@@ -6,11 +6,11 @@ The station provides the following interfaces accessible through your browser (a
+---------------------+---------+----------------------+-------------------+
|Interface |Subsystem|URL |Default credentials|
+=====================+=========+======================+===================+
-|Interactive scripting|Jupyter |http://localhost:8888 | |
+| :doc:`control` |Jupyter |http://localhost:8888 | |
+---------------------+---------+----------------------+-------------------+
-|Monitoring |Grafana |http://localhost:3000 |admin/admin |
+| :doc:`monitoring` |Grafana |http://localhost:3000 |admin/admin |
+---------------------+---------+----------------------+-------------------+
-|Logs |Kibana |http://localhost:5601 | |
+| :doc:`logs` |Kibana |http://localhost:5601 | |
+---------------------+---------+----------------------+-------------------+
Futhermore, there are some low-level interfaces: