@@ -7,7 +7,17 @@ The Grafana system exposed on http://localhost:3001 allows visualisation of the
* Each dashboard has a series of `panels <https://grafana.com/docs/grafana/latest/panels/>`_, often organised into collapsable *rows*. Each panel contains a specific visualisation, and can have alarms configured on them. The panels are tiled.
* Each panel has a set of *queries*, which describe the data to be visualised, and a single *visualization*, which is how the data is visualised.
The Grafana documentation will help you with using Grafana in general. Also be sure to check out the `webinars and videos <https://grafana.com/videos/>`_ provided by them.
The Grafana documentation will help you with using Grafana in general. Also useful are the following videos and posts:
* `Grafana Dashboard: Monitor CPU, Memory, Disk and Network Traffic Using Prometheus and Node Exporter <https://www.youtube.com/watch?v=YUabB_7H710>`_ [video, 26m], explains how to build dashboards in Grafana,
* `Guide to Grafana 101: Getting Started With (Awesome) Visualizations <https://www.youtube.com/watch?v=oPumWaoNw5s&t=1546s>`_ [video, 37m], explains setting up visualisations in Grafana (using a TimescaleDB data source),
* `Guide to Grafana 101 <https://www.youtube.com/playlist?list=PLsceB9ac9MHTjwvV18QJnPcLrTXm_Q-Ft>`_ [videos], series of videos explaining both beginner and advanced topics,
* `How to build a Prometheus query in Grafana <https://www.youtube.com/watch?v=5VrjOzIOJPw>`_ [video, 4m], explains how to use Prometheus in Grafana,
* `Using query builders for Grafana Loki and Prometheus <https://university.grafana.com/learn/course/83/play/711/video-using-query-builders-for-grafana-loki-and-prometheus;lp=22>`_ [e-learning, 15m], interactively helps you use the Grafana query builders to create Prometheus queries in Grafana,
* `Grafana Alerting: Explore our latest updates in Grafana 9 <https://grafana.com/blog/2022/06/14/grafana-alerting-explore-our-latest-updates-in-grafana-9/>`_ [blog, 6m], explains how Alerting works in Grafana,
* `FlowCharting Getting Started <https://algenty.github.io/flowcharting-repository/STARTED.html>`_ [blog], explains how to use the FlowCharting plugin to animate draw.io drawings in Grafana.
Finally, be sure to check out the `webinars and videos <https://grafana.com/videos/>`_ provided by the Grafana team.
Writing Queries
------------------------------------
...
...
@@ -160,7 +170,14 @@ In which `HBAT_PWR_on_R` represents the power of an HBAT element, which we sum p
The colour is configured in the *Data Layer* by setting `Marker Color` to `Value`, and configuring the `Thresholds` at the bottom for the actual colour to be used for each range of values.
SVG Panels
DrawIO Panels
------------------------------------
The `FlowCharting <https://grafana.com/grafana/plugins/agenty-flowcharting-panel/>`_ plugin allows you to create a drawing in `Draw.io <https://draw.io>`_, and replace values or colour elements according to query results.
See also the `FlowCharting documentation <https://algenty.github.io/flowcharting-repository/STARTED.html>`_.
Advanced: SVG Panels
------------------------------------
It is possible to display an SVG picture, and have areas of it highlight based on query results. We find the `ACE.SVG <https://grafana.com/grafana/plugins/aceiot-svg-panel/>`_ plugin to be useful.
...
...
@@ -223,3 +240,38 @@ An example setup can be constructed as follows:
console.log("refreshed")
Alerting
-------------------------------------------
Alerts on data are generated by Grafana, by periodically polling their underlying queries and triggering an alert once an alarm condition is met. These alerts are forwarded to *Alerta*, in which the user can manage (and annotate) them.
In Grafana, alerts are setup in the left-hand bar under ``Alerting -> Alerting Rules``. Each rule consists of:
* A *Query* that selects the results on which to trigger,
* An *Expression* that says when the alert should fire.
We typically want to trigger on the current value, and retain the labels of the input data, such that the individual lines in the graph are tracked as separate alerts. To do so, setup:
* A Query ``A`` to fetch the data,
* A Reduce Expression ``B`` to select the ``Last`` (=current) value of each line,
* A Math Expression ``C`` that contains the treshold, f.e. ``$B > 0.5``.
Grafana subsequently reevaluates the alert every given interval (10s minimum), and it holds for another interval (10s minimum), the alert will fire. The following *additional details* can additionally be configured to be sent along with the alert:
* ``Dashboard UID``: which dashboard to link to in the alert,
* ``Panel ID``: which panel to link to in the alert,
* ``Severity``: severity of the alert: ``critical``, ``major``, ``minor``, ``warning`` (default).
Alerta
```````````````````````````````````````````
The Alerta stack manages alerts that come from Grafana, and can be accessed through http://localhost:8082 (creds: admin/alerta). The main screen shows you an overview of the *unacknowledged alerts* that were generated by Grafana. Alerta allows an operator to track them using the `ISA 18.2 <http://www.tc.faa.gov/its/worldpac/Standards/isa/ISA_18.2[1].pdf>`_ alarm model, which has the following states per alert:
* ``NORM``: Condition is normal: alarm is not active, all past alarms were acknowledged,
* ``UNACK``: Alarm is active, and has not been acknowledged ("came"),
* ``RTNUN``: Alarm came and went, but has not been acknowledged ("went"),
* ``SHLVD``: Shelved: condition changes are ignored.
Alerts arrive in the ``UNACK`` state, and will alternate between ``UNACK`` and ``RTNUN`` until the user acknowledges the alert. Once acknowledged, the alert will not appear until it is triggered once again.
.. note:: Alerta will generate a message on *Slack* any time an alert is freshly generated (goes from `NORM`` to ``UNACK``).
