Alerts on data are generated by Grafana, by periodically polling their underlying queries and triggering an alert once an alarm condition is met. These alerts are forwarded to *Alerta*, in which the user can manage (and annotate) them.
In Grafana, alerts are setup in the left-hand bar under ``Alerting -> Alerting Rules``. Each rule consists of:
In Grafana, alerts can be setup in two ways:
* Attached to a panel: through the ``Alert`` tab when editing a panel,
* Free floating: through the left-hand bar under ``Alerting -> Alerting Rules``.
Each rule consists of:
* A *Query* that selects the results on which to trigger,
* An *Expression* that says when the alert should fire.
* *Expression(s)* that says when the alert should fire.
Grafana is capable of generating multiple *instances* of the same alert, each of which covering one of the query results. This allows us to track different sources of the same alert individually, yet grouped. To do so, we need to retain the *labels* of each result as returned by the query, as each unique set of labels results in a different instance of the alert.
We typically want to trigger on the current value, and retain the labels of the input data, such that the individual lines in the graph are tracked as separate alerts. To do so, setup:
As the "Classical condition" Expression advised by Grafana drops all labels, we need to do something different:
* A Query ``A`` to fetch the data,
* A Reduce Expression ``B`` to select the ``Last`` (=current) value of each line,
* A Reduce Expression ``B`` to select the ``Last`` (=current) values from query A,
* A Math Expression ``C`` that contains the treshold, f.e. ``$B > 0.5``.
Grafana subsequently reevaluates the alert every given interval (10s minimum), and it holds for another interval (10s minimum), the alert will fire. The following *additional details* can additionally be configured to be sent along with the alert:
...
...
@@ -274,4 +281,4 @@ The Alerta stack manages alerts that come from Grafana, and can be accessed thro
Alerts arrive in the ``UNACK`` state, and will alternate between ``UNACK`` and ``RTNUN`` until the user acknowledges the alert. Once acknowledged, the alert will not appear until it is triggered once again.
.. note:: Alerta will generate a message on *Slack* any time an alert is freshly generated (goes from `NORM`` to ``UNACK``).
.. note:: Alerta will generate a message on *Slack* any time an alert is freshly generated (goes from ``NORM`` to ``UNACK``).
