diff --git a/docs/source/elk_last_hour.png b/docs/source/elk_last_hour.png new file mode 100644 index 0000000000000000000000000000000000000000..d6f2a73c9ba754a5a6d5aeece1382906040acb15 Binary files /dev/null and b/docs/source/elk_last_hour.png differ diff --git a/docs/source/elk_log_fields.png b/docs/source/elk_log_fields.png new file mode 100644 index 0000000000000000000000000000000000000000..c5774931f23933be6033e396220b2459409b1def Binary files /dev/null and b/docs/source/elk_log_fields.png differ diff --git a/docs/source/index.rst b/docs/source/index.rst index 2705e130a3536d229ac6d6ff0626c93f29909ddb..b5281d49b2d0c08fcf3394e92858ae03d4c0e405 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -16,6 +16,7 @@ Welcome to LOFAR2.0 Station Control's documentation! devices/recv control monitoring + logs Indices and tables diff --git a/docs/source/logs.rst b/docs/source/logs.rst new file mode 100644 index 0000000000000000000000000000000000000000..de6ca454325bf966240b99d49cf4694d172dcddf --- /dev/null +++ b/docs/source/logs.rst @@ -0,0 +1,24 @@ +Logs +================== + +The devices, and the docker containers in general, produce logging output. The easiest way to access the logs of a specific container is to ask docker directly. For example, to access and follow the most recent logs of the `device-sdp` container, execute on the host:: + + docker logs -n 100 -f device-sdp + +This is mostly useful for interactive use. + +ELK stack +------------------ + +To monitor the logs remotely, or to browse older logs, use the *ELK stack* that is included on the station, and served on http://localhost:5601. ELK, or ElasticSearch + Logstash + Kibana, is a popular log collection and querying system. Currently, the following logs are collected in our ELK installation: + +- Logs of all devices, +- Logs of the Jupyter notebook server. + +If you browse to the ELK stack (actually, it is Kibana providing the GUI), your go-to is the *Discover* view at http://localhost:5601/app/discover. There, you can construct (and save, load) a dashboard that provides a custom view of the logs. For example, `this dashboard http://localhost:5601/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-60m,to:now))&_a=(columns:!(extra.tango_device,level,message),filters:!(),index:'1e8ca200-1be0-11ec-a85f-b97e4206c18b',interval:auto,query:(language:kuery,query:''),sort:!())` shows the logs of the last hour, with some useful columns added to the default timestamp and message columns. Expand the time range if no logs appear, to look further back. You should see something like: + +.. image:: elk_last_hour.png + +ELK allows you to filter, edit the columns, and a lot more. We enrich the log entries with several extra fields, for example the device that generated it, and stack traces if available. Click on the `>` before a log entry and the information expands, showing for example: + +.. image:: elk_log_fields.png