Merge branch 'L2SS-1186' into 'master'

L2SS-1186: Document services and remove deprecated services

Closes L2SS-1186

See merge request !544

.prepare.gitlab-ci.yml

@@ -9,11 +9,12 @@ build_ci_runner_image:
   script:
     - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
     - |
+      source docker-compose/.env
       if docker pull $CI_REGISTRY_IMAGE/ci-build-runner:$CI_COMMIT_REF_SLUG; then
-        docker build --cache-from $CI_REGISTRY_IMAGE/ci-build-runner:$CI_COMMIT_REF_SLUG --tag $CI_REGISTRY_IMAGE/ci-build-runner:$CI_COMMIT_REF_SLUG docker-compose/ci-runner
+        docker build --cache-from $CI_REGISTRY_IMAGE/ci-build-runner:$CI_COMMIT_REF_SLUG --tag $CI_REGISTRY_IMAGE/ci-build-runner:$CI_COMMIT_REF_SLUG --build-arg SOURCE_IMAGE=${LOCAL_DOCKER_REGISTRY_HOST}/${LOCAL_DOCKER_REGISTRY_USER}/tango-itango:${TANGO_ITANGO_VERSION} docker-compose/ci-runner
       else
         docker pull $CI_REGISTRY_IMAGE/ci-build-runner:latest || true
-        docker build --cache-from $CI_REGISTRY_IMAGE/ci-build-runner:latest --tag $CI_REGISTRY_IMAGE/ci-build-runner:$CI_COMMIT_REF_SLUG docker-compose/ci-runner
+        docker build --cache-from $CI_REGISTRY_IMAGE/ci-build-runner:latest --tag $CI_REGISTRY_IMAGE/ci-build-runner:$CI_COMMIT_REF_SLUG --build-arg SOURCE_IMAGE=${LOCAL_DOCKER_REGISTRY_HOST}/${LOCAL_DOCKER_REGISTRY_USER}/tango-itango:${TANGO_ITANGO_VERSION} docker-compose/ci-runner
       fi
     - docker push $CI_REGISTRY_IMAGE/ci-build-runner:$CI_COMMIT_REF_SLUG  # push the image
     - |

README.md

@@ -6,7 +6,8 @@
 
 # Tango Station Control
 
-Station Control software related to Tango devices.
+Station Control software related to Tango devices. \
+[Please consult the online documentation!](https://readthedocs.org/projects/lofar20-station-control/badge/?version=latest)
 
 # Index
 
@@ -14,7 +15,7 @@ Station Control software related to Tango devices.
     * [Prerequisites](#prerequisites)
     * [Bootstrap](#bootstrap)
 * [User documentation (ReadTheDocs (Sphinx / ReStructuredText))](tangostationcontrol/docs/README.md)
-* [Docker compose documentation](docker-compose/README.md)
+* [Docker compose & station services documentation](docker-compose/README.md)
     * [Timescaledb](docker-compose/timescaledb/README.md)
     * [Jupyter startup files](docker-compose/jupyterlab/ipython-profiles/stationcontrol-jupyter/startup/README.md)
     * [Tango Prometheus exporter](https://git.astron.nl/lofar2.0/ska-tango-grafana-exporter)

docker-compose/README.md

-# Docker Compose
+# Docker Compose & Station Services
 
 Documentation on how the LOFAR station control software utilizes docker-compose.
 This documentation is intended for developers listing strategies and their
 respective advantages and disadvantages. In addition, this documentation
 contains developer expectations that they should uphold to.
 
-## Image tagging and change detection
+## Index
 
-Preventing unnecessary builds of docker images reduces build times and increases
-iteration speed. In order to achieve this the project requires properly tagged
-images and mechanisms for change detection.
+1. [Station services]()
+   1. [Types of container services and specific strategies](#types-of-containers-and-specific-strategies)
+2. [HDB++ image updates](#hdb-image-updates)
+3. [Gitlab CI/CD](#gitlab-cicd)
+   1. [Image tagging and change detection](#image-tagging-and-change-detection)
+      1. [Setup and maintenance](#setup-and-maintenance)
+   2. [Gitlab CI phases](#gitlab-ci-phases)
 
-For change detection the system relies on git. Git is used to determine the
-directories and files that have changes between the current and previous commit.
-All image related change detection mechanisms are based on this difference.
+## Station Services
 
-Using docker cache within the dind service is impractical see:
-https://gitlab.com/gitlab-org/gitlab-foss/-/issues/17861
+The station offers different services, this documentation provides
+names and an overview of their behavior as well as if these services
+are used in production.
 
-### Types of containers and specific strategies.
+- [Devices \[external\]](https://lofar20-station-control.readthedocs.io/en/latest/devices/overview.html)
+- Simulators
+  - [sdptr \[external\]](https://git.astron.nl/lofar2.0/sdptr)
+    - [sdptr-sim](sdptr-sim.yml)
+  - [pypcc \[external\]](https://git.astron.nl/lofar2.0/pypcc)
+    -  [pypcc-sim-base](pypcc-sim-base/Dockerfile)
+      - [apsct-sim](apsct-sim.yml)
+      - [apspu-sim](apspu-sim.yml)
+      - [ccd-sim](ccd-sim.yml)
+      - [recv-sim](recv-sim.yml)
+      - [unb2-sim](unb2-sim.yml)
+- Base images
+  - [ci-runner](ci-runner/Dockerfile)
+  - [lofar-device-base](lofar-device-base.yml)
+- Services
+  - databases
+    - dsconfig
+    - [timescaledb](timescaledb/README.md)
+      - archiver-timescale
+      - hbdpp
+      - hbdpp-cm (ConfigurationManager)
+      - hbdpp-es (EventSubscriber)
+    - prometheus
+  - webservers / user interfaces
+    - jupyterlab
+    - [grafana](grafana/README.md)
+    - [http-json-schemas](http-json-schemas/README.md)
+  - logging / monitoring
+    - prometheus-node-exporter
+    - [tango-prometheus-exporter](tango-prometheus-exporter/README.md)
+    - logstash (grafana-logstash-output-loki)
+    - loki
+  - tango
+    - itango
+    - tango-rest
+
+### Types of containers and specific strategies
 
 - Devices
 - Simulators
@@ -41,22 +80,49 @@ Services, same mechanism as devices.
 
 ### HDB++ image updates
 
-The hdbpp Docker image is used as a base image for the hdbppts-cm (ConfigurationManager) and hdbppts-es (EventSubscriber)
-images. If one is developing on a branch and any updates is made in hdbpp/Dockerfile, those won't be automatically picked up from hdbppts-cm and hdbppts-es, because the argument SOURCE_IMAGE in the docker-compose yml file always refers to the remote hdbpp image in the repository. 
-
-A temporary workaround for locally testing on these archiving containers is the following:
-
-- Stop and remove any running hdbpp* container
-- In the archiver-timescale.yml file, replace the hdbppts-cm and hdbppts-es SOURCE_IMAGE tag 'latest' with the branch name 
-(e.g. SOURCE_IMAGE: ${LOCAL_DOCKER_REGISTRY_HOST}/${LOCAL_DOCKER_REGISTRY_USER}/hdbpp:l2ss-new-branch)
-- Rebuild all the hdbpp* container (make build hdbpp hdbppts-cm hdbppts-es), and then start them
+The hdbpp Docker image is used as a base image for the `hdbppts-cm`
+(ConfigurationManager) and `hdbppts-es` (EventSubscriber)
+images. If one is developing on a branch and any updates is made
+in hdbpp/Dockerfile, those won't be automatically picked up from `hdbppts-cm`
+and `hdbppts-es`, because the argument `SOURCE_IMAGE` in the docker-compose
+yml file always refers to the remote `hdbpp` image in the repository.
+
+A temporary workaround for locally testing on these archiving containers
+is the following:
+
+- Stop and remove any running `hdbpp*` container
+- In the archiver-timescale.yml file, replace the `hdbppts-cm` and `hdbppts-es`
+  `SOURCE_IMAGE` tag 'latest' with the branch name
+  (e.g. `SOURCE_IMAGE: ${LOCAL_DOCKER_REGISTRY_HOST}/${LOCAL_DOCKER_REGISTRY_USER}/hdbpp:l2ss-new-branch`)
+- Rebuild all the hdbpp* container (`make build hdbpp hdbppts-cm hdbppts-es`),
+  and then start them (`make start hdbpp hdbppts-cm hdbppts-es`)
 - Test the new features
 
-After the branch has been correctly developed, tested, the merge has been approved, and the new images have been built on the repository:
-- Put back 'latest' tag on the archiver-timescale.yml file, replacing the branch name
+After the branch has been correctly developed, tested, the merge has been
+approved, and the new images have been built on the repository:
+- Put back 'latest' tag on the `archiver-timescale.yml` file, replacing the branch name
 - Merge on master
 
-### Setup and maintenance
+## Gitlab CI/CD
+
+1. [Image tagging and change detection](#image-tagging-and-change-detection)
+    1. [Setup and maintenance](#setup-and-maintenance)
+2. [Gitlab CI phases](#gitlab-ci-phases)
+
+### Image tagging and change detection
+
+Preventing unnecessary builds of docker images reduces build times and increases
+iteration speed. In order to achieve this the project requires properly tagged
+images and mechanisms for change detection.
+
+For change detection the system relies on git. Git is used to determine the
+directories and files that have changes between the current and previous commit.
+All image related change detection mechanisms are based on this difference.
+
+Using docker cache within the dind service is impractical see:
+https://gitlab.com/gitlab-org/gitlab-foss/-/issues/17861
+
+#### Setup and maintenance
 
 All behavioral logic to orchestrate change detection and image pushing can be
 found in the sbin/tag_and_push_docker_images.sh script as well as the

docker-compose/ci-runner/Dockerfile

-FROM git.astron.nl:5000/lofar2.0/tango/tango-itango:9.3.10
+ARG SOURCE_IMAGE
+FROM ${SOURCE_IMAGE}
 
 RUN echo 'tzdata tzdata/Areas select Etc' | sudo debconf-set-selections
 RUN echo 'tzdata tzdata/Zones/Etc select UTC' | sudo debconf-set-selections

docker-compose/device-configuration.yml

@@ -60,4 +60,4 @@ services:
     stop_signal: SIGINT # request a graceful shutdown of Tango
     stop_grace_period: 2s
     depends_on:
-      - schemas
+      - http-json-schemas

docker-compose/device-observation-control.yml

@@ -59,4 +59,4 @@ services:
     stop_signal: SIGINT # request a graceful shutdown of Tango
     stop_grace_period: 2s
     depends_on:
-      - schemas
+      - http-json-schemas

docker-compose/schemas.yml → docker-compose/http-json-schemas.yml

@@ -6,22 +6,22 @@
 # created Observation devices.
 #
 # Defines:
-#   - schemas: LOFAR2.0 station
+#   - http-json-schemas: LOFAR2.0 station
 #
 version: '2.1'
 
 services:
-  schemas:
+  http-json-schemas:
     build:
-      context: schemas
-    container_name: schemas
+      context: http-json-schemas
+    container_name: http-json-schemas
     networks:
       - control
     # set the hostname, otherwise duplicate device registrations result every
     # time the hostname changes as the container is restarted.
-    hostname: schemas
+    hostname: http-json-schemas
     environment:
-      - NGINX_HOST=schemas
+      - NGINX_HOST=http-json-schemas
       - NGINX_PORT=80
     ports:
       - 9999:80

docker-compose/http-json-schemas/README.md

+# HTTP JSON Schemas
+
+Service to serve JSON schema files over HTTP. Used by configuration manager device.

docker-compose/schemas/definitions/station-configuration.json → docker-compose/http-json-schemas/definitions/station-configuration.json

 {
   "$schema": "http://json-schema.org/draft-07/schema#",
-  "$id": "http://schemas/station-configuration.json",
+  "$id": "http://http-json-schemas/station-configuration.json",
   "title": "Tango device generator JSON canonical format",
   "type": "object",
   "additionalProperties": false,

docker-compose/integration-test.yml

@@ -14,7 +14,7 @@ services:
       context: .
       dockerfile: ci-runner/Dockerfile
       args:
-        SOURCE_IMAGE: ${LOCAL_DOCKER_REGISTRY_HOST}/${LOCAL_DOCKER_REGISTRY_USER}/ci-runner:latest
+        SOURCE_IMAGE: ${LOCAL_DOCKER_REGISTRY_HOST}/${LOCAL_DOCKER_REGISTRY_USER}/tango-itango:${TANGO_ITANGO_VERSION}
     container_name: integration-test
     networks:
       - control

docker-compose/loki.yml

@@ -23,30 +23,3 @@ services:
       - "3100:3100"
     command: -config.file=/etc/loki/local-config.yaml
     restart: unless-stopped
-
-  promtail:
-    image: promtail
-    build:
-        context: promtail
-        args: 
-            SOURCE_IMAGE: grafana/promtail:main
-    container_name: promtail
-    logging:
-      driver: "json-file"
-      options:
-        max-size: "100m"
-        max-file: "10"
-    volumes:
-      - /var/log:/var/log
-      - lofar_log:/var/lofarlog
-      # Setting up the Docker Driver Client for Grafana requires a mapping with the O.S. Docker Data Path
-      # f.e. in Windows-WSL -> /mnt/wsl/docker-desktop-data/version-pack-data/community/docker:/var/lib/docker
-    command: -config.file=/etc/promtail/config.yml
-    networks:
-      - control
-    ports:
-      - "9080:9080"
-    restart: unless-stopped
-
-volumes:
-  lofar_log:

docker-compose/prometheus-node-exporter.yml

@@ -12,11 +12,6 @@ services:
   prometheus-node-exporter:
     image: prom/node-exporter
     container_name: prometheus-node-exporter
-    logging:
-      driver: "json-file"
-      options:
-        max-size: "100m"
-        max-file: "10"
     network_mode: host # run on the host to be able to access host network statistics
     logging:
       driver: syslog

docker-compose/prometheus.yml

@@ -18,11 +18,6 @@ services:
     build:
       context: prometheus
     container_name: prometheus
-    logging:
-      driver: "json-file"
-      options:
-        max-size: "100m"
-        max-file: "10"
     networks:
       - control
     extra_hosts:

docker-compose/promtail/Dockerfile

-ARG SOURCE_IMAGE
-FROM ${SOURCE_IMAGE}
-
-# Provide our promtail config
-COPY config.yml /etc/promtail/

docker-compose/promtail/README.md

-# Promtail
-
-Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance.
-
-## Usage and configuration
-
-Promtail can only be configured to scrape logs from a file, pod, or journal. 
-
-Since we do not save our TANGO device logs to file, it currently appears inconvenient using it for our lofar devices logs. 
-
-It can rather be used for systemlogs, server logs saved to disk, and docker container logs (but, in this case, GrafanaLabs have developed a tailored Docker plugin that allows to send logs directly to Loki - [https://grafana.com/docs/loki/latest/clients/docker-driver/])  
-
-Since our devices use a python log handler, this last one can be configured to send logs directly to Loki without passing through Promtail [https://medium.com/geekculture/pushing-logs-to-loki-without-using-promtail-fc31dfdde3c6]
-
-## Using Docker Driver Client
-
-To get the Docker Driver working, the steps are:
-
-* Install the Docker Driver Plugin
-  `docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions`
-  see [https://grafana.com/docs/loki/latest/clients/docker-driver/]
-* Change the default Logging Docker Driver, i.e. update the Docker daemon.json file
-  see [https://grafana.com/docs/loki/latest/clients/docker-driver/configuration/)]
-* Restart the Docker daemon and rebuild all the containers

docker-compose/promtail/config.yml

-# Configures the server for Promtail.
-server:
-  http_listen_port: 9080
-  grpc_listen_port: 0
-
-# Describes how to save read file offsets to disk
-positions:
-  filename: /tmp/positions.yaml
-
-# Describes how Promtail connects to multiple instances
-# of Grafana Loki, sending logs to each.
-# WARNING: If one of the remote Loki servers fails to respond or responds
-# with any error which is retryable, this will impact sending logs to any
-# other configured remote Loki servers.  Sending is done on a single thread!
-# It is generally recommended to run multiple Promtail clients in parallel
-# if you want to send to multiple remote Loki instances.
-clients:
-  - url: http://loki:3100/loki/api/v1/push  # Loki
-
-# Configures how Promtail can scrape logs from a series of targets 
-# using a specified discovery method
-
-# Local machine logs
-scrape_configs:
-- job_name: system
-  static_configs:
-  - targets:
-      - localhost
-    labels:
-      job: varlogs
-      __path__: /var/log/*log
-
-# Docker logs
-- job_name: docker
-  pipeline_stages:
-    - docker: {}
-  static_configs:
-    - labels:
-        job: docker
-        __path__: /var/lib/docker/containers/*/*-json.log
-
-# Tango logs (disabled)
-- job_name: lofar
-  static_configs:
-  - targets:
-      - localhost
-    labels:
-      job: lofar
-      __path__: /var/lofarlog/*
-