Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
  • L2SS-2154-migrate-common-to-lotus
  • L2SS-1509-move-grafana-rpc-to-opah
  • L2SS-2153--Improve-Error-Handling
  • L2SS-2153-Add-Grpc-Gateway-support
  • L2SS-1970-apsct-lol
  • DNM-pytango10.0.1rc1-test
  • fixes-for-black-25.0
  • v0.39.7-backports
  • westerbork-test-setup
  • L2SS-1970
  • merge-event-loops
  • add-old-statistics-attributes
  • L2SS-1957-add-support-for-new-pcon
  • L2SS-2008-unreliable-change-events
  • L2SS-2032-skip-consul-agent
  • pytango-10rc4
  • L2SS-1751-syslog_port_for_pis
  • L2SS-1902
  • release-grafana-fix-alarms
  • 0.47.0
  • v0.45.9 protected
  • v0.45.9-dev1 protected
  • v0.39.7-3 protected
  • v0.39.7-2 protected
  • v0.37.9 protected
  • v0.45.6 protected
  • jupyter-collaboration-test
  • v0.45.4 protected
  • v0.45.2 protected
  • v0.45.1 protected
  • v0.44.9 protected
  • v0.44.3-rc1 protected
  • l2ss-2062-test6
  • l2ss-2062-test5
  • l2ss-2062-test4
  • l2ss-2062-test3
  • l2ss-2062-test2
  • l2ss-2062-test1
  • v0.43.5 protected
40 results
Blame Permalink
Code owners
Assign users and groups as approvers for specific file changes. Learn more.
README.md 2.89 KiB

Docker Compose

Documentation on how the LOFAR station control software utilizes docker-compose. This documentation is intended for developers listing strategies and their respective advantages and disadvantages. In addition, this documentation contains developer expectations that they should uphold to.

Image tagging and change detection

Preventing unnecessary builds of docker images reduces build times and increases iteration speed. In order to achieve this the project requires properly tagged images and mechanisms for change detection.

For change detection the system relies on git. Git is used to determine the directories and files that have changes between the current and previous commit. All image related change detection mechanisms are based on this difference.

Using docker cache within the dind service is impractical see: https://gitlab.com/gitlab-org/gitlab-foss/-/issues/17861

Types of containers and specific strategies.

  • Devices
  • Simulators
  • Base images
  • Services

Devices, these are detected by changes to the .yml file or directory of the respective service inside the docker-compose directory.

Simulators, Since the source code for simulators is maintained by other teams we can not accurately determine from our repository if the simulator has changed. Instead, the images is build by the teams their respective CI pipelines. We simply pull these images as base images.

Base images, these are detected by changes to the .env file in the docker-compose directory. When changed they will be downloaded from the remote registry and uploaded to our own using matching tags.

Services, same mechanism as devices.

Setup and maintenance

All behavioral logic to orchestrate change detection and image pushing can be found in the sbin/tag_and_push_docker_images.sh script as well as the .gitlab-ci.yml. The shell script relies on the fact that each .yml file in the docker-compose directory corresponds to one image.

Gitlab CI phases

Docker images are managed in three phases. First is remote image storing, second is image building and change detection with finally image pulling.

Remote images are downloaded and stored on the local registry when the .env file for docker-compose has changes.

Local images are build when either the files in the base context directory change or if the docker compose file itself has changes. See the gitlab-ci.yml for how these changes are detected. All local images will be rebuild and tagged latest when a tagged commit is pushed to master.

Local images download the latest image from the registry as cache unless it is a tagged commit on master.

Finally, the integration test downloads all images from the registry either tagged with the current pipeline or with latest. Should both tags be unavailable than the integration test fails. Not all images are needed for the integration test. See sbin/tag_and_push_docker_image.sh for how these images are differentiated.