README.md

# Docker Compose

Documentation on how the LOFAR station control software utilizes docker-compose.
This documentation is intended for developers listing strategies and their
respective advantages and disadvantages. In addition, this documentation
contains developer expectations that they should uphold to.

## Image tagging and change detection

Preventing unnecessary builds of docker images reduces build times and increases
iteration speed. In order to achieve this the project requires properly tagged
images and mechanisms for change detection.

For change detection the system relies on git. Git is used to determine the
directories and files that have changes between the current and previous commit.
All image related change detection mechanisms are based on this difference.

Using docker cache within the dind service is impractical see:
https://gitlab.com/gitlab-org/gitlab-foss/-/issues/17861

### Types of containers and specific strategies.

- Devices
- Simulators
- Base images
- Services

Devices, these are detected by changes to the .yml file or directory of the
respective service inside the docker-compose directory.

Simulators, Since the source code for simulators is maintained by other teams
we can not accurately determine from our repository if the simulator has
changed. Instead, the images is build by the teams their respective CI
pipelines. We simply pull these images as base images.

Base images, these are detected by changes to the .env file in the
docker-compose directory. When changed they will be downloaded from the remote
registry and uploaded to our own using matching tags.

Services, same mechanism as devices.

### HDB++ image updates

The hdbpp Docker image is used as a base image for the hdbppts-cm (ConfigurationManager) and hdbppts-es (EventSubscriber)
images. If one is developing on a branch and any updates is made in hdbpp/Dockerfile, those won't be automatically picked up from hdbppts-cm and hdbppts-es, because the argument SOURCE_IMAGE in the docker-compose yml file always refers to the remote hdbpp image in the repository. 

A temporary workaround for locally testing on these archiving containers is the following:

- Stop and remove any running hdbpp* container
- In the archiver-timescale.yml file, replace the hdbppts-cm and hdbppts-es SOURCE_IMAGE tag 'latest' with the branch name 
(e.g. SOURCE_IMAGE: ${LOCAL_DOCKER_REGISTRY_HOST}/${LOCAL_DOCKER_REGISTRY_USER}/hdbpp:l2ss-new-branch)
- Rebuild all the hdbpp* container (make build hdbpp hdbppts-cm hdbppts-es), and then start them
- Test the new features

After the branch has been correctly developed, tested, the merge has been approved, and the new images have been built on the repository:
- Put back 'latest' tag on the archiver-timescale.yml file, replacing the branch name
- Merge on master 

### Setup and maintenance

All behavioral logic to orchestrate change detection and image pushing can be
found in the sbin/tag_and_push_docker_images.sh script as well as the
.gitlab-ci.yml. The shell script relies on the fact that each .yml file in the
docker-compose directory corresponds to one image.

### Gitlab CI phases

Docker images are managed in three phases. First is remote image storing, second
is image building and change detection with finally image pulling.

Remote images are downloaded and stored on the local registry when the .env
file for docker-compose has changes.

Local images are build when either the files in the base context directory
change or if the docker compose file itself has changes. See the gitlab-ci.yml
for how these changes are detected. All local images will be rebuild and tagged
latest when a tagged commit is pushed to master.

Local images download the latest image from the registry as cache unless it is
a tagged commit on master.

Finally, the integration test downloads all images from the registry either
tagged with the current pipeline or with latest. Should both tags be unavailable
than the integration test fails. Not all images are needed for the integration
test. See sbin/tag_and_push_docker_image.sh for how these images are
differentiated.