Rewrite of CI/CD pipeline

Problem description

The current CI/CD pipeline has a number of issues, when it comes to building a release. The most important ones are:

• The git tag is created on the release branch. As a result, this tag is not visible when the default branch is checked out, which causes git describe --tags to not return the desired result.
• There is no roll-back mechanism. Even if the pipeline fails, the git tag remains. This causes subsequent pipeline runs to fail when it tries to create an already existing git tag.
• The dockerPull URI in the CWL steps is not updated before the docker image is built, thus causing the image itself to contain the wrong URI. This is not so much of an issue, because you usually don't want to start LINC inside a docker container using docker, because running docker-in-docker poses its own challenges. It does however feel awkward.

A few issues that have only recently been addressed, and basically laid down the foundation for the work done in this MR:

• The deploy stage would already start once the build was complete; it should run only after all tests have passed. This issue has been resolved in !209 (merged) but has been present for a long while.
• The deploy_release job no longer worked, because it relied on being able to push to the git repository without credentials. This issue has been fixed in !223 (merged).

Solution

To resolve the remaining issues, the following changes were made:

• Two new stages, initialize and finalize, have been added. The jobs in these stages are only run when a release is created. Note that every branch whose name starts with releases/ is considered a release.
• The initialize stage contains one job, prepare_release.
  • Job prepare_release updates the dockerPull URI in the CWL steps, pushes those changes to the release branch, and adds a git tag (using as tag name the part after releases/ in the branch name). If the git tag already exists, the job will bail out with an error, since we do not want to clobber existing tags.
• The finalize stage contains two jobs, rollback_release and finalize_release.
  • Job finalize_release will only run if all the previous jobs succeeded. It will undo the changes made to the CWL steps in the prepare_release job, and will merge the release branch into the default branch. By doing this, we ensure that:
    • the release tag also become visible on the default branch, so that git describe --tags does return the expected result;
    • any bug-fixes made on the release branch will automatically be merged into the default branch, thus reducing the risk of regressions.
  • Job rollback_release will only run if any of the previous jobs failed. It will remove the git tag, if and only if it was created by the prepare_release job. Changes made on the release branch are not rolled back, because that's not really necessary.

With all these changes, the pipeline should now be ready for the future 😃