Skip to content
Snippets Groups Projects
Commit eb4db91f authored by Hannes Feldt's avatar Hannes Feldt
Browse files

Merge branch 'add-docker-image-build-2' into 'main'

Add docker image build and refactor template pipeline inclusion

See merge request !32
parents 829731d8 f0feadf8
No related branches found
No related tags found
1 merge request!32Add docker image build and refactor template pipeline inclusion
Pipeline #95512 passed
Pipeline: Python Package

#95514

    Pipeline: Python Package

    #95513

      .tox
      build
      *.egg-info
      venv
      .idea
      include: stages:
      - local: "{{cookiecutter.project_slug}}/.gitlab-ci.yml" - prepare
      - build
      - test
      trigger_prepare: trigger_prepare:
      stage: prepare stage: prepare
      ...@@ -7,41 +9,33 @@ trigger_prepare: ...@@ -7,41 +9,33 @@ trigger_prepare:
      strategy: depend strategy: depend
      include: "{{cookiecutter.project_slug}}/.prepare.gitlab-ci.yml" include: "{{cookiecutter.project_slug}}/.prepare.gitlab-ci.yml"
      default: # Generate template instance in my_awesome_app directory
      build-template:
      stage: build
      image: $CI_REGISTRY_IMAGE/ci-build-runner:$CI_COMMIT_REF_SLUG
      # Bootstrap Cookiecutter template to test provided ci pipeline template # Bootstrap Cookiecutter template to test provided ci pipeline template
      before_script: script:
      - python --version # For debugging - python --version # For debugging
      - cookiecutter --no-input --overwrite-if-exists --output-dir . . - cookiecutter --no-input --overwrite-if-exists --output-dir . .
      - cd my_awesome_app - cd my_awesome_app
      - git init - git init
      # cannot use needs, for artifacts on child pipeline so must regenerate template!
      # Override semgrep-sast before script
      sast:
      before_script:
      - python --version # For debugging
      # Override unit test before script
      .run_unit_test_version_base:
      before_script:
      - pip install cookiecutter
      - !reference [default, before_script]
      - python -m pip install --upgrade pip
      - pip install --upgrade tox twine
      # Override artifact directories
      run_unit_tests_coverage:
      artifacts: artifacts:
      reports:
      coverage_report:
      coverage_format: cobertura
      path: my_awesome_app/coverage.xml
      paths: paths:
      - my_awesome_app/htmlcov/* - my_awesome_app/*
      - project.gitlab-ci.yml
      # Override artifact directories # Spawn pipeline using the gitlab-ci.yml from generated template instance
      package_docs: # use project.gitlab.ci.yml for necessary job overrides from this template instance
      stage: package # (due to changes in directories and paths etc)
      artifacts: project-pipeline:
      expire_in: 1w stage: test
      paths: trigger:
      - my_awesome_app/docs/build/* strategy: depend
      \ No newline at end of file include:
      - artifact: my_awesome_app/.gitlab-ci.yml
      job: build-template
      - artifact: project.gitlab-ci.yml
      job: build-template
      variables:
      PARENT_PIPELINE_ID: $CI_PIPELINE_ID
      # Example Python Package # Python Package Template
      ![Build status](https://git.astron.nl/templates/python-package/badges/main/pipeline.svg) ![Build status](https://git.astron.nl/templates/python-package/badges/main/pipeline.svg)
      ![Test coverage](https://git.astron.nl/templates/python-package/badges/main/coverage.svg) ![Test coverage](https://git.astron.nl/templates/python-package/badges/main/coverage.svg)
      An example repository of an CI/CD pipeline for building, testing and publishing a python package. Template to create Python repositories with CI/CD pipeline for building, testing and publishing a python package.
      If you find some missing functionality with regards to CI/CD, testing, linting or something else, feel free to make a merge request with the proposed changes. If you find some missing functionality with regards to CI/CD, testing, linting or something else, feel free to make a merge request with the proposed changes.
      ...@@ -30,5 +30,13 @@ pages to configure Gitlab appropriately: ...@@ -30,5 +30,13 @@ pages to configure Gitlab appropriately:
      1. [Gitlab Repository Configuration](https://git.astron.nl/groups/templates/-/wikis/Gitlab-Repository-Configuration) 1. [Gitlab Repository Configuration](https://git.astron.nl/groups/templates/-/wikis/Gitlab-Repository-Configuration)
      2. [Continuous delivery guideline](https://git.astron.nl/groups/templates/-/wikis/Continuous%20Delivery%20Guideline) 2. [Continuous delivery guideline](https://git.astron.nl/groups/templates/-/wikis/Continuous%20Delivery%20Guideline)
      ## Setup
      Once you have used the template there are some additional steps to fully use this
      repository on Gitlab.
      1. [Cleanup Docker Registry Images](https://git.astron.nl/groups/templates/-/wikis/Cleanup-Docker-Registry-Images)
      2. [Setup Protected Verson Tags](https://git.astron.nl/groups/templates/-/wikis/Setting-up-Protected-Version-Tags)
      ## License ## License
      This project is licensed under the Apache License Version 2.0 This project is licensed under the Apache License Version 2.0
      FROM python:3.12 FROM python:3.12
      RUN python -m pip install --upgrade pip RUN python -m pip install --upgrade pip
      RUN pip install --upgrade cookiecutter tox twine RUN python -m pip install --upgrade cookiecutter tox twine
      # This file overrides all the jobs defined in {{cookiecutter.project_slug}}/.gitlab.ci-yml
      # this is to ensure they depend on the template installation artifact of the root
      # .gitlab-ci.yml from job `build-template`
      # The generated gitlab-ci.yml from this `build-template` job is used for the actual
      # trigger include to prevent including jobs that still contain template arguments
      trigger_prepare:
      rules:
      - if: $CI_PIPELINE_SOURCE == "parent_pipeline"
      when: never
      default:
      # Bootstrap Cookiecutter template to test provided ci pipeline template
      before_script:
      - cd my_awesome_app
      run_black:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      run_flake8:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      run_pylint:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      sast:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      dependency_scanning:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      secret_detection:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      .run_unit_test_version_base:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      # Run all unit tests for Python versions except the base image
      run_unit_tests:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      # Run code coverage on the base image thus also performing unit tests
      run_unit_tests_coverage:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      artifacts:
      reports:
      coverage_report:
      coverage_format: cobertura
      path: my_awesome_app/coverage.xml
      paths:
      - my_awesome_app/htmlcov/*
      package_files:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      artifacts:
      expire_in: 1w
      paths:
      - my_awesome_app/dist/*
      package_docs:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      artifacts:
      expire_in: 1w
      paths:
      - my_awesome_app/docs/build/*
      docker_build:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      - package_files
      before_script:
      - cd my_awesome_app
      run_integration_tests:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      - package_files
      publish_on_gitlab:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      - package_files
      publish_on_test_pypi:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      - package_files
      publish_on_pypi:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      - package_files
      publish_to_readthedocs:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      - package_docs
      release_job:
      needs:
      - pipeline: $PARENT_PIPELINE_ID
      job: build-template
      \ No newline at end of file
      ...@@ -14,6 +14,7 @@ stages: ...@@ -14,6 +14,7 @@ stages:
      # - build_extensions # - build_extensions
      - test - test
      - package - package
      - images
      - integration - integration
      - publish # publish instead of deploy - publish # publish instead of deploy
      ...@@ -77,7 +78,7 @@ secret_detection: ...@@ -77,7 +78,7 @@ secret_detection:
      before_script: before_script:
      - python --version # For debugging - python --version # For debugging
      - python -m pip install --upgrade pip - python -m pip install --upgrade pip
      - pip install --upgrade tox twine - python -m pip install --upgrade tox twine
      # Run all unit tests for Python versions except the base image # Run all unit tests for Python versions except the base image
      run_unit_tests: run_unit_tests:
      ...@@ -122,6 +123,20 @@ package_docs: ...@@ -122,6 +123,20 @@ package_docs:
      script: script:
      - tox -e docs - tox -e docs
      docker_build:
      stage: images
      image: docker:latest
      needs:
      - package_files
      tags:
      - dind
      before_script: []
      script:
      - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
      - docker build -f docker/{{cookiecutter.project_slug}}/Dockerfile . --build-arg BUILD_ENV=copy --tag $CI_REGISTRY_IMAGE/{{cookiecutter.project_slug}}:$CI_COMMIT_REF_SLUG
      # enable this push line once you have configured docker registry cleanup policy
      # - docker push $CI_REGISTRY_IMAGE/{{cookiecutter.project_slug}}:$CI_COMMIT_REF_SLUG
      run_integration_tests: run_integration_tests:
      stage: integration stage: integration
      allow_failure: true allow_failure: true
      ......
      ...@@ -11,6 +11,17 @@ An example repository of an CI/CD pipeline for building, testing and publishing ...@@ -11,6 +11,17 @@ An example repository of an CI/CD pipeline for building, testing and publishing
      pip install . pip install .
      ``` ```
      ## Setup
      One time template setup should include configuring the docker registry to regularly cleanup old images of
      the CI/CD pipelines. And you can consider creating protected version tags for software releases:
      1. [Cleanup Docker Registry Images](https://git.astron.nl/groups/templates/-/wikis/Cleanup-Docker-Registry-Images)
      2. [Setup Protected Verson Tags](https://git.astron.nl/groups/templates/-/wikis/Setting-up-Protected-Version-Tags)
      Once the cleanup policy for docker registry is setup you can uncomment the `docker push` comment in the `.gitlab-ci.yml`
      file from the `docker_build` job. This will allow to download minimal docker images with your Python package installed.
      ## Usage ## Usage
      ```python ```python
      from {{cookiecutter.project_slug}} import cool_module from {{cookiecutter.project_slug}} import cool_module
      ......
      FROM python:3.12 FROM python:3.12
      RUN python -m pip install --upgrade pip RUN python -m pip install --upgrade pip
      RUN pip install --upgrade tox twine RUN python -m pip install --upgrade tox twine
      ARG BUILD_ENV=no_copy
      FROM python:3.11 AS build_no_copy
      ADD ../../requirements.txt .
      COPY ../.. /work
      RUN rm -r /work/dist | true
      RUN python -m pip install --user tox
      WORKDIR /work
      RUN python -m tox -e build
      FROM python:3.11 AS build_copy
      COPY dist /work/dist
      FROM build_${BUILD_ENV} AS build
      FROM python:3.11-slim
      COPY --from=build /work/dist /dist
      RUN python -m pip install /dist/*.whl
      ...@@ -28,7 +28,7 @@ classifiers = ...@@ -28,7 +28,7 @@ classifiers =
      [options] [options]
      include_package_data = true include_package_data = true
      packages = find: packages = find:
      python_requires = >=3.7 python_requires = >=3.8
      install_requires = file: requirements.txt install_requires = file: requirements.txt
      [flake8] [flake8]
      ......
      ...@@ -53,4 +53,5 @@ commands = ...@@ -53,4 +53,5 @@ commands =
      [testenv:build] [testenv:build]
      usedevelop = False usedevelop = False
      deps = build
      commands = {envpython} -m build commands = {envpython} -m build
      0% Loading or .
      You are about to add 0 people to the discussion. Proceed with caution.
      Please register or to comment