Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
  • MAM-90-executor-with-cwltool
  • MAM-88-change-status-names
  • MAM-50-specification-GUI
  • MAM-44-build-specification-functionality
  • MAM-42-ATDBspec-add-endpoints
  • MAM-41-ATDBspec-database-changes
  • bugfix-ingested-sizes
  • SDC-1663-inputs-validation
  • SDC-1649-compression-pipeline-quality-indicators
  • SDC-1635-monitoring-page
  • split-pipeline-for-gitlab
  • SDC-1590-database-changes
  • SDC-1580-adapt-configuration-to-multi-service
  • multi-edit
  • SDC-1552-datamodel-add-service-host
  • logentry_graph
  • SDC-1549-aggregation-task-ux
  • imaging-compression-pipeline-commissioning
  • unit-tests
20 results

atdb-ldv

  • Clone with SSH
  • Clone with HTTPS
    • Name Last commit Last update
    .idea
    atdb
    .gitlab-ci.yml
    LICENSE
    README.md
    pip-selfcheck.json
    README.md

    ATDB-LDV

    ATDB-LDV is the Django backend application for LDV. It's main purpose is to hold the state of all the tasks that are in progress for LDV. ATDB does not control the flow itself, that is handled by a cloud of micro services. Communication is done through the REST API.

    Micro Services (in separate repo)

    Project Documentation (Confluence)

    Confluence Page:

    Specifications:

    These diagrams roughly serves as the specifications for adapting ATDB for LDV.

    Overview Diagrams

    Datamodel:

    Workflow/Status Diagram:

    Test Environment on sdc-dev.astron.nl:

    main GUI:

    admin interface:

    REST API

    serializers:

    get_size:

    Deployment Diagram:

    CI/CD (semi) automatic deploy in Docker

    For the master branch there is a CI/CD pipeline in place which builds and deploys the backend at https://sdc.astron.nl:5554/atdb/

    The deploy step requires pushing the 'play' button in the gitlab pipelines section. This is done to not have a mandatory deploy for every minor commmit.

    Manual deploy in Docker (alternative to CI/CD)

    initial

    > cd ~/my_docker
> cd atdb-ldv
> git clone https://git.astron.nl/astron-sdc/atdb-ldv.git

    update:

    > export DOCKER_BUILD_DIR=$HOME/my_docker/atdb-ldv/atdb-ldv/atdb
> export DOCKER_COMPOSE_DIR=$DOCKER_BUILD_DIR/docker
> cd $DOCKER_BUILD_DIR
> git pull
> docker build -t atdb-ldv:latest .
> cd $DOCKER_COMPOSE_DIR
> docker-compose -p atdb up -d

    Database changes and migrations

    When the models.py is changed, then the database must be migrated. This is the procedure for that.

    on local dev:

    > python manage.py makemigrations --settings=atdb.settings.dev
> python manage.py migrate --settings=atdb.settings.dev
- add new migration file to git
- commit & push

    on CI/CD page: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines

    - when automatic build is finished, push >> to deploy

    on sdc machine:

    > docker exec -it atdb-ldv python manage.py makemigrations --settings atdb.settings.docker_sdc
> docker exec -it atdb-ldv python manage.py migrate --settings atdb.settings.docker_sdc

    Postgres Database operations

    dump a copy of the database

    > cd $HOME/shared
> docker exec -it atdb-ldv-postgres pg_dump -U postgres -O atdbldv > atdb_ldv.sql

    load a database (warning, this overwrites the database)

    > docker exec -it atdb-ldv-postgres psql --command "drop database atdbldv" 
> docker exec -it atdb-ldv-postgres psql --command "create database atdbldv"
> docker exec -it atdb-ldv-postgres psql -U postgres atdbldv -f /shared/atdb_ldv.sql
> docker exec atdb-ldv-postgres psql --command "grant ALL ON DATABASE atdbldv TO dbadmin;"

    Original ATDB Context diagrams

    Similar diagram for ATDB-LDV will follow, showing the relationships with external services