ATDB-LDV

ATDB-LDV is the Django backend application for LDV. It's main purpose is to hold the state of all the tasks that are in progress for LDV. ATDB does not control the flow itself, that is handled by a cloud of micro services. Communication is done through the REST API.

Micro Services (in separate repo)

• https://git.astron.nl/astron-sdc/ldv-services

Project Documentation (Confluence)

Confluence Page:

• https://support.astron.nl/confluence/display/LDV/LOFAR+Data+Valorization+Home

Specifications:

These diagrams roughly serves as the specifications for adapting ATDB for LDV.

• workflow: https://support.astron.nl/confluence/display/LDV/WORKFLOW
• datamodel: https://dbdiagram.io/d/5ffc5fb180d742080a35d560

Overview Diagrams (current implementation)

These diagrams show the current implementation and are kept up-to-date.

Datamodel:

• https://drive.google.com/file/d/1v5hMBQS0jT8DQJwySVISfRa1zF4o0fCQ/view?usp=sharing

Workflow/Status Diagram:

• https://drive.google.com/file/d/1BCrIwUfacOVqSN73VIIjUuBCXNYA67iK/view?usp=sharing

GUI implementation

• https://app.diagrams.net/#G16R8L06OFiKHFHBUA6FhrNVZVAaQBC2tU

Deployed Instances

main GUI:

• test: https://sdc-dev.astron.nl:5554/atdb/
• acc : http://dop457.astron.nl:5554/atdb/
• prod: https://sdc.astron.nl:5554/atdb/

admin interface:

• test: https://sdc-dev.astron.nl:5554/atdb/admin/
• acc : http://dop457.astron.nl:5554/atdb/admin/
• prod: https://sdc.astron.nl:5554/atdb/admin/

REST API (prod)

serializers:

• workflows: http://sdc.astron.nl:5554/atdb/workflows/
• tasks: http://sdc.astron.nl:5554/atdb/tasks/

get_size: Return the sum the sizes of all tasks with a given list of statusses

• https://sdc.astron.nl:5554/atdb/tasks/get_size
• https://sdc.astron.nl:5554/atdb/tasks/get_size?status__in=staged,processing,processed,validating,validated,ingesting,removing,removed

Build & Deploy

Deployment Diagram:

• https://drive.google.com/file/d/1_j9Fp505pZTxcmzAEdgftdPkoIFrKfAX/view?usp=sharing

CI/CD (semi) automatic deploy in Docker

A gitlab CI/CD pipeline will automatically build and semi-automatically deploy ATDB on several targets, depending on the name of the branch.

• master => test: https://sdc-dev.astron.nl:5554/atdb/
• acceptance=> acceptance: http://dop457.astron.nl:5554/atdb/
• release => prod: https://sdc.astron.nl:5554/atdb/

The deploy step requires pushing the 'play' button in the gitlab pipelines section. This is done to not have a mandatory deploy for every minor commmit.

The Procedure:

• commit changes to 'feature branch'

• gitlab: merge with master

  • pipeline will start running: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines
  • if 'passed' then click >> on the 'deploy' stage, this will deploy to test on sdc-dev

• gitlab: merge with acceptance

  • pipeline will start running: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines
  • if 'passed' then click >> on the 'deploy' stage, this will deploy to acceptance on dop457

• gitlab: merge with release

  • pipeline will start running: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines
  • if 'passed' then click >> on the 'deploy' stage, this will deploy to production on sdc

Manual deploy in Docker (alternative to CI/CD)

initial

> cd ~/my_docker
> cd atdb-ldv
> git clone https://git.astron.nl/astron-sdc/atdb-ldv.git

update:

> export DOCKER_BUILD_DIR=$HOME/my_docker/atdb-ldv/atdb-ldv/atdb
> export DOCKER_COMPOSE_DIR=$DOCKER_BUILD_DIR/docker
> cd $DOCKER_BUILD_DIR
> git pull
> docker build -t atdb-ldv:latest .
> cd $DOCKER_COMPOSE_DIR
> docker-compose -p atdb up -d

Database changes and migrations

When the models.py is changed, then the database must be migrated. This is the procedure for that.

on local dev:

> python manage.py makemigrations --settings=atdb.settings.dev
> python manage.py migrate --settings=atdb.settings.dev
- add new migration file to git
- commit & push
- apply 'The Procedure' (see above) to deploy in test, acceptance and production.

on CI/CD page: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines

- when automatic build is finished, push >> to deploy

on 'test' (sdc@dop814), 'acc' (sdc@dop457) and 'prod' (sdco@dop821) machine's:

> docker exec -it atdb-ldv python manage.py makemigrations --settings atdb.settings.docker_sdc
  (this should say 'No changes detected', but do this step anyway as a check) 
> docker exec -it atdb-ldv python manage.py migrate --settings atdb.settings.docker_sdc