ATDB-LDV

ATDB-LDV is the Django backend application for LDV. It's main purpose is to hold the state of all the tasks that are in progress for LDV. ATDB does not control the flow itself, that is handled by a cloud of micro services. Communication is done through the REST API.

Micro Services (in separate repo)

• https://git.astron.nl/astron-sdc/ldv-services

Project Documentation (Confluence)

Confluence Page:

• https://support.astron.nl/confluence/display/LDV/LOFAR+Data+Valorization+Home

Specifications:

These diagrams roughly serves as the specifications for adapting ATDB for LDV.

• workflow: https://support.astron.nl/confluence/display/LDV/WORKFLOW
• datamodel: https://dbdiagram.io/d/5ffc5fb180d742080a35d560

Overview Diagrams (current implementation)

These diagrams show the current implementation and are kept up-to-date.

Datamodel:

• https://drive.google.com/file/d/1v5hMBQS0jT8DQJwySVISfRa1zF4o0fCQ/view?usp=sharing

Workflow/Status Diagram:

• https://drive.google.com/file/d/1BCrIwUfacOVqSN73VIIjUuBCXNYA67iK/view?usp=sharing

Test Environment on sdc-dev.astron.nl:

main GUI:

• https://sdc.astron.nl:5554/atdb/

admin interface:

• https://sdc.astron.nl:5554/atdb/admin/

REST API

serializers:

• workflows: http://sdc.astron.nl:5554/atdb/workflows/
• tasks: http://sdc.astron.nl:5554/atdb/tasks/

get_size: Return the sum the sizes of all tasks with a given list of statusses

• https://sdc.astron.nl:5554/atdb/tasks/get_size
• https://sdc.astron.nl:5554/atdb/tasks/get_size?status__in=staged,processing,processed,validating,validated,ingesting,removing,removed

Build & Deploy

Deployment Diagram:

• https://drive.google.com/file/d/1_j9Fp505pZTxcmzAEdgftdPkoIFrKfAX/view?usp=sharing

CI/CD (semi) automatic deploy in Docker

For the master branch there is a CI/CD pipeline in place which builds and deploys the backend at https://sdc.astron.nl:5554/atdb/

The deploy step requires pushing the 'play' button in the gitlab pipelines section. This is done to not have a mandatory deploy for every minor commmit.

Manual deploy in Docker (alternative to CI/CD)

initial

> cd ~/my_docker
> cd atdb-ldv
> git clone https://git.astron.nl/astron-sdc/atdb-ldv.git

update:

> export DOCKER_BUILD_DIR=$HOME/my_docker/atdb-ldv/atdb-ldv/atdb
> export DOCKER_COMPOSE_DIR=$DOCKER_BUILD_DIR/docker
> cd $DOCKER_BUILD_DIR
> git pull
> docker build -t atdb-ldv:latest .
> cd $DOCKER_COMPOSE_DIR
> docker-compose -p atdb up -d

Database changes and migrations

When the models.py is changed, then the database must be migrated. This is the procedure for that.

on local dev:

> python manage.py makemigrations --settings=atdb.settings.dev
> python manage.py migrate --settings=atdb.settings.dev
- add new migration file to git
- commit & push

on CI/CD page: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines

- when automatic build is finished, push >> to deploy

on sdc machine:

> docker exec -it atdb-ldv python manage.py makemigrations --settings atdb.settings.docker_sdc
> docker exec -it atdb-ldv python manage.py migrate --settings atdb.settings.docker_sdc

Postgres Database operations

dump a copy of the database

> cd $HOME/shared
> docker exec -it atdb-ldv-postgres pg_dump -U postgres -O atdbldv > atdb_ldv.sql

load a database (warning, this overwrites the database)

> docker exec -it atdb-ldv-postgres psql --command "drop database atdbldv" 
> docker exec -it atdb-ldv-postgres psql --command "create database atdbldv"
> docker exec -it atdb-ldv-postgres psql -U postgres atdbldv -f /shared/atdb_ldv.sql
> docker exec atdb-ldv-postgres psql --command "grant ALL ON DATABASE atdbldv TO dbadmin;"    

Original ATDB Context diagrams

Similar diagram for ATDB-LDV will follow, showing the relationships with external services