ATDB-LDV

Apertif Task Database for LOFAR Data Valorization

Test Environment on sdc.astron.nl:

main GUI:

• https://sdc.astron.nl:5554/atdb/

admin interface:

• https://sdc.astron.nl:5554/atdb/admin/

REST API

• workflows: http://localhost:8000/atdb/workflows/
• tasks: http://localhost:8000/atdb/tasks/

Micro Services (in separate repo)

• https://git.astron.nl/astron-sdc/ldv-services

Confluence

Confluence Page:

• https://support.astron.nl/confluence/display/LDV/LOFAR+Data+Valorization+Home

Specifications: These diagrams roughly serves as the specifications for adapting ATDB for LDV.

• workflow: https://support.astron.nl/confluence/display/LDV/WORKFLOW
• datamodel: https://dbdiagram.io/d/5ffc5fb180d742080a35d560

Overview Diagrams (under construction)

These diagrams are the core of the ATDB documentation.

Datamodel:

• https://drive.google.com/file/d/1v5hMBQS0jT8DQJwySVISfRa1zF4o0fCQ/view?usp=sharing

Workflow/Status Diagram:

• https://drive.google.com/file/d/1BCrIwUfacOVqSN73VIIjUuBCXNYA67iK/view?usp=sharing

Deployment Diagram:

• https://drive.google.com/file/d/1_j9Fp505pZTxcmzAEdgftdPkoIFrKfAX/view?usp=sharing

CI/CD (semi) automatic deploy in Docker

For the `master' branch there is a CI/CD pipeline in place which builds and deploys the backend at https://sdc.astron.nl:5554/atdb/

The deploy step requires pushing the 'play' button in the gitlab pipelines section. This is done to not have a mandatory deploy for every minor commmit.

Manual deploy in Docker (alternative to CI/CD)

initial

> cd ~/my_docker
> cd atdb-ldv
> git clone https://git.astron.nl/astron-sdc/atdb-ldv.git

update:

> export DOCKER_BUILD_DIR=$HOME/my_docker/atdb-ldv/atdb-ldv/atdb
> export DOCKER_COMPOSE_DIR=$DOCKER_BUILD_DIR/docker
> cd $DOCKER_BUILD_DIR
> git pull
> docker build -t atdb-ldv:latest .
> cd $DOCKER_COMPOSE_DIR
> docker-compose -p atdb up -d

Database changes and migrations

When the models.py is changed, then the database must be migrated. This is the procedure for that.

on local dev:

> python manage.py makemigrations --settings=atdb.settings.dev
> python manage.py migrate --settings=atdb.settings.dev
- add new migration file to git
- commit & push

on CI/CD page: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines

- when automatic build is finished, push >> to deploy

on sdc machine:

> docker exec -it atdb-ldv python manage.py makemigrations --settings atdb.settings.docker_sdc
> docker exec -it atdb-ldv python manage.py migrate --settings atdb.settings.docker_sdc

Postgres Database operations

dump a copy of the database

> cd $HOME/shared
> docker exec -it atdb-ldv-postgres pg_dump -U postgres -O atdbldv > atdb_ldv.sql

load a database (warning, this overwrites the database)

> docker exec -it atdb-ldv-postgres psql --command "drop database atdbldv" 
> docker exec -it atdb-ldv-postgres psql --command "create database atdbldv"
> docker exec -it atdb-ldv-postgres psql -U postgres atdbldv -f /shared/atdb_ldv.sql
> docker exec atdb-ldv-postgres psql --command "grant ALL ON DATABASE atdbldv TO dbadmin;"    

Original ATDB diagrams

These diagrams can be used as inspiration and guidance to the original ATDB services, which can be harvested for functionality.