ATDB-LDV

Apertif Task Database for LOFAR Data Valorization

Test Environment on sdc.astron.nl:

main GUI:

• https://sdc.astron.nl:5554/atdb/

admin interface:

• https://sdc.astron.nl:5554/atdb/admin/

REST API

• workflows: http://localhost:8000/atdb/workflows/
• tasks: http://localhost:8000/atdb/tasks/

Micro Services (in separate repo)

• https://git.astron.nl/astron-sdc/ldv-services

Confluence

Confluence Page:

• https://support.astron.nl/confluence/display/LDV/LOFAR+Data+Valorization+Home

Workflow: This diagram roughly serves as the specs for adapting ATDB for LDV.

• https://support.astron.nl/confluence/display/LDV/WORKFLOW

Overview Diagrams (under construction)

These diagrams are the core of the ATDB documentation. Currently they are still mostly the original ATDB diagrams. They will be adapted and kept in sync once the ATDB-LDV structure solifies a bit more.

Datamodel:

• https://drive.google.com/file/d/1v5hMBQS0jT8DQJwySVISfRa1zF4o0fCQ/view?usp=sharing

Workflow/Status Diagram:

• https://drive.google.com/file/d/1BCrIwUfacOVqSN73VIIjUuBCXNYA67iK/view?usp=sharing

Deployment Diagram:

• https://drive.google.com/file/d/1_j9Fp505pZTxcmzAEdgftdPkoIFrKfAX/view?usp=sharing

CI/CD (semi) automatic deploy in Docker

For the `master' branch there is a CI/CD pipeline in place which builds and deploys the backend at https://sdc.astron.nl:5554/atdb/

The deploy step requires pushing the 'play' button in the gitlab pipelines section. This is done to not have a mandatory deploy for every minor commmit.

Manual deploy in Docker (alternative to CI/CD)

initial

> cd ~/my_docker
> cd atdb-ldv
> git clone https://git.astron.nl/astron-sdc/atdb-ldv.git

update:

> export DOCKER_BUILD_DIR=$HOME/my_docker/atdb-ldv/atdb-ldv/atdb
> export DOCKER_COMPOSE_DIR=$DOCKER_BUILD_DIR/docker
> cd $DOCKER_BUILD_DIR
> git pull
> docker build -t atdb-ldv:latest .
> cd $DOCKER_COMPOSE_DIR
> docker-compose -p atdb up -d

Database migrations

When the models.py is changed, then the database must be migrated. This is how to do that manually for a persistent database in a Docker container.

> docker exec -it atdb-ldv python manage.py makemigrations --settings atdb.settings.docker_sdc
> docker exec -it atdb-ldv python manage.py migrate --settings atdb.settings.docker_sdc    

Postgres Database operations

dump a copy of the database

> cd $HOME/shared
> docker exec -it atdb-ldv-postgres pg_dump -U postgres -O atdbldv > atdb_ldv.sql

load a database (warning, this overwrites the database)

> docker exec -it atdb-ldv-postgres psql --command "drop database atdbldv" 
> docker exec -it atdb-ldv-postgres psql --command "create database atdbldv"
> docker exec -it atdb-ldv-postgres psql -U postgres atdbldv -f /shared/atdb_ldv.sql
> docker exec atdb-ldv-postgres psql --command "grant ALL ON DATABASE atdbldv TO dbadmin;"    

Original ATDB diagrams

These diagrams can be used as inspiration and guidance to the original ATDB services, which can be harvested for functionality.