ATDB-LDV
ATDB-LDV is the Django backend application for LDV. It's main purpose is to hold the state of all the tasks that are in progress for LDV. ATDB does not control the flow itself, that is handled by a cloud of micro services. Communication is done through the REST API.
Micro Services (in separate repo)
Project Documentation (Confluence)
Confluence Page:
Specifications:
These diagrams roughly serves as the specifications for adapting ATDB for LDV.
- workflow: https://support.astron.nl/confluence/display/LDV/WORKFLOW
- datamodel: https://dbdiagram.io/d/5ffc5fb180d742080a35d560
Overview Diagrams (current implementation)
These diagrams show the current implementation and are kept up-to-date.
Datamodel:
Workflow/Status Diagram:
GUI implementation
Deployed Instances
main GUI:
- test: https://sdc-dev.astron.nl:5554/atdb/
- acc : http://dop457.astron.nl:5554/atdb/
- prod: https://sdc.astron.nl:5554/atdb/
admin interface:
- test: https://sdc-dev.astron.nl:5554/atdb/admin/
- acc : http://dop457.astron.nl:5554/atdb/admin/
- prod: https://sdc.astron.nl:5554/atdb/admin/
REST API (prod)
serializers:
get_size: Return the sum the sizes of all tasks with a given list of statusses
- https://sdc.astron.nl:5554/atdb/tasks/get_size
- https://sdc.astron.nl:5554/atdb/tasks/get_size?status__in=staged,processing,processed,validating,validated,ingesting,removing,removed
Build & Deploy
Deployment Diagram:
CI/CD (semi) automatic deploy in Docker
A gitlab CI/CD pipeline will automatically build and semi-automatically deploy ATDB on several targets, depending on the name of the branch.
-
master=> test: https://sdc-dev.astron.nl:5554/atdb/ -
acceptance=> acceptance: http://dop457.astron.nl:5554/atdb/ -
release=> prod: https://sdc.astron.nl:5554/atdb/
The deploy step requires pushing the 'play' button in the gitlab pipelines section. This is done to not have a mandatory deploy for every minor commmit.
The Procedure:
-
commit changes to 'feature branch'
-
gitlab: merge with
master- pipeline will start running: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines
- if 'passed' then click
>>on the 'deploy' stage, this will deploy to test on sdc-dev
-
gitlab: merge with
acceptance- pipeline will start running: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines
- if 'passed' then click
>>on the 'deploy' stage, this will deploy to acceptance on dop457
-
gitlab: merge with
release- pipeline will start running: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines
- if 'passed' then click
>>on the 'deploy' stage, this will deploy to production on sdc
Manual deploy in Docker (alternative to CI/CD)
initial
> cd ~/my_docker
> cd atdb-ldv
> git clone https://git.astron.nl/astron-sdc/atdb-ldv.gitupdate:
> export DOCKER_BUILD_DIR=$HOME/my_docker/atdb-ldv/atdb-ldv/atdb
> export DOCKER_COMPOSE_DIR=$DOCKER_BUILD_DIR/docker
> cd $DOCKER_BUILD_DIR
> git pull
> docker build -t atdb-ldv:latest .
> cd $DOCKER_COMPOSE_DIR
> docker-compose -p atdb up -dDatabase changes and migrations
When the models.py is changed, then the database must be migrated.
This is the procedure for that.
on local dev:
> python manage.py makemigrations --settings=atdb.settings.dev
> python manage.py migrate --settings=atdb.settings.dev
- add new migration file to git
- commit & push
- apply 'The Procedure' (see above) to deploy in test, acceptance and production.on CI/CD page: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines
- when automatic build is finished, push >> to deployon 'test' (sdc@dop814), 'acc' (sdc@dop457) and 'prod' (sdco@dop821) machine's:
> docker exec -it atdb-ldv python manage.py makemigrations --settings atdb.settings.docker_sdc
(this should say 'No changes detected', but do this step anyway as a check)
> docker exec -it atdb-ldv python manage.py migrate --settings atdb.settings.docker_sdc