Skip to content
Snippets Groups Projects
Select Git revision
  • MAM-42-ATDBspec-add-endpoints
  • master default protected
  • MAM-41-ATDBspec-database-changes
  • bugfix-ingested-sizes
  • SDC-1663-inputs-validation
  • SDC-1649-compression-pipeline-quality-indicators
  • SDC-1635-monitoring-page
  • split-pipeline-for-gitlab
  • SDC-1590-database-changes
  • SDC-1580-adapt-configuration-to-multi-service
  • multi-edit
  • SDC-1552-datamodel-add-service-host
  • logentry_graph
  • SDC-1549-aggregation-task-ux
  • imaging-compression-pipeline-commissioning
  • unit-tests
  • master-backup-september-2024
  • profiling
18 results
Compare
Name Last commit Last update
.idea
atdb
.gitignore
.gitlab-ci.yml
LICENSE
README.md
pip-selfcheck.json
README.md

ATDB-LDV

ATDB-LDV is the Django backend application for LDV. It's main purpose is to hold the state of all the tasks that are in progress for LDV. ATDB does not control the flow itself, that is handled by a cloud of micro services. Communication is done through the REST API.

Micro Services (in separate repo)

Project Documentation (Confluence)

Confluence Page:

Specifications:

These diagrams roughly serves as the specifications for adapting ATDB for LDV.

Overview Diagrams (current implementation)

These diagrams show the current implementation and are kept up-to-date.

Datamodel:

Workflow/Status Diagram:

GUI implementation

Deployed Instances

main GUI:

admin interface:

REST API (prod)

serializers:

get_size: Return the sum the sizes of all tasks with a given list of statusses

Build & Deploy

Deployment Diagram:

CI/CD (semi) automatic deploy in Docker

A gitlab CI/CD pipeline will automatically build after every commit. Branches can be deployed to sdc-dev (test) Master can be deployed to sdc-dev (test) or sdc (production)

Manual deploy in Docker (alternative to CI/CD)

initial

> cd ~/my_docker
> cd atdb-ldv
> git clone https://git.astron.nl/astron-sdc/atdb-ldv.git

update:

> export DOCKER_BUILD_DIR=$HOME/my_docker/atdb-ldv/atdb-ldv/atdb
> export DOCKER_COMPOSE_DIR=$DOCKER_BUILD_DIR/docker
> cd $DOCKER_BUILD_DIR
> git pull
> docker build -t atdb-ldv:latest .
> cd $DOCKER_COMPOSE_DIR
> docker-compose -p atdb up -d

Database changes and migrations

When the models.py is changed, then the database must be migrated. This is the procedure for that.

on local dev:

> python manage.py makemigrations --settings=atdb.settings.dev
> python manage.py migrate --settings=atdb.settings.dev
- add new migration file to git
- commit & push

on CI/CD page: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines

- when automatic build is finished, push >> to deploy

on 'sdc-dev.astron.nl' (sdc@dop814) and 'sdc.astron.nl' (sdco@dop821) machine's:

> docker exec -it atdb-ldv python manage.py makemigrations --settings atdb.settings.docker_sdc
  (this should say 'No changes detected', but do this step anyway as a check) 
> docker exec -it atdb-ldv python manage.py migrate --settings atdb.settings.docker_sdc

Integrating Astronauth

See the integration instructions in the astronauth repo

Running tests

To run tests, you can spin up a dedicated test database locally using the the provided docker-compose-test-local.yml file. This test database will not interfere with your local development database. For example, you can run this command from the atdb/docker folder:

docker compose -f .\docker-compose-test-local.yml up -d

After spinning up the database, you can execute the tests with the following command:

python manage.py test --settings atdb.settings.test_local

Dedicated settings for running the tests are provided in the atdb/settings/test_local.dev file. For convenience, test.bat is provided to run the above command (Windows only).

Finally, these tests are also executed in the CI pipeline through the test stage in the .gitlab-ci.yml file. For running the tests in the CI pipeline, the settings file atdb/settings/test_ci.dev is used.