Skip to content
Snippets Groups Projects
Select Git revision
  • d337a00f4ac3e3e103e6e24cbfc71863f91f52f2
  • master default protected
  • MAM-110-propagate-output-sasid
  • MAM-109-specify-ingest-location
  • master-backup-september-2024
5 results

atdb-ldv

  • Clone with SSH
  • Clone with HTTPS
  • ATDB-LDV

    Apertif Task Database for LOFAR Data Valorization

    Test Environment on sdc.astron.nl:

    main GUI:

    admin interface:

    REST API

    Micro Services (in separate repo)

    Confluence

    Confluence Page:

    Specifications: These diagrams roughly serves as the specifications for adapting ATDB for LDV.

    Overview Diagrams (under construction)

    These diagrams are the core of the ATDB documentation.

    Datamodel:

    Workflow/Status Diagram:

    Deployment Diagram:

    CI/CD (semi) automatic deploy in Docker

    For the `master' branch there is a CI/CD pipeline in place which builds and deploys the backend at https://sdc.astron.nl:5554/atdb/

    The deploy step requires pushing the 'play' button in the gitlab pipelines section. This is done to not have a mandatory deploy for every minor commmit.

    Manual deploy in Docker (alternative to CI/CD)

    initial

    > cd ~/my_docker
    > cd atdb-ldv
    > git clone https://git.astron.nl/astron-sdc/atdb-ldv.git

    update:

    > export DOCKER_BUILD_DIR=$HOME/my_docker/atdb-ldv/atdb-ldv/atdb
    > export DOCKER_COMPOSE_DIR=$DOCKER_BUILD_DIR/docker
    > cd $DOCKER_BUILD_DIR
    > git pull
    > docker build -t atdb-ldv:latest .
    > cd $DOCKER_COMPOSE_DIR
    > docker-compose -p atdb up -d

    Database changes and migrations

    When the models.py is changed, then the database must be migrated. This is the procedure for that.

    on local dev:

    > python manage.py makemigrations --settings=atdb.settings.dev
    > python manage.py migrate --settings=atdb.settings.dev
    - add new migration file to git
    - commit & push

    on CI/CD page: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines

    - when automatic build is finished, push >> to deploy

    on sdc machine:

    > docker exec -it atdb-ldv python manage.py makemigrations --settings atdb.settings.docker_sdc
    > docker exec -it atdb-ldv python manage.py migrate --settings atdb.settings.docker_sdc

    Postgres Database operations

    dump a copy of the database

    > cd $HOME/shared
    > docker exec -it atdb-ldv-postgres pg_dump -U postgres -O atdbldv > atdb_ldv.sql

    load a database (warning, this overwrites the database)

    > docker exec -it atdb-ldv-postgres psql --command "drop database atdbldv" 
    > docker exec -it atdb-ldv-postgres psql --command "create database atdbldv"
    > docker exec -it atdb-ldv-postgres psql -U postgres atdbldv -f /shared/atdb_ldv.sql
    > docker exec atdb-ldv-postgres psql --command "grant ALL ON DATABASE atdbldv TO dbadmin;"    

    Original ATDB diagrams

    These diagrams can be used as inspiration and guidance to the original ATDB services, which can be harvested for functionality.