Skip to content
Snippets Groups Projects
Commit 1e6864d5 authored by Alissa Cheng's avatar Alissa Cheng
Browse files

update readme, add set up instructions

parent b5f98e8b
No related branches found
No related tags found
1 merge request!337Upgrade to django 5
Pipeline #69256 passed
......@@ -3,3 +3,4 @@
**/.idea/
**/__pycache__/
*.sql
/atdb/run.sh
......@@ -8,7 +8,7 @@ Communication is done through the REST API.
## Micro Services (in separate repo)
* https://git.astron.nl/astron-sdc/ldv-services
## Project Documentation (Confluence)
## Project Documentation
### Confluence Page:
* https://support.astron.nl/confluence/display/LDV/LOFAR+Data+Valorization+Home
......@@ -17,9 +17,6 @@ These diagrams roughly serves as the specifications for adapting ATDB for LDV.
* workflow: https://support.astron.nl/confluence/display/LDV/WORKFLOW
* datamodel: https://dbdiagram.io/d/5ffc5fb180d742080a35d560
## Overview Diagrams (current implementation)
These diagrams show the current implementation and are kept up-to-date.
### Datamodel:
* https://drive.google.com/file/d/1v5hMBQS0jT8DQJwySVISfRa1zF4o0fCQ/view?usp=sharing
......@@ -37,21 +34,21 @@ These diagrams show the current implementation and are kept up-to-date.
## Deployed Instances
### main GUI:
### Main GUI:
* test: https://sdc-dev.astron.nl:5554/atdb/
* prod: https://sdc.astron.nl:5554/atdb/
### admin interface:
### Admin interface:
* test: https://sdc-dev.astron.nl:5554/atdb/admin/
* prod: https://sdc.astron.nl:5554/atdb/admin/
### REST API (prod)
serializers:
#### serializers:
* workflows: http://sdc.astron.nl:5554/atdb/workflows/
* tasks: http://sdc.astron.nl:5554/atdb/tasks/
get_size:
Return the sum the sizes of all tasks with a given list of statusses
#### get_size:
Return the sum the sizes of all tasks with a given list of statuses
* https://sdc.astron.nl:5554/atdb/tasks/get_size
* https://sdc.astron.nl:5554/atdb/tasks/get_size?status__in=staged,processing,processed,validating,validated,ingesting,removing,removed
......@@ -90,13 +87,6 @@ Master can be deployed to sdc-dev (test) or sdc (production)
When the `models.py` is changed, then the database must be migrated.
This is the procedure for that.
on local dev:
> python manage.py makemigrations --settings=atdb.settings.dev
> python manage.py migrate --settings=atdb.settings.dev
- add new migration file to git
- commit & push
on CI/CD page: https://git.astron.nl/astron-sdc/atdb-ldv/-/pipelines
- when automatic build is finished, push >> to deploy
......@@ -107,21 +97,72 @@ This is the procedure for that.
(this should say 'No changes detected', but do this step anyway as a check)
> docker exec -it atdb-ldv python manage.py migrate --settings atdb.settings.docker_sdc
#### Integrating Astronauth
See the integration instructions in the [astronauth repo](https://git.astron.nl/astron-sdc/astronauth)
#### CI tests
Tests are executed in the CI pipeline through the test stage in the `.gitlab-ci.yml` file.
For running the tests in the CI pipeline, the settings file `atdb/settings/test_ci.dev` is used.
## Local development
### Set up
1. Clone the repo
<br/><br/>
2. Install Postgres and make a database:
```shell
sudo apt install postgresql postgresql-contrib libpq-dev python3-dev
sudo -u postgres psql
```
The following database credentials should be identical to those in `atdb/atdb/settings/dev.py`:
```postgresql
CREATE DATABASE atdb_ldv_12jan2024;
CREATE USER atdb_admin WITH PASSWORD 'atdb123';
ALTER ROLE atdb_admin SET client_encoding TO 'utf8';
ALTER ROLE atdb_admin SET default_transaction_isolation TO 'read committed';
ALTER ROLE atdb_admin SET timezone TO 'UTC';
GRANT ALL PRIVILEGES ON DATABASE atdb_ldv_12jan2024 TO atdb_admin;
CREATE ROLE ldvstats;
\q
```
Import the database dump `atdbdb.sql` from its parent directory:
```shell
sudo pg_restore -h localhost -U atdb_admin -d atdb_ldv_12jan2024 < atdbdb.sql
```
Note: The required password here is `atdb123`, not your sudo password.
<br/><br/>
3. Make sure you have Python installed. Then go to the `atdb` directory (not the second `atdb/atdb`) - it's where the `manage.py` lives. Create a virtual environment and install dependencies:
```shell
python3 -m venv venv
source venv/bin/activate
python -m pip install -r requirements/dev.txt
python manage.py migrate --settings=atdb.settings.dev
```
### Start the server
To start the server, run:
```shell
python manage.py runserver --settings=atdb.settings.dev
```
However, if you need to login to the local server, then you will need some environment variables for Keycloak.
In `atdb` directory where the file `run.sh.example` lives, duplicate it and rename it `run.sh`.
Fill in the Keycloak client secret, you can find that in the Keycloak dashboard. Do not commit this file.
Then you can start the server by running the file: `source ./run.sh`.
### Migrations
When the models.py is changed, then the database must be migrated.
In the `atdb` directory, run:
```shell
python manage.py makemigrations --settings=atdb.settings.dev
python manage.py migrate --settings=atdb.settings.dev
```
Then add and commit the resulting migration file.
### Running tests
To run tests, you can spin up a dedicated test database locally using the the provided `docker-compose-test-local.yml` file. This test database will not interfere with your local development database. For example, you can run this command from the `atdb/docker` folder:
` docker compose -f .\docker-compose-test-local.yml up -d`
After spinning up the database, you can execute the tests with the following command:
`python manage.py test --settings atdb.settings.test_local`
To run tests, you can spin up a dedicated test database locally with docker.
This test database will not interfere with your local development database.
```shell
docker compose -f docker/docker-compose-test-local.yml up -d
python manage.py test --settings atdb.settings.test_local
```
Dedicated settings for running the tests are provided in the `atdb/settings/test_local.dev` file. For convenience, `test.bat` is provided to run the above command (Windows only).
Finally, these tests are also executed in the CI pipeline through the test stage in the `.gitlab-ci.yml` file. For running the tests in the CI pipeline, the settings file `atdb/settings/test_ci.dev` is used.
\ No newline at end of file
export KEYCLOAK_URL=https://keycloak-sdc.astron.nl
export KEYCLOAK_CLIENT_ID=ATDB-LDV-DEV
export KEYCLOAK_CLIENT_SECRET=
python manage.py runserver --settings=atdb.settings.dev
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment