Skip to content
Snippets Groups Projects
Roy de Goei's avatar
Merge branch 'SDC-1526_macaroons_documentation' into 'master'
Roy de Goei authored
SDC-1526: Add content to the Macaroons part and remove the Monitoring...

Closes SDC-1526

See merge request !72
4d7d626f
History

StageIT

New implementation for the LOFAR stager in the Django framework. Code quality check report at Sonarcloud

Version dependencies

The StageIT docker containers has python 3.9 as base. This version is at least supported until October 2025 See Status of python versions We can not upgrade to a higher python version due to the gfal2-python3 package (CERN) which is only supported until python version 3.9 As a result we are currently stuck to Django version 4.2 at this moment. See Status of Django versions The Django 4.2 support ends in April 2026. The result of Django 4.2 version is using Postgres version 12 or larger, currently version 13.

Contributing

Installing a development version can be done via. Activating a virtual environment is required:

pip install -r requirements.txt

Load environment variables:

source ./tests/files/.env

If you have created the stageit_postgres docker container before you most likely wanted to clean all the data as well:

docker volume rm deploy_pgdata

If you get an error that the volume is still in use

docker ps -a --filter volume=deploy_pgdata
docker stop CONTAINER_ID
docker rm CONTAINER_ID

and try again

You can create the database container by running:

# This is required for the provided docker-compose.yaml files to work, but is not used
# in development.
docker network create traefik_proxy

# replace `docker compose` by `docker-compose` if necessary
docker compose -f deploy/docker-compose.yaml -f deploy/docker-compose.development.yaml up -d --force-recreate postgres && sleep 2

Note: If you get something like:

    ● postgresql.service - PostgreSQL RDBMS
     Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2023-10-09 07:33:47 CEST; 3 weeks 3 days ago
   Main PID: 1811 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 18708)
     Memory: 0B
     CGroup: /system.slice/postgresql.service

You have to switch off the service by systemctl stop postgresql.service or systemctl disable postgresql.service

Fill the database with some content -> obsolete since it will break the DB structure (just here for info how you can load from sql): docker exec -i stageit_postgres /bin/bash -c "PGPASSWORD=stageit psql --username stageit stageit" < ./tests/files/stageit_dump.sql

The next commands should be performed in preferably the virtual environment you have been activated

Then the database can be migrated by running (in the virtual env):

python manage.py migrate

Add user local as superuser (in the virtual env):

DJANGO_SUPERUSER_PASSWORD=local python manage.py createsuperuser --username local --email no-reply@example.com --noinput

Now we have the database tables created, but no content. The only content that is initial relevant is the LTASite table and the email templates This can be obtained by loading two fixture files (in the virtual env):

python manage.py loaddata fixture/lta_sites fixture/email_templates

And finally a development instance can be run at local host (in the virtual env), see http://localhost:8000/stageit:

python manage.py runserver

Running the unittests (in the virtual env):

python manage.py test

If you want to create/update a diagram of the datamodel you can run (in the virtual env):

python manage.py graph_models -g -o documentation/static/img/stageit_models.png --pydot stage_api auth

If you want to have some small set of test data (having a staging request available), you can load the test fixture file (in the virtual env):

python manage.py loaddata ./tests/files/test_ltarequest.json

Macaroons

To simplify the LTA data access of the staged dataproducts, dCache provides support for data access based on the WEBDAV protocol and token-based access authorization called 'Macaroon'. A valid Macaroon will be stored per LTA site per LTA request and can be used to stage data via webdav. The macaroon will be sent within the notification email or the user has the ability to download it from the user request view .

So for every surl of the LTARequest: Check if surl (LTASite) does not have a LTAMacaroon, then

  1. Determine the root_caveat_path of the surl; This is the path after surl_pattern and before the <project_code> See some examples
  2. Send POST request for macaroon using root_caveat_path
  3. Create LTAMacaroon object for request and store in the DB

The macaroon is valid for 6 days (this is configurable per LTA Site) There is one exception to NOT generate macaroons! For LDV Staging service, the generation of macaroons is superfluous and should be skipped. Unfortunately the user is configured as 'admin' (not LDV). It a kind of quick and dirty, but if user is 'admin' then no macaroon will be created/stored.

Acknowledgements

EU Flag EOSC-hub logo This work is co-funded by the EOSC-hub project (Horizon 2020) under Grant number 777536.