StageIT
New implementation for the LOFAR stager in the Django framework. Code quality check report at Sonarcloud
Version dependencies
The StageIT docker containers has python 3.9 as base. This version is at least supported until October 2025 See Status of python versions We can not upgrade to a higher python version due to the gfal2-python3 package (CERN) which is only supported until python version 3.9 As a result we are currently stuck to Django version 4.2 at this moment. See Status of Django versions The Django 4.2 support ends in April 2026. The result of Django 4.2 version is using Postgres version 12 or larger, currently version 13.
Contributing
Installing a development version can be done via. Activating a virtual environment is required:
pip install -r requirements.txt
Load environment variables:
source ./tests/files/.env
If you have created the stageit_postgres docker container before you most likely wanted to clean all the data as well:
docker volume rm deploy_pgdata
If you get an error that the volume is still in use
docker ps -a --filter volume=deploy_pgdata
docker stop CONTAINER_ID
docker rm CONTAINER_ID
and try again
You can create the database container by running:
# This is required for the provided docker-compose.yaml files to work, but is not used
# in development.
docker network create traefik_proxy
# replace `docker compose` by `docker-compose` if necessary
docker compose -f deploy/docker-compose.yaml -f deploy/docker-compose.development.yaml up -d --force-recreate postgres && sleep 2
Note: If you get something like:
● postgresql.service - PostgreSQL RDBMS
Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
Active: active (exited) since Mon 2023-10-09 07:33:47 CEST; 3 weeks 3 days ago
Main PID: 1811 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 18708)
Memory: 0B
CGroup: /system.slice/postgresql.service
You have to switch off the service by
systemctl stop postgresql.service
or systemctl disable postgresql.service
Fill the database with some content -> obsolete since it will break the DB structure (just here for info how you can load from sql): docker exec -i stageit_postgres /bin/bash -c "PGPASSWORD=stageit psql --username stageit stageit" < ./tests/files/stageit_dump.sql
The next commands should be performed in preferably the virtual environment you have been activated
Then the database can be migrated by running (in the virtual env):
python manage.py migrate
Add user local as superuser (in the virtual env):
DJANGO_SUPERUSER_PASSWORD=local python manage.py createsuperuser --username local --email no-reply@example.com --noinput
Now we have the database tables created, but no content. The only content that is initial relevant is the LTASite table and the email templates This can be obtained by loading two fixture files (in the virtual env):
python manage.py loaddata fixture/lta_sites fixture/email_templates
And finally a development instance can be run at local host (in the virtual env), see http://localhost:8000/stageit:
python manage.py runserver
Running the unittests (in the virtual env):
python manage.py test
If you want to create/update a diagram of the datamodel you can run (in the virtual env):
python manage.py graph_models -g -o documentation/static/img/stageit_models.png --pydot stage_api auth
If you want to have some small set of test data (having a staging request available), you can load the test fixture file (in the virtual env):
python manage.py loaddata ./tests/files/test_ltarequest.json
Macaroons
To simplify the LTA data access of the staged dataproducts, dCache provides support for data access based on the WEBDAV protocol and token-based access authorization called 'Macaroon'. A valid Macaroon will be stored per LTA site per LTA request and can be used to stage data via webdav. The macaroon will be sent within the notification email or the user has the ability to download it from the user request view .
So for every surl of the LTARequest: Check if surl (LTASite) does not have a LTAMacaroon, then
- Determine the root_caveat_path of the surl; This is the path after surl_pattern and before the <project_code> See some examples
- Send POST request for macaroon using root_caveat_path
- Create LTAMacaroon object for request and store in the DB
The macaroon is valid for 6 days (this is configurable per LTA Site) There is one exception to NOT generate macaroons! For LDV Staging service, the generation of macaroons is superfluous and should be skipped. Unfortunately the user is configured as 'admin' (not LDV). It a kind of quick and dirty, but if user is 'admin' then no macaroon will be created/stored.
Acknowledgements

