ESAP has the capability to run “asynchronous” jobs — that is, jobs which do not return a result immediately, but which are run in the background over a period of time with the expectation that the user will eventually return to collect output. Asynchronous jobs are appropriate in various situations, including:
- Long running database queries;
- Batch processing tasks.
This document provides a brief overview of the Asynchronous ESAP system. The deployment information is intended to be complementary to the information on deployment elsewhere in this wiki.
System components
Asynchronous ESAP consists of at least 4 applications:
- ESAP Front-end
- ESAP API Gateway
- RabbitMQ instance
- One or more Worker instances
Implementation diagram
For an interactive diagram, see here
Running a development build environment:
For RabbitMQ, you can use the following docker-compose.yml
which makes it available on localhost
which is the default used in development.
services:
rabbitmq:
image: rabbitmq:3.9-alpine
restart: always # persist over reboot
ports:
- "5672:5672"
You can bring the container up via:
docker-compose -f docker-compose.yml up -d
Alternatively, you can set the CELERY_BROKER_URL
environment variable for both the API and the Worker application to point to another host such as a remote system or another docker container name.
Order of operations
- Make sure that RabbitMQ service is running.
- Migrate (only needed the first time or when the database has been changed)
- Run API and Worker and Front-end
API
The API can be found in the API Gateway repository
The UWS app is published on Pypi and can be installed via pip install django-uws
. It is already included in the requirements.txt
files.
You can safely ignore the instructions in the README.md
; this already has been configured for ESAP.
Make sure that all python dependencies are installed by using the esap/requirements/dev.txt
file, run this from the root of the repository.
Before starting the API, make sure that all migrations have been done.
Instructions for the migration can be found here.
In addition for the uws
app you should run the following migration:
python3 manage.py migrate uws --database=uws --settings=esap.settings.dev --no-input
Then the backend can be started by running:
python3 manage.py runserver 5555 --settings=esap.settings.dev
Worker
An example hello world
query worker can be found here, please look at the README.md for instructions on how to run it.
Adding worker types
Please read the documentation here.
Debugging a worker:
To run a worker in an interactive debugging session, you can use the following .vscode/launch.json
configuration in VS Code (PyCharm should be able to be configured in a similar way):
{
"version": "0.2.0",
"configurations": [
{
"name": "Python: Worker",
"type": "python",
"request": "launch",
"env": {
"DJANGO_SETTINGS_MODULE": "esap.settings.async"
},
"module": "celery",
"args": [
"-A",
"esap",
"worker",
"-l",
"INFO"
],
"cwd": "${workspaceFolder}/esap/",
"console": "integratedTerminal",
"justMyCode": false
}
]
}
Testing the API:
You will need to have an active session to be able to interact with the API.
The easiest way to do this is a run a local instance of ESAP (both GUI and API) and access the Asynchronous Jobs
tab.
Alternatively you can copy the JWT token from from the GUI and interact via the shell or other tools. You can use the following shell commands to create and start a job (change curl
to your tool of preference, name of the arguments may differ in that case):
# List all jobs
curl -s http://localhost:5555/esap-api/uws/jobs/ \
-H "Authorization: Bearer <jwt_token>"
# Create a job
curl -s \
http://localhost:5555/esap-api/uws/jobs/ \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <jwt_token>" \
-d '{"runId": "test", "parameters": [{"key": "type", "value": "query.http"}, {"key": "someparam", "value": "somevalue"}]}' | jq
# Start a job
curl -s -d "PHASE=RUN" http://localhost:5555/esap-api/uws/jobs/1/phase/ \
-H "Authorization: Bearer <jwt_token>"
# Check the output
curl -s -H "Authorization: Bearer <jwt_token>" http://localhost:5555/esap-api/uws/jobs/1/