Asynchronous Processing

ESAP has the capability to run “asynchronous” jobs — that is, jobs which do not return a result immediately, but which are run in the background over a period of time with the expectation that the user will eventually return to collect output. Asynchronous jobs are appropriate in various situations, including:

• Long running database queries;
• Batch processing tasks.

This document provides a brief overview of the Asynchronous ESAP system. The deployment information is intended to be complementary to the information on deployment elsewhere in this wiki.

System components

Asynchronous ESAP consists of at least 4 applications:

• ESAP Front-end
• ESAP API Gateway
• RabbitMQ instance
• One or more Worker instances

Implementation diagram

For an interactive diagram, see here

Running a development build environment:

For RabbitMQ, you can use the following docker-compose.ymlwhich makes it available on localhost which is the default used in development.

services:
  rabbitmq:
    image: rabbitmq:3.9-alpine
    restart: always # persist over reboot
    ports:
      - "5672:5672"

You can bring the container up via:

docker-compose -f docker-compose.yml up -d

Alternatively, you can set the CELERY_BROKER_URL environment variable for both the API and the Worker application to point to another host such as a remote system or another docker container name.

Order of operations

1. Make sure that RabbitMQ service is running.
2. Migrate (only needed the first time or when the database has been changed)
3. Run API and Worker and Front-end

API

The API can be found in the API Gateway repository

The UWS app is published on Pypi and can be installed via pip install django-uws. It is already included in the requirements.txt files. You can safely ignore the instructions in the README.md; this already has been configured for ESAP.

Make sure that all python dependencies are installed by using the esap/requirements/dev.txt file, run this from the root of the repository.

Before starting the API, make sure that all migrations have been done. Instructions for the migration can be found here. In addition for the uws app you should run the following migration:

python3 manage.py migrate uws --database=uws --settings=esap.settings.dev --no-input

Then the backend can be started by running:

python3 manage.py runserver 5555 --settings=esap.settings.dev

Worker

An example hello world query worker can be found here, please look at the README.md for instructions on how to run it.

Adding worker types

Please read the documentation here.

Debugging a worker:

To run a worker in an interactive debugging session, you can use the following .vscode/launch.json configuration in VS Code (PyCharm should be able to be configured in a similar way):

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Worker",
            "type": "python",
            "request": "launch",
            "env": {
                "DJANGO_SETTINGS_MODULE": "esap.settings.async"
            },
            "module": "celery",
            "args": [
                "-A",
                "esap",
                "worker",
                "-l",
                "INFO"
            ],
            "cwd": "${workspaceFolder}/esap/",
            "console": "integratedTerminal",
            "justMyCode": false
        }
    ]
}

Testing the API:

You will need to have an active session to be able to interact with the API. The easiest way to do this is a run a local instance of ESAP (both GUI and API) and access the Asynchronous Jobs tab.

Alternatively you can copy the JWT token from from the GUI and interact via the shell or other tools. You can use the following shell commands to create and start a job (change curl to your tool of preference, name of the arguments may differ in that case):

# List all jobs
curl -s http://localhost:5555/esap-api/uws/jobs/ \
    -H "Authorization: Bearer <jwt_token>"

# Create a job
curl -s \
    http://localhost:5555/esap-api/uws/jobs/ \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer <jwt_token>" \
    -d '{"runId": "test", "parameters": [{"key": "type", "value": "query.http"}, {"key": "someparam", "value": "somevalue"}]}' | jq

# Start a job
curl -s -d "PHASE=RUN" http://localhost:5555/esap-api/uws/jobs/1/phase/ \
    -H "Authorization: Bearer <jwt_token>"

# Check the output
curl -s -H "Authorization: Bearer <jwt_token>" http://localhost:5555/esap-api/uws/jobs/1/