These are a kind of a mix of an overview and instructions on setting things
up. Given time, we'd probably want to split those aspects; for now, just put
them in the overview.
ESAP has the capability to run “asynchronous” jobs — that is, jobs which do not return a result immediately, but which are run in the background over a period of time with the expectation that the user will eventually return to collect output.
Asynchronous jobs are appropriate in various situations, including:
- Long running database queries;
- Batch processing tasks.
This document provides a brief overview of the Asynchronous ESAP system.
The deployment information is intended to be complementary to the information on deployment elsewhere in this wiki.
## System components
Asynchronous ESAP consists of at least 4 applications:
* ESAP Front-end
* ESAP API Gateway
* RabbitMQ instance
* One or more Worker instances
## Implementation diagram
For an interactive diagram, see [here](https://drive.google.com/file/d/1L9hlhJarf0Cvpe8Fik75xU0JkPjxenha/view?usp=sharing)
For RabbitMQ, you can use the following `docker-compose.yml`which makes it available on `localhost` which is the default used in development.
```
services:
rabbitmq:
image: rabbitmq:3.9-alpine
restart: always # persist over reboot
ports:
- "5672:5672"
```
You can bring the container up via:
```bash
docker-compose -f docker-compose.yml up -d
```
Alternatively, you can set the `CELERY_BROKER_URL` environment variable for both the API and the Worker application to point to another host such as a remote system or another docker container name.
### Order of operations
1. Make sure that RabbitMQ service is running.
2. Migrate (only needed the first time or when the database has been changed)
3. Run API and Worker and Front-end
### API
The API can be found in the [API Gateway repository](https://git.astron.nl/astron-sdc/escape-wp5/esap-api-gateway)
The [UWS app](https://git.astron.nl/astron-sdc/django-uws) is published on Pypi and can be installed via `pip install django-uws`. It is already included in the `requirements.txt` files.
You can safely ignore the instructions in the `README.md`; this already has been configured for ESAP.
Make sure that all python dependencies are installed by using the `esap/requirements/dev.txt` file, run this from the root of the repository.
Before starting the API, make sure that all migrations have been done.
Instructions for the migration can be found [here](https://git.astron.nl/astron-sdc/escape-wp5/esap-api-gateway/-/wikis/Build-&-Deploy-ESAP/Development-mode-(localhost)).
In addition for the `uws` app you should run the following migration:
An example `hello world` query worker can be found [here](https://git.astron.nl/astron-sdc/escape-wp5/esap-worker), please look at the [README.md](https://git.astron.nl/astron-sdc/escape-wp5/esap-worker/-/blob/main/README.md) for instructions on how to run it.
#### Adding worker types
Please read the documentation [here](https://git.astron.nl/astron-sdc/escape-wp5/esap-api-gateway/-/wikis/Adding-a-new-worker-type)
#### Debugging a worker:
To run a worker in an interactive debugging session, you can use the following `.vscode/launch.json` configuration in VS Code (PyCharm should be able to be configured in a similar way):
```
{
"version": "0.2.0",
"configurations": [
{
"name": "Python: Worker",
"type": "python",
"request": "launch",
"env": {
"DJANGO_SETTINGS_MODULE": "esap.settings.async"
},
"module": "celery",
"args": [
"-A",
"esap",
"worker",
"-l",
"INFO"
],
"cwd": "${workspaceFolder}/esap/",
"console": "integratedTerminal",
"justMyCode": false
}
]
}
```
## Testing the API:
You will need to have an active session to be able to interact with the API.
The easiest way to do this is a run a local instance of ESAP (both GUI and API) and access the `Asynchronous Jobs` tab.
Alternatively you can copy the JWT token from from the GUI and interact via the shell or other tools. You can use the following shell commands to create and start a job (change `curl` to your tool of preference, name of the arguments may differ in that case):
