Batch Service Category

Summary

Batch computing has been implemented with the ability to launch singularity jobs on a DIRAC server.

Workflow

• Select Batch from main page.
  • Search for workflow.
  • Workflows are saved as JSON files, see below.
  • (In future it would be nice to be able to search for these in a repository).
• Select Workload system
  • Currently only CTADIRAC implemented.
• Edit Parameters
  • Edit the parameters which are pulled from the JSON file. Submit the job and then start it running.
• Monitoring
  • It is possible to monitor the asynchronous jobs from the main Asynchronous tab in ESAP.

Example JSON steering file

Example helloworld.json:

{
   "uuid": "fa2b5ebc-ee25-10ec-7ea1-0247ac120006",
   "type": "URL",
   "title": "ESAP CONCORDIA",
   "description": "This workflow is the example",
   "workflow": "CONCORDIA",
   "Parameters": {
     "singularity_image": "LOCATION of image (url)",
     "output_file_name ": "output.tar",
     "cputime": "500",
     "job_name": "ESAPTEST",
     "group_name": "ESAPTEST_GROUP",
     "dirac_path": "/vo.cta.in2p3.fr/user/g/ghughes/",
     "dirac_SE": "CC-IN2P3-USER",
     "arguement1": "1",
     "arguement2": "2",
     "arguement3": "3"
   }
}

Required parameters:

• cputime
• singularity_image
• output_file_name
• job_name
• group_name
• dirac_SE
• dirac_path

Implementation

GUI

Batch.js

Main batch routine for the GUI. Handles monitoring and submission. Loads both the JSON file workflows and the server locations from the batch YAML file It sends request directly to the ESAP-Worker, via axios posts directly to the uws/jobs:

axios.post(api_host + "uws/jobs/", content, {headers})

BatchParametersComponent.js

Paginates the parameters from the JSON file.

API

Batch implementation in the API is based on the interactive analysis implementation.

Worker

workers/batch.py

Needs to run in an environment with the DIRAC client. Currently an x509 certificate is also currently required and it must be initialized. The best way to do this currently is run the worker in a container along with the DIRAC client.

The worker sends jobs to DIRAC to submit and monitor jobs.

To monitor a job only the DIRAC jobID number is required.

When a job is submitted the parameters from the JSON file are also used.

If the Singularity file required is not yet downloaded the worker will also download the file.

Diagram