This note was drafted following discussion at ASTRON on 2021-10-21. Participants in the discussion were:
- Roy de Goei
- Yan Grange
- Klaas Kliffen
- Mattia Mancini
- John Swinbank
- Nico Vermaas
- Dany Vohl
This page proposes an update to the design of the ESAP system to resolve ongoing discussions around the purpose and use of the “shopping basket” and to properly incorporate asynchronous workflows. Specifically, it addresses the following questions:
- How can the shopping basket scale to handle bulk query results (thousands, millions, or even billions of rows)?
- Should the shopping basket store query results directly, or pointers/URLs/queries/other artefacts that make it possible to access the results of the query?
- How can asynchronous services be incorporated into the ESAP design?
- How should non-query-driven services interact with the shopping basket?
The ESAP interaction model
The basic model for user interaction with the ESAP system is that the user dispatches a request to a service, the service optionally ingests part or all of the contents of the shopping basket, and the service appends results to the shopping basket.
For example, this model applies to an archive query: the user sends a request to the service with some set of query parameters, the service does not refer to other shopping basket entries, and the service returns a set of results which are appended to the shopping basket.
However, this model also addresses more general services, including:
- The user sends a request to an interactive data analysis service (e.g. BinderHub) to start a Jupyter notebook session. The contents of the user's shopping basket are loaded into the notebook. Results generated by the user while using the notebook are appended to the shopping basket, and are so available to other services in ESAP.
- The user sends a request to a “positional cross match” service, which identifies sources which appear in multiple source catalogues based on positional coincidence (i.e. if two astronomical surveys see something at the same place in the sky, it's probably the same object). The cross-match service reads input catalogues from the user's shopping basket. The service then appends a new catalogue to the shopping basket, which consists of the resultant matched source list.
A variety of external archive services --- e.g. IVOA TAP, SDSS CasJobs --- already support asynchronous queries, and it is likely that this will become increasingly prevalent in future: it is impractical to expect database queries to return on interactive timescales as the number of sources grows extremely large. For example, the Rubin/LSST database, Qserv, explicitly adopts a “shared scan” model whereby queries are batched and executed periodically.
Supporting asynchronous queries of this sort requires some redesign of the ESAP execution model. Specifically, ESAP be able to dispatch queries asynchronously, await the results, and notify the user when those results become available.
This document proposes that --- for consistency and simplicity --- that asynchronous model be adopted when accessing every service. That is, even if a service returns results quickly and does not in itself provide an asynchronous API, ESAP should not handle it as part of its core loop. Rather, the query is sent to an ESAP-worker process which communicates asynchronously with the ESAP core. That worker is responsible for all communications with the external service, and notifies the ESAP core when results are ready.
Note that this model applies to services which are not performing archive queries. Referring to the same examples as above:
- The IDA service is intrinsically asynchronous as far as ESAP is concerned. Optionally, a space in the shopping basket could be reserved for results that the service may ultimately return.
- The positional cross match service will likely take some time to compute. Blocking the ESAP core loop on this would be unfortunate: better to ensure this is handled by an asynchronous worker.
As the user navigates through ESAP, dispatching queries or performing other actions, a number of asynchronous “jobs” will be started on their behalf. ESAP will maintain a list of the jobs which have been started for each user, and enable the user to view that list. Each entry in the list should provide the user with meaningful information the status of the job. This might include, for example, the submission time, the time at which the query was completed, and the total number of results returned. The user should be able to cancel any job which has not yet been started.
The list of jobs is persistent: the user can log off from ESAP and jobs will continue being executed in the background. When the user returns, they can log back in and check on the current status of their jobs.
Future versions of ESAP should provide a notification system to inform users when jobs have been completed. This could include --- for example --- sending mail, or pushing a system notification from the web. This functionality is not required for a “minimum viable” asynchronous ESAP system.
When a job has been completed, the user can inspect its outputs in through the job listing. If the results are of interest, the user can request that they be transferred to the shopping basket for further analysis.
Hierarchical shopping basket
As described above, entries are added to the shopping basket on user request, after they have first been inspected in the jobs listing. Each entry in the shopping basket refers to the total results of an interaction with an external service. That is, if a query to a source list returns multiple sources, a single entry is added to the shopping basket which contains details of all of those sources. This entry should maintain all metadata or additional information returned by the service. This means that, for example, the complete metadata returned by IVOA TAP services should be preserved intact.
Each shopping basket entry should be linked to the entry in the job listing which corresponds to the action which produced the data captured in that entry.
Where possible, the user should be able to hierarchically “drill down” into query results. For example, at the highest level, the shopping basket might record that the query returned a VOTable with 100 entries; the user should be able to click through to examine those entries. Some query results might be opaque; in these cases, no drill down is available. Where appropriate, the drill-down should be paginated, and only fetched from the query service on demand (that is, rather than trying to load billions of sources into memory, we should fetch only the first e.g. 30 results for display). This will require support from the upstream service, so may be possible in every case.
The figure below shows a mock-up of how the interface to this shopping basket might appear.
Editing basket entries; provenance
An important aspect of the current (i.e. prior to this proposal) ESAP experience is that the user can edit the contents of their shopping basket: they can select individual elements from the shopping basket for further processing. The new system should continue to make this possible whenever practical, e.g. by providing an interface which enables users to select individual rows within the drilled-down user interface. Some upstream services may not make this possible; ESAP should degrade gracefully (e.g. by not showing the corresponding UI elements) when this functionality is not available.
This system extends naturally to make it possible to track the provenance of shopping basket entries. Each high-level entry is tagged with the name of the service that provided it and any inputs that were provided to that service. For a simple query service, that might include the user-supplied parameters; for a service that takes inputs from the shopping basket (like the cross-match service), it would include the name of those inputs. Any user-driven selections or modifications would similarly be logged, so that the complete origin and history of each entry is tracked.