Skip to content
Snippets Groups Projects
Commit 34fc6826 authored by John Swinbank's avatar John Swinbank
Browse files

Import old capabilities description

parent 4da13ab9
Branches
No related tags found
No related merge requests found
...@@ -33,7 +33,7 @@ Specifically, we will consider: ...@@ -33,7 +33,7 @@ Specifically, we will consider:
Broadly, this report is structured in two parts. Broadly, this report is structured in two parts.
In the first part, we provide essential context by describing the \pgls{ESAP} system. In the first part, we provide essential context by describing the \pgls{ESAP} system.
In \cref{sec:vision} we the overall vision for the \pgls{ESAP} system: what did we set out to build? In \cref{sec:vision} we the overall vision for the \pgls{ESAP} system: what did we set out to build?
\Cref{sec:current} takes that further by describing what is currently available: if one were to deploy and operate an \pgls{ESAP} system today, what would it consist of? \Cref{sec:delivered} takes that further by describing what is currently available: if one were to deploy and operate an \pgls{ESAP} system today, what would it consist of?
The second part of this document moves on to asses how successful that vision and implementation have been in addressing user --- and in particular \gls{ESFRI} --- needs. The second part of this document moves on to asses how successful that vision and implementation have been in addressing user --- and in particular \gls{ESFRI} --- needs.
In particular, \cref{sec:requirements} tabulates the documented requirements and compares them to delivered product; \cref{sec:usecases} discusses how \pgls{ESAP} relates to all the use cases collected by the \gls{ESCAPE} project; and \cref{sec:esfri} describes current \gls{ESFRI} engagement with the platform.. In particular, \cref{sec:requirements} tabulates the documented requirements and compares them to delivered product; \cref{sec:usecases} discusses how \pgls{ESAP} relates to all the use cases collected by the \gls{ESCAPE} project; and \cref{sec:esfri} describes current \gls{ESFRI} engagement with the platform..
......
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
This section presents a brief overview of the vision for and design of \pgls{ESAP}. This section presents a brief overview of the vision for and design of \pgls{ESAP}.
It supplements and expands upon earlier discussions \cite{ESCAPE-GA, ESCAPE-D5.2, ESCAPE-D5.3} to describe current thinking about how \pgls{ESAP} can best meet its goals. It supplements and expands upon earlier discussions \cite{ESCAPE-GA, ESCAPE-D5.2, ESCAPE-D5.3} to describe current thinking about how \pgls{ESAP} can best meet its goals.
Note that not all of the capabilities described in this section are currently available in the \pgls{ESAP} codebase: see \cref{sec:current} for a description of the current state of the art, and \cref{sec:future} for future development plans. Note that not all of the capabilities described in this section are currently available in the \pgls{ESAP} codebase: see \cref{sec:delivered} for a description of the current state of the art, and \cref{sec:future} for future development plans.
\subsection{High-Level Summary} \subsection{High-Level Summary}
\label{sec:vision:summary} \label{sec:vision:summary}
...@@ -248,4 +248,4 @@ Instead, the team has focused on: ...@@ -248,4 +248,4 @@ Instead, the team has focused on:
\end{itemize} \end{itemize}
The services which are supported at the completion of the \gls{ESCAPE} project are described in \cref{sec:current}, but furthers will be added by the future \pgls{ESAP} open source project (described in \cref{sec:future}). The services which are supported at the completion of the \gls{ESCAPE} project are described in \cref{sec:delivered}, but furthers will be added by the future \pgls{ESAP} open source project (described in \cref{sec:future}).
\section{Current Capabilities}
\label{sec:current}
\section{Delivered Capabilities}
\label{sec:delivered}
This section provides an overview of the \pgls{ESAP} capabilities delivered within the scope of the \gls{ESCAPE} project, and describes how they relate to the vision described in \cref{sec:vision}.
The delivered version of \gls{ESAP} does not meet the full scope of the vision: this is expected, and reflects the wide scope of the problem domain that \gls{ESAP} addresses and the potential for future development (discussed in \cref{sec:future}).
However, these delivered capabilities do provide a flexible and compelling system that \glspl{ESFRI} and other projects can build upon, as discussed in \cref{sec:esfri}.
Most of the capabilities described here are available through a test deployment of \pgls{ESAP} at ASTRON.
This system may be accessed at \url{https://sdc-dev.astron.nl/esap-gui/}.
Note that this system is provided without support to facilitate development and demonstration; the service is not expected to be reliable, nor to be available indefinitely.
\subsection{User Interface}
\label{sec:delivered:ui}
The front page of the \pgls{ESAP} test deployment at ASTRON is shown in \cref{fig:esap:front}.
It presents an attractive and usable web-based interface that enables the user to rapidly discover which services are configured in this \pgls{ESAP} instance (in the top bar; here, \texttt{Archives}, \texttt{Interactive Analysis} and \texttt{IVOA-SAMP}).
The interface highlights the available archives: in this case, corresponding to Apertif \autocite{2018wtfa.confE..16O}, the ASTRON \gls{VO} system\footnote{\url{https://vo.astron.nl/}}, and the Zooniverse citizen science system\footnote{\url{https://www.zooniverse.org/}}.
As discussed in \cref{sec:vision:capabilities:ui}, the user interface shown is a separate process from the core “API Gateway” back-end system for \pgls{ESAP}.
The interface is a cross-platform web application implemented in React\footnote{\url{https://reactjs.org/}}, which communicates with the API Gateway over a \gls{REST} interface.
The API Gateway itself is written in Python, using Django\footnote{\url{https://www.djangoproject.com/}} and its companion \gls{REST} framework\footnote{\url{https://www.django-rest-framework.org/}}.
In principle, it would be possible for alternative or specialist user interfaces to communicate with the API Gateway using the same interface; as of now, however, the authors are aware of no other \pgls{ESAP} interfaces.
\begin{figure}
\centering
\includegraphics[width=0.66\textwidth]{images/esap-front.png}
\caption{The front page of the \glsentrytext{ESAP} test deployment at ASTRON.}
\label{fig:esap:front}
\end{figure}
\subsection{Authentication and Authorization}
\label{sec:delivered:auth}
\begin{figure}
\centering
\includegraphics[width=0.66\textwidth]{images/esap-auth.png}
\caption{Using \glsentrytext{ESCAPE} \glsentrytext{IAM} to authenticate with \glsentrytext{ESAP}.}
\label{fig:esap:auth}
\end{figure}
\pgls{ESAP} is fully integrated with the \gls{ESCAPE} project's \gls{IAM} service.
\cref{fig:esap:auth} shows an example of a user authenticating with the \pgls{ESAP} test system using \gls{ESCAPE} \gls{IAM}.
After the user has been authenticated, \pgls{ESAP} should automatically be able to forward their credentials to downstream services, to prevent the user from having to re-authenticate multiple times.
Final integration of this capability is still ongoing.
\subsection{Data Orchestration within \glsentrytext{ESAP}}
\label{sec:delivered:dm}
\begin{figure}
\centering
\includegraphics[width=0.66\textwidth]{images/esap-basket.png}
\caption{The “shopping basket” viewed through the \glsentrytext{ESAP} web interface.}
\label{fig:esap:basket}
\end{figure}
Shopping basket capabilities, as described in \cref{sec:vision:capabilities:orch}, are available in the current version of \gls{ESAP} to users who have authenticated through \gls{ESCAPE} \gls{IAM} (\cref{sec:delivered:auth}).
\cref{fig:esap:basket} shows an example of the shopping basket visualized through the web interface.
Note that the formatting of the contents of the basket varies depending on the source of the data.
Future work will provide additional flexibility for managing the contents of the shopping basket.
\subsection{Data Discovery}
\label{sec:delivered:data}
\begin{figure}
\centering
\begin{subfigure}{0.48\textwidth}
\includegraphics[width=\textwidth]{images/esap-apertif-results.png}
\subcaption{Apertif.}
\label{sub:esap:query:apertif}
\end{subfigure}
\begin{subfigure}{0.48\textwidth}
\includegraphics[width=\textwidth]{images/esap-zooniverse-results.png}
\subcaption{Zooniverse.}
\label{sub:esap:query:zooniverse}
\end{subfigure}
\caption{Query results displayed though \glsentrytext{ESAP}.}
\label{fig:esap:query}
\end{figure}
Interfaces between \pgls{ESAP} and a variety of archive systems have been implemented, to varying degrees of polish and reliability.
At time of writing, the Apertif, ASTRON \gls{VO} and Zooniverse interfaces are the most reliable and functional.
Examples are shown in \cref{fig:esap:query}.
Note that the query interface and the form of the results returned adapt appropriately depending on the nature of the archive in question, so that --- for example --- the user is given the opportunity to specify celestial coordinates when querying the Apertif archive, but not when querying the Zooniverse system.
The leftmost columns in the result interface give the user the opportunity to select individual rows from the results to add them to their shopping basket (\cref{sec:delivered:dm}).
Future versions of \pgls{ESAP} will offer more flexibility here; this might include, for example, bulk addition of many rows to the basket without selecting each individually.
Currently, archives are queried individually and synchronously.
That is, the query is targeted towards one archive, and the user interface blocks until a response is received.
This may not be appropriate for all archives, in particular those which offer access to extremely large catalogues.
Future upgrades are expected to include dispatching a query to multiple archives (with appropriate conversion from the form in which it is specified by the user to service-specific interfaces), and asynchronous queries (in which the user can log off and return later to inspect the query results), perhaps based on the \gls{IVOA} \gls{UWS} pattern \autocite{2016ivoa.spec.1024H}.
\subsection{\glsentrytext{SAMP}}
\label{sec:delivered:samp}
\pgls{ESAP} incorporates support for \gls{SAMP} using the sampjs library\footnote{\url{https://github.com/astrojs/sampjs}}.
Users can transmit data to the ESAP web interface from other \gls{SAMP}-enabled applications, and from there add it to their shopping basket.
Transmission in the other direction --- from \pgls{ESAP} to other applications --- is not yet available.
\subsection{\glsentrydesc{IDA}}
\label{sec:delivered:ida}
\begin{figure}
\centering
\begin{subfigure}[t]{0.48\textwidth}
\includegraphics[width=\textwidth]{images/esap-workflow-select.png}
\subcaption{Workflow selection.}
\label{sub:esap:ida:basket}
\end{subfigure}
\begin{subfigure}[t]{0.48\textwidth}
\includegraphics[width=\textwidth]{images/esap-analysis.png}
\subcaption{Jupyter notebook.}
\label{sub:esap:ida:notebook}
\end{subfigure}
\caption{The \glsentrytext{ESAP} \glsentrytext{IDA} workflow.}
\label{fig:esap:ida}
\end{figure}
The current version of \pgls{ESAP} directly embeds information which enables it to execute a limited number of workflows, described in the form of Jupyter notebooks.
A user interface exists to make it possible to execute these notebooks on a number of different analysis systems, but the existing test system only provides access to MyBinder\footnote{\url{https://www.mybinder.org/}}.
Upon choosing an appropriate workflow and analysis service, the user is redirected to the notebook environment.
In that environment, a Python library --- initially developed specifically to address Zooniverse classification data, but now adapted to a wide range of data types --- makes it possible for them to access their \pgls{ESAP} shopping basket, and hence to download or otherwise manipulate the data that they have selected.
Selecting the workflow and execution service, then working in the notebook, are shown in \cref{fig:esap:ida}.
There some usability and polish issues remain outstanding regarding shopping basket integration in the notebook: currently, the user must manually use the Python \gls{API} to access data, including supplying an appropriate authentication token (copied-and-pasted from the \pgls{ESAP} interface).
Ultimately, this process should be improved: data should be automatically staged to the notebook system, and no additional authentication should be required.
This system is not yet integrated with the \gls{WP}3 \gls{OSSR}, or any other external software or service collections.
Discussions are currently underway between \gls{WP}3 and \gls{WP}5 to clarify these interfaces.
When complete, this integration will provide a much richer experience for the user to discover and schedule workflows.
Although the current \gls{IDA} system focuses on Jupyter notebooks, we expect to extend this service to address other forms of interaction in future work, likely making use of the Rosetta system (\cref{sec:vision:extensibility}).
\subsection{Managed Database}
Development of the Managed Database service, described in \cref{sec:vision:capabilities:db} started in summer 2021.
A functional prototype is now available, and development is progressing rapidly.
As yet, however, the service is not fully integrated with the \pgls{ESAP} deployment at ASTRON.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment