diff --git a/contents/1-intro.tex b/contents/1-intro.tex index 9b3a2fbd9d817a21f09f31f914c14c861103e50f..a069159ec4012dd9f1193aa91b895654d898a992 100644 --- a/contents/1-intro.tex +++ b/contents/1-intro.tex @@ -33,7 +33,7 @@ Specifically, we will consider: Broadly, this report is structured in two parts. In the first part, we provide essential context by describing the \pgls{ESAP} system. In \cref{sec:vision} we the overall vision for the \pgls{ESAP} system: what did we set out to build? -\Cref{sec:current} takes that further by describing what is currently available: if one were to deploy and operate an \pgls{ESAP} system today, what would it consist of? +\Cref{sec:delivered} takes that further by describing what is currently available: if one were to deploy and operate an \pgls{ESAP} system today, what would it consist of? The second part of this document moves on to asses how successful that vision and implementation have been in addressing user --- and in particular \gls{ESFRI} --- needs. In particular, \cref{sec:requirements} tabulates the documented requirements and compares them to delivered product; \cref{sec:usecases} discusses how \pgls{ESAP} relates to all the use cases collected by the \gls{ESCAPE} project; and \cref{sec:esfri} describes current \gls{ESFRI} engagement with the platform.. diff --git a/contents/2-vision.tex b/contents/2-vision.tex index 81566453e3712cbf9f54bab9f0c27e134105dc7a..b1f9efba6831f2827ea7c40137f601395cb9e79d 100644 --- a/contents/2-vision.tex +++ b/contents/2-vision.tex @@ -3,7 +3,7 @@ This section presents a brief overview of the vision for and design of \pgls{ESAP}. It supplements and expands upon earlier discussions \cite{ESCAPE-GA, ESCAPE-D5.2, ESCAPE-D5.3} to describe current thinking about how \pgls{ESAP} can best meet its goals. -Note that not all of the capabilities described in this section are currently available in the \pgls{ESAP} codebase: see \cref{sec:current} for a description of the current state of the art, and \cref{sec:future} for future development plans. +Note that not all of the capabilities described in this section are currently available in the \pgls{ESAP} codebase: see \cref{sec:delivered} for a description of the current state of the art, and \cref{sec:future} for future development plans. \subsection{High-Level Summary} \label{sec:vision:summary} @@ -248,4 +248,4 @@ Instead, the team has focused on: \end{itemize} -The services which are supported at the completion of the \gls{ESCAPE} project are described in \cref{sec:current}, but furthers will be added by the future \pgls{ESAP} open source project (described in \cref{sec:future}). +The services which are supported at the completion of the \gls{ESCAPE} project are described in \cref{sec:delivered}, but furthers will be added by the future \pgls{ESAP} open source project (described in \cref{sec:future}). diff --git a/contents/3-current.tex b/contents/3-current.tex deleted file mode 100644 index c9f0c84b1fb1cfd5dfd2f89c1d23a09417c9935f..0000000000000000000000000000000000000000 --- a/contents/3-current.tex +++ /dev/null @@ -1,2 +0,0 @@ -\section{Current Capabilities} -\label{sec:current} diff --git a/contents/3-delivered.tex b/contents/3-delivered.tex new file mode 100644 index 0000000000000000000000000000000000000000..8dcdb9397cf378309a9f7501966c18e4f57d1d9c --- /dev/null +++ b/contents/3-delivered.tex @@ -0,0 +1,143 @@ +\section{Delivered Capabilities} +\label{sec:delivered} + +This section provides an overview of the \pgls{ESAP} capabilities delivered within the scope of the \gls{ESCAPE} project, and describes how they relate to the vision described in \cref{sec:vision}. +The delivered version of \gls{ESAP} does not meet the full scope of the vision: this is expected, and reflects the wide scope of the problem domain that \gls{ESAP} addresses and the potential for future development (discussed in \cref{sec:future}). +However, these delivered capabilities do provide a flexible and compelling system that \glspl{ESFRI} and other projects can build upon, as discussed in \cref{sec:esfri}. + +Most of the capabilities described here are available through a test deployment of \pgls{ESAP} at ASTRON. +This system may be accessed at \url{https://sdc-dev.astron.nl/esap-gui/}. +Note that this system is provided without support to facilitate development and demonstration; the service is not expected to be reliable, nor to be available indefinitely. + +\subsection{User Interface} +\label{sec:delivered:ui} + +The front page of the \pgls{ESAP} test deployment at ASTRON is shown in \cref{fig:esap:front}. +It presents an attractive and usable web-based interface that enables the user to rapidly discover which services are configured in this \pgls{ESAP} instance (in the top bar; here, \texttt{Archives}, \texttt{Interactive Analysis} and \texttt{IVOA-SAMP}). +The interface highlights the available archives: in this case, corresponding to Apertif \autocite{2018wtfa.confE..16O}, the ASTRON \gls{VO} system\footnote{\url{https://vo.astron.nl/}}, and the Zooniverse citizen science system\footnote{\url{https://www.zooniverse.org/}}. + +As discussed in \cref{sec:vision:capabilities:ui}, the user interface shown is a separate process from the core “API Gateway” back-end system for \pgls{ESAP}. +The interface is a cross-platform web application implemented in React\footnote{\url{https://reactjs.org/}}, which communicates with the API Gateway over a \gls{REST} interface. +The API Gateway itself is written in Python, using Django\footnote{\url{https://www.djangoproject.com/}} and its companion \gls{REST} framework\footnote{\url{https://www.django-rest-framework.org/}}. +In principle, it would be possible for alternative or specialist user interfaces to communicate with the API Gateway using the same interface; as of now, however, the authors are aware of no other \pgls{ESAP} interfaces. + +\begin{figure} +\centering +\includegraphics[width=0.66\textwidth]{images/esap-front.png} +\caption{The front page of the \glsentrytext{ESAP} test deployment at ASTRON.} +\label{fig:esap:front} +\end{figure} + +\subsection{Authentication and Authorization} +\label{sec:delivered:auth} + +\begin{figure} +\centering +\includegraphics[width=0.66\textwidth]{images/esap-auth.png} +\caption{Using \glsentrytext{ESCAPE} \glsentrytext{IAM} to authenticate with \glsentrytext{ESAP}.} +\label{fig:esap:auth} +\end{figure} + +\pgls{ESAP} is fully integrated with the \gls{ESCAPE} project's \gls{IAM} service. +\cref{fig:esap:auth} shows an example of a user authenticating with the \pgls{ESAP} test system using \gls{ESCAPE} \gls{IAM}. + +After the user has been authenticated, \pgls{ESAP} should automatically be able to forward their credentials to downstream services, to prevent the user from having to re-authenticate multiple times. +Final integration of this capability is still ongoing. + +\subsection{Data Orchestration within \glsentrytext{ESAP}} +\label{sec:delivered:dm} + +\begin{figure} +\centering +\includegraphics[width=0.66\textwidth]{images/esap-basket.png} +\caption{The “shopping basket” viewed through the \glsentrytext{ESAP} web interface.} +\label{fig:esap:basket} +\end{figure} + +Shopping basket capabilities, as described in \cref{sec:vision:capabilities:orch}, are available in the current version of \gls{ESAP} to users who have authenticated through \gls{ESCAPE} \gls{IAM} (\cref{sec:delivered:auth}). + +\cref{fig:esap:basket} shows an example of the shopping basket visualized through the web interface. +Note that the formatting of the contents of the basket varies depending on the source of the data. + +Future work will provide additional flexibility for managing the contents of the shopping basket. + +\subsection{Data Discovery} +\label{sec:delivered:data} + +\begin{figure} +\centering +\begin{subfigure}{0.48\textwidth} +\includegraphics[width=\textwidth]{images/esap-apertif-results.png} +\subcaption{Apertif.} +\label{sub:esap:query:apertif} +\end{subfigure} +\begin{subfigure}{0.48\textwidth} +\includegraphics[width=\textwidth]{images/esap-zooniverse-results.png} +\subcaption{Zooniverse.} +\label{sub:esap:query:zooniverse} +\end{subfigure} +\caption{Query results displayed though \glsentrytext{ESAP}.} +\label{fig:esap:query} +\end{figure} + +Interfaces between \pgls{ESAP} and a variety of archive systems have been implemented, to varying degrees of polish and reliability. +At time of writing, the Apertif, ASTRON \gls{VO} and Zooniverse interfaces are the most reliable and functional. +Examples are shown in \cref{fig:esap:query}. +Note that the query interface and the form of the results returned adapt appropriately depending on the nature of the archive in question, so that --- for example --- the user is given the opportunity to specify celestial coordinates when querying the Apertif archive, but not when querying the Zooniverse system. + +The leftmost columns in the result interface give the user the opportunity to select individual rows from the results to add them to their shopping basket (\cref{sec:delivered:dm}). +Future versions of \pgls{ESAP} will offer more flexibility here; this might include, for example, bulk addition of many rows to the basket without selecting each individually. + +Currently, archives are queried individually and synchronously. +That is, the query is targeted towards one archive, and the user interface blocks until a response is received. +This may not be appropriate for all archives, in particular those which offer access to extremely large catalogues. +Future upgrades are expected to include dispatching a query to multiple archives (with appropriate conversion from the form in which it is specified by the user to service-specific interfaces), and asynchronous queries (in which the user can log off and return later to inspect the query results), perhaps based on the \gls{IVOA} \gls{UWS} pattern \autocite{2016ivoa.spec.1024H}. + +\subsection{\glsentrytext{SAMP}} +\label{sec:delivered:samp} + +\pgls{ESAP} incorporates support for \gls{SAMP} using the sampjs library\footnote{\url{https://github.com/astrojs/sampjs}}. +Users can transmit data to the ESAP web interface from other \gls{SAMP}-enabled applications, and from there add it to their shopping basket. +Transmission in the other direction --- from \pgls{ESAP} to other applications --- is not yet available. + +\subsection{\glsentrydesc{IDA}} +\label{sec:delivered:ida} + +\begin{figure} +\centering +\begin{subfigure}[t]{0.48\textwidth} +\includegraphics[width=\textwidth]{images/esap-workflow-select.png} +\subcaption{Workflow selection.} +\label{sub:esap:ida:basket} +\end{subfigure} +\begin{subfigure}[t]{0.48\textwidth} +\includegraphics[width=\textwidth]{images/esap-analysis.png} +\subcaption{Jupyter notebook.} +\label{sub:esap:ida:notebook} +\end{subfigure} +\caption{The \glsentrytext{ESAP} \glsentrytext{IDA} workflow.} +\label{fig:esap:ida} +\end{figure} + +The current version of \pgls{ESAP} directly embeds information which enables it to execute a limited number of workflows, described in the form of Jupyter notebooks. +A user interface exists to make it possible to execute these notebooks on a number of different analysis systems, but the existing test system only provides access to MyBinder\footnote{\url{https://www.mybinder.org/}}. + +Upon choosing an appropriate workflow and analysis service, the user is redirected to the notebook environment. +In that environment, a Python library --- initially developed specifically to address Zooniverse classification data, but now adapted to a wide range of data types --- makes it possible for them to access their \pgls{ESAP} shopping basket, and hence to download or otherwise manipulate the data that they have selected. + +Selecting the workflow and execution service, then working in the notebook, are shown in \cref{fig:esap:ida}. + +There some usability and polish issues remain outstanding regarding shopping basket integration in the notebook: currently, the user must manually use the Python \gls{API} to access data, including supplying an appropriate authentication token (copied-and-pasted from the \pgls{ESAP} interface). +Ultimately, this process should be improved: data should be automatically staged to the notebook system, and no additional authentication should be required. + +This system is not yet integrated with the \gls{WP}3 \gls{OSSR}, or any other external software or service collections. +Discussions are currently underway between \gls{WP}3 and \gls{WP}5 to clarify these interfaces. +When complete, this integration will provide a much richer experience for the user to discover and schedule workflows. + +Although the current \gls{IDA} system focuses on Jupyter notebooks, we expect to extend this service to address other forms of interaction in future work, likely making use of the Rosetta system (\cref{sec:vision:extensibility}). + +\subsection{Managed Database} + +Development of the Managed Database service, described in \cref{sec:vision:capabilities:db} started in summer 2021. +A functional prototype is now available, and development is progressing rapidly. +As yet, however, the service is not fully integrated with the \pgls{ESAP} deployment at ASTRON.