Skip to content
Snippets Groups Projects
Commit 6b5212b4 authored by John Swinbank's avatar John Swinbank
Browse files

Respond to feedback from Gareth Hughes

parent 1b272d10
Branches
Tags
No related merge requests found
...@@ -8,7 +8,7 @@ A fundamentally web-based application, it serves a key part of the interface bet ...@@ -8,7 +8,7 @@ A fundamentally web-based application, it serves a key part of the interface bet
At the time of writing, the funded \gls{ESCAPE} project is reaching its conclusion. At the time of writing, the funded \gls{ESCAPE} project is reaching its conclusion.
Substantial development effort has been invested into \pgls{ESAP} and the surrounding ecosystem of software and services. Substantial development effort has been invested into \pgls{ESAP} and the surrounding ecosystem of software and services.
At the same time, we do not believe that the development of \pgls{ESAP} itself is coming to an end: the technologies developed, and the workflows pioneered, have shown real potential for widespread future application. However, we do not believe that the development of \pgls{ESAP} itself is coming to an end: the technologies developed, and the workflows pioneered, have shown real potential for widespread future application.
This makes it an opportune moment to step back and assess the current state of and future plans for \pgls{ESAP}. This makes it an opportune moment to step back and assess the current state of and future plans for \pgls{ESAP}.
What have we achieved within the context of \gls{ESCAPE} so far? What have we achieved within the context of \gls{ESCAPE} so far?
How can we build upon these achievements in the future? How can we build upon these achievements in the future?
...@@ -36,4 +36,4 @@ In \cref{sec:vision} we the overall vision for the \pgls{ESAP} system: what did ...@@ -36,4 +36,4 @@ In \cref{sec:vision} we the overall vision for the \pgls{ESAP} system: what did
The second part of this document moves on to asses how successful that vision and implementation have been in addressing user --- and in particular \gls{ESFRI} --- needs. The second part of this document moves on to asses how successful that vision and implementation have been in addressing user --- and in particular \gls{ESFRI} --- needs.
In particular, \cref{sec:requirements} tabulates the documented requirements and compares them to delivered product, while \cref{sec:esfri} describes how some of the major \gls{ESCAPE} project \gls{ESFRI}s are engaging with and building upon the work carried out by \gls{WP}5. In particular, \cref{sec:requirements} tabulates the documented requirements and compares them to delivered product, while \cref{sec:esfri} describes how some of the major \gls{ESCAPE} project \gls{ESFRI}s are engaging with and building upon the work carried out by \gls{WP}5.
Finally, we consider potential future enhancements in \cref{sec:future}, discuss the availability of \gls{ESAP} and its integration with \gls{EOSC} in \cref{sec:access}, and provide a summary of the results of this analysis is presented in \cref{sec:conclusions}. Finally, we consider potential future enhancements in \cref{sec:future}, discuss the availability of \gls{ESAP} and its integration with \gls{EOSC} in \cref{sec:access}, and provide a summary of the results of this analysis in \cref{sec:conclusions}.
...@@ -17,7 +17,7 @@ By \emph{science platform}, we mean an interactive web-based environment which s ...@@ -17,7 +17,7 @@ By \emph{science platform}, we mean an interactive web-based environment which s
By \emph{toolkit}, we mean that \pgls{ESAP} is not, in itself, intended to be deployed independently. By \emph{toolkit}, we mean that \pgls{ESAP} is not, in itself, intended to be deployed independently.
Rather, \pgls{ESAP} provides a collection of software components, design patterns, and best practices that enable \glspl{ESFRI} or other groups that have a library of bespoke services (for example data archives, or interactive notebook environments) to make them available to a user community in a coherent, consistent, and integrated way. Rather, \pgls{ESAP} provides a collection of software components, design patterns, and best practices that enable \glspl{ESFRI} or other groups that have a library of bespoke services (for example data archives, or interactive notebook environments) to make them available to a user community in a coherent, consistent, and integrated way.
Using the \pgls{ESAP} toolkit, \gls{ESCAPE} project partners can make assist users in engaging with the services provided in the other ESCAPE work packages by: Using the \pgls{ESAP} toolkit, \gls{ESCAPE} project partners provide assistance to users in engaging with the services provided in the other ESCAPE work packages by:
\begin{itemize} \begin{itemize}
...@@ -102,7 +102,7 @@ The current \pgls{ESAP} system provides no concept of federation between instanc ...@@ -102,7 +102,7 @@ The current \pgls{ESAP} system provides no concept of federation between instanc
\subsubsection{User Interface} \subsubsection{User Interface}
\label{sec:vision:capabilities:ui} \label{sec:vision:capabilities:ui}
\gls{ESAP} is primarily a web application: the central hub (the “\gls{API} Gateway”, or “back-end”) runs on one or more servers, and users interact with it by making \gls{HTTP} requests. \gls{ESAP} is primarily a web application: the central hub (the “\gls{API} Gateway”) runs on one or more servers, and users interact with it by making \gls{HTTP} requests.
The \gls{ESCAPE} project delivery includes a customizable front-end application (“ESAP-GUI”) which runs in the browser and communicates with the back-end. The \gls{ESCAPE} project delivery includes a customizable front-end application (“ESAP-GUI”) which runs in the browser and communicates with the back-end.
This separation of concerns is illustrated in \cref{fig:vision:capabilities:ui}. This separation of concerns is illustrated in \cref{fig:vision:capabilities:ui}.
In principle, it is possible to support alternative \glspl{GUI} which communicate with the same back-end, although none have yet been implemented. In principle, it is possible to support alternative \glspl{GUI} which communicate with the same back-end, although none have yet been implemented.
...@@ -124,8 +124,7 @@ Users may be asked to log in to access \pgls{ESAP} itself, or to use some or all ...@@ -124,8 +124,7 @@ Users may be asked to log in to access \pgls{ESAP} itself, or to use some or all
This step is not required: if both the owner of the \pgls{ESAP} instance and the owner of any services being accessed make them available to the general public, then \pgls{ESAP} need not force the user to log in. This step is not required: if both the owner of the \pgls{ESAP} instance and the owner of any services being accessed make them available to the general public, then \pgls{ESAP} need not force the user to log in.
In general, however, users are expected to log in before using the data orchestration services (\cref{sec:vision:capabilities:orch}). In general, however, users are expected to log in before using the data orchestration services (\cref{sec:vision:capabilities:orch}).
\pgls{ESAP} as delivered by this work package supports user authentication through the \gls{ESCAPE} \gls{IAM} service\footnote{\url{https://iam-escape.cloud.cnaf.infn.it/login}}. \pgls{ESAP} as delivered by this work package is designed to support authentication through any \gls{OIDC}-compliant system, although it has only been extensively tested and integrated with the \gls{ESCAPE} \gls{IAM} service\footnote{\url{https://iam-escape.cloud.cnaf.infn.it/login}}.
Where possible, \pgls{ESAP} is designed to be flexible and adaptable to other systems, but explicit support for other systems is outside the scope of this work package.
\subsubsection{Data Orchestration within \glsentrytext{ESAP}} \subsubsection{Data Orchestration within \glsentrytext{ESAP}}
\label{sec:vision:capabilities:orch} \label{sec:vision:capabilities:orch}
...@@ -185,7 +184,7 @@ Batch data processing describes a situation which is in many ways similar to \gl ...@@ -185,7 +184,7 @@ Batch data processing describes a situation which is in many ways similar to \gl
\item{the user does not interact with the computing systems while processing takes place;} \item{the user does not interact with the computing systems while processing takes place;}
\item{processing generally happens at scale, perhaps being distributed over multiple computing systems.} \item{processing generally happens at scale, with workflows operating in parallel and often being distributed over multiple computing systems.}
\end{itemize} \end{itemize}
...@@ -202,7 +201,7 @@ Batch data processing describes a situation which is in many ways similar to \gl ...@@ -202,7 +201,7 @@ Batch data processing describes a situation which is in many ways similar to \gl
\subsubsection{Service and Software Discovery} \subsubsection{Service and Software Discovery}
\label{sec:vision:capabilities:discovery} \label{sec:vision:capabilities:discovery}
\pgls{ESAP} provides deep integration with the \gls{OSSR}, and other repositories of software and services if appropriate. \pgls{ESAP} makes use of the eOSSR library developed in \gls{WP}3 \autocite{vuillaume_thomas_2022_6826881} to provide deep integration with the \gls{OSSR}, and could be adapted to use other software repositories where necessary.
This will make it possible for users to discover capabilities which are of relevance to them. This will make it possible for users to discover capabilities which are of relevance to them.
In particular, \pgls{ESAP} helps users discover software workflows and compute and storage infrastructure that can be used to execute both \gls{IDA} and batch processing tasks (as described in \cref{sec:vision:capabilities:ida,sec:vision:capabilities:batch}). In particular, \pgls{ESAP} helps users discover software workflows and compute and storage infrastructure that can be used to execute both \gls{IDA} and batch processing tasks (as described in \cref{sec:vision:capabilities:ida,sec:vision:capabilities:batch}).
......
...@@ -170,7 +170,7 @@ Although the current \gls{IDA} system focuses on Jupyter notebooks, we expect to ...@@ -170,7 +170,7 @@ Although the current \gls{IDA} system focuses on Jupyter notebooks, we expect to
\subsection{Batch Processing} \subsection{Batch Processing}
\label{sec:delivered:batch} \label{sec:delivered:batch}
\pgls{ESAP} builds on the asynchronous job management system described in \cref{sec:delivered:gateway} to provide a batch computing environment that is capable of launching jobs to a DIRAC \autocite{dirac:2022} environment, monitoring their progress, and retrieving their results. \pgls{ESAP} builds on the asynchronous job management system described in \cref{sec:delivered:gateway} to provide a batch computing environment that is capable of launching jobs to a DIRAC \autocite{dirac:2022} environment, monitoring their progress.
This system currently works with a limited number of pre-defined batch processing workflows. This system currently works with a limited number of pre-defined batch processing workflows.
Future work (\cref{sec:future:batch}) may include extension to a more flexible job definition system and integration with additional batch processing middleware. Future work (\cref{sec:future:batch}) may include extension to a more flexible job definition system and integration with additional batch processing middleware.
......
...@@ -21,7 +21,7 @@ It is currently under construction at two sites --- the \gls{CTAO} Northern Arra ...@@ -21,7 +21,7 @@ It is currently under construction at two sites --- the \gls{CTAO} Northern Arra
\gls{CTAO} will consist of three classes of telescopes, covering a broad energy range from 20\,GeV to 300\,TeV the Large-Sized Telescope (LST), the Medium-Sized Telescope (MST) and the Small-Sized Telescope (SST). \gls{CTAO} will consist of three classes of telescopes, covering a broad energy range from 20\,GeV to 300\,TeV the Large-Sized Telescope (LST), the Medium-Sized Telescope (MST) and the Small-Sized Telescope (SST).
During the Construction Phase, the plan for CTAO Northern Array includes 4 LSTs and 9 MSTs, whilst the CTAO Southern Array will feature 14 MSTs and 37 SSTs. During the Construction Phase, the plan for CTAO Northern Array includes 4 LSTs and 9 MSTs, whilst the CTAO Southern Array will feature 14 MSTs and 37 SSTs.
\gls{CTAO} is expected to drive a five- to ten-fold increase in the number of known astronomical gamma ray sources, potentially detecting as many as 1,000 new previously-unknown objects. \gls{CTAO} is expected to drive a five- to ten-fold increase in the number of known astronomical gamma-ray sources, potentially detecting as many as 1,000 new previously-unknown objects.
It is particularly suited to observing short (less than a minute) timescale phenomena, making it a key instrument in the future multi-wavelength and multi-messenger infrastructure of time domain astronomy. It is particularly suited to observing short (less than a minute) timescale phenomena, making it a key instrument in the future multi-wavelength and multi-messenger infrastructure of time domain astronomy.
The primary scientific drivers of \gls{CTAO} are: The primary scientific drivers of \gls{CTAO} are:
...@@ -50,7 +50,7 @@ Specifically, \gls{CTAO} looks to \gls{ESAP} as the potential basis for a scienc ...@@ -50,7 +50,7 @@ Specifically, \gls{CTAO} looks to \gls{ESAP} as the potential basis for a scienc
The \gls{CTAO} team identify \gls{ESAP}'s modular and extensible nature --- described in \cref{sec:vision:extensibility} --- as important to their needs: it makes it possible to extend the platform to add new services and archives by building on \gls{REST} interfaces. The \gls{CTAO} team identify \gls{ESAP}'s modular and extensible nature --- described in \cref{sec:vision:extensibility} --- as important to their needs: it makes it possible to extend the platform to add new services and archives by building on \gls{REST} interfaces.
This helps enable the identified \gls{CTAO} goals of common access mechanisms. This helps enable the identified \gls{CTAO} goals of common access mechanisms.
The project has already put this into practice, for example by linking \pgls{ESAP} with the \gls{OSSR}, as described in \cref{sec:delivered:ida} using the eOSSR software developed in \gls{WP}3 \autocite{vuillaume_thomas_2022_6826881}. The project has already put this into practice, for example by linking \pgls{ESAP} with the \gls{OSSR}, as described in \cref{sec:delivered:ida}.
\paragraph{Analysis Workflows} \paragraph{Analysis Workflows}
......
...@@ -77,6 +77,8 @@ There are numerous improvements that could be made here, including: ...@@ -77,6 +77,8 @@ There are numerous improvements that could be made here, including:
\end{itemize} \end{itemize}
Further, we note that the DIRAC system itself is under active development and there will soon be opportunities for \gls{ESAP} to take advantage of its new capabilities including token-based authorization and a \gls{REST} \gls{API}.
\subsubsection{Advanced Metadata for Job and Infrastructure Selection} \subsubsection{Advanced Metadata for Job and Infrastructure Selection}
\label{sec:future:metadata} \label{sec:future:metadata}
......
...@@ -9,7 +9,7 @@ ...@@ -9,7 +9,7 @@
\setDocTitle{Final Assessment of the Performance of the Science Platform Prototype} \setDocTitle{Final Assessment of the Performance of the Science Platform Prototype}
\setWorkPackage{WP5, ESFRI Science Analysis Platform} \setWorkPackage{WP5, ESFRI Science Analysis Platform}
\setLeadAuthor{John D. Swinbank (ASTRON)} \setLeadAuthor{John D. Swinbank (ASTRON)}
\setOtherAuthors{James Collinson (SKAO), Stelios Voutsinas (U.~Edinburgh)} \setOtherAuthors{James Collinson (SKAO), Gareth Hughes (CTAO), Stelios Voutsinas (U.~Edinburgh)}
\setDueDate{2022-11-30} \setDueDate{2022-11-30}
\setDueMonth{M46} \setDueMonth{M46}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment