Skip to content
Snippets Groups Projects
Commit 228eeb2a authored by John Swinbank's avatar John Swinbank
Browse files

Respond to feedback from Sara Bertocco

parent 6b5212b4
No related branches found
No related tags found
No related merge requests found
Subproject commit 477d155500a1dcdf5bb114e21c541960dd27494d Subproject commit 3141cf803b6961491167b1c43803c51db9988cf7
...@@ -30,7 +30,7 @@ Specifically, we will consider: ...@@ -30,7 +30,7 @@ Specifically, we will consider:
Broadly, this report is structured in two parts. Broadly, this report is structured in two parts.
In the first part, we provide essential context by describing the \pgls{ESAP} system. In the first part, we provide essential context by describing the \pgls{ESAP} system.
In \cref{sec:vision} we the overall vision for the \pgls{ESAP} system: what did we set out to build? In \cref{sec:vision} we the address the overall vision for the \pgls{ESAP} system: what did we set out to build?
\Cref{sec:delivered} takes that further by describing what is currently available: if one were to deploy and operate an \pgls{ESAP} system today, what would it consist of? \Cref{sec:delivered} takes that further by describing what is currently available: if one were to deploy and operate an \pgls{ESAP} system today, what would it consist of?
The second part of this document moves on to asses how successful that vision and implementation have been in addressing user --- and in particular \gls{ESFRI} --- needs. The second part of this document moves on to asses how successful that vision and implementation have been in addressing user --- and in particular \gls{ESFRI} --- needs.
......
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
This section presents a brief overview of the vision for and design of \pgls{ESAP}. This section presents a brief overview of the vision for and design of \pgls{ESAP}.
It supplements and expands upon earlier discussions \cite{ESCAPE-GA, ESCAPE-D5.2, ESCAPE-D5.3} to describe current thinking about how \pgls{ESAP} can best meet its goals. It supplements and expands upon earlier discussions \cite{ESCAPE-GA, ESCAPE-D5.2, ESCAPE-D5.3} to describe current thinking about how \pgls{ESAP} can best meet its goals.
The vision is inspired by and aims to address a wide rage of \glspl{ESFRI} use cases that have been expressed by the project partners through the \gls{ESCAPE} Project Platform.\footnote{\url{https://project.escape2020.de}} The vision is inspired by and aims to address a wide range of \glspl{ESFRI} use cases that have been expressed by the project partners through the \gls{ESCAPE} Project Platform.\footnote{\url{https://project.escape2020.de}}
Note that not all of the capabilities described in this section are currently available in the \pgls{ESAP} codebase: see \cref{sec:delivered} for a description of the current state of the art, and \cref{sec:future} for future development plans. Note that not all of the capabilities described in this section are currently available in the \pgls{ESAP} codebase: see \cref{sec:delivered} for a description of the current state of the art, and \cref{sec:future} for future development plans.
\subsection{High-Level Summary} \subsection{High-Level Summary}
...@@ -124,7 +124,7 @@ Users may be asked to log in to access \pgls{ESAP} itself, or to use some or all ...@@ -124,7 +124,7 @@ Users may be asked to log in to access \pgls{ESAP} itself, or to use some or all
This step is not required: if both the owner of the \pgls{ESAP} instance and the owner of any services being accessed make them available to the general public, then \pgls{ESAP} need not force the user to log in. This step is not required: if both the owner of the \pgls{ESAP} instance and the owner of any services being accessed make them available to the general public, then \pgls{ESAP} need not force the user to log in.
In general, however, users are expected to log in before using the data orchestration services (\cref{sec:vision:capabilities:orch}). In general, however, users are expected to log in before using the data orchestration services (\cref{sec:vision:capabilities:orch}).
\pgls{ESAP} as delivered by this work package is designed to support authentication through any \gls{OIDC}-compliant system, although it has only been extensively tested and integrated with the \gls{ESCAPE} \gls{IAM} service\footnote{\url{https://iam-escape.cloud.cnaf.infn.it/login}}. \pgls{ESAP} as delivered by this work package is designed to support authentication through any \gls{OIDC}-compliant system, although it has only been extensively tested and integrated with the \gls{ESCAPE} \gls{IAM} service.\footnote{\url{https://iam-escape.cloud.cnaf.infn.it/login}; \url{https://indigo-iam.github.io/v/v1.8.0/docs/overview}}
\subsubsection{Data Orchestration within \glsentrytext{ESAP}} \subsubsection{Data Orchestration within \glsentrytext{ESAP}}
\label{sec:vision:capabilities:orch} \label{sec:vision:capabilities:orch}
...@@ -220,7 +220,7 @@ That is, based on metadata sourced from the \gls{OSSR}, \pgls{ESAP} helps the us ...@@ -220,7 +220,7 @@ That is, based on metadata sourced from the \gls{OSSR}, \pgls{ESAP} helps the us
\subsubsection{Managed Database} \subsubsection{Managed Database}
\label{sec:vision:capabilities:db} \label{sec:vision:capabilities:db}
A managed database service provides users with with the capability to define and use their own relational databases directly within the \gls{ESAP} system. A managed database service provides users with the capability to define and use their own relational databases directly within the \gls{ESAP} system.
It is possible to directly load the results of queries against external archives into the user's database space, and then to submit complex \gls{SQL} queries to the database system. It is possible to directly load the results of queries against external archives into the user's database space, and then to submit complex \gls{SQL} queries to the database system.
This provides the user with advanced data analysis capabilities --- for example, the ability to perform complex catalogue cross-matching --- without requiring that they set up and administer their own database system. This provides the user with advanced data analysis capabilities --- for example, the ability to perform complex catalogue cross-matching --- without requiring that they set up and administer their own database system.
Further, it opens the prospect of integrating \pgls{ESAP} with external \gls{SQL} federation services such as Trino\footnote{\url{https://trino.io}} or openLooKeng\footnote{\url{https://openlookeng.io}}. Further, it opens the prospect of integrating \pgls{ESAP} with external \gls{SQL} federation services such as Trino\footnote{\url{https://trino.io}} or openLooKeng\footnote{\url{https://openlookeng.io}}.
......
...@@ -29,7 +29,7 @@ The interface is a cross-platform web application implemented in React\footnote{ ...@@ -29,7 +29,7 @@ The interface is a cross-platform web application implemented in React\footnote{
\label{sec:delivered:gateway} \label{sec:delivered:gateway}
As discussed in \cref{sec:vision:capabilities:ui}, the “API Gateway” provides back-end capabilities for the \pgls{ESAP} system. As discussed in \cref{sec:vision:capabilities:ui}, the “API Gateway” provides back-end capabilities for the \pgls{ESAP} system.
The API Gateway is written in Python, using using Django\footnote{\url{https://www.djangoproject.com/}} and its companion \gls{REST} framework\footnote{\url{https://www.django-rest-framework.org/}}. The API Gateway is written in Python, using Django\footnote{\url{https://www.djangoproject.com/}} and its companion \gls{REST} framework\footnote{\url{https://www.django-rest-framework.org/}}.
The API Gateway provides a rich, plugin-based system for adding new service integrations to \gls{ESAP}. The API Gateway provides a rich, plugin-based system for adding new service integrations to \gls{ESAP}.
It also provides an asynchronous job control system, shown in \cref{fig:esap:async}, which is used to manage long-running tasks like some queries (\cref{sec:delivered:data}) and batch processing jobs (\cref{sec:delivered:batch}). It also provides an asynchronous job control system, shown in \cref{fig:esap:async}, which is used to manage long-running tasks like some queries (\cref{sec:delivered:data}) and batch processing jobs (\cref{sec:delivered:batch}).
......
...@@ -54,7 +54,7 @@ The project has already put this into practice, for example by linking \pgls{ESA ...@@ -54,7 +54,7 @@ The project has already put this into practice, for example by linking \pgls{ESA
\paragraph{Analysis Workflows} \paragraph{Analysis Workflows}
The \gls{CTAO} Science Portal will offer data and services to users its users and staff, with the aim of enabling user interaction with data products and cross-collaboration and joint analysis of data products. The \gls{CTAO} Science Portal will offer data and services to its users and staff, with the aim of enabling user interaction with data products and cross-collaboration and joint analysis of data products.
These workflows will build upon new and existing tools, including: These workflows will build upon new and existing tools, including:
\begin{description} \begin{description}
......
...@@ -19,7 +19,7 @@ The engagement of research infrastructures and other facilities with \gls{ESAP} ...@@ -19,7 +19,7 @@ The engagement of research infrastructures and other facilities with \gls{ESAP}
In the first, major infrastructures --- such as \gls{SKAO} and \gls{CTAO}, as described in \cref{sec:esfri} --- see \gls{ESAP} as a toolkit or library of potentially reusable components. In the first, major infrastructures --- such as \gls{SKAO} and \gls{CTAO}, as described in \cref{sec:esfri} --- see \gls{ESAP} as a toolkit or library of potentially reusable components.
For example, the distributed and extraordinarily data-intensive nature of the SKA Regional Centres Network means that \gls{ESAP} cannot simply be deployed in its current form: that environment is more complex and specialized than even a tool as generic as \gls{ESAP} can address without modification. For example, the distributed and extraordinarily data-intensive nature of the SKA Regional Centres Network means that \gls{ESAP} cannot simply be deployed in its current form: that environment is more complex and specialized than even a tool as generic as \gls{ESAP} can address without modification.
However, it is likely that that the Regional Centres effort will build on existing libraries and toolkits --- like \pgls{ESAP} --- to realise its goals (although we note that multiple technologies are under consideration, and the use of \pgls{ESAP}-derived code is not certain). However, it is likely that the Regional Centres effort will build on existing libraries and toolkits --- like \pgls{ESAP} --- to realise its goals (although we note that multiple technologies are under consideration, and the use of \pgls{ESAP}-derived code is not certain).
A similar story applies to other major research infrastructures, certainly including \gls{CTAO}, but also perhaps \gls{ESO}, \gls{JIVE}, and other research infrastructures both within \gls{ESCAPE} and across the wider scientific ecosystem. A similar story applies to other major research infrastructures, certainly including \gls{CTAO}, but also perhaps \gls{ESO}, \gls{JIVE}, and other research infrastructures both within \gls{ESCAPE} and across the wider scientific ecosystem.
A second community of potential users are those research infrastructures which have less exacting needs, and instead are looking for a “plug and play” solution that they can use to rapidly provide functionality to their user community: rather than seeing \gls{ESAP} as components upon which they might build, they look to \gls{ESAP} to provide a ready-to-use solution that can be deployed with minimal modification. A second community of potential users are those research infrastructures which have less exacting needs, and instead are looking for a “plug and play” solution that they can use to rapidly provide functionality to their user community: rather than seeing \gls{ESAP} as components upon which they might build, they look to \gls{ESAP} to provide a ready-to-use solution that can be deployed with minimal modification.
......
...@@ -12,4 +12,4 @@ However, it also presents an opportunity to go further: the \gls{ESAP} system pr ...@@ -12,4 +12,4 @@ However, it also presents an opportunity to go further: the \gls{ESAP} system pr
Therefore, in \cref{sec:future} we described potential approaches to future work, both in terms of the organization of the project and the technical challenges which could be addressed. Therefore, in \cref{sec:future} we described potential approaches to future work, both in terms of the organization of the project and the technical challenges which could be addressed.
The conclusion of the \gls{ESCAPE} project marks the release of the first stable version of \gls{ESAP}, which may be accessed as described in \cref{sec:access}. The conclusion of the \gls{ESCAPE} project marks the release of the first stable version of \gls{ESAP}, which may be accessed as described in \cref{sec:access}.
However, we are optimistic that the system will continue to grown and evolve into the future. However, we are optimistic that the system will continue to grown and evolve into the future, and we look forward to future funding opportunities which might support that evolution.
...@@ -18,6 +18,6 @@ Over the course of the \Acrshort{ESCAPE} Project, \Acrshort{ESAP} has grown from ...@@ -18,6 +18,6 @@ Over the course of the \Acrshort{ESCAPE} Project, \Acrshort{ESAP} has grown from
Based on the material presented here, we conclude that \Acrshort{ESAP} has broadly achieved the goals that were set out at the beginning of the \Acrshort{ESCAPE} project, and is starting to see uptake in the \Acrshort{ESFRI} community. Based on the material presented here, we conclude that \Acrshort{ESAP} has broadly achieved the goals that were set out at the beginning of the \Acrshort{ESCAPE} project, and is starting to see uptake in the \Acrshort{ESFRI} community.
It has also become clear that much of its potential remains to be tapped: major \Acrshort{ESFRI}s can build upon the systems delivered by \Acrshort{ESCAPE} to address the extreme requirements of the next generation of data- and compute-intensive infrastructures. It has also become clear that much of its potential remains to be tapped: major \Acrshort{ESFRI}s can build upon the systems delivered by \Acrshort{ESCAPE} to address the extreme requirements of the next generation of data- and compute-intensive infrastructures.
We look forward to seeing \Acrshort{ESAP} continue to grow as an open-source project even as the Horizon 2020 funding draws to a close. We look forward to seeing \Acrshort{ESAP} continue to grow as an open-source project even as the Horizon 2020 funding draws to a close, and we will continue to seek appropriate funding opportunities to support that effort.
This document is submitted as \Acrshort{ESCAPE} project deliverable \docNumber{}, \emph{\docTitle{}}. This document is submitted as \Acrshort{ESCAPE} project deliverable \docNumber{}, \emph{\docTitle{}}.
Subproject commit 802b1a1c12201d1e6573de0af2a40e08329a410c Subproject commit 2ccbbc617af57f6470780e853c673c169df8ea31
...@@ -9,7 +9,7 @@ ...@@ -9,7 +9,7 @@
\setDocTitle{Final Assessment of the Performance of the Science Platform Prototype} \setDocTitle{Final Assessment of the Performance of the Science Platform Prototype}
\setWorkPackage{WP5, ESFRI Science Analysis Platform} \setWorkPackage{WP5, ESFRI Science Analysis Platform}
\setLeadAuthor{John D. Swinbank (ASTRON)} \setLeadAuthor{John D. Swinbank (ASTRON)}
\setOtherAuthors{James Collinson (SKAO), Gareth Hughes (CTAO), Stelios Voutsinas (U.~Edinburgh)} \setOtherAuthors{Sara Bertocco (INAF), James Collinson (SKAO), Gareth Hughes (CTAO), Stelios Voutsinas (U.~Edinburgh)}
\setDueDate{2022-11-30} \setDueDate{2022-11-30}
\setDueMonth{M46} \setDueMonth{M46}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment