diff --git a/proceedings/Makefile b/proceedings/Makefile index 43675d7b993e6d7ca88e6558ff631b236a223019..c0193a99996d9e1f63567cdd211265215dc310c5 100644 --- a/proceedings/Makefile +++ b/proceedings/Makefile @@ -116,7 +116,7 @@ $(P).pdf: $(P).dvi $(FIGS) $(P).bib: # bootstrap if you don't have one touch $(P).bib -$(P).dvi: $(P).tex $(P).bib +$(P).dvi: $(P).tex $(P).bib $(FIGS) latex $(P) if [ -s $(P).bib ]; then bibtex $(P); fi latex $(P) diff --git a/proceedings/X0-010.tex b/proceedings/X0-010.tex index fee3daec09e99f1dba1338a354db9ad66be1d755..bdcd4fa1d52831a4025f4f43c2a77c014ad2be9b 100644 --- a/proceedings/X0-010.tex +++ b/proceedings/X0-010.tex @@ -21,7 +21,7 @@ % Use mixed case type for the shortened title % Ensure shortened title does not cause an overfull hbox LaTeX error % See ASPmanual2010.pdf 2.1.4 and ManuscriptInstructions.pdf for more details -\markboth{Author1, Author2, and Author3}{Short Title} +\markboth{Swinbank}{ESAP} \begin{document} @@ -41,8 +41,8 @@ %\affil{$^1$Institution Name, Institution City, State/Province, Country; \email{AuthorEmail@email.edu}} %\affil{$^2$Institution Name, Institution City, State/Province, Country} -\author{John~D.~Swinbank$^1$} -\affil{$^1$ASTRON, Oude Hoogeveensedijk 4, 7991 PD Dwingeloo The Netherlands; \email{swinbank@astron.nl}} +\author{John~D.~Swinbank} +\affil{ASTRON, Oude Hoogeveensedijk 4, 7991 PD Dwingeloo The Netherlands; \email{swinbank@astron.nl}} % This section is for ADS Processing. There must be one line per author. paperauthor has 9 arguments. \paperauthor{John~D.~Swinbank}{swinbank@astron.nl}{https://orcid.org/0000-0001-9445-1846}{ASTRON}{}{Dwingeloo}{State/Province}{3571\,KD}{The Netherlands} @@ -58,55 +58,92 @@ %\aindex{Swinbank,~J.~D.} \begin{abstract} -This paper provides an overview of ESAP, the ESCAPE ESFRI Science Analysis Platform. +ESAP, the ESCAPE ESFRI Science Analysis Platform, is being developed to provide a flexible toolkit for constructing science platforms. +It provides capabilities to quickly unite a range of data access and analysis services --- in particular those developed by the ESCAPE project --- behind a single, consistent but customizable, user interface. +This paper provides a brief overview of ESAP's aims, its architecture, and its current and expected future capabilities. \end{abstract} -\section{ESCAPE} -The ESCAPE project brings together the astronomy, astroparticle, and particle physics communities to address fundamental challenges in data-driven research, driven by the goals and needs of major European research infrastructures. +\section{ESCAPE and ESAP} -ESCAPE aims to produce versatile solutions to support the implementation of the European Open Science Cloud (EOSC) by fostering a multi-disciplinary environment including open data management according to FAIR principles. Our goal is to enable interoperability between facilities, encourage cross-fertilization, and develop joint multiwavelength and multimessenger capabilities. +The ESCAPE Project\footnote{\url{https://www.projectescape.eu}} brings together the astronomy, astroparticle, and particle physics communities to address fundamental challenges in data-driven research, driven by the goals and needs of major European research infrastructures, or ESFRIs. -In support of this goal, ESCAPE is developing a range of services, including data infrastructure, a software repository, virtual observatory integration, and a citizen science programme. +ESCAPE aims to produce versatile solutions to support the implementation of the European Open Science Cloud by fostering a multi-disciplinary environment, including open data management according to FAIR principles. -This poster describes ESAP, the science platform functionality being developed in the context of ESCAPE. +Our goal is to enable interoperability between facilities, encourage cross-fertilization, and develop joint multiwavelength and multimessenger data processing and analysis capabilities. +We are addressing this by developing a range of services, including data infrastructure, a software repository, virtual observatory integration, and a citizen science programme. + +ESAP, the ESFRI Science Analysis Platform, is under development by ESCAPE's Work Package 5, to facilitate access to the full range of ESCAPE services. +ESAP aims to provide a comfortable and consistent gateway to the full range of project-provided functionality, while being customizable and adaptable to the needs of particular science goals or research infrastructures. \section{The ESAP Mission} -ESAP aims to help users engage with ESCAPE and other services by providing: +\articlefigure{esap-interfaces.eps}{fig:esap-interfaces}{ESAP provides a single, consistent, interface and point of access to a variety services drawn from a range of providers. +Links to the other work packages within ESCAPE are indicated.} + +ESAP will help users engage with the services provided by the other ESCAPE work packages --- and with services sourced from elsewhere -- by: \begin{itemize} -\item{Data discovery and retrieval from a range of archives and data repositories.} -\item{Exploration and discovery of relevant tools in the ESCAPE software repository.} -\item{Access to a range of compute and analysis services provided by project partners and other facilities.} -\item{Orchestration of data, services, and software to help users create and access research environments that meet their needs.} +\item{Providing a flexible interface for querying and retrieving data from a variety of archives and data repositories;} + +\item{Enabling users to explore the software repositories to identify and select analysis tools and workflows which are appropriate to their needs;} + +\item{Helping users to identify interactive data analysis and batch computing facilities which are accessible to them;} + +\item{Facilitating the staging of data, software, and workflows to compute facilities, providing access to those facilities for end users, and subsequently retrieving the results of processing.} \end{itemize} +This model is shown schematically in Figure \ref{fig:esap-interfaces}, which illustrates the range of services which ESAP can help the user access. -ESAP is not a single instance of a science platform, but rather a ``toolkit'' that research infrastructures, institutions, projects, and others can use to create and manage analysis environments tailored to their particular needs. +Our aim is to meet the needs of the widest possible range of scientists, using a variety of different research infrastructures. +To make this possible, we do not envision ESAP as being operated as a single integrated platform to which all researchers must adapt. +Instead, ESAP provides a ``toolkit'', from various communities and research infrastructures can assemble a customized analysis platform geared to their specific needs. +We envision instances of ESAP being deployed at a variety of scales, from providing services to just a few users within a small project, to supporting major pieces of infrastructure. \section{Architecture} -ESAP consists of an API Gateway, which brokers requests across multiple independent, infrastructure provided services, and a web-based user interface. +\articlefigure{esap-high-level-architecture.eps}{fig:esap-arch}{A conceptual view of the ESAP architecture, showing a range of possible service integrations.} + +ESAP is a web application, implemented in Python, Django, and React.\footnote{\url{https://www.python.org}; \url{https://www.djangoproject.com}; \url{https://reactjs.org}} +It is designed around the \emph{API Gateway}, which brokers requests across a range of independent services, and a web-based \emph{user interface}. +This is shown schematically in Figure \ref{fig:esap-arch}. + +The API Gateway communicates with external services using REST APIs. +New service integrations can easily be added by writing ``plug-in'' modules which extend the Gateway's capabilities. +The ESCAPE team is writing a number of service integration plugins, which are supplied with ESAP; we hope that the wider community will contribute many more. -New capabilities are easily added by writing ``plug-in'' modules which integrate external services with the API Gateway. A number of service integration plugins are provided with ESAP. +Internally, ESAP uses the model of a ``shopping basket'' to help users manage their data and workflows. +As they interact with external services, they can select data or other products of interest, and add them to their basket. +They can edit the contents of their basket from within the ESAP interface itself, and take the contents of their basket with them as they move to analysis services. +While the basket is designed to be flexible in terms of the types of data it stores, it is normally used to store references to bulk data products, rather than the products themselves, to minimize data transport issues between services -\section{Current Capabilities} +\section{Capabilities} + +ESAP is still very much under development at time of writing, and its range of capabilities continues to expand. +Some highlights of current capabilities include: \begin{itemize} -\item{Query multiple archives with an adaptable interface.} -\item{Load software from the ESCAPE repository.} -\item{integration with ESCAPE Identity and Access Management.} -\item{Built with Python, Django, and React.} -\item{Data orchestration across multiple services.} -\item{Interactive data analysis through BinderHub services.} +\item{Access to a variety of bespoke archives, including Apertif and Zooniverse;\footnote{\url{http://www.apertif.nl/}; \url{https://www.zooniverse.org/}}} +\item{Powerful tools for searching the Virtual Observatory through IVOA-compliant interfaces, and for integrating with VO tools through SAMP, the Simple Application Messaging Protocol;} +\item{The capability to search and discover data on the Rucio\footnote{\url{https://rucio.cern.ch}}-based ESCAPE data lake;} +\item{The ability to locate software in the ESCAPE project software repository, and to stage that software to BinderHub\footnote{\url{https://binderhub.readthedocs.io/}}-based interactive data analysis services;} +\item{Bi-directional data transfer between the ESAP ``shopping basket'' and Python-based analysis services;} +\item{Integration with the ESCAPE Project's Identity \& Access Management system.} \end{itemize} +Future work is expected to focus on: -\section{Downloads and Further Information} +\begin{itemize} + +\item{Batch computing capabilities;} +\item{A managed database system;} +\item{Closer interaction between the analysis environment and the Rucio data lake, building on ``data lake as a service'' technology developed in ESCAPE Work Package 2;} +\item{Tighter integration with the virtual observatory, thought ESCAPE WP4, and the ESCAPE software repository, developed by WP3.} -Information about the ESCAPE Project and details of the capabilities it provides are available from \url{http://www.projectescape.eu/}. +\end{itemize} + +\section{Downloads and Further Information} ESAP is available under the terms of the Apache license, version 2.0, from \url{https://git.astron.nl/astron-sdc/esap-api-gateway} and \url{https://git.astron.nl/astron-sdc/esap-gui}. diff --git a/proceedings/makedefs b/proceedings/makedefs index 0ec7de5fedc13fbd79f127da38d2fa417cc1d72a..790a3846a65007a466d5f15472875d29269d3a1e 100644 --- a/proceedings/makedefs +++ b/proceedings/makedefs @@ -3,4 +3,9 @@ P = X0-010 V = $(shell git describe --always --dirty) A = Swinbank E = swinbank@astron.nl -#FIGS = P13-20_*.eps +FIGS = esap-high-level-architecture.eps esap-interfaces.eps + +vpath %.png ../screenshots + +%.eps: %.png + gm convert -resize 2000 -resample 72 $< $@