Skip to content
Snippets Groups Projects
Verified Commit 722941d7 authored by John Swinbank's avatar John Swinbank
Browse files

Merge branch 'tickets/SDC-198'

parents 3a3bd97a 946bbfed
No related branches found
No related tags found
1 merge request!4Respond to comments from van der Vlugt
Pipeline #13792 passed
...@@ -11,5 +11,6 @@ SDC-006.out ...@@ -11,5 +11,6 @@ SDC-006.out
SDC-006.pdf SDC-006.pdf
SDC-006.toc SDC-006.toc
SDC-006.run.xml SDC-006.run.xml
changes.tex
meta.tex meta.tex
texput.log texput.log
...@@ -34,7 +34,7 @@ A non-exhaustive list includes: ...@@ -34,7 +34,7 @@ A non-exhaustive list includes:
\begin{itemize} \begin{itemize}
\item The Application Repository, which will store and make available software components for use in Data Processing Service and Portal. \item The Application Repository, which will store and make available software components for use in Data Processing Service and Portal.
\item The Staging, Data Transfer, and Archiver services, which make data available to the Data Processing Service and ingest results back into the Data Product Repository. \item The Staging, Data Transfer, and Archiver services, which make data available to the Data Processing Service and ingest results back into the Data Product Repository.
\item The Science Data Repository and Virtual Observatory services, which provide metadata and interfaces that enable the Data Repository to be searched, indexed, and made accessible using standard tooling, enforcing the data access policies. \item The Science Data Repository and Virtual Observatory services, which provide metadata and interfaces that enable the Data Repository to be searched, indexed, and made accessible using standard tooling, while ensuring that appropriate data access policies are enforced.
\item The Federated Authentication and Authorization Infrastructure and the Community Management Service, which provide control over user access rights. \item The Federated Authentication and Authorization Infrastructure and the Community Management Service, which provide control over user access rights.
\end{itemize} \end{itemize}
...@@ -75,13 +75,13 @@ In general, however, the Repository is expected to take an hierarchical approach ...@@ -75,13 +75,13 @@ In general, however, the Repository is expected to take an hierarchical approach
The distributed nature of the Repository represents a challenge in terms of data locality: in general, it is cheaper and more efficient to process data close to where it is stored, rather than transmitting it over a long-haul network for analysis. The distributed nature of the Repository represents a challenge in terms of data locality: in general, it is cheaper and more efficient to process data close to where it is stored, rather than transmitting it over a long-haul network for analysis.
The Repository will therefore cooperate with the Data Processing Service and related ancillary services to route processing and analytics jobs to compute systems which are as close to the data as possible. The Repository will therefore cooperate with the Data Processing Service and related ancillary services to route processing and analytics jobs to compute systems which are as close to the data as possible.
To facilitate data sharing (\Cref{sec:features:sharing}), the will provide support for automatically associating \glspl{PID} with published data products. To facilitate data sharing (\Cref{sec:features:sharing}), the Repository will provide support for automatically associating \glspl{PID} with published data products.
Ancillary services will ensure that appropriate data published to the Repository is made available to the \gls{VO} (\Cref{sec:features:vo}). Ancillary services will ensure that appropriate data published to the Repository is made available to the \gls{VO} (\Cref{sec:features:vo}).
The Repository will provide data management functionality capable of implementing the capabilities described in \Cref{sec:features:drm}. The Repository will provide the data management functionality required to implement the capabilities described in \Cref{sec:features:drm}.
In particular, this will include support for the concept of ownership and rights to data. In particular, this will include support for the concept of ownership and rights to data.
That is, when appropriate, it will be clear what organization produced and is responsible for a piece of data (for example, the \gls{ILT}), and it will be possible to define and enforce policies regarding which users have permission to access it. That is, when appropriate, it will be clear what organization produced and is responsible for a piece of data (for example, the \gls{ILT}), and it will be possible to define and enforce policies regarding which users have permission to access it.
These capabilities will be generic: it should be possible to use them to implement the policies of multiple data owners where appropriate. These capabilities will be generic: it will be possible to use them to implement the policies of multiple data owners where appropriate.
\subsubsection{Data Processing Service} \subsubsection{Data Processing Service}
...@@ -209,7 +209,7 @@ Providing documentation appropriate to all levels of users is a core goal of the ...@@ -209,7 +209,7 @@ Providing documentation appropriate to all levels of users is a core goal of the
Documentation will be tightly integrated with and published through the Portal. Documentation will be tightly integrated with and published through the Portal.
Responsibility for generating documentation is shared by all aspects of the \gls{SDC}. Responsibility for generating documentation is shared by all aspects of the \gls{SDC}.
In particular, developers are responsible for providing \gls{API} and code documentation to accompany their software, while the operations team should provide provide higher-level guides and descriptions of the instruments, available datasets, processing pipelines, and analysis techniques. In particular, developers are responsible for providing \gls{API} and code documentation to accompany their software, while the operations team should provide higher-level guides and descriptions of the instruments, available datasets, processing pipelines, and analysis techniques.
\subsubsection{Communications Channels} \subsubsection{Communications Channels}
......
...@@ -6,11 +6,11 @@ This section aims to present a high-level overview of our current understanding ...@@ -6,11 +6,11 @@ This section aims to present a high-level overview of our current understanding
\subsection{Data Storage} \subsection{Data Storage}
The \gls{SDC} will store and offer to the community multiple petabytes of data (\Cref{sec:features:raw,sec:features:srdp,sec:features:simple}). The \gls{SDC} will store and offer to the community multiple petabytes of data (as described in \Cref{sec:features:raw,sec:features:srdp,sec:features:simple}).
Our current data holdings, and their predicted growth rates, are: Current data holdings, and their predicted growth rates, are estimated as:
\begin{itemize} \begin{itemize}
\item 50\,PB of \gls{LOFAR} data distributed over three archive sites, growing at a rate of about 7\,PB a year \item 50\,PB of \gls{LOFAR} data distributed over three archive sites, growing at a rate of about 7\,PB a year;
\item 4\,PB of Apertif data, which will increase to 7.5\,PB by the end of the Apertif surveys. \item 4\,PB of Apertif data, which will increase to 7.5\,PB by the end of the Apertif surveys.
\end{itemize} \end{itemize}
......
DOCNAME=SDC-006 DOCNAME=SDC-006
export TEXMFHOME ?= astron-texmf/texmf export TEXMFHOME ?= astron-texmf/texmf
$(DOCNAME).pdf: $(DOCNAME).tex meta.tex $(DOCNAME).pdf: $(DOCNAME).tex meta.tex changes.tex
xelatex $(DOCNAME) xelatex $(DOCNAME)
makeglossaries $(DOCNAME) makeglossaries $(DOCNAME)
biber $(DOCNAME) biber $(DOCNAME)
xelatex $(DOCNAME) xelatex $(DOCNAME)
xelatex $(DOCNAME) xelatex $(DOCNAME)
include astron-texmf/vcs-meta.make include astron-texmf/make/vcs-meta.make
include astron-texmf/make/changes.make
...@@ -3,6 +3,7 @@ ...@@ -3,6 +3,7 @@
\usepackage{glossary-mcols} \usepackage{glossary-mcols}
\input{meta} \input{meta}
\input{changes}
\setDocTitle{ASTRON Science Data Centre Vision} \setDocTitle{ASTRON Science Data Centre Vision}
\setDocNumber{SDC-006} \setDocNumber{SDC-006}
...@@ -10,12 +11,6 @@ ...@@ -10,12 +11,6 @@
\setDocDate{\vcsDate} \setDocDate{\vcsDate}
\setDocProgram{SDC} \setDocProgram{SDC}
\setDocChangeRecord{
\addChangeRecord{0.3}{2021-05-07}{Detailed response to comments from Pizzo}
\addChangeRecord{0.2}{2021-04-16}{Revised draft for distribution to A\&O}
\addChangeRecord{0.1}{2021-02-18}{Initial draft for distribution}
}
\setDocAuthors{ \setDocAuthors{
\addPerson{Roberto Pizzo}{ASTRON}{\vcsDate} \addPerson{Roberto Pizzo}{ASTRON}{\vcsDate}
\addPerson{John D. Swinbank}{ASTRON}{\vcsDate} \addPerson{John D. Swinbank}{ASTRON}{\vcsDate}
......
...@@ -118,7 +118,7 @@ They will therefore be consumers of documentation and software. ...@@ -118,7 +118,7 @@ They will therefore be consumers of documentation and software.
\paragraph{Prioritization} \paragraph{Prioritization}
Deep integration with the \gls{SRC} network is important to ASTRON's ambition of running a its own Regional Centre. Deep integration with the \gls{SRC} network is important to ASTRON's ambition of directly operating a Regional Centre.
\subsubsection{Institutional Partners} \subsubsection{Institutional Partners}
\label{sec:goals:users:institutions} \label{sec:goals:users:institutions}
......
Subproject commit 0555476206ac766cde6163a8d07ee06ba65e80d3 Subproject commit 6764795b003822e61cfd8e62e1f55d263d919b4f
...@@ -164,7 +164,7 @@ As for \cref{sec:features:sharing}, all data access services --- including both ...@@ -164,7 +164,7 @@ As for \cref{sec:features:sharing}, all data access services --- including both
Standard data analysis tooling are packages widely used in the wider (radio) astronomy community for data analysis. Standard data analysis tooling are packages widely used in the wider (radio) astronomy community for data analysis.
This might include, for example, CASA\footnote{\url{https://casa.nrao.edu}}, TOPCAT\footnote{\url{http://www.star.bris.ac.uk/~mbt/topcat/}}, Aladin\footnote{\url{https://aladin.u-strasbg.fr}}, among a wide range of other packages. This might include, for example, CASA\footnote{\url{https://casa.nrao.edu}}, TOPCAT\footnote{\url{http://www.star.bris.ac.uk/~mbt/topcat/}}, Aladin\footnote{\url{https://aladin.u-strasbg.fr}}, among a wide range of other packages.
Users are familiar with these packages and know how to use them to quickly obtain the results they nee. Users are familiar with these packages and know how to use them to quickly obtain the results they need.
These tools might be access directly within the \gls{SDC} (e.g. in a web-based environment, or running on a \gls{VM}), or may be packaged for convenient download and offline use. These tools might be access directly within the \gls{SDC} (e.g. in a web-based environment, or running on a \gls{VM}), or may be packaged for convenient download and offline use.
The tooling should interoperate seamlessly with the data products provided by the \gls{SDC} (\cref{sec:features:srdp,sec:features:raw,sec:features:simple}): having obtained the software and the data through \gls{SDC}-sanctioned channels, they should immediately be able to load and work with the data in their tool of choice without further adaptation. The tooling should interoperate seamlessly with the data products provided by the \gls{SDC} (\cref{sec:features:srdp,sec:features:raw,sec:features:simple}): having obtained the software and the data through \gls{SDC}-sanctioned channels, they should immediately be able to load and work with the data in their tool of choice without further adaptation.
...@@ -291,7 +291,7 @@ This is fundamental to enable the community to engage with the facility, and is ...@@ -291,7 +291,7 @@ This is fundamental to enable the community to engage with the facility, and is
\paragraph{System Design} \paragraph{System Design}
The \gls{SDC} will make extensive use off-the-shelf tooling for providing these capabilities. The \gls{SDC} will make extensive use of off-the-shelf tooling for providing these capabilities.
As appropriate, these will be integrated with the \portal{}. As appropriate, these will be integrated with the \portal{}.
\subsubsection{Technical and \Acrshort{API} Documentation} \subsubsection{Technical and \Acrshort{API} Documentation}
...@@ -308,7 +308,7 @@ It is also essential to the \gls{SDC} development workflow. ...@@ -308,7 +308,7 @@ It is also essential to the \gls{SDC} development workflow.
\paragraph{System Design} \paragraph{System Design}
The \gls{SDC} will make extensive use off-the-shelf tooling for providing these capabilities. The \gls{SDC} will make extensive use of off-the-shelf tooling for providing these capabilities.
As appropriate, these will be integrated with the \portal{}. As appropriate, these will be integrated with the \portal{}.
\subsubsection{Access to Source Code and Software} \subsubsection{Access to Source Code and Software}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment