diff --git a/Makefile b/Makefile index 6f26aeeb20e6ab3c934271a61eda3ec84cd7c398..0fc3cc797b5d3413ecad3266ca55ff0581c5717e 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -DOCHANDLE=SDC-example +DOCHANDLE=SDC-011 export TEXMFHOME ?= astron-texmf/texmf $(DOCHANDLE).pdf: main.tex meta.tex changes.tex diff --git a/main.tex b/main.tex index b3a08a8b38664c760bcd9224d9dcf03125e9d693..732d2a2a274457826bcbd33c3965302032284ff8 100644 --- a/main.tex +++ b/main.tex @@ -1,14 +1,9 @@ -% documentclass{} takes the following mutually-exclusive options: -% -% note -- formats the document as a technical note. -% proposed -- formats the document as a proposed change to the baseline. -% baseline -- formats the document as part of the technical baseline. \documentclass[note]{astron} \input{meta} \input{changes} -\setDocTitle{ASTRON SDC Document Example} +\setDocTitle{SDC Program Activities During 2021} \setDocProgram{SDC} \setDocAuthors{ \addPerson{John D. Swinbank}{ASTRON}{\vcsDate} @@ -21,34 +16,77 @@ \begin{document} -\maketitle +%\maketitle -\printglossary[title=List of Abbreviations] -\clearpage +%\printglossary[title=List of Abbreviations] +%\clearpage -\section{Getting Started} -\label{sec:getting-started} +\section*{\docTitle} -\begin{enumerate} +\subsection*{Science Data Centre Context \& Governance} - \item{Edit \verb|Makefile| and set \verb|DOCHANDLE| to the appropriate handle for your document. - Refer to the \href{https://support.astron.nl/confluence/display/SDCP/List+of+SDC-Series+Documents}{list of SDC series documents} which have been issued to date, and choose the next one in sequence.} +The Science Data Centre is a new facility which aims to be the single point of contact for users of instrumentation operated by ASTRON. +It will ultimately provide services including --- but not limited to --- proposal preparation and submission; archiving and providing access to data; generation of science-ready data products; user-friendly interfaces to discover and access data; and on-line interactive analysis facilities. - \item{Check the preamble of this file to ensure that the document title, programme, and author(s) have been set correctly.} +As an facility, the SDC is run by an operational unit within ASTRON's Astronomy \& Operations department: the SDC Operations team. +The facility is developed and supported by the SDC Development Program, which draws on staff from across the institute. - \item{Insert your text here.} +The SDC was formed in 2020, inheriting a complex service portfolio from its predecessors within ASTRON. +During 2021, our activities have included both establishing the organizational structures on which the SDC will be built and managed, on a number of high-impact technical deliverables, and on engagement with a number of major international projects. - \item{Use the \verb|\gls| command to refer to abbreviations, like \gls{SDC}.} +\subsection*{Scaled Agile Framework} - \item{Use the \verb|\autocite| command to cite other documents, like \autocite{SDC-006}.} +The SDC Program has adopted an organization based on the \emph{Scaled Agile Framework}\footnote{\url{https://scaledagileframework.com}}, or SAFe. +The SAFe system was chosen because it provides best-practice structures for organizing agile software development at scale in a way which focuses both on quality and on responsiveness to customer needs. +Further, SAFe has been shown to be very successful in the scientific context by our peers and colleagues in the Square Kilometre Array project\footnote{\url{https://www.skatelescope.org/}}. - \item{Use the \verb|\cref| command to refer to sections, tables, or other material in this document, like \cref{sec:getting-started}.} +Adopting SAFe has required a substantial effort in training and adapting our ways of working during 2021, but initial results have been very positive and the SDC Program enters 2022 with a solid organizational structure and confidence that we are ready to tackle the challenges of SDC construction! - \item{Run \verb|make| to compile the document.} +\subsection*{Pipeline Execution Framework} -\end{enumerate} +During early 2021, the SDC Program delivered the \emph{ATDB-LDV} pipeline execution framework to operations. +\emph{ATDB-LDV} provides a standardized systems for processing data within the operational SDC facility. +Throughout 2021, it has been used by the \emph{LOFAR Data Valorisation} project in support of ongoing processing campaigns. +We expect that development of the framework will continue, and further pipelines will be integrated with it, during 2022. -\clearpage -\printbibliography +\subsection*{Prefactor Pipeline} + +\emph{Prefactor} is a pipeline for direction-independent calibration and imaging of LOFAR data. +It is the result of several years of development effort, both within ASTRON and across the broader community of LOFAR users, and draws on a number of standard software components developed by ASTRON's Smart Backend group and its partners. + +During 2021, the SDC team accepted responsibility for continued development and maintenance of Prefactor. +It has been updated so that it is now defined using the Common Workflow Language\footnote{\url{https://www.commonwl.org}}, which forms the basis of the pipeline execution system used by the SDC facility. + +This updated Prefactor pipeline was made available in late 2021, and, at time of writing, is undergoing integration with the \emph{ATDB-LDV} execution framework. +During 2022, it will form the basis of processing carried out by the \emph{LOFAR Data Valorisation} project. + +\subsection*{SKA Regional Centres} + +When it becomes operational in the second half of this decade, the Square Kilometre Array will generate enormous quantities of data. +The SKA Observatory itself will not make this data directly available to end users. +Rather, it will be supplied to a globally-distributed network of \emph{SKA Regional Centres} which are responsible for archiving the data and supporting end-user science activities. + +Construction of this SRC network is a major international effort, involving stakeholders and technical teams from across the various members of the SKA. +The SDC Program, in collaboration with the SDCO Operations team and ASTRON's SKA-SDC Focus Group, are leading ASTRON's contribution to the SRC design effort. +During 2021, this has involved collecting system requirements and developing initial architectural concepts. +This work will continue into 2022, as we move towards a large-scale prototyping effort. + +\subsection*{Science Analysis Platform} + +The ESCAPE Project\footnote{European Science Cluster of Astronomy \& Particle Physics ESFRI Research Infrastructures; \url{https://www.projectescape.eu}} is developing common digital infrastructure for use by major research infrastructure projects across Europe. +The SDC Program leads ESCAPE's Work Package 5, which is developing a science analysis platform framework. +This provides data discovery, access, and analysis services across a wide range of science cases and communities. + +During 2021, development of this system continued rapidly, reaching a culmination in an international workshop in August which demonstrated an impressive range of functionality and received positive feedback from the community. +A test version of the platform is has been made available online available\footnote{\url{https://sdc-dev.astron.nl/esap-gui/}, but note that this is strictly a development system and availability and functionality is not guaranteed.}, and work is now underway to develop an ASTRON-specific version which will be deployed within the operational SDC facility (known as ADEX, the ASTRON Data Explorer). + +\subsection*{LoTSS Data Release 2 and Scientific Data Repositories} + +LoTSS, the LOFAR Two-metre Sky Survey, is a major wide-area imaging survey carried out in the LOFAR high band. +Data processing for the survey has been coordinated by the LOFAR Surveys Key Science Project. +The SDC team has coordinated hosting and publication of the second data release through a scientific data repository hosted at SURF (although at time of writing the data is embargoed pending publication of an accompanying scientific paper). + +Deployment of this form of repository at SURF is a key technological step for our SDC plans, and paves the way to our support of future data releases. +It is also a great example of cooperation between the ASTRON SDC, the scientific community using LOFAR, and SURF, the national e-infrastructure provider, a mode of operation which we regard as vital to the success of the SDC effort and the scientific productivity of the community. \end{document}