diff --git a/.gitmodules b/.gitmodules index a2f4555068035b87770fffbe020469436ef2ac93..b12766ceb8764f1ea9ebe1fd2c29fcbfa4e2e1df 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,3 +1,6 @@ [submodule "astron-texmf"] path = astron-texmf url = https://git.astron.nl/swinbank/astron-texmf.git +[submodule "figures"] + path = figures + url = https://git.astron.nl/swinbank/doc-sdc-figures.git diff --git a/SDC-009.tex b/SDC-009.tex index ed642c3f947feccd873c12c599b4c696c69e02e3..4859393f095002df826ed3c48b3430cba7741079 100644 --- a/SDC-009.tex +++ b/SDC-009.tex @@ -24,9 +24,9 @@ \glsfindwidesttoplevelname %\setglossarystyle{mcolindex} \printglossary[title=List of Abbreviations] -\clearpage -\section{Introduction} +\clearpage +\input{content} \clearpage \printbibliography diff --git a/content.tex b/content.tex new file mode 100644 index 0000000000000000000000000000000000000000..6bcfc7d13076779d755934cba5449c787fe244b2 --- /dev/null +++ b/content.tex @@ -0,0 +1,180 @@ +\section{Introduction} +\label{sec:governance} + +This document addresses the way in which the effort to construct and operate the \gls{SDC} will be organized. +This includes both the structures which have been established within ASTRON, and the expected interactions with other stakeholders and the wider community. + +\section{Structure} +\label{sec:governance:structure} + +The ASTRON \gls{MT} has established two bodies to construct and operate the \gls{SDC}: + +\begin{itemize} + + \item{The \emph{\acrlong{SDCP}} (\acrshort{SDCP}) is responsible for developing the software and services that are necessary to operate the \gls{SDC}.} + \item{\emph{\acrlong{SDCO}} (\acrshort{SDCO}) is responsible for operating the \gls{SDC}, defining and prioritizing a service offering to meet community needs, and providing support to members of the community.} + +\end{itemize} + +It is expected that these bodies will draw on both other staff within ASTRON --- members of the various Competence and Focus Groups --- and from the external scientific community. +They are overseen by a Steering Committee, appointed by the ASTRON \gls{MT}. +The relationship between all of these groups is illustrated in \cref{fig:sdc-hierarchy}, while their various responsibilities are described below. + +\subsection{\acrlong{SDCP}} +\label{sec:governance:structure:sdcp} + +\renewcommand{\floatpagefraction}{.8} +\begin{figure} +\begin{center} +\includegraphics[width=0.66\textwidth]{figures/SDC/sdc-hierarchy} +\end{center} +\caption{% +Relationships between core stakeholders in the \gls{SDC} effort. +Arrows indicate expected exchanges of information, expertise, and effort, rather than organizational hierarchy, which is discussed in the text. +} +\label{fig:sdc-hierarchy} +\end{figure} + +The \gls{SDCP} is envisioned as a \emph{development programme} within ASTRON. +As such, it is formally a temporary organization which exists to develop certain defined functionality; in this case, the software required to operate the \gls{SDC}. + +The \gls{SDCP} is jointly responsible with \gls{SDCO} for developing the overall architecture of the \gls{SDC} facility. +It is expected that the \gls{SDCP} will bring software development and architectural expertise to this discussion, which will be used to design the overall system based on the available infrastructure and operational needs of \gls{SDCO}. + +The \gls{SDCP} is responsible for the delivery of operable \emph{software products} to the \gls{SDCO} team. +These products are combined with infrastructure --- provided by the \gls{SDCO} team --- and appropriate configuration to deploy and operate the services which comprise the \gls{SDC}. +The \gls{SDCP} has ultimate responsibility for the software used to implement all core and ancillary (but not support \autocite{SDC-006}) services deployed within the \gls{SDC} facility\footnote{It does not follow that the \gls{SDCP} must implement this software in-house; indeed, it is expected that much of the \gls{SDC} software infrastructure will be based on third-party tooling.}. + +The \gls{SDCP} is further responsible for the delivery of the scientific pipelines which are used as part of regular facility operations to generate the core \glspl{SRDP}. +This includes pipelines specifically identified in \citetitle[the][]{SDC-006} \autocite{SDC-006} or other baselined documentation as well as any other pipelines which are identified as strategically important by the \gls{SDCO} team \emph{and} which are agreed as within scope by \gls{SDCP} management. + +The \gls{SDCP} is expected to arrange its work to address the scientific and operational requirements, and their relative priorities, expressed by \gls{SDCO}. +Detailed scheduling of work will remain the domain of \gls{SDCP}, and it is acknowledged that development or management considerations must be accounted for in parallel with \gls{SDCO} needs. + +The \gls{SDCP} is responsible for defining the working methods around the software products it delivers. +This includes, for example, adopting appropriate agile team working practices, software release methodologies and the like. +These methods will make it possible for the \gls{SDCP} to provide appropriate support to the \gls{SDCO} team when necessary. +Wherever possible, these methods will be aligned with approaches adopted across ASTRON more generally. + +The \gls{SDCP} is responsible for delivering software products of high quality. +That is, the products are appropriately tested, suitably documented, and packaged adequately for distribution. +It is expected that the \gls{SDCP} will collaborate with the Software Delivery Competence Group to develop coding, testing, and packaging standards which can be applied both to the \gls{SDCP} itself and, ultimately, across ASTRON's software development effort. + +The \gls{SDCP} is primarily funded and supported through executing projects which are sponsored by external organizations such as the \acrlong{EC} and \acrshort{NWO}. +The \gls{SDCP} is responsible for coordinating efforts across ASTRON to compete for and win funding in this way. +These projects have their own deliverables and milestones, and the \gls{SDCP} is responsible for ensuring that these are met and the projects are completed successfully. +Projects will be chosen, and the work will be managed, such that project deliverables contribute to the software products that \gls{SDCP} will deliver to \gls{SDCO}. +Some level of additional funding to support integration and oversight activities will also be required; this will be drawn from the ASTRON base budget and in-kind contributions as appropriate. +This structure is illustrated in \cref{fig:sdc-organization}. + +According to the ``ASTRON 2.0'' organizational model, as a programme the \gls{SDCP} does not have any permanent staff. +Rather, these are sourced from Competence and Focus Groups. +Additional effort may be drawn from in-kind contributions to the \gls{ILT} or other projects, with the agreement of the \gls{ILT} Board where appropriate. +The \gls{SDCP} is responsible for coordinating with the various group leaders to ensure that the development teams are adequately resourced. + +\gls{SDCP} is led by a Programme Manager, who is functionally responsible to the ASTRON \gls{MT}. + +\begin{figure} +\begin{center} +\includegraphics[width=\textwidth]{figures/SDC/sdc-organization} +\end{center} +\caption{% +Structure of the \gls{SDC} development and operations effort. +The bulk of development effort is funded through externally-sponsored projects, with their own deliverables, shown at left. +The \gls{SDCP} integrates functionality developed for these projects and augments it with internal, ASTRON-funded work to create the \gls{SDC} Software Products. +This software is then deployed to form operational services by the \gls{SDCO} team. +Generation of \glspl{SRDP} is also the responsibility of \gls{SDCO}. +} +\label{fig:sdc-organization} +\end{figure} + +\subsection{\acrlong{SDCO}} +\label{sec:governance:structure:sdco} + +The \gls{SDC} is intended to be a long-term facility within ASTRON, broadly equivalent to a major piece of instrumentation like \gls{LOFAR}. +As such, it will be run by the \gls{SDCO} Operational Unit within ASTRON's \gls{AO} Department. +The overriding goal of the \gls{SDCO} team is to deliver effective services and data product to the community. + +\gls{SDCO} is responsible for day-to-day facility operations. +This includes configuring, resourcing, and managing all services provided by the facility, as well as identifying and responding to problems. +Where those problems are software bugs, they may either be addressed directly, or, when necessary, referred to the \gls{SDCP}, as described in \cref{sec:governance:structure:sdcp}. + +\gls{SDCO} will provide ``helpdesk''-type support to facility users. +This will include establishing and operating ticketing systems, discussion forums, mailing lists, etc, as appropriate. + +\gls{SDCO} will provide training, tutorials, documentation, and other material to end users. + +\gls{SDCO} is responsible for provisioning the infrastructure on which the \gls{SDC} facility will run. +This infrastructure may be directly purchased or leased, sourced from cloud or other vendors, or from in-kind contributions to the \gls{ILT} or other projects, with the agreement of the \gls{ILT} Board where appropriate. +Infrastructure will be consistent with the agreed-upon \gls{SDC} system architecture. + +\gls{SDCO} is jointly responsible with the \gls{SDCP} for developing the overall architecture of the \gls{SDC} facility. +It is expected that this architecture will reflect current and likely future infrastructure available to the \gls{SDCO} team, as well as their operational requirements. + +\gls{SDCO} will provide resources for maintenance of the software products delivered by \gls{SDCP} on which the \gls{SDC} facility relies, with the ultimate aim of ensuring that both the software itself and the associated developer expertise is sustainable beyond the duration of the \gls{SDCP} itself. +Even when funded by \gls{SDCO}, developers working on \gls{SDCP} products will follow the working practices, norms, policies, and procedures established by \gls{SDCP}. + +\gls{SDCO} will work with the community and other stakeholders to identify both long- and short-term scientific and functional priorities for the \gls{SDC}. +These will be communicated to the \gls{SDCP}, and form the basis of a collaborative process of defining detailed requirements and prioritizing work. + +\gls{SDCO} is responsible for acceptance testing and commissioning of the products delivered by \gls{SDCP}. + +\gls{SDCO} is responsible for using the functionality provided by the \gls{SDC} facility to generate a set of standardized data products and to make them available to end users. + +The \gls{SDCO} Operational Unit has some permanent staffing funded by the ASTRON base budget. +Additional members of the operations team will be drawn from Focus and Competence Groups as required. + +The \gls{SDCO} Operational Unit Lead is responsible to the Head of \gls{AO}. + +\subsection{\acrshort{SDC} Steering Committee} +\label{sec:governance:structure:steering} + +The \gls{SDC} Steering Committee is a standing committee which was established by the ASTRON \gls{MT} to oversee all aspects of the \gls{SDC}. +In particular, it will provide a forum for resolving any differences or inconsistencies between the approaches taken by \gls{SDCO} and \gls{SDCP}. + +\section{Community Engagement} + +Beyond the formal management structure outlined above, we regard it as vital that the SDC be responsive to the current and emergent needs of the scientific community and other facilities. +By engaging with them directly, we can best ensure that the SDC is achieving its aims, and plan for and prioritise future work. +We therefore expect the SDC team to maintain regular contact with: + +\begin{itemize} +\item the Board of the \gls{ILT}; +\item the \gls{LUC}; +\item the various \gls{LOFAR} \glspl{KSP}; +\item other relevant organizations representing sections of the scientific community; +\item the \gls{SRC} network. +\end{itemize} + +Engagement with \gls{ILT} Board is realised through a regular dialogue every trimester. +The \gls{SDCO} team provides updates about the activities relevant to \gls{LOFAR} and receives feedback from the Board concerning the areas that should receive focus in the future. +The \gls{SDCO} also has a regular dialogue with the \gls{LUC} every quarter, during which they offer a comprehensive overview of the achieved results, active development areas, and planning. +Following this, the \gls{LUC} will summarize the feedback from its members, as well as that the \glspl{KSP} it represents, and share it with \gls{SDCO}. +Dialogue with the \gls{SRC} network is realised through the engagement and contribution of various \gls{SDC} representatives to the \gls{SRCSC} working groups. + +In addition, we expect to hold a programme of \gls{SDC} schools, traineeships, workshops, and other educational events which will serve not only to inform the community about the capabilities available through the \gls{SDC} but also to provide fora through which the \gls{SDC} team can solicit feedback directly from community members. + +\section{Access Policy and Data Rights} + +We envision the SDC as being accessible to all comers, with no restrictions on who is able to sign up for an account and log in to the basic service. +However: + +\begin{itemize} + +\item Policies will be defined for the use of resources such as storage, computing, and data transfer: simply having the ability to access the \gls{SDC} will not guarantee users with substantial capacity. + +\item Users will have to submit proposals for substantial requests or, when requesting observing time, for the generation of advanced data products through the standard ASTRON pipelines. + If awarded, these requests will be handled by \gls{SDCO}\footnote{Expert users may be asked to participate in this following the current model of ‘user shared-support’ mode.}, who will deliver the final data products to the users. + For processing requests within a certain quota (to be defined), users will be able to directly use the \gls{SDC} resources and will not need to go through the proposal submission process. + +\item The policy detailing the resource allocation will need further definition beyond its current concept state. + However, we note here simply that \gls{SDC} systems must be able to effectively implement this policy. + +\item Data stored within and made available through the \gls{SDC} will be subject to ownership and data rights policies which will vary based on its origin. + Again, \gls{SDC} systems must be capable of recognizing and enforcing these policies. + +\end{itemize} + +Given an increasing move towards a model of cloud-based compute and storage, members of the user community may be able to provision their own compute and/or storage capacity on systems co-located with the SDC. +As such, users could directly access SDC resources at their own cost. +We suggest that this might be a worthwhile avenue for future development, but is not a priority for the SDC at the present time. diff --git a/figures b/figures new file mode 160000 index 0000000000000000000000000000000000000000..0f0651f746665aa7587a654cfa0705c1484362bf --- /dev/null +++ b/figures @@ -0,0 +1 @@ +Subproject commit 0f0651f746665aa7587a654cfa0705c1484362bf