Merge branch 'tickets/SDC-232'

.gitlab-ci.yml

@@ -4,6 +4,8 @@ variables:
   GIT_SUBMODULE_STRATEGY: normal
 
 build:
+  before_script:
+    - apt-get update && apt-get install -y git-lfs
   script:
     - make
   artifacts:

1-intro.tex

@@ -29,8 +29,8 @@ This analysis is briefly summarized in \cref{sec:analysis}, with much more exten
 The capabilities thus identified are used to motivate the overall design of the \gls{SDC} facility, which is summarized in \cref{sec:product}.
 In \cref{sec:sizing} we briefly consider whether it is plausible to develop a system at the scale --- in terms of compute and storage --- required to meaningfully address user needs.
 
-Having thus established our goals, the remainder of the document then addresses practical aspects of constructing the facility.
-Specifically, \cref{sec:governance} describes the structure of the work: where the responsibility for building and operating the \gls{SDC} lies, and the ways in which it is accountable within ASTRON and to the wider community.
+Having thus established our goals, the remainder of the document then addresses the interactions between the facility and its environment.
+Specifically, in \cref{sec:community} we describe how the \gls{SDC} will relate to the wider community: who will have access? what will their rights be? how will the facility incorporate feedback from its users and other stakeholders?
 Finally, in \cref{sec:sustainability}, we describe how the project will be made sustainable, both in terms of its environmental impacts, and of its place within the wider landscape of astronomical software and data management systems.
 
 This is a ``living'' document, which we expect to develop and refine as our plans for the \gls{SDC} mature, and as the technical and scientific landscape in which we are operating evolves.

3-analysis.tex

@@ -14,7 +14,7 @@ A three-step process has been undertaken to develop the architecture for the \gl
 \end{enumerate}
 
 This flow is illustrated in \cref{fig:sdc-vision-flow}.
-The architecture arrived at in this way is described in \cref{sec:product}, while the remainder of this document provides background material describing the structure, governance, and sustainability of the \gls{SDC} effort.
+The architecture arrived at in this way is described in \cref{sec:product}, while the remainder of this document provides background material describing the structure and sustainability of the \gls{SDC} effort.
 
 \begin{figure}
 \begin{center}
@@ -83,5 +83,5 @@ This mapping is indicative: it illustrates the range of users being considered a
 The procedure described in this section is indented to bootstrap the development of the \gls{SDC}: we have considered a range of user profiles and the capabilities that they require to get their work done.
 Based on this work, it is possible to start the process of designing the \gls{SDC} and planning for its operation.
 However, it is not expected that this analysis be exhaustive or immutable.
-Rather, the \gls{SDCO} team --- introduced in detailed in \cref{sec:governance} --- will develop a detailed set of use cases and operational concepts that enabled detailed design work to be undertaken and, ultimately, the complete operational model for the \gls{SDC} facility to be developed.
+Rather, the \gls{SDCO} team will develop a detailed set of use cases and operational concepts that enabled detailed design work to be undertaken and, ultimately, the complete operational model for the \gls{SDC} facility to be developed.
 Thus, the work presented here will be substantially augmented and expanded upon by future \gls{SDCO} documentation.

5-sizing.tex

@@ -64,5 +64,5 @@ The \gls{SDCO} team will engage with the \gls{ILT} Board and, where appropriate,
 The \gls{SDC} will make both storage and computing resources available to end users (\Cref{sec:features:standardpipes,sec:features:custompipes,sec:features:interactive}).
 In both cases, it is an operational and practical necessary that these resources be limited to avoid resource exhaustion which may interfere with other users' activities or even with the operation of the facility itself.
 
-Resource allocation will be a matter of policy which is set by \gls{SDC} leadership in conjunction with, and under the guidance of, various stakeholders, as discussed in \cref{sec:governance}.
+Resource allocation will be a matter of policy which is set by \gls{SDC} leadership in conjunction with, and under the guidance of, various stakeholders, as discussed in \cref{sec:community:engagement}.
 It is outside the scope of this document to define that policy; we simply note here that mechanisms for enforcing policy must be provided.

6-community.tex

+\section{Community Interactions}
+\label{sec:community}
+
+This section describes the ways in which the \gls{SDC} will interact with its community.
+This includes not only policies around access to the facility and the ways in which data rights are enforced, but also the mechanisms by which feedback from the astronomical community and key stakeholders will be addressed.
+
+\subsection{Access Policy and Data Rights}
+
+We envision the SDC as being accessible to all comers, with no restrictions on who is able to sign up for an account and log in to the basic service.
+However:
+
+\begin{itemize}
+
+\item Policies will be defined for the use of resources such as storage, computing, and data transfer: simply having the ability to access the \gls{SDC} will not guarantee users with substantial capacity.
+
+\item Users will have to submit proposals for substantial requests or, when requesting observing time, for the generation of advanced data products through the standard ASTRON pipelines.
+      If awarded, these requests will be handled by \gls{SDCO}\footnote{Expert users may be asked to participate in this following the current model of ‘user shared-support’ mode.}, who will deliver the final data products to the users.
+      For processing requests within a certain quota (to be defined), users will be able to directly use the \gls{SDC} resources and will not need to go through the proposal submission process.
+
+\item The policy detailing the resource allocation will need further definition beyond its current concept state.
+      However, we note here simply that \gls{SDC} systems must be able to effectively implement this policy.
+
+\item Data stored within and made available through the \gls{SDC} will be subject to ownership and data rights policies which will vary based on its origin.
+      Again, \gls{SDC} systems must be capable of recognizing and enforcing these policies.
+
+\end{itemize}
+
+Given an increasing move towards a model of cloud-based compute and storage, members of the user community may be able to provision their own compute and/or storage capacity on systems co-located with the SDC.
+As such, users could directly access SDC resources at their own cost.
+We suggest that this might be a worthwhile avenue for future development, but is not a priority for the SDC at the present time.
+
+\subsection{Community Engagement}
+\label{sec:community:engagement}
+
+We regard it as vital that the SDC be responsive to the current and emergent needs of the scientific community and other facilities.
+By engaging with them directly, we can best ensure that the SDC is achieving its aims, and plan for and prioritise future work.
+We therefore expect the SDC team to maintain regular contact with:
+
+\begin{itemize}
+\item the Board of the \gls{ILT};
+\item the \gls{LUC};
+\item the various \gls{LOFAR} \glspl{KSP};
+\item other relevant organizations representing sections of the scientific community;
+\item the \gls{SRC} network.
+\end{itemize}
+
+Engagement with \gls{ILT} Board is realised through a regular dialogue every trimester.
+The \gls{SDCO} team provides updates about the activities relevant to \gls{LOFAR} and receives feedback from the Board concerning the areas that should receive focus in the future.
+The \gls{SDCO} also has a regular dialogue with the \gls{LUC} every quarter, during which they offer a comprehensive overview of the achieved results, active development areas, and planning.
+Following this, the \gls{LUC} will summarize the feedback from its members, as well as that the \glspl{KSP} it represents, and share it with \gls{SDCO}.
+Dialogue with the \gls{SRC} network is realised through the engagement and contribution of various \gls{SDC} representatives to the \gls{SRCSC} working groups.
+
+In addition, we expect to hold a programme of \gls{SDC} schools, traineeships, workshops, and other educational events which will serve not only to inform the community about the capabilities available through the \gls{SDC} but also to provide fora through which the \gls{SDC} team can solicit feedback directly from community members.

6-governance.tex

-\section{Organization and Governance}
-\label{sec:governance}
-
-This section addresses the way in which the effort to construct and operate the \gls{SDC} will be organized.
-This includes both the structures which have been established within ASTRON, and the expected interactions with other stakeholders and the wider community.
-
-\subsection{Structure}
-\label{sec:governance:structure}
-
-The ASTRON \gls{MT} has established two bodies to construct and operate the \gls{SDC}:
-
-\begin{itemize}
-
-  \item{The \emph{\acrlong{SDCP}} (\acrshort{SDCP}) is responsible for developing the software and services that are necessary to operate the \gls{SDC}.}
-  \item{\emph{\acrlong{SDCO}} (\acrshort{SDCO}) is responsible for operating the \gls{SDC}, defining and prioritizing a service offering to meet community needs, and providing support to members of the community.}
-
-\end{itemize}
-
-It is expected that these bodies will draw on both other staff within ASTRON --- members of the various Competence and Focus Groups --- and from the external scientific community.
-They are overseen by a Steering Committee, appointed by the ASTRON \gls{MT}.
-The relationship between all of these groups is illustrated in \cref{fig:sdc-hierarchy}, while their various responsibilities are described below.
-
-\subsubsection{\acrlong{SDCP}}
-\label{sec:governance:structure:sdcp}
-
-\renewcommand{\floatpagefraction}{.8}
-\begin{figure}
-\begin{center}
-\includegraphics[width=0.66\textwidth]{figures/SDC/sdc-hierarchy}
-\end{center}
-\caption{%
-Relationships between core stakeholders in the \gls{SDC} effort.
-Arrows indicate expected exchanges of information, expertise, and effort, rather than organizational hierarchy, which is discussed in the text.
-}
-\label{fig:sdc-hierarchy}
-\end{figure}
-
-The \gls{SDCP} is envisioned as a \emph{development programme} within ASTRON.
-As such, it is formally a temporary organization which exists to develop certain defined functionality; in this case, the software required to operate the \gls{SDC}.
-
-The \gls{SDCP} is jointly responsible with \gls{SDCO} for developing the overall architecture of the \gls{SDC} facility.
-It is expected that the \gls{SDCP} will bring software development and architectural expertise to this discussion, which will be used to design the overall system based on the available infrastructure and operational needs of \gls{SDCO}.
-
-The \gls{SDCP} is responsible for the delivery of operable \emph{software products} to the \gls{SDCO} team.
-These products are combined with infrastructure --- provided by the \gls{SDCO} team --- and appropriate configuration to deploy and operate the services which comprise the \gls{SDC}.
-The \gls{SDCP} has ultimate responsibility for the software used to implement all core and ancillary (but not support; \cref{sec:product:arch}) services deployed within the \gls{SDC} facility\footnote{It does not follow that the \gls{SDCP} must implement this software in-house; indeed, it is expected that much of the \gls{SDC} software infrastructure will be based on third-party tooling.}.
-
-The \gls{SDCP} is further responsible for the delivery of the scientific pipelines which are used as part of regular facility operations to generate the core \glspl{SRDP} (\cref{sec:product:science:pipelines}), as well as any other pipelines which are identified as strategically important by the \gls{SDCO} team \emph{and} which are agreed as within scope by \gls{SDCP} management.
-
-The \gls{SDCP} is expected to arrange its work to address the scientific and operational requirements, and their relative priorities, expressed by \gls{SDCO}.
-Detailed scheduling of work will remain the domain of \gls{SDCP}, and it is acknowledged that development or management considerations must be accounted for in parallel with \gls{SDCO} needs.
-
-The \gls{SDCP} is responsible for defining the working methods around the software products it delivers.
-This includes, for example, adopting appropriate agile team working practices, software release methodologies and the like.
-These methods will make it possible for the \gls{SDCP} to provide appropriate support to the \gls{SDCO} team when necessary.
-Wherever possible, these methods will be aligned with approaches adopted across ASTRON more generally.
-
-The \gls{SDCP} is responsible for delivering software products of high quality.
-That is, the products are appropriately tested, suitably documented, and packaged adequately for distribution.
-It is expected that the \gls{SDCP} will collaborate with the Software Delivery Competence Group to develop coding, testing, and packaging standards which can be applied both to the \gls{SDCP} itself and, ultimately, across ASTRON's software development effort.
-
-The \gls{SDCP} is primarily funded and supported through executing projects which are sponsored by external organizations such as the \acrlong{EC} and \acrshort{NWO}.
-The \gls{SDCP} is responsible for coordinating efforts across ASTRON to compete for and win funding in this way.
-These projects have their own deliverables and milestones, and the \gls{SDCP} is responsible for ensuring that these are met and the projects are completed successfully.
-Projects will be chosen, and the work will be managed, such that project deliverables contribute to the software products that \gls{SDCP} will deliver to \gls{SDCO}.
-Some level of additional funding to support integration and oversight activities will also be required; this will be drawn from the ASTRON base budget and in-kind contributions as appropriate.
-This structure is illustrated in \cref{fig:sdc-organization}.
-
-According to the ``ASTRON 2.0'' organizational model, as a programme the \gls{SDCP} does not have any permanent staff.
-Rather, these are sourced from Competence and Focus Groups.
-Additional effort may be drawn from in-kind contributions to the \gls{ILT} or other projects, with the agreement of the \gls{ILT} Board where appropriate.
-The \gls{SDCP} is responsible for coordinating with the various group leaders to ensure that the development teams are adequately resourced.
-
-\gls{SDCP} is led by a Programme Manager, who is functionally responsible to the ASTRON \gls{MT}.
-
-\begin{figure}
-\begin{center}
-\includegraphics[width=\textwidth]{figures/SDC/sdc-organization}
-\end{center}
-\caption{%
-Structure of the \gls{SDC} development and operations effort.
-The bulk of development effort is funded through externally-sponsored projects, with their own deliverables, shown at left.
-The \gls{SDCP} integrates functionality developed for these projects and augments it with internal, ASTRON-funded work to create the \gls{SDC} Software Products.
-This software is then deployed to form operational services by the \gls{SDCO} team.
-Generation of \glspl{SRDP} is also the responsibility of \gls{SDCO}.
-}
-\label{fig:sdc-organization}
-\end{figure}
-
-\subsubsection{\acrlong{SDCO}}
-\label{sec:governance:structure:sdco}
-
-The \gls{SDC} is intended to be a long-term facility within ASTRON, broadly equivalent to a major piece of instrumentation like \gls{LOFAR}.
-As such, it will be run by the \gls{SDCO} Operational Unit within ASTRON's \gls{AO} Department.
-The overriding goal of the \gls{SDCO} team is to deliver effective services and data product to the community.
-
-\gls{SDCO} is responsible for day-to-day facility operations.
-This includes configuring, resourcing, and managing all services provided by the facility, as well as identifying and responding to problems.
-Where those problems are software bugs, they may either be addressed directly, or, when necessary, referred to the \gls{SDCP}, as described in \cref{sec:governance:structure:sdcp}.
-
-\gls{SDCO} will provide ``helpdesk''-type support to facility users.
-This will include establishing and operating ticketing systems, discussion forums, mailing lists, etc, as appropriate.
-
-\gls{SDCO} will provide training, tutorials, documentation, and other material to end users.
-
-\gls{SDCO} is responsible for provisioning the infrastructure on which the \gls{SDC} facility will run.
-This infrastructure may be directly purchased or leased, sourced from cloud or other vendors, or from in-kind contributions to the \gls{ILT} or other projects, with the agreement of the \gls{ILT} Board where appropriate.
-Infrastructure will be consistent with the agreed-upon \gls{SDC} system architecture.
-
-\gls{SDCO} is jointly responsible with the \gls{SDCP} for developing the overall architecture of the \gls{SDC} facility.
-It is expected that this architecture will reflect current and likely future infrastructure available to the \gls{SDCO} team, as well as their operational requirements.
-
-\gls{SDCO} will provide resources for maintenance of the software products delivered by \gls{SDCP} on which the \gls{SDC} facility relies, with the ultimate aim of ensuring that both the software itself and the associated developer expertise is sustainable beyond the duration of the \gls{SDCP} itself.
-Even when funded by \gls{SDCO}, developers working on \gls{SDCP} products will follow the working practices, norms, policies, and procedures established by \gls{SDCP}.
-
-\gls{SDCO} will work with the community and other stakeholders to identify both long- and short-term scientific and functional priorities for the \gls{SDC}.
-These will be communicated to the \gls{SDCP}, and form the basis of a collaborative process of defining detailed requirements and prioritizing work.
-
-\gls{SDCO} is responsible for acceptance testing and commissioning of the products delivered by \gls{SDCP}.
-
-\gls{SDCO} is responsible for using the functionality provided by the \gls{SDC} facility to generate a set of standardized data products and to make them available to end users.
-
-The \gls{SDCO} Operational Unit has some permanent staffing funded by the ASTRON base budget.
-Additional members of the operations team will be drawn from Focus and Competence Groups as required.
-
-The \gls{SDCO} Operational Unit Lead is responsible to the Head of \gls{AO}.
-
-\subsubsection{\acrshort{SDC} Steering Committee}
-\label{sec:governance:structure:steering}
-
-The \gls{SDC} Steering Committee is a standing committee which was established by the ASTRON \gls{MT} to oversee all aspects of the \gls{SDC}.
-In particular, it will provide a forum for resolving any differences or inconsistencies between the approaches taken by \gls{SDCO} and \gls{SDCP}.
-
-\subsection{Community Engagement}
-
-Beyond the formal management structure outlined above, we regard it as vital that the SDC be responsive to the current and emergent needs of the scientific community and other facilities.
-By engaging with them directly, we can best ensure that the SDC is achieving its aims, and plan for and prioritise future work.
-We therefore expect the SDC team to maintain regular contact with:
-
-\begin{itemize}
-\item the Board of the \gls{ILT};
-\item the \gls{LUC};
-\item the various \gls{LOFAR} \glspl{KSP};
-\item other relevant organizations representing sections of the scientific community;
-\item the \gls{SRC} network.
-\end{itemize}
-
-Engagement with \gls{ILT} Board is realised through a regular dialogue every trimester.
-The \gls{SDCO} team provides updates about the activities relevant to \gls{LOFAR} and receives feedback from the Board concerning the areas that should receive focus in the future.
-The \gls{SDCO} also has a regular dialogue with the \gls{LUC} every quarter, during which they offer a comprehensive overview of the achieved results, active development areas, and planning.
-Following this, the \gls{LUC} will summarize the feedback from its members, as well as that the \glspl{KSP} it represents, and share it with \gls{SDCO}.
-Dialogue with the \gls{SRC} network is realised through the engagement and contribution of various \gls{SDC} representatives to the \gls{SRCSC} working groups.
-
-In addition, we expect to hold a programme of \gls{SDC} schools, traineeships, workshops, and other educational events which will serve not only to inform the community about the capabilities available through the \gls{SDC} but also to provide fora through which the \gls{SDC} team can solicit feedback directly from community members.
-
-\subsection{Access Policy and Data Rights}
-
-We envision the SDC as being accessible to all comers, with no restrictions on who is able to sign up for an account and log in to the basic service.
-However:
-
-\begin{itemize}
-
-\item Policies will be defined for the use of resources such as storage, computing, and data transfer: simply having the ability to access the \gls{SDC} will not guarantee users with substantial capacity.
-
-\item Users will have to submit proposals for substantial requests or, when requesting observing time, for the generation of advanced data products through the standard ASTRON pipelines.
-      If awarded, these requests will be handled by \gls{SDCO}\footnote{Expert users may be asked to participate in this following the current model of ‘user shared-support’ mode.}, who will deliver the final data products to the users.
-      For processing requests within a certain quota (to be defined), users will be able to directly use the \gls{SDC} resources and will not need to go through the proposal submission process.
-
-\item The policy detailing the resource allocation will need further definition beyond its current concept state.
-      However, we note here simply that \gls{SDC} systems must be able to effectively implement this policy.
-
-\item Data stored within and made available through the \gls{SDC} will be subject to ownership and data rights policies which will vary based on its origin.
-      Again, \gls{SDC} systems must be capable of recognizing and enforcing these policies.
-
-\end{itemize}
-
-Given an increasing move towards a model of cloud-based compute and storage, members of the user community may be able to provision their own compute and/or storage capacity on systems co-located with the SDC.
-As such, users could directly access SDC resources at their own cost.
-We suggest that this might be a worthwhile avenue for future development, but is not a priority for the SDC at the present time.

SDC-006.tex

@@ -52,7 +52,7 @@
 \clearpage
 \input{5-sizing}
 \clearpage
-\input{6-governance}
+\input{6-community}
 \clearpage
 \input{7-sustainability}
 \clearpage