EGI-Engage:WP5 (SA1) Operations
WP leader: Peter Solagna/EGI.eu
WP contact: egi-engage-wp5@mailman.egi.eu
Objective
This work package will:
- coordinate the operational activities of the EGI production infrastructure, ensuring a secure and reliable provisioning of grid, cloud and storage resources, harmonised between resource providers and peer e-Infrastructures.
- Evolve the security activities in EGI to support the new technologies and resource provisioning paradigms, maintaining a secure and trustworthy infrastructure while supporting new use cases and new ways to access the resources;
- Integrate and deploy platforms on cloud and grid resources to support new use cases for the existing and new EGI users. These platforms will include services for the long-tail of science that will reduce both the barriers for new users to access EGI resources and the learning curve to efficiently use them.
Task Leaders
To contact all task leaders (see below), send mail to
Task | Name | Task Leader | Deputy | |
---|---|---|---|---|
SA1.1 |
Operations Coordination |
Peter Solagna (EGI.eu) | Małgorzata Krakowian (EGI.eu) | |
SA1.2 |
Development of Security Operations |
David Kelsey (STFC) | ||
SA1.3 |
Integration, Deployment of Grid and Cloud Platforms |
TBC (CNR) |
Involved partners
- EGI.eu
- CESNET
- CNRS
- INFN
- CNR
- SURF sara
- CYFRONET
- UU-SNIC
- STFC
- CERN
Tasks
TASK SA1.1 Operations Coordination
(Lead: EGI.eu, M1 – M30)
Estimated task effort: 60PM
The main goal of this task is to coordinate the EGI infrastructure operational activities, liaising with the NGI Operations Managers, to ensure that the operations of the partners providing resources and services to the users are harmonised among each other and with the e-Infrastructure Commons services. The operational procedures and processes will be extended to support diverse technologies and paradigms for service provisioning such as IaaS, PaaS and SaaS.
Operations coordination will ensure that the core services (funded by the NGIs through participation in EGI.eu) are delivered and fulfil their targets, and that their evolution is aligned with the evolution of the EGI production infrastructure. Requirements will be gathered from the service providers and the user communities.
Other activities of this task include: to evolve the portfolio of technologies offered through the Unified Middleware Distribution (UMD), which is used by the resource providers to deploy the software, and to liaise with UMD software contributors and initiate agreements with new technology providers, in particular to include the software products that are needed to support the new upcoming technologies in the Federated Cloud Platform.
TASK SA1.2 Development of Security Operations
(Lead: STFC, M1 – M30)
Estimated task effort: 24PM
This task will develop security operations, including policies, procedures and best practices, to meet the requirements of new trust models, new developments and new usage scenarios as these evolve in EGI-Engage. The work is split into the following activities:
Security requirements and risk assessment for new services, technology, and deployments
The new developments and evolving usage scenarios in EGI-Engage will involve trust models different from the core infrastructure used in EGI-InSPIRE. The task will ensure that the security requirements and the trust model are defined. Any security problems will be addressed and risk assessment associated with new deployments will be developed, to drive operational security in the evolved environment, to keep services secure and available and to mitigate the serious risks
The evolution of operational security procedures, including forensics
Refine and extend the current security procedures and tools for incident response and forensics, for example: to take into account new kinds of players (e.g. cloud resource providers), or to extend the emergency suspension mechanism to cover new kinds of services. The security procedures related to other EGI operational procedures will also be modified as required.
Develop a new trust framework and develop new policies
In collaboration with other infrastructures, we will define new additions to a new policy framework to handle the new deployment and usage scenarios as they evolve in EGI-Engage.
In collaboration with JRA1.1 the task will validate the architecture assumptions through testing in partnership with user communities under realistic production conditions and provide support on AAI security issues in close coordination with the EGI CSIRT and SVG. The task will provide recommendations on how best to sustain this important activity beyond the end of EGI-Engage.
Develop the security challenge framework
Experience from EGI-InSPIRE has shown that performing security service challenges on the operational infrastructure is useful confirm that there is sufficient audit information for traceability of any incident, that procedures and tools are sufficient and that participants are trained and aware of the need to participate in incident response. The framework for these security challenges will be modified and extended to meet the evolving scenarios.
Develop the software vulnerability handling process to adapt to new technology and deployments
Software vulnerability issues in the EGI core infrastructure have been handled through a close relationship with the technology providers, many of whom supply members of the Software Vulnerability Group (SVG). The general principles will remain, including the assessment of risks and the issuing of advisories. In the evolving scenarios of EGI-Engage there are, however, likely to be different types of relationship with the technology providers, especially when this does not involve membership of SVG. The procedures and methods for handling vulnerabilities in EGI-Engage will evolve accordingly.
TASK SA1.3 Integration, Deployment of Grid and Cloud Platforms
(Lead: CNR, M1 – M30)
Estimated task effort: 22PM
This task will deploy, operate and integrate new platforms and frameworks, in the EGI production infrastructure, to enable new use cases and support new user communities.
Fishery and Marine Sciences VREs deployment and operations (CNR) (M6 - M30)
Deploy and maintain a set of fishery and marine sciences VREs to offer innovative working environments with the as-a-Service paradigm of both gCube and EGI. These facilities are offered by exploiting EGI resources, namely hosting of services and data. A number of representative VREs will be pro-actively created to act as a sort of typical environment ready to use for a class of users (e.g. biodiversity students, data managers). In addition VREs will be specifically created to serve the needs of specific use cases identified during the project. This activity will also support the adaptation of existing applications and data to the VREs.
The e-Collaboration for Earth Observation (e-CEO) platform
The e-Collaboration for Earth Observation (e-CEO) platform developed by the European Space Agency (ESA) to support online contexts where researchers can work collaboratively and compare and evaluation different problem-solving approaches. EGI will support these challenges by provision capacity to the e-CEO platform onto the EGI Federated Cloud. The required capacity will be negotiated between ESA and the resource providers via EGI.eu for each challenge and provided as unfunded contribution by the EGI Community as described in the section 3.4.1.
Platform for the long-tail of science (CESNET, CNRS, CYFRONET, INFN)
This task will also develop services and tools to support the long-tail of science, removing the barriers that prevent new users to quickly start using the EGI resources and decreasing the learning curve.
The services are:
- Short-term credential service. This service will provide to the long-tail of science users with short-term X509 credentials, in case of difficulties to access to an IGTF Certification Authority. With this service users will be able to access the EGI resources using credentials that they already own, without the need to start a process for new credentials.
- User Management Portal. This portal will be deployed to allow users to require access to the EGI resources, and to submit a request for resources. The requests will be approved by NGI representatives or outreach teams. The portal will allow the users to use short-term X509 credentials and will prevent users to submit more tasks to the EGI services once they consumed all the capacity allocated for their request or for security reasons.
- Integrated VREs, to lower the learning curve of the EGI services and reduce the time required for new users to become productive and make the most out of the granted resources. The VREs to be integrated are services extensively used by several communities (e.g. DIRAC and SCI-BUS).
Deliverables
The following gives an overview of deliverables scheduled
Code | Title | Delivery PM | Delivery CM | Delivered date | Status |
---|---|---|---|---|---|
D5.1 |
Report on the evolution of the EGI Operations (R) |
M12 |
02.2016 | ||
D5.2 | Platforms for the long-tail of science (R) | M12 |
02.2016 | ||
D5.3 | Evolution of security policies, procedures and best practices (R) |
M25 |
03.2017 | ||
D5.4 | Roadmap of the evolution of the EGI Operations beyond EGI-Engage (R) |
M30 |
08.2017 |
Milestones
The following gives an overview of milestones scheduled
Milestone | Title | Lead-Task | Delivery PM | Delivery CM | Delivered | Status |
---|---|---|---|---|---|---|
M5.1 |
Platform for the long-tail of science is available |
M10 |
12.2015 | |||
M5.2 |
e-CEO challenges run on EGI resources |
M15 |
05.2016 | |||
M5.3 | VREs for the fishery and marine sciences community are deployed in production | M18 | 08.2016 |
Metrics
KPIs
Metrics |
Description | Type | How measured | Target PM12 | Target PM24 | Target PM30 |
---|---|---|---|---|---|---|
KPI.1.JAR2.OpenData | Number of open research datasets that can be published, discovered, used and reused by EGI applications/tools | Cumulative | Number of open datasets published in the EGI Application DB and/or Market Place plus number of open data archives used by applications/tools run in EGI (the latter requires a survey to VOs and VRCs) | 0 |
10 |
20 |
KPI.2.SA1.Intergation | Number of RIs and e-Infrastructures integrated with EGI | Cumulative | Number of RIs and e-Infrastructures that are NOT participants of egi using at least one service from either Core, Collaboration or a Community Platform (via MoU or OLA) | 9 |
11 |
13 |
KPI.3.SA1.Software | Number of new registered software items and VM appliances | Per period | Numbers of new registered software and VM Appliances in AppDB | 50/50 |
60/60 |
70/70 |
KPI.4.SA1.Cloud | Number of providers offering compute and storage capacity accessible through open standard interfaces | Cumulative | Number of Cloud resource centres registering in GOCDB interfaces exposing standard API: OCCI, CDMI... | 25 |
30 |
35 |
KPI.5.SA2.Users | Number of researchers served by EGI | Cumulative | Number of users registered in VOs | 40 000 |
45 000 |
47 000 |
KPI.6.JRA1.AAI | Number of users adopting federated IdP | Cumulative | Number of users accessing EGI services through the IdP Proxy/broker | TBD |
TBD | TBD |
KPI.7.SA2.Users | Number of research communities served | Per period | Number of international and national VOs | 20 |
20 |
20 |
KPI.8.SA1.Users | Number of VO SLAs established | Cumulative | Number of VO SLAs established regarding to HTC, Cloud and Operations tools | 4 |
8 |
10 |
KPI.9.NA2.Communication | Number of scientific publications supported by EGI | Cumulative | The Communication Team requests NGIs to provide a list of publications; the publications are then aggregated in a master list and categorised by NGI | NA |
NA |
NA |
KPI.10.NA2.Communication | Number of relevant authorities informed of the policy paper on procurement | Cumulative | Number of authorities that confirmed reception of the document | 5 |
20 |
25 |
KPI.11.SA1.Users | User satisfaction | Average | Satisfaction of Long tail of science and VO managers with whom EGI has SLA (1 to 5 scale ) | 4 |
5 |
5 |
KPI.12.NA2.Industry | Number of services, demonstrators and project ideas running on EGI for SMEs and industry | Cumulative | RT (dedicated queue for business engagement) | 2 |
7 |
10 |
KPI.13.SA2.Support | Number of delivered knowledge transfer events | Cumulative | Internal registry | 15 |
30 |
45 |
KPI.14.SA1.Size | Number of compute available to international research communities and long tail of science | Per period | Accouting portal | TBD | TBD | TBD |
KPI.15.SA1.Size | Number of storage available to international research communities and long tail of science | Per period | Accouting portal | TBD | TBD | TBD |
KPI.16.SA2.Support | Number of international support cases (for/with RIs, projects, industry) | Cumulative | Number of tickets in technical-support-cases RT queue | 30 |
60 |
90 |
KPI.17.SA1.Size | Number of compute resources available to the long tail of science | Cumulative | Amount of resources (Cores) supporting the long-tail VO | 300 |
500 |
500 |
Activity Metrics
Metrics |
Description | Type | Task |
---|---|---|---|
M.NA1.Quality.1 | Percentage of deliverables and milestones delivered on | Per period | 1.3 |
M.NA2.Communication.1 | Percentage of articles, news, blog posts about or contributed by user communities and NGIs/EIROs with respect to the total of items published in EGI’s channels | Per period | 2.1 |
M.NA2.Communication.2 | Number of unique visitors to the website | Per period | 2.1 |
M.NA2.Communication.3 | Number of pageviews on the website | Per period | 2.1 |
M.NA2.Communication.4 | Number of news items published | Per period | 2.1 |
M.NA2.Communication.5 | Number of events with participation of EGI Champions | Per period | 2.1 |
M.NA2.Communication.6 | Number of case studies published | Per period | 2.1 |
M.NA2.Communication.7 | Attendee-days per event | Per period | 2.1 |
M.NA2.Strategy.1 | Number of EGI impact assessment reports circulated to the stakeholders | Cumulative | 2.2 |
M.NA2.Strategy.2 | Number of MoUs involving EGI.eu or EGI-Engage as a project | Cumulative | 2.2 |
M.NA2.Strategy.3 | Number of SLAs established paying customers | Cumulative | 2.2 |
M.NA2.Industry.1 | Number of engaged SMEs/Industry contacts | Cumulative | 2.3 |
M.NA2.Industry.2 | Number of establish collaborations with SMEs/Industry (with MoU) | Per period | 2.3 |
M.NA2.Industry.3 | Number of requirements gathered from market analysis activities | Per period | 2.3 |
M.JRA1.AAI.1 | Number of communities whose IdP framework integrates with EGI AAI | Cumulative | 3.1 |
M.JRA1.Marketplace.1 | Number of entries in the EGI Marketplace (i.e. services, applications etc.) | Cumulative | 3.2 |
M.JRA1.Accounting.1 | Number of kinds of data repository systems integrated with the EGI accounting software | Cumulative | 3.3 |
M.JRA1.Accounting.2 | Number of kinds of storage systems integrated with the EGI accounting software | Cumulative | 3.3 |
M.JRA1.OpsTools.1 | Number of new requirements introduced in the roadmap | Cumulative | 3.4 |
M.JRA1.OpsTools.2 | Number of probes developed to monitor cloud resources | Per period | 3.4 |
M.JRA1.eGrant.1 | Number of user requests handled in e-GRANT | Per period | 3.5 |
M.JRA2.Cloud.1 | Number of VM instances managed through AppDB GUI | Average |
4.2 |
M.JRA2.Cloud.2 | Percentage of cloud providers providing snapshot support | Per period | 4.2 |
M.JRA2.Cloud.3 | Percentage of cloud providers providing VM resizing support | Per period | 4.2 |
M.JRA2.Cloud.4 | Number of OCCI implementation supporting OCCI 1.2 | Per period | 4.2 |
M.JRA2.Cloud.5 | Number of new OCCI implementations for existing or new CMFs. | Per period | 4.2 |
M.JRA2.Integration.1 | Number of European cloud providers in the federated Astronomy community cloud | Cumulative | 4.3 |
M.JRA2.Integration.2 | Number of virtual appliances shared | Cumulative | 4.3 |
M.JRA2.Integration.3 | Number of different datasets replicated across CADC and EGI | Cumulative | 4.3 |
M.JRA2.Integration.4 | Number of EUDAT services integrated with the HTC and Cloud platforms of EGI | Cumulative | 4.3 |
M.JRA2.Integration.5 | Number of open research datasets replicated in the federated cloud for scalable access by iMARINE VREs | Cumulative | 4.3 |
M.JRA2.Integration.6 | Number of research clouds that interoperate with EGI federated cloud: community clouds, integrated, peer | Cumulative | 4.3 |
M.JRA2.AcceleratedComputing.1 | Number of batch systems for which GPGPU integration is possible to be supported through CREAM | Cumulative | 4.4 |
M.JRA2.AcceleratedComputing.2 | Number of Cloud Middleware Frameworks for which GPGPU integration is supported and implemented | Cumulative | 4.4 |
M.JRA2.AcceleratedComputing.3 | Number of level 3 disciplines with user applications that can use federated accelerated computing | Cumulative | 4.4 |
M.SA1.Operations.1 | Amount of federated HTC compute capacity (EGI participants and integrated) | Cumulative | 5.1 |
M.SA1.Operations.2 | Amount of federated HTC storage capacity (EGI participants and integrated): (Disk, Tape) | Cumulative | 5.1 |
M.SA1.Operations.3 | Amount of allocated resources (storage) allocated through a EGI centrally managed pool of resources | Cumulative | 5.1 |
M.SA1.Operations.4 | Amount of allocated resources (logical cores) allocated through a EGI centrally managed pool of resources | Cumulative | 5.1 |
M.SA1.Operations.5 | Number of new products distributed with UMD | Per period |
5.1 |
M.SA1.SecurityOperations.1 | Number of security policies and procedures updated reviewed and adapted to support new services | Per period | 5.2 |
M.SA1.Platforms.1 | Number of gCUBE VREs instantiated on the Federated Cloud for the iMARINE community | Cumulative | 5.3 |
M.SA1.Platforms.2 | Number of CPU time consumed by e-CEO challenges (hours * cores) | Per period | 5.3 |
M.SA2.UserSupport.1 | Number of training modules produced and kept up-to-date | Cumulative | 6.2 |
M.SA2.UserSupport.2 | HTC Absolute normalized time to a reference value of HEPSPEC06 (excluding OPS and dteam) per 1 level disciplines | Cumulative | 6.2 |
M.SA2.UserSupport.3 | HTC Relative increase normalized time to a reference value of HEPSPEC06 (excluding OPS and dteam) per 1 level disciplines | Per period | 6.2 |
M.SA2.UserSupport.4 | Relative increase of users per 1 level disciplines | Per period | 6.2 |
M.SA2.UserSupport.5 | HTC Number of Low/Medium/High Activity VOs and total | Per period | 6.2 |
M.SA2.UserSupport.6 | Number of VM instantiated in Federated Cloud per 1 level discipline | Per period | 6.2 |
Yearly plan
Internal Documents