Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Core Activities and Services PHASE III"

From EGIWiki
Jump to navigation Jump to search
(37 intermediate revisions by 3 users not shown)
Line 4: Line 4:
<br> This page will list the EGI.eu core activities provided by EGI.eu partners for the period: '''January 2018 - December 2020'''  
<br> This page will list the EGI.eu core activities provided by EGI.eu partners for the period: '''January 2018 - December 2020'''  


Proposals are in evaluation phase.
Proposals have been evaluated and the results have been used as inputs for the EOSC-hub project (funded).
 
'''Bidding phase III is now closed'''


= List of the bidding activities  =
= List of the bidding activities  =
Line 32: Line 34:
The Accounting Portal receives and stores the site, user, and VO level summaries generated by the Accounting Repository and provides views via a web portal, for example, by aggregating sites in a country on custom time intervals. The databases are organized into a CPU record database, a User record database, and a topology database.
The Accounting Portal receives and stores the site, user, and VO level summaries generated by the Accounting Repository and provides views via a web portal, for example, by aggregating sites in a country on custom time intervals. The databases are organized into a CPU record database, a User record database, and a topology database.


'''Consortium''':
'''Consortium''': CESGA, STFC


'''Contacts''':
'''Contacts''': Adrian Coveney - adrian.coveney<AT>stfc.ac.uk, Iván Díaz Álvarez - idiaz<AT>cesga.es


== Application DB (virtual appliances and applications library) ==
== Application DB (virtual appliances and applications library) ==
'''Overview''': The EGI Applications Database (AppDB) is a central service that provides:
*Information about software solutions in the form of native software products and virtual appliances, linking the programmers and the scientists who are involved, and the publications derived from the registered solutions
*The tools for the distribution of the virtual machine images in the cloud sites part of the the federated cloud
* A dashboard to operate virtual machines in the fedcloud sites
Three types of software solutions are offered through the EGI Applications Database:
*Software items, in its classical sense, i.e. applications, tools, utilities, etc..,
*Virtual Appliances: composed by one or more pre-configured virtual machine images packaged with an operating system and software application(s)
*Software Appliances: one or more a set pairs of a virtual appliance and a contextualization script. A Contextualization Script (CS) is the script launched on VM boot time and could be used for installing, configuring and preparing software upon boot time on a pre-defined virtual machine image
'''Consortium''': IASA
'''Contacts''': Marios Chatziangelou (mhaggel<AT>iasa.gr), Paraskevas Sphicas (sphicas<AT>iasa.gr)
== Collaboration tools ==
== Collaboration tools ==
'''Overview''':
Collaborations tools are services needed by the EGI back-office and supporting EGI collaboration.
'''Consortium''': CESNET
'''Contacts''': Martin Kuba (makub<AT>ics.mun i.cz) main and administrative contact, Michal Stava (michal.stava<AT>cesnet.cz
== DIRAC4EGI ==
== DIRAC4EGI ==
'''Overview''':
DIRAC service is provided to the EGI community as a workload management service used to distribute the users' computing tasks among the available resources both HTC and cloud. The service will support established Virtual Organization as well as the long tail of users.
Given the critical nature of the activity bid must contain an availability and continuity plan for the service.
'''Consortium''': CNRS, CYFRONET
'''Contacts''': Andrei Tsaregorodtsev (atsareg@in2p3.fr), Tomasz Szepieniec (t.szepieniec@cyfronet.pl)
== Helpdesk (GGUS) ==
== Helpdesk (GGUS) ==
'''Overview''': EGI provides support to users and operators through a distributed helpdesk with central coordination (GGUS). The central helpdesk provides a single interface for support. The central system is interfaced to a variety of other ticketing systems at the NGI level in order to allow a bi-directional exchange of tickets.
'''Consortium''': KIT
'''Contacts''': Andreas Heiss (andreas.heiss<AT>kit.edu), Günter Grein (guenter.grein<AT>kit.edu)
== Helpdesk human support ==
== Helpdesk human support ==
'''Overview''': EGI provides support to users and operators through a distributed helpdesk with central coordination (GGUS). The central helpdesk provides a single interface for support. The helpdesk support activities are grouped in 1st and 2nd level support.
'''Consortium''': CESNET
'''Contacts''': Zdeněk Šustr (sustr4@cesnet.cz), Petr Hanousek (petr.hanousek@cesnet.cz)
== Marketplace and resource allocation ==
== Marketplace and resource allocation ==
'''Overview''': This activity will run the EGI systems and underpinning human processes necessary to enable discovery, selection and access to services of the EGI external catalogue (https://www.egi.eu/services/). This will comprise handling of service requests including pay-for-use services, authentication and authorization processes of long-tail of science, users brokering, compute/storage resource matchmaking and SLA/OLA negotiations.
'''Consortium''': CYFRONET, VUB
'''Contacts''':
Tomasz Szepieniec (t.szepieniec<AT>cyfronet.pl), Stéphane Gérard (stephane.gerard<AT>vub.ac.be)
== Message brokers ==
== Message brokers ==
'''Overview''':
The message broker network is a fundamental part of the operations infrastructure ensuring message exchange for monitoring, the operations dashboard and accounting. As such it is a critical core infrastructure platform service component whose continuity and high availability configuration must be ensured.
'''Consortium''': GRNET, SRCE
'''Contacts''': Christos Kanellopoulos (skanct<AT>grnet.gr), Kostas Koumantaros (kkoum<AT>grnet.gr) <br>
== Monitoring ==
== Monitoring ==
'''Overview''':
Monitoring services archive and provide access to the infrastructure monitoring results of the services. These data are accessible at many levels (Resource Centres, Operations Centres and EGI.EU), and it is used for the generation of service level reports, and for the central monitoring of EGI.eu operational tools and other central monitoring needs. Infrastructure operations require in some cases monitoring activities created ad-hoc to support specific operational activities, for example UserDN publishing in accounting records and of software versions of deployed middleware.
'''Consortium''': GRNET, SRCE, CNRS
'''Contact''': Kostas Koumantaros (kkoum<AT>grnet.gr)
== Online CA ==
== Online CA ==
'''Overview''':
The Online CA will generate X.509 certificates upon user request making available through the delegation service long-lived X.509 proxies.
The Online CA is a critical component to enable access to the EGI infrastructure to a wider range of users: the operation of this activity will be executed in tight collaboration
with the CheckIn Activity.
The RCauth CA is already accredited as an IOTA CA in IGTF and the delegation portal is
and will remain R&S and SIRTFI compliant.
'''Consortium''': GRNET, NIKHEF, RCUK
'''Contacts''': Kostas Koumantaros (kkoum<AT>grnet.gr), David Groep (davidg<AT>nikhef.nl), Jens Jensen (jens.jensen<AT>stfc.ac.uk)
== Operations Portal ==
== Operations Portal ==
'''Overview''':
EGI.eu provides a central portal for the operations community that offers a bundle of different capabilities, such as the broadcast tool, VO management facilities, a security dashboard and an operations dashboard that is used to display information about failing monitoring probes and to open tickets to the Resource Centres affected. The dashboard also supports the central infrastructure oversight activities. It is fully interfaced with the EGI Helpdesk and the monitoring system through messaging. It is a critical component as it is used by all EGI Operations Centres to provide support to the respective Resource Centres. Moreover, the Operations Portal provides tools supporting security operations, VO management, broadcast, availability reporting. The Operations Portal includes also a new tool, VAPOR, which displays information gathered by the BDII, among which the amount of resources of the EGI RCs.
'''Consortium''': CNRS
'''Contacts''': Cyril L’Orphelin (cyril.lorphelin@cc.in2p3.fr)
== Security coordination and security tools ==
== Security coordination and security tools ==
'''Overview''': Security is recognised as an important aspect of e-Infrastructures and requires coordination between the EGI participants at various levels, in particular for the prevention and handling of incidents.
To keep a distributed infrastructure secure there is need for a coordination activity of the security effort at NGI and resource center level, and for tools that automatically test the EGI sites for vulnerabilities.
The activity ensures the central coordination of security activities, including incident response, vulnerabilities handling and security policies development.
'''Consortium''': STFC (UK), FOM-Nikhef (NL), CERN, CESNET (CZ), GRNET (GR)
'''Contact''': David Kelsey (david.kelsey<AT>stfc.ac.uk), Kostas Koumantaros (kkoum<AT>grnet.gr)
== Service registry (GOCDB) ==
== Service registry (GOCDB) ==
'''Overview''':
EGI relies on a central registry (GOCDB) to record information about different entities such as the Operations Centres, the Resource Centres, service endpoints and the contact information and roles of people responsible for operations at different levels. GOCDB is a source of information for many other operational tools, such as the broadcast tool, the Aggregated Topology Provider, the Accounting Portal, etc.
'''Consortium''': STFC
'''Contacts''': Ian Collier (ian.collier@stfc.ac.uk), George Ryall (George.ryall@stfc.ac.uk)
== Services for AAI (CheckIn) ==
== Services for AAI (CheckIn) ==
'''Overview''':
The CheckIn service is the AAI Platform for the EGI infrastructure. The CheckIn service provides the following capabilities:
*Integration of IdPs (from eduGAIN and individual institutions) with the EGI services through an IdP/SP proxy
*Credential translation service:
**SAML2 &lt;--&gt; SAML2
**SAML2 &lt;--&gt; OIDC
**SAML2/OIDC --&gt; X.509 through the connection with the RC Auth online-CA
*Attribute harmonization and policy enforcing
The activity supports as well some legacy services for the authorization and authentication of users in EGI:
* Classic catch-all CA
* Catch-all and for dteam VO VOMS
PERUN is a group management system, developed, maintained and operated by CESNET, that is in use by some
scientific communities.
'''Consortium''': CESNET, GRNET
'''Contacts''': Michal Prochazka - michal.prochazka<AT>cesnet.cz, Nicholas Liampotis - nliam<AT>grnet.gr)
== UMD and CMD quality assurance ==
== UMD and CMD quality assurance ==
'''Overview''':
The quality criteria are the functional and nonfunctional requirements that a product must fulfill to be released in UMD and CMD; these include generic requirements applicable to every product, and specific requirements applicable to the capabilities supported by a component.
The Staged Rollout is a procedure by which certified updates of the supported middleware are first tested by Early Adopter (EA) sites before being made available to all sites through the production repositories. This procedure permits to test an update in a production environment that exposes the product to more heterogeneous use cases than the certification and verification phase. This allows the discovery of potential issues and the addition of corresponding mitigation information to the UMD and CMD release notes.
The Quality Criteria and the Staged Rollout are applied to UMD and CMD and all the distributions that are managed within the UMD and CMD software provisioning infrastructure, with particular reference to the CMD (Cloud Middleware Distribution) releases.
'''Consortium''': IBERGRID (LIP, IFCA, CESGA, CSIC, UPV)
'''Contacts''': Jorge Gomes (jorge@lip.pt), Mário David (david@lip.pt), Joao Pina (jpina@lip.pt), Isabel Campos (isabel@campos-it.es ), Pablo Orviz (orviz@cern.ch), Carlos Fernandez (carlosf@cesga.es), Ignacio Blanquer (iblanque@dsic.upv.es)
== UMD and CMD software provisioning infrastructure ==
== UMD and CMD software provisioning infrastructure ==
'''Overview''':
The software provisioning infrastructure provides the technical tools to support the UMD release process, used for UMD (Unified Middleware Distribution) and CMD (Cloud Middleware Distribution) from pulling packages from the developers repositories to the build of a release. CMD follows exactly the same process as for UMD; the only difference is in the content (CMD contains cloud-oriented software) and in the release cycle (months for CMD against years for UMD).
'''Consortium''': IASA, - maybe also KIT?
'''Contacts''': Marios Chatziangelou (mhaggel@iasa.gr), Paraskevas Sphicas (sphicas@iasa.gr)

Revision as of 15:05, 3 April 2019

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security



EGI Core services menu: Services PHASE I Services PHASE II Services PHASE III Bids Payments Travel procedure Performance


This page will list the EGI.eu core activities provided by EGI.eu partners for the period: January 2018 - December 2020

Proposals have been evaluated and the results have been used as inputs for the EOSC-hub project (funded).

Bidding phase III is now closed

List of the bidding activities

Please, find the information about the biddings in the Bidding page


Documentation:

  • EGI.eu Confirmation letters ...
  • Operation Level Agreements

Each activity is obligated to deliver every 6 months report.

Contact:

  • internal communication across activity leaders: core-egi-activities [at] mailman.egi.eu
  • to contact activity separately please use GGUS Support Unit
  • in case of any general questions: operations [at] egi.eu

Service/Activities

Accounting repositories and portal

Overview: The Accounting repositories store computing (serial and parallel jobs), storage, and cloud resources accounting data collected from Resource Centres of the EGI Federation. Accounting information is gathered from distributed sensors into a central accounting repository where it is processed to generate summaries that are available through the EGI Accounting Portal. The Accounting Repository, based on the APEL software, has a MySQL database backend and needs to ensure the exchange of accounting information with peer e-Infrastructures.

The Accounting Portal receives and stores the site, user, and VO level summaries generated by the Accounting Repository and provides views via a web portal, for example, by aggregating sites in a country on custom time intervals. The databases are organized into a CPU record database, a User record database, and a topology database.

Consortium: CESGA, STFC

Contacts: Adrian Coveney - adrian.coveney<AT>stfc.ac.uk, Iván Díaz Álvarez - idiaz<AT>cesga.es

Application DB (virtual appliances and applications library)

Overview: The EGI Applications Database (AppDB) is a central service that provides:

  • Information about software solutions in the form of native software products and virtual appliances, linking the programmers and the scientists who are involved, and the publications derived from the registered solutions
  • The tools for the distribution of the virtual machine images in the cloud sites part of the the federated cloud
  • A dashboard to operate virtual machines in the fedcloud sites

Three types of software solutions are offered through the EGI Applications Database:

  • Software items, in its classical sense, i.e. applications, tools, utilities, etc..,
  • Virtual Appliances: composed by one or more pre-configured virtual machine images packaged with an operating system and software application(s)
  • Software Appliances: one or more a set pairs of a virtual appliance and a contextualization script. A Contextualization Script (CS) is the script launched on VM boot time and could be used for installing, configuring and preparing software upon boot time on a pre-defined virtual machine image

Consortium: IASA

Contacts: Marios Chatziangelou (mhaggel<AT>iasa.gr), Paraskevas Sphicas (sphicas<AT>iasa.gr)

Collaboration tools

Overview: Collaborations tools are services needed by the EGI back-office and supporting EGI collaboration.

Consortium: CESNET

Contacts: Martin Kuba (makub<AT>ics.mun i.cz) main and administrative contact, Michal Stava (michal.stava<AT>cesnet.cz

DIRAC4EGI

Overview: DIRAC service is provided to the EGI community as a workload management service used to distribute the users' computing tasks among the available resources both HTC and cloud. The service will support established Virtual Organization as well as the long tail of users.

Given the critical nature of the activity bid must contain an availability and continuity plan for the service.

Consortium: CNRS, CYFRONET

Contacts: Andrei Tsaregorodtsev (atsareg@in2p3.fr), Tomasz Szepieniec (t.szepieniec@cyfronet.pl)

Helpdesk (GGUS)

Overview: EGI provides support to users and operators through a distributed helpdesk with central coordination (GGUS). The central helpdesk provides a single interface for support. The central system is interfaced to a variety of other ticketing systems at the NGI level in order to allow a bi-directional exchange of tickets.

Consortium: KIT

Contacts: Andreas Heiss (andreas.heiss<AT>kit.edu), Günter Grein (guenter.grein<AT>kit.edu)

Helpdesk human support

Overview: EGI provides support to users and operators through a distributed helpdesk with central coordination (GGUS). The central helpdesk provides a single interface for support. The helpdesk support activities are grouped in 1st and 2nd level support.

Consortium: CESNET

Contacts: Zdeněk Šustr (sustr4@cesnet.cz), Petr Hanousek (petr.hanousek@cesnet.cz)

Marketplace and resource allocation

Overview: This activity will run the EGI systems and underpinning human processes necessary to enable discovery, selection and access to services of the EGI external catalogue (https://www.egi.eu/services/). This will comprise handling of service requests including pay-for-use services, authentication and authorization processes of long-tail of science, users brokering, compute/storage resource matchmaking and SLA/OLA negotiations.

Consortium: CYFRONET, VUB

Contacts: Tomasz Szepieniec (t.szepieniec<AT>cyfronet.pl), Stéphane Gérard (stephane.gerard<AT>vub.ac.be)

Message brokers

Overview:

The message broker network is a fundamental part of the operations infrastructure ensuring message exchange for monitoring, the operations dashboard and accounting. As such it is a critical core infrastructure platform service component whose continuity and high availability configuration must be ensured.

Consortium: GRNET, SRCE

Contacts: Christos Kanellopoulos (skanct<AT>grnet.gr), Kostas Koumantaros (kkoum<AT>grnet.gr)

Monitoring

Overview: Monitoring services archive and provide access to the infrastructure monitoring results of the services. These data are accessible at many levels (Resource Centres, Operations Centres and EGI.EU), and it is used for the generation of service level reports, and for the central monitoring of EGI.eu operational tools and other central monitoring needs. Infrastructure operations require in some cases monitoring activities created ad-hoc to support specific operational activities, for example UserDN publishing in accounting records and of software versions of deployed middleware.

Consortium: GRNET, SRCE, CNRS

Contact: Kostas Koumantaros (kkoum<AT>grnet.gr)

Online CA

Overview: The Online CA will generate X.509 certificates upon user request making available through the delegation service long-lived X.509 proxies.

The Online CA is a critical component to enable access to the EGI infrastructure to a wider range of users: the operation of this activity will be executed in tight collaboration with the CheckIn Activity.

The RCauth CA is already accredited as an IOTA CA in IGTF and the delegation portal is and will remain R&S and SIRTFI compliant.

Consortium: GRNET, NIKHEF, RCUK

Contacts: Kostas Koumantaros (kkoum<AT>grnet.gr), David Groep (davidg<AT>nikhef.nl), Jens Jensen (jens.jensen<AT>stfc.ac.uk)

Operations Portal

Overview: EGI.eu provides a central portal for the operations community that offers a bundle of different capabilities, such as the broadcast tool, VO management facilities, a security dashboard and an operations dashboard that is used to display information about failing monitoring probes and to open tickets to the Resource Centres affected. The dashboard also supports the central infrastructure oversight activities. It is fully interfaced with the EGI Helpdesk and the monitoring system through messaging. It is a critical component as it is used by all EGI Operations Centres to provide support to the respective Resource Centres. Moreover, the Operations Portal provides tools supporting security operations, VO management, broadcast, availability reporting. The Operations Portal includes also a new tool, VAPOR, which displays information gathered by the BDII, among which the amount of resources of the EGI RCs.

Consortium: CNRS

Contacts: Cyril L’Orphelin (cyril.lorphelin@cc.in2p3.fr)

Security coordination and security tools

Overview: Security is recognised as an important aspect of e-Infrastructures and requires coordination between the EGI participants at various levels, in particular for the prevention and handling of incidents. To keep a distributed infrastructure secure there is need for a coordination activity of the security effort at NGI and resource center level, and for tools that automatically test the EGI sites for vulnerabilities. The activity ensures the central coordination of security activities, including incident response, vulnerabilities handling and security policies development.

Consortium: STFC (UK), FOM-Nikhef (NL), CERN, CESNET (CZ), GRNET (GR)

Contact: David Kelsey (david.kelsey<AT>stfc.ac.uk), Kostas Koumantaros (kkoum<AT>grnet.gr)

Service registry (GOCDB)

Overview: EGI relies on a central registry (GOCDB) to record information about different entities such as the Operations Centres, the Resource Centres, service endpoints and the contact information and roles of people responsible for operations at different levels. GOCDB is a source of information for many other operational tools, such as the broadcast tool, the Aggregated Topology Provider, the Accounting Portal, etc.

Consortium: STFC

Contacts: Ian Collier (ian.collier@stfc.ac.uk), George Ryall (George.ryall@stfc.ac.uk)

Services for AAI (CheckIn)

Overview: The CheckIn service is the AAI Platform for the EGI infrastructure. The CheckIn service provides the following capabilities:

  • Integration of IdPs (from eduGAIN and individual institutions) with the EGI services through an IdP/SP proxy
  • Credential translation service:
    • SAML2 <--> SAML2
    • SAML2 <--> OIDC
    • SAML2/OIDC --> X.509 through the connection with the RC Auth online-CA
  • Attribute harmonization and policy enforcing

The activity supports as well some legacy services for the authorization and authentication of users in EGI:

  • Classic catch-all CA
  • Catch-all and for dteam VO VOMS

PERUN is a group management system, developed, maintained and operated by CESNET, that is in use by some scientific communities.

Consortium: CESNET, GRNET

Contacts: Michal Prochazka - michal.prochazka<AT>cesnet.cz, Nicholas Liampotis - nliam<AT>grnet.gr)

UMD and CMD quality assurance

Overview: The quality criteria are the functional and nonfunctional requirements that a product must fulfill to be released in UMD and CMD; these include generic requirements applicable to every product, and specific requirements applicable to the capabilities supported by a component.

The Staged Rollout is a procedure by which certified updates of the supported middleware are first tested by Early Adopter (EA) sites before being made available to all sites through the production repositories. This procedure permits to test an update in a production environment that exposes the product to more heterogeneous use cases than the certification and verification phase. This allows the discovery of potential issues and the addition of corresponding mitigation information to the UMD and CMD release notes.

The Quality Criteria and the Staged Rollout are applied to UMD and CMD and all the distributions that are managed within the UMD and CMD software provisioning infrastructure, with particular reference to the CMD (Cloud Middleware Distribution) releases.

Consortium: IBERGRID (LIP, IFCA, CESGA, CSIC, UPV)

Contacts: Jorge Gomes (jorge@lip.pt), Mário David (david@lip.pt), Joao Pina (jpina@lip.pt), Isabel Campos (isabel@campos-it.es ), Pablo Orviz (orviz@cern.ch), Carlos Fernandez (carlosf@cesga.es), Ignacio Blanquer (iblanque@dsic.upv.es)

UMD and CMD software provisioning infrastructure

Overview: The software provisioning infrastructure provides the technical tools to support the UMD release process, used for UMD (Unified Middleware Distribution) and CMD (Cloud Middleware Distribution) from pulling packages from the developers repositories to the build of a release. CMD follows exactly the same process as for UMD; the only difference is in the content (CMD contains cloud-oriented software) and in the release cycle (months for CMD against years for UMD).

Consortium: IASA, - maybe also KIT?

Contacts: Marios Chatziangelou (mhaggel@iasa.gr), Paraskevas Sphicas (sphicas@iasa.gr)