Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "2019-bidding/datamanagement"

From EGIWiki
Jump to navigation Jump to search
Line 17: Line 17:
== Technical description ==
== Technical description ==


Platforms must comply with the following requirements:
This bid  welcomes any Data Management Service compatible with the EGI Computing and Storage services, able to authenticate users with Check-in and covering totally or partially the following features:
* The platform integrates with Check-in for authenticating users, and delegates those identities as needed for interacting with other EGI services on behalf of the users.
* Digital repository for annotating, publishing, sharing, cataloguing, discovering, and using data.
* The platform can exploit computing and storage resources from EGI services. Ideally the platform should dynamically allocate those resources by using the services APIs.
* Asynchronous and reliable file transfer across distributed infrastructures.
* Provide probes for monitoring the Avalaibility and Reliablity of the platform with the EGI Monitoring service.
* On-demand and automatic replication and caching of data.
* Exposing data to computing services with POSIX semantics.


The platform should be offered as a centrally managed instance that will be run by the project and will provide access to ‘long-tail of science’ type use cases on ‘generic clouds’ for any user, and to thematic groups on the ‘thematic clouds’. Additionally, the platform provider should be capable of setting up dedicated instances to specific communities as needed upon request.
The Data Management service should be offered as a centrally managed instance that will be run by the project and will provide access to ‘long-tail of science’ type use cases on ‘generic clouds’ for any user, and to thematic groups on the ‘thematic clouds’. Additionally, the platform provider should be capable of setting up dedicated instances to specific communities as needed upon request. These instances must include support for monitoring via the EGI Monitoring service and provide accounting records as specified by the EGI Accounting service.


Optionally, platforms should also be available as ready-to-use artifacts (VM images, containers, helm charts) in the EGI AppDB, docker or helm repositories alongside with complete documentation for the deployment and operation of the platform so dedicated local instances can be setup on cloud providers as needed.
Example of technologies that power services that may apply to this bid are: Invenio, CKAN, FTS, Onedata, Rucio.
 
Below follows a list of platforms that are expected to be covered by applications to this bid (applications may cover one or several of the platforms below):
*Workload manager (DIRAC): A job manager system compatible with EGI HTC and Cloud resources.
* EC3: A VM orchestrator system compatible with EGI Cloud resources
* Notebooks (JupyterHub): A web-based interface to define and run computational notebooks.
* Binder: A web-based environment to reproduce and/or recreate computational environments as notebooks.
* FutureGateway Science Gateway (FGSG): a web-based gateway that provides general purpose scientific applications on demand.
* DODAS: Can turn IaaS cloud sites into ‘batch job execution’ facilities.
* PROMINENCE: Can turn IaaS cloud sites into ‘batch job execution’ facilities and can support clouds to burst out jobs to other clouds.


== Coordination ==
== Coordination ==

Revision as of 15:58, 15 November 2019

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security



EGI Services and Service Management Support menu: Bids Old Bids Performance

Go back to the EGI Services Bidding page.

Data Management Services

Introduction

EGI Data Management Services support the management of large volumes of data spread across the infrastructure following FAIR principles, including:

  • Discovery and publication of data sets to specific communities or worldwide.
  • Transfer of large sets of data.
  • Metadata management
  • Manual and policy-based replication for resiliency and availability purposes.
  • Efficient exploitation of data by bringing it close to computing.
  • On-demand cache of data to avoid full replication.

Technical description

This bid welcomes any Data Management Service compatible with the EGI Computing and Storage services, able to authenticate users with Check-in and covering totally or partially the following features:

  • Digital repository for annotating, publishing, sharing, cataloguing, discovering, and using data.
  • Asynchronous and reliable file transfer across distributed infrastructures.
  • On-demand and automatic replication and caching of data.
  • Exposing data to computing services with POSIX semantics.

The Data Management service should be offered as a centrally managed instance that will be run by the project and will provide access to ‘long-tail of science’ type use cases on ‘generic clouds’ for any user, and to thematic groups on the ‘thematic clouds’. Additionally, the platform provider should be capable of setting up dedicated instances to specific communities as needed upon request. These instances must include support for monitoring via the EGI Monitoring service and provide accounting records as specified by the EGI Accounting service.

Example of technologies that power services that may apply to this bid are: Invenio, CKAN, FTS, Onedata, Rucio.

Coordination

Operations

Maintenance

Software Compliance

  • Unless explicitly agreed, software being used and developed to provide the service should:
    • Be licensed under an open source and permissive license (like MIT, BSD, Apache 2.0,...).
      • The license should provide unlimited access rights to the EGI Foundation and EGI federation member organisations.
    • Have source code publicly available via a public source code repository (if needed a mirror can be put in place under the EGI organisation in GitHub.) All releases should be appropriately tagged.
    • Adopt best practices:
      • Defining and enforcing code style guidelines.
      • Using Semantic Versioning.
      • Using a Configuration Management frameworks such as Ansible.
      • Taking security aspects into consideration through at every point in time.
      • Having automated testing in place.
      • Using code reviewing.
      • Treating documentation as code.
        • Documentation should be available for Developers, administrators and end users.

IT Service Management compliance

  • Key staff who deliver services should have foundation or basic level ITSM training and certification.
    • ITSM training and certification could include FitSM, ITIL, ISO 20000 etc.
  • Key staff and service owners should have advanced/professional training and certification covering the key processes for their services.
  • Providers should have clear interfaces with the EGI SMS processes and provide the required information.
  • Providers should commit to improving their management system used to support the services they provide.

Support

  • Support is provided through the EGI Helpdesk.
  • Support hours: eight hours a day (for example 9-17 CE(S)T), Monday to Friday – excluding public holidays of the hosting organisation.

Service level targets

Platforms must have 95% A/R on a monthly basis.

Support should be provided according to the medium Quality of Support level described in the GGUS QoS levels.

Expressions of interest

?

Selection criteria

?