Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

EGI-Engage:WP3 (JRA1) E-Infrastructure Commons

From EGIWiki
Jump to navigation Jump to search
EGI-Engage project: Main page WP1(NA1) WP3(JRA1) WP5(SA1) PMB Deliverables and Milestones Quality Plan Risk Plan Data Plan
Roles and
responsibilities
WP2(NA2) WP4(JRA2) WP6(SA2) AMB Software and services Metrics Project Office Procedures



WP leader: Diego Scardaci/EGI.eu at INFN

WP contact:egi-engage-wp3@mailman.egi.eu


Objective

This workpackage will coordinate the development of the e-Infrastructure Commons - an ecosystem of services that constitute the foundation layer of any distributed e-Infrastructures. The technical development of the e-Infrastructure Commons services will be user-driven to ensure full interoperability with other e-Infrastructures and RIs. The current set of services will be expanded to include a Service Registry and Marketplace. The main targets of this task are:

  • Provide viable methods for authentication and authorisation in the EGI ecosystem.
  • Simplify the access to the infrastructure services through technological innovation and new services in the area of Service Registry and Marketplace and resource allocation.
  • Evolve the EGI accounting system to manage the data deluge expected over the next years, including new types of accounting metric (e.g. data accounting) and redesigning of the presentation layer to improve the user experience.
  • Adapt the operations tools to new technologies and to satisfy new requirements emerging from service providers and user communities.
  • Define interfaces to create a network of analogue tools that provides users with integrated view of all the infrastructures involved.

Task Leaders

To contact all task leaders (see below), send mail to 

Task Name Task Leader Deputy
JRA1.1
Authentication and Authorisation Infrastructure
Christos Kanellopoulos, GRNET

JRA1.2 Service Registry and Marketplace
Dean Flanders, SWING

JRA1.3 Accounting


JRA1.4 Operations Tools
Cyril Lorphelin, CNRS

JRA1.5 Resource Allocation – e-GRANT


Involved partners

  • EGI.eu
  • SWING
  • CESNET
  • CSIC
  • CNRS
  • GRNET
  • SRCE
  • INFN
  • SURF sara
  • CYFRONET
  • STFC

Tasks

TASK JRA1.1 Authentication and Authorisation Infrastructure

(Lead: GRNET, M3 – M27)

Estimated task effort: 24.5PM
Provide viable methods for authentication and authorisation (AA) in the EGI ecosystem, addressing current shortcomings. The task will explore how to integrate suggested AA methods with current middleware and community services, guaranteeing a sufficient Level of Assurance, and the use of credentials issued by other infrastructures and services:

  • Explore approaches to easier safe management of user credentials, including dedicated services such as, for example, trusted stores.
  • Identify possibilities and requirements for user authentication against both web and non web-based applications. Develop a solution for authorisation, which will be based on information provided by VOs and/or additional sources. Validate the solutions via their integration with the middleware and services used by the community.
  • Develop authentication solutions for use cases, identified among the requirements generated by the communities supported by the SA2 tasks, which will then provide guidelines for other typical EGI scenarios. The solutions can combine usage of multiple credentials for a user.
  • Identify user registration and management requirements from a VO perspective, by analysing the needs of users, VO managers, infrastructure operations, and resource providers. Design a solution for user registration and management, which will cover current shortcomings.
  • Explore current technical possibilities and the usability of existing infrastructures covering identity management, such as eduGain, eduRoam, or STORK and their large-scale combined usage.
  • Investigate alternative identity-vetting approaches to current practices, such as the IGTF IOTA profile or other means, such as reputation-based methods. Evaluate the necessity of changing existing policies covering user identity management. Identify security risks and aspects affecting the secure operations of the developed solutions. Ensure that proper security mechanisms to ban users with immediate effect are available.
  • Develop solution prototypes and provide them for integration with the overall AAI solution developed by the CC.
  • Liaise with other projects focusing on AAI to share know-how and best practices.

TASK JRA1.2 Service Registry and Marketplace

(Lead: SWING, M1 – M30)

Estimated task effort: 22.5PM
The main target of this task is the design and the development of a proof-of-concept of a Service Registry and a Marketplace for the EGI infrastructure. The Service Registry will record the list of all the services offered by EGI and will be available for consultation by the EGI customers. The Marketplace will allow customers to acquire services through a payment or for free, through SLAs with service providers. An important role will be covered also by EGI.eu, as broker/intermediary to define SLA and policies and to identify the best services for the customers and the best users for the service providers. This design will be used to drive the development of the Service Registry and Marketplace proof-of-concept prototype. The implementation will be based on already existing open-source tools, identified during the analysis phase, and extensions of EGI tools (AppDB, GOCDB and e-GRANT).

TASK JRA1.3 Accounting

(Lead: STFC, M1 – M30)

Estimated task effort: 33PM
The target of this task is the evolution of the EGI accounting system. Its main components are the accounting repository and the Accounting Portal. The development guidelines will be:

  • Evolve the accounting system to be able manage big data: the accounting team will investigate techniques to manage huge amount of data as, for example, Hadoop or Cassandra;
  • Include new types of data accounting to record who accesses data, how often, how much is transferred and where to;
  • Extend the current accounting measurement:
    • Cloud accounting: the current system will be extended adding features to normalise the CPU usage on different kind of cloud resources and to account the usage of the cloud storages supported in the EGI Federated Cloud;
    • Storage accounting: the number of the supported storage systems will be extended;
    • GPU accounting: extended the number of batch systems supported;
  • Improve the portal designing new and easier way to access and visualise data for the final users;
  • Develop unified views in the portal for different kind of resources (e.g. CPU usage for grid and cloud resources);
  • Create new views to show the new types of data available in the accounting repository (e.g. data accounting);
  • Expose a complete API allowing third parties to gather accounting data from the system.

This task will collaborate with the OGF Usage Record Working Group, in particular to agree a schema for a data usage record. Moreover, the support for the OGF standard UR2 will be improved.

TASK JRA1.4 Operations Tools

(Lead: CNRS, M1 – M30)

Estimated task effort: 56PM
The operations tools have to be continued improved to adapt them to technology evolution and to satisfy new requirements emerging from service providers and user communities. The aim of this task is to coordinate tool development to:

  • Implement a modular architecture to manage AAI allowing to create easy-to-develop plugins for each AAI supported by EGI in the near future;
  • Display publicly all the non-confidential information;
  • Make them able to serve any research infrastructure;
  • Evolve and improve their support for cloud resources;
  • Define interfaces to exchange information with analogue tools belonging to other e-Infrastructures or RIs.
  • Expose internal data through a REST API interface.

The work plan, described below, includes a set of subtasks, focused on specific tool components.
Operations Portal
The Operations Portal provides information to various actors along with related facilities, such as the VO administration tool and the VO management module, the different dashboards (Operations dashboard, security dashboard, VO Operations dashboard) and different communication tools: broadcast tool and downtime system announcement. The activity will:

  • Integrate the VO Administration and operations PORtal (VAPOR) into the Operations Portal: VAPOR has been developed to address VO operation tasks and will allow the Operations Portal to become a unique, one-stop tool for resource monitoring, issues management and user community management, either from the resource centre perspective or from the VO perspective.
  • Monitor infrastructure resources: the Operations Portal and VAPOR provide complementary tools and views dedicated to either resource centres or VOs. These tools will be evolved to be easily extended and support any type of resources (e.g. cloud). The resource distribution browser and the dashboard will be updated to better serve the cloud resources, giving more details (e.g. OS, number of cores, capacity) and allowing the monitoring. Furthermore, the main GSTAT features will be added to this module and APIs will be provided for the resource distribution browser and more generally for the VO information. VAPOR monitoring features will be integrated as part of the existing VO Operations dashboard.

GOCDB
GOCDB is an information system for recording and managing e-Infrastructure topology data. The GOCDB evolution foresees work to:

  • Extend the authentication system to simplify user-access, integrating different AAI systems, especially for Federated Identity Management (FIM). This would also include an open-access mode allowing un-authenticated users to browse the public-facing services/resources in read-only mode.
  • Add a writeable REST API to provide a scriptable interface to input/edit data.
  • Investigate if the GOCDB business rules could be abstracted using a Business Rules Management Engines (BRMS), to more effectively address the requirements of different infrastructures hosted in the same GOCDB instance, and assess feasibility and benefit.
  • Extend the data model to more effectively support cloud resources supporting new object types and a greater range of attributes (e.g. taken from the GLUE2 standard and the currently evolving GLUE2.1 cloud extensions).
  • Introduce a more capable MVC framework to improve the UI and user experience.

Monitoring
The ARGO platform is the continuation and evolution of the SAM framework. The development of ARGO in EGI-Engage will focus in three main branches: modularise the Core Monitoring Engine, monitoring availability & reliability as a Cloud Service and provide clean interfaces for end-users, operations and developers. The first branch, the Core Monitoring Engine, includes the following activities:

  • Improve the ARGO SW architecture to make easier and faster updates and developments;
  • Extending monitoring capabilities developing new probes for monitoring cloud resources;
  • Improvements in test/probe management enabling easier deployment/upgrades of new probes;
  • Support multiple authentication methods allowing to associate different authentication mechanism to each test;
  • Improvements in communication protocols between different ARGO components to be ready to manage bigger workloads.

Messaging
The Messaging Service is a critical core service provided through a Network of Message Brokers resilient to isolated failures. The Messaging Service will be evolved providing a Restful HTTP API as a layer on top of the existing Message Broker Network. The change will be backwards compatible, as the operation of the STOMP interfaces for direct usage of the Message Broker Network will be continued. Developers, which will move to the new Restful Service layer, simplify the maintenance of their client implementations. The introduction of a Restful Service layer on top of the Message Broker Network will give EGI the ability to implement also other requested features such as access controls or message accounting.
Security Monitoring
Security monitoring tools developed in the scope of EGI focused mainly on traditional grid sites and operations, which is not enough nowadays. Current technologies, namely federated clouds, bring new security challenges that must be addressed by new approaches. In this task we will identify the new areas and provide solutions for proper monitoring of them. The solutions delivered by the task will be modular and individual components could be utilised independently and on several levels in the infrastructure.

TASK JRA1.5 Resource Allocation – e-GRANT

(Lead: CYFRONET, M1 – M30)

Estimated task effort: 17PM
e-GRANT is a platform that enables EGI customers to apply and get allocation of compute and storage resources. In this task, e-GRANT functionalities will be extended to cover the complete SLA lifecycle-enabling activities such as tracing of resource centre configuration, monitoring of resources delivery, availability analysis for agreed services, capacity reports analysis and billing. Another area of e-GRANT development includes facilitating activities taking place before customer request resources. The platform will enable customers to investigate the services and resources available to them by publishing this information on the EGI Service Registry and Marketplace, including special offers or promotion actions, such as seed-resources for new communities, demonstrators or hackathons. Furthermore, e-GRANT will be extended to play an important role in the new mode of service delivery like pay-for-use, which would require extension towards negotiating the price. e-Grant will support the integration of EGI with other e-Infrastructures enabling customers to jointly request resources owning to different infrastructures.

Deliverables

The following gives an overview of deliverables scheduled

Code Title Delivery PM Delivery CM Delivered date Status
D3.1
Technical design of the new Accounting Portal and implementation plan (R)
M06



D3.2 Design of the EGI Service Registry and Marketplace (R)
M12



D3.3 Accounting Repository release (OTHER)
M12



D3.4 First release of the Operational tools (OTHER)
M12



D3.6 (3.5)
Analysis on techniques to manage big data on the EGI accounting system (R) M15
D3.5 (3.6) First release of the new Accounting Portal deployed in production (OTHER)
M14



D3.7 First release of the EGI Service Registry and Marketplace prototype (DEM)
M18



D3.8 First data accounting prototype (DEM)
M20



D3.9 Identity Management for Distributed User Communities report (R)
M24



D3.10 Second release of the new Accounting Portal deployed in production (OTHER)
M24



D3.11 Second release of the Operational tools (OTHER)
M24



D3.12 Second release of the Accounting Repository (OTHER)
M24



D3.14 (3.13) Report on Data Accounting (R) M24


D3.13 (3.14) Second release of the EGI Service Registry and Marketplace prototype (DEM)
M26



D3.15 Second data accounting prototype (DEM)
M28



D3.16 Final report on EGI Service Registry and Marketplace (R)
M30



D3.17 Final release of the Accounting Repository (OTHER)
M30



D3.18 Final release of the Operational tools (OTHER)
M30



D3.19 Final release of the new Accounting Portal deployed in production (OTHER)
M30



Milestones

The following gives an overview of milestones scheduled

Milestone Title Lead-Task Delivery PM Delivery CM Delivered Status
M3.1 Operational tools development roadmap agreed   
M04


M3.2 User requirements on data accounting collected
M12


M3.3 Operational tools development roadmap revised
M16


M3.4 Pilot services and best practices to enable federated AAI solutions released
M16


M3.5 The first version of the EGI Marketplace is demonstrated
M18


M3.6 The second version of the EGI Marketplace is demonstrated
M26


Internal Documents