EGI-Engage:WP3 (JRA1) E-Infrastructure Commons
|EGI-Engage project:||Main page||WP1(NA1)||WP3(JRA1)||WP5(SA1)||PMB||Deliverables and Milestones||Quality Plan||Risk Plan||Data Plan|
| Roles and
|WP2(NA2)||WP4(JRA2)||WP6(SA2)||AMB||Software and services||Metrics||Project Office||Procedures|
WP leader: Diego Scardaci/EGI.eu at INFN
WP contact (all members):egi-engage-wp3 AT mailman.egi.eu
WP taskleaders: egi-engage-wp3-taskleaders AT mailman.egi.eu
This workpackage will coordinate the development of the e-Infrastructure Commons - an ecosystem of services that constitute the foundation layer of any distributed e-Infrastructures. The technical development of the e-Infrastructure Commons services will be user-driven to ensure full interoperability with other e-Infrastructures and RIs. The current set of services will be expanded to include a Service Registry and Marketplace. The main targets of this task are:
- Provide viable methods for authentication and authorisation in the EGI ecosystem.
- Simplify the access to the infrastructure services through technological innovation and new services in the area of Service Registry and Marketplace and resource allocation.
- Evolve the EGI accounting system to manage the data deluge expected over the next years, including new types of accounting metric (e.g. data accounting) and redesigning of the presentation layer to improve the user experience.
- Adapt the operations tools to new technologies and to satisfy new requirements emerging from service providers and user communities.
- Define interfaces to create a network of analogue tools that provides users with integrated view of all the infrastructures involved.
To contact all task leaders (see below), send mail to
|Task||Name||Task Leader||Deputy||Mailing list|
|| Authentication and Authorisation Infrastructure
|| Christos Kanellopoulos, GRNET
|| Peter Solagna, EGI.eu
|JRA1.2|| Service Registry and Marketplace
|| Dean Flanders, SWING
|| Stuart Pullinger, STFC
|| Adrian Coveney
|JRA1.4|| Operations Tools
|| Cyril Lorphelin, CNRS
||Christos Kanellopoulos, GRNET|| email@example.com|
|JRA1.5|| Resource Allocation – e-GRANT
|| Tomasz Szepieniec, CYFRONET
|| Roksana Różańska
- SURF sara
(Lead: GRNET, M3 – M27)
Estimated task effort: 24.5PM
Provide viable methods for authentication and authorisation (AA) in the EGI ecosystem, addressing current shortcomings. The task will explore how to integrate suggested AA methods with current middleware and community services, guaranteeing a sufficient Level of Assurance, and the use of credentials issued by other infrastructures and services:
- Explore approaches to easier safe management of user credentials, including dedicated services such as, for example, trusted stores.
- Identify possibilities and requirements for user authentication against both web and non web-based applications. Develop a solution for authorisation, which will be based on information provided by VOs and/or additional sources. Validate the solutions via their integration with the middleware and services used by the community.
- Develop authentication solutions for use cases, identified among the requirements generated by the communities supported by the SA2 tasks, which will then provide guidelines for other typical EGI scenarios. The solutions can combine usage of multiple credentials for a user.
- Identify user registration and management requirements from a VO perspective, by analysing the needs of users, VO managers, infrastructure operations, and resource providers. Design a solution for user registration and management, which will cover current shortcomings.
- Explore current technical possibilities and the usability of existing infrastructures covering identity management, such as eduGain, eduRoam, or STORK and their large-scale combined usage.
- Investigate alternative identity-vetting approaches to current practices, such as the IGTF IOTA profile or other means, such as reputation-based methods. Evaluate the necessity of changing existing policies covering user identity management. Identify security risks and aspects affecting the secure operations of the developed solutions. Ensure that proper security mechanisms to ban users with immediate effect are available.
- Develop solution prototypes and provide them for integration with the overall AAI solution developed by the CC.
- Liaise with other projects focusing on AAI to share know-how and best practices.
(Lead: SWING, M1 – M30)
Estimated task effort: 22.5PM (SWISS 7PM; GR NGI (IASA) 3PM; IT NGI (INFN) 6.5PM; CYFRONET 3PM; STFC 3PM)
The main target of this task is the design and the development of a proof-of-concept of a Service Registry and a Marketplace for the EGI infrastructure. The Service Registry will record the list of all the services offered by EGI and will be available for consultation by the EGI customers. The Marketplace will allow customers to acquire services through a payment or for free, through SLAs with service providers. An important role will be covered also by EGI.eu, as broker/intermediary to define SLA and policies and to identify the best services for the customers and the best users for the service providers. This design will be used to drive the development of the Service Registry and Marketplace proof-of-concept prototype. The implementation will be based on already existing open-source tools, identified during the analysis phase, and extensions of EGI tools (AppDB, GOCDB and e-GRANT).
(Lead: STFC, M1 – M30)
Estimated task effort: 33PM
The target of this task is the evolution of the EGI accounting system. Its main components are the accounting repository and the Accounting Portal. The development guidelines will be:
- Evolve the accounting system to be able manage big data: the accounting team will investigate techniques to manage huge amount of data as, for example, Hadoop or Cassandra;
- Include new types of data accounting to record who accesses data, how often, how much is transferred and where to;
- Extend the current accounting measurement:
- Cloud accounting: the current system will be extended adding features to normalise the CPU usage on different kind of cloud resources and to account the usage of the cloud storages supported in the EGI Federated Cloud;
- Storage accounting: the number of the supported storage systems will be extended;
- GPU accounting: extended the number of batch systems supported;
- Improve the portal designing new and easier way to access and visualise data for the final users;
- Develop unified views in the portal for different kind of resources (e.g. CPU usage for grid and cloud resources);
- Create new views to show the new types of data available in the accounting repository (e.g. data accounting);
- Expose a complete API allowing third parties to gather accounting data from the system.
This task will collaborate with the OGF Usage Record Working Group, in particular to agree a schema for a data usage record. Moreover, the support for the OGF standard UR2 will be improved.
(Lead: CNRS, M1 – M30)
Estimated task effort: 56PM
The operations tools have to be continued improved to adapt them to technology evolution and to satisfy new requirements emerging from service providers and user communities. The aim of this task is to coordinate tool development to:
- Implement a modular architecture to manage AAI allowing to create easy-to-develop plugins for each AAI supported by EGI in the near future;
- Display publicly all the non-confidential information;
- Make them able to serve any research infrastructure;
- Evolve and improve their support for cloud resources;
- Define interfaces to exchange information with analogue tools belonging to other e-Infrastructures or RIs.
- Expose internal data through a REST API interface.
The work plan, described below, includes a set of subtasks, focused on specific tool components.
The Operations Portal provides information to various actors along with related facilities, such as the VO administration tool and the VO management module, the different dashboards (Operations dashboard, security dashboard, VO Operations dashboard) and different communication tools: broadcast tool and downtime system announcement. The activity will:
- Integrate the VO Administration and operations PORtal (VAPOR) into the Operations Portal: VAPOR has been developed to address VO operation tasks and will allow the Operations Portal to become a unique, one-stop tool for resource monitoring, issues management and user community management, either from the resource centre perspective or from the VO perspective.
- Monitor infrastructure resources: the Operations Portal and VAPOR provide complementary tools and views dedicated to either resource centres or VOs. These tools will be evolved to be easily extended and support any type of resources (e.g. cloud). The resource distribution browser and the dashboard will be updated to better serve the cloud resources, giving more details (e.g. OS, number of cores, capacity) and allowing the monitoring. Furthermore, the main GSTAT features will be added to this module and APIs will be provided for the resource distribution browser and more generally for the VO information. VAPOR monitoring features will be integrated as part of the existing VO Operations dashboard.
GOCDB is an information system for recording and managing e-Infrastructure topology data. The GOCDB evolution foresees work to:
- AAI: Extend authentication to support different AAI systems, especially for Federated Identity Management (FIM). Depends on AAI taskforce but a likely need is to support multiple federations and account-linking (i.e. link different accounts to one Gocdb user-profile/account considering different levels of assurance). Include fine-grained content rendering based on authenticated roles such as permitAll and protected pages/regions.
- Rules/Roles: Further abstract the roles/business-rule logic so different projects hosted in same instance can apply different rule/roles/role-types per project. Investigate feasibility and benefit of using a Business Rules Management Engines (BRMS) to address these requirements.
- Auditing: Enhance audit logging to record more actions (who did what and when). Include role-action log to record role approval/denial/revoke and diffs when editing objects (pre/post change).
- Domain Model: Extend the data model to support changing infrastructure and service marketplace requirements. Likely need to support new object types and a greater range of attributes (e.g. taken from the GLUE2 standard and the currently evolving GLUE2.1 cloud extensions).
- MVC and GUI: Replace GOCDB’s proprietary MVC with a more capable framework and refactor GUI to improve the UI and user experience (lower priority).
- CRUD API: Add a writeable REST API to provide a scriptable interface to input/edit data (lower priority).
The ARGO platform is the continuation and evolution of the SAM framework. The development of ARGO in EGI-Engage will focus in three main branches: modularise the Core Monitoring Engine, monitoring availability & reliability as a Cloud Service and provide clean interfaces for end-users, operations and developers. The first branch, the Core Monitoring Engine, includes the following activities:
- Improve the ARGO SW architecture to make easier and faster updates and developments;
- Extending monitoring capabilities developing new probes for monitoring cloud resources;
- Improvements in test/probe management enabling easier deployment/upgrades of new probes;
- Support multiple authentication methods allowing to associate different authentication mechanism to each test;
- Improvements in communication protocols between different ARGO components to be ready to manage bigger workloads.
The Messaging Service is a critical core service provided through a Network of Message Brokers resilient to isolated failures. The Messaging Service will be evolved providing a Restful HTTP API as a layer on top of the existing Message Broker Network. The change will be backwards compatible, as the operation of the STOMP interfaces for direct usage of the Message Broker Network will be continued. Developers, which will move to the new Restful Service layer, simplify the maintenance of their client implementations. The introduction of a Restful Service layer on top of the Message Broker Network will give EGI the ability to implement also other requested features such as access controls or message accounting.
Security monitoring tools developed in the scope of EGI focused mainly on traditional grid sites and operations, which is not enough nowadays. Current technologies, namely federated clouds, bring new security challenges that must be addressed by new approaches. In this task we will identify the new areas and provide solutions for proper monitoring of them. The solutions delivered by the task will be modular and individual components could be utilised independently and on several levels in the infrastructure.
(Lead: CYFRONET, M1 – M30)
Estimated task effort: 17PM
e-GRANT is a platform that enables EGI customers to apply and get allocation of compute and storage resources. In this task, e-GRANT functionalities will be extended to cover the complete SLA lifecycle-enabling activities such as tracing of resource centre configuration, monitoring of resources delivery, availability analysis for agreed services, capacity reports analysis and billing. Another area of e-GRANT development includes facilitating activities taking place before customer request resources. The platform will enable customers to investigate the services and resources available to them by publishing this information on the EGI Service Registry and Marketplace, including special offers or promotion actions, such as seed-resources for new communities, demonstrators or hackathons. Furthermore, e-GRANT will be extended to play an important role in the new mode of service delivery like pay-for-use, which would require extension towards negotiating the price. e-Grant will support the integration of EGI with other e-Infrastructures enabling customers to jointly request resources owning to different infrastructures.
The following gives an overview of deliverables. Schedule
|| Technical design of the new Accounting Portal and implementation plan (R)|
|D3.2|| Design of the EGI Service Registry and Marketplace (R)|
|D3.3|| Accounting Repository release (OTHER)|
|D3.4|| First release of the Operational tools (OTHER)|
| D3.6 (3.5)
||Analysis on techniques to manage big data on the EGI accounting system (R)|
|D3.5 (3.6)|| First release of the new Accounting Portal deployed in production (OTHER)|
|D3.7|| First release of the EGI Service Registry and Marketplace prototype (DEM)|
|D3.8|| First data accounting prototype (DEM)|
|D3.9|| Identity Management for Distributed User Communities report (R)|
|D3.10|| Second release of the new Accounting Portal deployed in production (OTHER)|
|D3.11|| Second release of the Operational tools (OTHER)|
|D3.12|| Second release of the Accounting Repository (OTHER)|
|D3.14 (3.13)||Report on Data Accounting (R)|
|D3.13 (3.14)|| Second release of the EGI Service Registry and Marketplace prototype (DEM)|
|D3.15|| Second data accounting prototype (DEM) |
|D3.16|| Final report on EGI Service Registry and Marketplace (R)|
|D3.17|| Final release of the Accounting Repository (OTHER)|
|D3.18|| Final release of the Operational tools (OTHER)|
|D3.19|| Final release of the new Accounting Portal deployed in production (OTHER)|
The following gives an overview of milestones. Schedule
|M3.1||Operational tools development roadmap agreed|
|M3.2||User requirements on data accounting collected|
|M3.3||Operational tools development roadmap revised|
|M3.4||Pilot services and best practices to enable federated AAI solutions released|
|M3.5||The first version of the EGI Marketplace is demonstrated|
|M3.6||The second version of the EGI Marketplace is demonstrated|
||Description||Type||How measured||Target PM12||Target PM24||Target PM30|
|KPI.6.JRA1.AAI||Number of users adopting federated IdP||Cumulative||Number of users accessing EGI services through the IdP Proxy/broker|| TBD
|M.JRA1.AAI.1||Number of communities whose IdP framework integrates with EGI AAI||Cumulative|| 3.1|
|M.JRA1.Marketplace.1||Number of entries in the EGI Marketplace (i.e. services, applications etc.)||Cumulative|| 3.2|
|M.JRA1.Accounting.1||Number of kinds of data repository systems integrated with the EGI accounting software||Cumulative|| 3.3|
|M.JRA1.Accounting.2||Number of kinds of storage systems integrated with the EGI accounting software||Cumulative|| 3.3|
|M.JRA1.OpsTools.1||Number of new requirements introduced in the roadmap||Per period|| 3.4|
|M.JRA1.OpsTools.2||Number of probes developed to monitor cloud resources||Per period||3.4|
|M.JRA1.eGrant.1||Number of user requests handled in e-GRANT||Per period||3.5|