Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

VT VAPOR

From EGIWiki
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Check out VAPOR

VAPOR is deployed at the following URL: https://operations-portal.egi.eu/vapor/

General Project Information

  • Leader: Franck Michel, CNRS
  • Mailing List: vt-vapor@mailman.egi.eu
  • Meetings: https://indico.egi.eu/indico/categoryDisplay.py?categId=100
  • Status: In progress
  • Start Date: 15/04/2013
  • Duration: 12 months
  • Customer: as a first use case, the biomed VO is also the first user, represented by Franck Michel. Other partners of the project have a major role in the specification of the project features, but they may also be customers of the project, by starting to use it in a time framed to be defined: France Grille VO (Hèlene Cordier), WeNMR (Alexandre Bovin), CompChem VO (Alessandro Costantini), VLEMED VO (Silvia Olabarriaga).

Deployed portal URL: https://operations-portal.egi.eu/vapor/

Source code and documentation repository: https://redmine.i3s.unice.fr/projects/vapor

Material

Slides: Presentation of the project at the EGI Community Forum 2013 (Apr. 2013).

Slides: VAPOR mini-project progress update at the EGI Technical Forum 2013 (Sept. 2013).

Slides: Using VAPOR to improve VO administration and operations at the EGI Community Forum 2014 (May 2014).

Project description

Overview

This project intends to help small to medium-size grid user communities perform daily administrative and operational tasks, by developing VAPOR, the Vo Administration and operations PORtal, a generic tool to assist community managers and support teams in performing their daily activities. Such communities may typically have no or few dedicated IT support, have scattered scientific activities or fragmented user groups, and may possibly (although not necessarily) make an opportunistic usage of the resources.

This portal is expected to

  • help such communities sustain their model by mutualising the administrative and operational cost at the level of a Virtual Research Community (VRC) or beyond,
  • facilitates the outreach of new user communities by making it easier to start with the administration and operations of a VO. This portal is available to any community wishing to deploy it, and licensed as open source software.

The primary use case of this project will be the Life-Science Grid Community (LSGC), starting with the biomed Virtual Organization (VO), and mutualising this activity with other VOs of the LSGC.

Motivations

Operating large subsets of the European Grid Infrastructure (EGI) with reasonable quality of service requires substantial effort to cope with runtime issues like hardware failures or configuration flaws. While this cost is acceptable for large and structured communities such as High Energy Physics backed with a solid IT support, it is often out of reach for small VOs and fragmented communities. The Virtual Research Community (VRC) model has encouraged such communities to organize and structure themselves, as illustrated by the Life-Science VRC (LSGC) that gathers five VOs related to the Life-Science field.

Among them, the biomed VO has set up a volunteering-based technical support team, which goal is to be the technical liaison between the VO users, the resources providers, and the EGI bodies (UCB, OMB, UCST). Most active users of the VO contribute their time and experience to perform this job on a volunteering basis, and relay each other during duty shifts. However, the two-years experience of this team shows that the effort required to operate such a VO remains high even when mutualised with members the VO, and it is most likely unreachable for new emerging communities.

Finally, this experience suggests that, to be sustainable, the volunteering model should be mutualised not only within the VO, but also with other VOs, in order to reduce its cost while improving its efficiency.

Some communities require operation and administration tools tailored to their model

A large variety of existing tools and portals (SAM, VO Admin Dashboard, VO Operations Portal, VOMS) are available to assist VO managers and support teams. While some are very generic, others are designed to meet somewhat specific contexts. An example is the CERN Experiment Dashboard, which, although generic and customizable, is tailored for large VOs with common needs, and does not easily fit in the context of smaller "opportunistic" VOs with heterogeneous user groups.

The insight gained in biomed attests that mutualising the effort at the VRC level still requires efforts. In particular, some administration and operational tasks need to be facilitated or automated in different areas: community users management, community-wide accounting, data management, quality of service (QoS) monitoring. Incidentally, the VRC model having no technical existence in the middleware, VRCs lack tools able to provide "VRC-aware" views. From these observations, in 2011 the LSGC members agreed on the need for a tool primarily tailored to the LSGC VOs, though generic enough to meet needs of other similar communities.

Objectives

VAPOR targets existing communities as well as new user communities, with no or few dedicated IT support, possibly relying on the opportunistic usage of resources. VAPOR complements existing tools (SAM, VO Admin Dashboard, VO Operations Portal, VOMS) with novel services, which main characteristics are described below:

  • Help communities to sustain their model by mutualising support and admin effort: VAPOR is generic, independent of scientific applications, and designed to help communities to sustain their model by assisting VO managers and VO support teams in their daily tasks, and making it possible to mutualise those tasks between several VOs.
  • Help handle the community users and apprehend their diversity
    • Users database, handling and follow-up of user registration lifecycle
    • Keep track of real users "hidden behind" a robot certificate, while tracking their articles in order to spur the acknowledgment of EGI support in publications.
    • Interface with external services: VOMS, LFC, EGI Applications Database
    • Collection of publications and scientific production achieved using the infrastructure
    • Maintain VRC-wide and VO-wide mailing lists in sync with VOMS
  • Providing a better understanding of the operational issues in the subset of the infrastructure used. Frequently, indeed, for a VO relying on the opportunistic use of resources, the understanding of operational issues is mostly limited to a snapshot of the alarms and the number of tickets submitted. VAPOR intends to issue usage and QoS reports to allow VO managers get a global picture of how things work for their community.
    • Resources status indicators & statistical reports: (i) issue reports on resources availability (free storage space, jobs success rate), (ii) report GOCDB and BDII status to exclude resources not in production
    • Data Management: files migration before storage decommissioning or in case of SE filling-up, clean-up files of former users
  • Community Accounting tools:
    • On-demand generation of reports about global community resource usage, per VO resource usage, per VO sub-group resource usage
    • Projection of future needs for storage and computing resources: estimating the future needs for resources is roughly impossible in VOs with fragmented user groups. VAPOR proposes accounting projections based on the extrapolation of passed resource usage data that should help steer discussions with the Resource Allocation Work Group.

Initially, the portal will be used to administrate and operate the biomed VO. Then, the effort will be mutualised with other VOs of the LSGC and the France Grille VO. The feedback collected during the project will help assess if the portal reaches its objectives in terms of improving the efficiency and facilitating the work of the support team. Eventually, the features specifications will be updated to improve functionalities. Based on the experience gained, the portal will be proposed as open source to any other community willing to use it.

Tasks

Below we provide a description of the main tasks of the project.

Task 1: Features specification (M1 to M3): all partners

The design of VAPOR will be based on the features scoped during former activity of the LSGC members. A technical feasibility study will review existing tools and portals, either external to the LSGC community or produced internally, in particular by the biomed support team. The priority will be given to the integration of existing software as part of the portal, rather than fresh developments.

From this material, the consortium will prioritize the features. The resulting features specification might be updated later on during the project, and should help to continuously adapt the work plan.

Deliverable D1: VAPOR Functional Specification.

Task 2: Software development (M4 to M10): CNRS I3S

This phase will produce code for the three major items identified in 3.2. The development will be handled keeping in mind agile methodologies by being responsive to the changes that may occur along the project, by running short development cycles, and applying continuous integration and frequent deployments to maintain a working software.

Deliverables D2, D3 and D4 correspond to the three major software packages: Community users management, Operations management, Accounting

Task 3: Platform deployment and exploitation (M6 to M12) : all partners

The experience gained by the biomed VO technical support team for the last couple of years, along with the biomed model relying on the opportunistic use of resources, make this support team a real-world use case for VAPOR. For this reason, biomed support team members will be beta users of the portal. Software packages will be deployed on existing platforms at the earliest to allow for early feedback, bug fixing and improvements. Each deployment will include writing documentation, the update of existing administration and operational procedures and the training of the support team members.

Deliverable D5: final software package along with guidelines and documentation.

Deliverables

  • Deliverable D1 (M3): VAPOR Functional Specification, M3 => delivered on date, see specification
  • Deliverable D2 (M10): Community users management software package
  • Deliverable D3.1 (M6): Resource status indicators, reports and white list software package => delivered at M7, see code repository
  • Deliverable D3.2 (M9): VO Data Management software package => partial delivery at M9, last part at M12, see code repository
  • Deliverable D4: Accounting software package => finally agreed to not develop.
  • Deliverable D5 (M12): final software package along with guidelines and documentation => Documentation available in the docs repository.

The definition of priorities with partner VOs changed the order of development initially proposed in the project description. As a consequence, D2 is postponed after D3. D4 is deemed less useful and is postponed at the end of the project, if time remains. During the development, D3 appeared to be a bigger work than expected, it is therefore split into two deliverables: D3.1 (Resource status indicators, statistical reports and white list) and D3.2 (VO Data Management).

Members

The project involves the partners described below.

I3S is a joint research unit of the CNRS and the University of Nice - Sophia Antipolis. The MODALIS team involved is specialized in distributed computing and software engineering. It has been involved in EGI-InSPIRE SA3 activities, and in the management of the biomed VO since 2003 (DataGrid project).

CREATIS is a joint CNRS research unit working on medical image analysis. It has been involved in the management of the biomed VO since 2003 (DataGrid project). It is using the infrastructure to simulate medical image acquisition and cancer treatment. It contributes to the biomed technical support team and LSGC.

GRyCAP is the Grid and High Performance Computing Group of the Universitat Politècnica de València (UPVLC). It has a strong background in Grid and Cloud Computing and Data Management, especially on the area of Biomedicine. It has participated in many European and Latin American projects, with an important experience in technology dissemination and e-infrastructures (EGEE I to III, EELA 1 and 2, EGI-InSPIRE, VENUS-C, CyTED-GRiD, EU-BrazilOpenBio). GRyCAP coordinated the Spanish Network for e-Science which is the tool to implement the Spanish NGI. It contributes to the biomed technical support team and LSGC.

IPHC (Institut Pluridisciplinaire Hubert Curien) is a multi-disciplinary institute gathering eco-physiology, chemistry and subatomic physics activities. It operates one resource centre of the French NGI, offering computing and storage resources to national and international VOs. It is involved in biomed VO administration and operations.

France Grilles, the French NGI, has a long-standing commitment in the LSGC support. It operates a national catch-all VO which activity is de facto scattered. The characteristics of the France Grille VO are close to those targeted by VAPOR, it is thus an interesting use case. France Grille is also a strong vector for promoting VAPOR to other VO administrators.

AMC is the Academic Medical Center of the University of Amsterdam. The e-science group of the Bioinformatics Laboratory has long experience in the integration, development and provision of science gateways for biomedical research. The group has actively participated in Dutch grid-related initiatives, such as the LSGC and EGI virtual teams. It also operates the VLEMED VO (part of the LSGC) which may be an interesting use case of VAPOR.

COMPCHEM has been established in the EGEE Production Grid environment to support the Computational Chemistry applications. Starting from May 2010 the VO is running on the EGI Production Grid. It is coordinated by the Computational Dynamics and Kinetics group of the Department of Chemistry at the University of Perugia (UNIPG) and by the High Performance Computing Lab. of the Department of Mathematics and Informatics on the same University.

WeNMR is both a three years project funded under the European Commission’s 7th Framework Programme and a Virtual Research Community supported by EGI, within the life science area. WeNMR aims at bringing together complementary research teams in the structural biology and life science area into a virtual research community at a worldwide level and provide them with a platform integrating and streamlining the computational approaches necessary for NMR and SAXS data analysis and structural modelling.

Resources

I3S, CREATIS, GRyCAP and IPHC will directly and technically contribute to the project, for instance through software developments, or providing expertise on existing tools to be adapted.

France Grille, AMC, CompChem and WeNMR are interesting use cases that wish to contribute to the design of the project. They may also be using the project, in a time framed to be defined, and then provide feed-back about VAPOR.

Member Affiliation Involved Work load
Franck MICHEL, research engineer I3S 33% 4 months
Flavien Forestier, recruited research engineer I3S 100% 12 months
Sorina Camarasu-Pop, researcher CREATIS 8.3% 1 month
Patrick Guterl and Jérôme Pansanel, operations managers, IPHC 8.3% 1 month
Ignácio Blanquer, professor and Enrique Bayonne, engineer GRyCAP 8.3% 1 month

The project budget essentially funds the recruitement of Flavien Forestier, CNRS I3S, for 12 months. The other human resources are contributed by the consortium partners.

Project follow-up

Progress Report

Features specification

Priorities

Meetings