VT VAPOR

General Project Information

• Leader: Franck Michel, CNRS
• Mailing List:
• Meetings: <link to Indico container>
• Status: In progress
• Start Date: 15/04/2013
• Duration: 12 months
• Customer: Virtual Organizations

Project description

Overview

This project intends to help small to medium-size grid user communities perform daily administrative and operational tasks, by developing VAPOR, the Vo Administration and operations PORtal, a generic tool to assist community managers and support teams in performing their daily activities. Such communities may typically have no or few dedicated IT support, have scattered scientific activities or fragmented user groups, and may possibly (although not necessarily) make an opportunistic usage of the resources.

This portal is expected to

• help such communities sustain their model by mutualising the administrative and operational cost at the level of a Virtual Research Community (VRC) or beyond,
• facilitates the outreach of new user communities by making it easier to start with the administration and operations of a VO. This portal will be available to any community wishing to deploy it, and licensed as open source software.

The primary use case of this project will be the Life-Science Grid Community (LSGC), starting with the biomed Virtual Organization (VO), and mutualising this activity with other VOs of the LSGC.

Motivation

Operating large subsets of the European Grid Infrastructure (EGI) with reasonable quality of service requires substantial effort to cope with runtime issues like hardware failures or configuration flaws. While this cost is acceptable for large and structured communities such as High Energy Physics backed with a solid IT support, it is often out of reach for small VOs and fragmented communities. The Virtual Research Community (VRC) model has encouraged such communities to organize and structure themselves, as illustrated by the Life-Science VRC (LSGC) that gathers five VOs related to the Life-Science field.

Among them, the biomed VO has set up a volunteering-based technical support team, which goal is to be the technical liaison between the VO users, the resources providers, and the EGI instances (UCB, OMB, UCST). Most active users of the VO contribute their time and experience to perform this job on a volunteering basis, and relay each other during duty shifts. However, the two-years experience of this team shows that the effort required to operate such a VO remains high even when mutualised with members the VO, and it is most likely unreachable for new emerging communities.

Finally, this experience suggests that, to be sustainable, the volunteering model should be mutualised not only within the VO, but also with other VOs, in order to reduce its cost while improving its efficiency.

Some communities require operation and administration tools tailored to their model

A large variety of existing tools and portals (SAM, VO Admin Dashboard, VO Operations Portal, VOMS) are available to assist VO managers and support teams. While some are very generic, others are designed to meet somewhat specific contexts. An example is the CERN Experiment Dashboard, which, although generic and customizable, is tailored for large VOs with common needs, and does not easily fit in the context of smaller "opportunistic" VOs with heterogeneous user groups.

The insight gained in biomed attests that mutualising the effort at the VRC level still requires efforts. In particular, some administration and operational tasks need to be facilitated or automated in different areas: community users management, community-wide accounting, data management, quality of service (QoS) monitoring. Incidentally, the VRC model having no technical existence in the middleware, VRCs lack tools able to provide "VRC-aware" views. From these observations, in 2011 the LSGC members agreed on the need for a tool primarily tailored to the LSGC VOs, though generic enough to meet needs of other similar communities.

Objectives

VAPOR targets existing communities as well as new user communities, with no or few dedicated IT support, possibly relying on the opportunistic usage of resources. VAPOR main characteristics are:

VAPOR complements existing tools (SAM, VO Admin Dashboard, VO Operations Portal, VOMS) with novel features sorted in three categories:

• Help communities to sustain their model: VAPOR is generic, independent of scientific applications, and designed to help communities to sustain their model by assisting VO managers and VO support teams in their daily tasks, and making it possible to mutualise those tasks.
• Help handle the community users and apprehend their diversity
  • Users database, handling and follow-up of user registration lifecycle
  • Keep track of real users "hidden behind" a robot certificate, while tracking their articles in order to spur the acknowledgment of EGI support in publications.
  • Interface with external services: VOMS, LFC, EGI Applications Database
  • Collection of publications and scientific production achieved using the infrastructure
  • Maintain VRC-wide and VO-wide mailing lists in sync with VOMS
• Providing a better understanding of the operational issues in the subset of the infrastructure used. Frequently, indeed, for a VO relying on the opportunistic use of resources, the understanding of operational issues is mostly limited to a snapshot of the alarms and the number of tickets submitted. VAPOR intends to issue usage and QoS reports to allow VO managers get a global picture of how things work for their community.
  • Resources status indicators & statistical reports: (i) issue reports on resources availability (free storage space, jobs success rate), (ii) report GOCDB and BDII status to exclude resources not in production
  • Data Management: files migration before storage decommissioning or in case of SE filling-up, clean-up files of former users
• Community Accounting tools:
  • On-demand generation of reports about global community resource usage, per VO resource usage, per VO sub-group resource usage
  • Projection of future needs for storage and computing resources: estimating the future needs for resources is roughly impossible in VOs with fragmented user groups. VAPOR proposes accounting projections based on the extrapolation of passed resource usage data that should help steer discussions with the Resource Allocation Work Group.

Tasks

List and describe the specific tasks with target dates for completion.

Outcomes/Deliverables

Initially, the portal will be used to administrate and operate the biomed VO. Then, the effort will be mutualised with other VOs of the LSGC and the France Grille VO. The feedback collected during the project will help assess if the portal reaches its objectives in terms improving the efficiency and facilitating the work of the support team. Eventually, the features specifications will be updated to improve functionalities. Based on the experience gained, the portal will be proposed as open source to any other community willing to use it.

• Deliverable 1 description and target date
• Deliverable 2 description

Members

• Member 1 Name and Role in team
• Member 2 Name and Role in team

• Stake holder 1 and reason for interest
• Stake holder 1 and reason for interest

Emerging Information

if relevant - may change the nature and objectives of the project

Resources

How is the Project Team to be resourced and how will members work? How much effort will be required from each individual, how will this effort impact contributing organisations? – do not confuse VT participants with stakeholders. VT participants do work for the project – others don’t have to do any work for the project. How will your funds be consumed?

Progress Reporting

How will you measure your progress and how will you report this? To whom? The objective here is to assure others that your project is on track, but if there are problems then this is the path to getting help and more resources (time, funding, effort).

The Project Leader will provide a short emailed progress report on a weekly basis. The report will be due by 17:00 on Fridays and is to contain details of: ● Work achieved that week ● Work planned for next week ● Progress against the goals in the project plan ● Issues that the virtual team leader needs help with (e.g. non-responsive partners, more resources, support from EGI.eu teams, etc.)

The Team project manager will also provide a more comprehensive input for the EGI InSPIRE quarterly reporting.