Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "VO Services"

From EGIWiki
Jump to navigation Jump to search
 
(119 intermediate revisions by 2 users not shown)
Line 1: Line 1:
__TOC__
{{Template:Op menubar}}
{{Template:Doc_menubar}}
[[Category:Operations Documentation]]
<br />
[[File:Vo_services_logo.png|right|600px]]
{{TOC_right}}


= Introduction =
= Introduction =


[[File:VOServicesWikiFig1.png|right]]
Users of the European Grid Infrastructure are organised into Virtual Organisations (VO). A VO is a group of people who share similar interests, have similar goals, and need to work collaboratively sharing resources (e.g. data, software, expertise, CPU, storage space) regardless of their geographical location. There are already more than [http://operations-portal.egi.eu/vo <big>'''200 VOs hosted on EGI'''</big>]. Users are invited to join these VOs, or establish new ones for their own communities.


Setting up and operating a VO is a complex task that requires an important effort for ensuring a high quality of services. Many tools and services are available in EGI that rely on the VO information, and sometimes, procedures are neither easily available nor complete. The EGI VO Services subtask aims at supporting VOs in the whole process of start-up, management and operation, pointing out to tools, services, documentation and procedural guidelines to maximize the usage of the resources.


<br />
<br />
= Main activities =


[[File:VOServicesWikiFig1_2.png|right|400px]]
The VO services activities are summarized in four main points:
The VO services activities are summarized in four main points:
* '''<big>VO Management Support</big>''': To set up a central documentation point with procedures, FAQs and a single contact point.
* '''Evaluation of VO and VRC support services and tools''': To identify and evaluate tools / services which could be considered valuable by VOs
* '''<big>Promotion of VO activities</big>''': To ease the access to tools and services to foster production quality by the users of a VO.
* '''Provision of services and tools for VOs and VRCS''': To provide or help setting up tools / services for VOs aiming to foster production quality by VO users and VO infrastructure providers
* '''<big>VO Infrastructure Monitoring</big>''': To aid setting up tools for monitoring the resources under the point of view of the VO.
* '''Consultancy for VO managers''': To provide consultancy services to VO managers, clarifying procedures and gathering documentation aiming to optimize the VO managers daily work.
* '''<big>VO Infrastructure Usage</big>''': To support the monitoring on the usage of the resources by the users from a VO or a VRC.
* '''Helpdesk for VO managers''': To provide a first line support to VO managers.


<br />
<br />


= VO Management Support =
= Evaluation of support tools for VOs and VRCs =


[[File:VOServicesWikiFig2.png|right]]
Members of VRCs and VOs should be able to assess how reliable their production infrastructure is. In order to achieve this VRCs and VO members must make use of tools that run VO specific tests on their infrastructure on a regular basis, and at same time make the results visible to all VO and/or VRC members. Such tools typically consist of multiple modules:
* '''An infrastructure testing framework:''' A component that can host VO/VRC specific tests and run these tests on a regular basis on the infrastructure
* '''A monitor system:''' A system that collects test results and makes them visible for VO/VRC members
* '''An alarm system:''' A component that can raise alarms whenever a site/service/resource fails a VO-specific test.


== Consultancy and Support ==
<br />


The [https://gus.fzk.de/pages/ggus-docs/PDF/18000_FAQ_for_vo_services.pdf '''GGUS VO Services Support Unit'''] is one of the official communication channel at VO Managers diposal to receive support regarding procedures, services and tools they operate in their daily work. It has been established to provide a closer link with the technical services staff so that VO managers can:
The VO Service team has investigated tools and services available within the EGI community that could be adapted to multi-VO environments. The investigated tools cover areas from job management to resource monitoring. The final goal is to easy the access to those applications aiming to foster production quality by the users of a VO.
* '''ask for consultancy services regarding tools and services'''
* '''ask for support while operating their VO'''
* '''ask for clarification of procedures or documentation improvements'''
* '''provide feedback regarding specific tools and/or services'''
 
The established workflow assumes that, while interacting with the VO managers, VO Services staff can involve different experts in the discussions, and bridge VO managers requests with the appropriate support units.
 
Through the interactions with the VO managers, different requirements can emerge. In this case, it is the VO Services Support Unit responsibility to summarize the discussion into a requirement, and insert it in [https://rt.egi.eu EGI Request Tracker]. USCT is then responsible for the requirement follow up.  


<br />
<br />


== Documentation for the VO Manager daily work==
'''All communities (VRCs, VOs, NGIs, ...) are invited to recommend new tools for evaluation or share their own evaluation / experience with the VO Services team. The obvious advantage is that those surveys become available to the whole EGI community. Please contact the VO Service team via its GGUS support unit, or directly via  <big>vo-services@mailman.egi.eu</big>'''.
* <big>[https://documents.egi.eu/document/278 '''VO Registration Process in EGI''']</big>
* <big>[https://wiki.egi.eu/wiki/VO_Managment_FAQS '''VO Management Frequently Asked Questions''']</big>
 
<br />
<br />


= Promotion of VO activities =
== Provision of services and tools for VOs and VRCs ==
VO activities normally consist of a very well defined set of procedures executed on a specific defined order. To minimize time and optimize success, the VO Service team has investigated tools and services available within the EGI community that could be adapted to multi-VO environments. The final goal is to easy the access to those applications aiming to foster production quality by the users of a VO. The following sections present a list of tools and services that could be offered as Emerging VOs.
 
== Services and Tools ==
=== Ganga ===
GANGA aims to be an easy tool for job submission and management. It is built on python and provides client command tools, a Graphical User Interface (GUI), and a WebGUI. A job in GANGA is constructed from a set of building blocks. All jobs must specify the software to be run (application) and the processing system (backend) to be used. Many jobs will specify an input dataset to be read and/or an output dataset to be produced. Therefore, GANGA provides a framework for handling different types of applications, backends and datasets, implemented as plugin classes. Pragmatically, this means that GANGA can be used to submit jobs to the localhost where it is installed, to a local farm or to a computing grid such as LCG/EGI, as long as the appropriate clients command tools are available to GANGA. From the list of offered default backends, it seems that for grid infrastructures, only the gLite middleware is available. Nevertheless, since it seems to be a very modular tool, other extensions (middlewares) could be easily integrated. GANGA is presently used by ATLAS and LHCb users among other collaborations. For complete details on how GANGA is used in the framework of those collaborations, please consult [3].
 
{|border="1" cellspacing="0" cellpadding="1"
|'''Home web page:'''
|http://ganga.web.cern.ch/ganga/
|-
|'''Installation guide:'''
| http://ganga.web.cern.ch/ganga/user/installation/index.php
|-
|'''Evaluation Document:'''
|[[Media:Diane_Ganga_Evaluation_v2.pdf|Ganga Evaluation Document]]
|}


There are two different kinds of tools which can be considered useful for VOs:
# global tools supported under the EGI project (operations portal, accounting portal, gocdb, ...)
# specific tools developed and supported by communities (VO, VRCs or NGIs).
<br />
<br />


=== Diane ===
=== Tools developed and supported by communities ===
Evaluation Document: [[Media:Diane_Ganga_Evaluation_v2.pdf|'''Ganga + Diane + minidashboard evaluation (version 1.0.2)''']]
=== Minidashboard ===
Evaluation Document: [[Media:Diane_Ganga_Evaluation_v2.pdf|'''Ganga + Diane + minidashboard evaluation (version 1.0.2)''']]


== Complementary Documentation ==
<big>
* [[Media:Setting_up_VO_Services_in_EGI_v2.pdf|'''Core VO services evaluation (version 1.0.2)''']]
* [[Media:Monitoring_tools_V5.pdf|'''Survey on monitoring tools (version 1.0.5)''']]
* [[Media:Diane_Ganga_Evaluation_v2.pdf|'''Ganga + Diane + minidashboard evaluation (version 1.0.2)''']]
</big>
<br />
<br />
<!--


The VO Services team already evaluated a high number of services and tools. Making use of that information the VO Services compiled a list of services and tools useful to VO and/or VRC's members. which can be consulted at:  [[VO_Services_and_Tools_Portfolio |<big>'''Services and Tools portfolio wiki page'''</big>]]




Before taking the decision of installing them under a production environment VOs should take into account the effort needed for the operation and maintenance of those tools. Three different scenarios are foreseen:
* The VO installs and operate those tools using their own resources
* The VO uses a service instance installed elsewhere during a temporary period while it gains expertise to deploy and operate the service.
* The VO makes permanent use of the tools operated elsewhere and offered as a service to the VO or VO users
<br />


== GANGA-DIANE mini-Dashboard ==
= Consultancy for VO Managers =


ACTUALLY, MOST OF THE INFORMATION ABOUT THE TOOL SHOULD BE STORED IN THE APPLICATION DATABASE AND THIS PAGE SHOULD INCLUDE ONLY LINKS, PLUS A SHORT DESCRIPTION TO PUT THE APPLICATION IN THE RIGHT CONTEXT HERE AND TO EMPHASISE ITS VO MONITORING FEATURES. AKS THE DEVELOPER OF THE TOOL TO REGISTER THE TOOL IN APPDB!
[[File:VOServicesWikiFig2.png|right]]
 
The mini-Dashboard monitoring service provides a web-based interface to keep track of and monitor jobs submitted to EGI with the GANGA job submission framework. The framework can be setup for an individual user, or centrally for one or multiple VOs. In case of a central installation that tool provides overview of the jobs that were submitted by members of the VO through GANGA. ...
 
=== Key features related to monitoring:===
*...
*...


The [[GGUS:VO_Services_FAQ |<big>'''GGUS VO Services Support Unit'''</big>]] is the official communication channel between VO managers and VO Services staff. One of its main objectives is to provide a direct link to VO managers so that they can ask for consultancy services on VO management issues, like:


===Evaluation===
* '''ask for help on operations through different scenarios'''
* '''provide feedback regarding specific tools and services'''
* '''discuss and assess requests and enhancements'''


add link to Goncalo's document?
While interacting with the VO managers, VO Services staff can involve different experts in the discussions, and bridge VO managers requests with the appropriate support units. Through the interactions with the VO managers, different requirements can also emerge. In this case, it is the VO Services Support Unit responsibility to summarize the discussion into a requirement, and insert it in [https://rt.egi.eu EGI Request Tracker]. USCT is then responsible for the requirement follow up.


===Further information about the tool===
Also, through this subtask, the VO Service staff is responsible for evaluating and gathering documentation found relevant for VO managers daily activities. All communities (VRCs, VOs, NGIs, ...) are invited to recommend topics about which new materials should be prepared (manuals, presentations, etc...) or even to share their own materials about VO management related topics. Please contact the VO Service team via its GGUS support unit, or directly via  <big>'''vo-services@mailman.egi.eu'''</big>'''.
 
- Insert link once the tool is registered in AppDB
- Add link to homepage
 
===Need this tool for your VO===
 
- Describe whether the tool can be provided by VO Services group, an NGI, or whether it must be installed by the VO. Point to installation manual in this latter case.
-->
 
= VO Infrastructure Monitoring =
 
Members of VRCs and VOs should be able to assess how reliable their production infrastructure is. VRCs and VO members should be able to use integrated tools that run VO specific tests on their infrastructure on a regular basis, and make the results visible for VO and VRC members. Such tools tipically consist of multiple modules:
* '''An infrastructure testing framework:''' A component that can host VO/VRC specific tests and run these tests on a regular basis on the infrastructure
* '''A monitor system:''' A system that collects test results and makes them visible for VO/VRC members
* '''An alarm system:''' A component that can raise alarms whenever a site/service/resource fails a VO-specific test.


<br />
<br />


== Services and Tools ==
== Documentation ==
The following proposed tool aim to acomplish the previous defined objectives:
The <big>[[User_Documentation|'''Relevant documentation for VO activities''']]</big> wiki page presents a snapshot of the current aggregated material.
* [https://wiki.egi.eu/wiki/VO_Service_Availability_Monitoring <big>'''VO Service Availability Monitoring'''</big>]
<br />


= Activity documentation =
== Workplans ==
* [https://documents.egi.eu/document/277 '''Activity Summary and 6 months Workplan (Month 6)''']
<br />
<br />


== USAG Presentations ==
= Helpdesk for VO managers =
* [https://www.egi.eu/indico/materialDisplay.py?contribId=3&materialId=slides&confId=311 '''USAG Fev 2011''']
A first line service to support VO managers is consolidated through the [https://gus.fzk.de/pages/ggus-docs/PDF/18000_FAQ_for_vo_services.pdf <big>'''GGUS VO Services Support Unit'''</big>]. Through this service, the VO managers can:
* [https://www.egi.eu/indico/getFile.py/access?contribId=2&resId=0&materialId=slides&confId=223 '''USAG Nov 2010''']
* '''ask for support while operating their VO'''
<br/>
* '''ask for clarification of procedures'''
*''' ask for documentation improvements'''
The VO Service team seats behind the [https://gus.fzk.de/pages/ggus-docs/PDF/18000_FAQ_for_vo_services.pdf <big>'''GGUS VO Services Support Unit'''</big>] and is responsible for evaluating VO managers requests and treat them in a timely manner. Because this support unit is fully integrated in GGUS user support workflows, it is possible to reassign requests to other support units and/or involved other experts in the evaluation of a given request. For the previous reasons, the [https://gus.fzk.de/pages/ggus-docs/PDF/18000_FAQ_for_vo_services.pdf <big>'''GGUS VO Services Support Unit'''</big>] represents the primordial point of contact for VO managers support. Nevertheless, the VO Services mailing list is also available through <big>'''vo-services@mailman.egi.eu'''</big>'''


== Internal schedule ==
* [https://wiki.egi.eu/wiki/VO_Services_Internal_Schedule '''Activity reports schedule''']
<br />
<br />



Latest revision as of 11:41, 23 November 2012

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home Manuals Procedures Training Other Contact For: VO managers Administrators


Vo services logo.png


Introduction

Users of the European Grid Infrastructure are organised into Virtual Organisations (VO). A VO is a group of people who share similar interests, have similar goals, and need to work collaboratively sharing resources (e.g. data, software, expertise, CPU, storage space) regardless of their geographical location. There are already more than 200 VOs hosted on EGI. Users are invited to join these VOs, or establish new ones for their own communities.



Main activities

VOServicesWikiFig1 2.png

The VO services activities are summarized in four main points:

  • Evaluation of VO and VRC support services and tools: To identify and evaluate tools / services which could be considered valuable by VOs
  • Provision of services and tools for VOs and VRCS: To provide or help setting up tools / services for VOs aiming to foster production quality by VO users and VO infrastructure providers
  • Consultancy for VO managers: To provide consultancy services to VO managers, clarifying procedures and gathering documentation aiming to optimize the VO managers daily work.
  • Helpdesk for VO managers: To provide a first line support to VO managers.


Evaluation of support tools for VOs and VRCs

Members of VRCs and VOs should be able to assess how reliable their production infrastructure is. In order to achieve this VRCs and VO members must make use of tools that run VO specific tests on their infrastructure on a regular basis, and at same time make the results visible to all VO and/or VRC members. Such tools typically consist of multiple modules:

  • An infrastructure testing framework: A component that can host VO/VRC specific tests and run these tests on a regular basis on the infrastructure
  • A monitor system: A system that collects test results and makes them visible for VO/VRC members
  • An alarm system: A component that can raise alarms whenever a site/service/resource fails a VO-specific test.


The VO Service team has investigated tools and services available within the EGI community that could be adapted to multi-VO environments. The investigated tools cover areas from job management to resource monitoring. The final goal is to easy the access to those applications aiming to foster production quality by the users of a VO.


All communities (VRCs, VOs, NGIs, ...) are invited to recommend new tools for evaluation or share their own evaluation / experience with the VO Services team. The obvious advantage is that those surveys become available to the whole EGI community. Please contact the VO Service team via its GGUS support unit, or directly via vo-services@mailman.egi.eu.

Provision of services and tools for VOs and VRCs

There are two different kinds of tools which can be considered useful for VOs:

  1. global tools supported under the EGI project (operations portal, accounting portal, gocdb, ...)
  2. specific tools developed and supported by communities (VO, VRCs or NGIs).


Tools developed and supported by communities


The VO Services team already evaluated a high number of services and tools. Making use of that information the VO Services compiled a list of services and tools useful to VO and/or VRC's members. which can be consulted at: Services and Tools portfolio wiki page


Before taking the decision of installing them under a production environment VOs should take into account the effort needed for the operation and maintenance of those tools. Three different scenarios are foreseen:

  • The VO installs and operate those tools using their own resources
  • The VO uses a service instance installed elsewhere during a temporary period while it gains expertise to deploy and operate the service.
  • The VO makes permanent use of the tools operated elsewhere and offered as a service to the VO or VO users


Consultancy for VO Managers

VOServicesWikiFig2.png

The GGUS VO Services Support Unit is the official communication channel between VO managers and VO Services staff. One of its main objectives is to provide a direct link to VO managers so that they can ask for consultancy services on VO management issues, like:

  • ask for help on operations through different scenarios
  • provide feedback regarding specific tools and services
  • discuss and assess requests and enhancements

While interacting with the VO managers, VO Services staff can involve different experts in the discussions, and bridge VO managers requests with the appropriate support units. Through the interactions with the VO managers, different requirements can also emerge. In this case, it is the VO Services Support Unit responsibility to summarize the discussion into a requirement, and insert it in EGI Request Tracker. USCT is then responsible for the requirement follow up.

Also, through this subtask, the VO Service staff is responsible for evaluating and gathering documentation found relevant for VO managers daily activities. All communities (VRCs, VOs, NGIs, ...) are invited to recommend topics about which new materials should be prepared (manuals, presentations, etc...) or even to share their own materials about VO management related topics. Please contact the VO Service team via its GGUS support unit, or directly via vo-services@mailman.egi.eu.


Documentation

The Relevant documentation for VO activities wiki page presents a snapshot of the current aggregated material.


Helpdesk for VO managers

A first line service to support VO managers is consolidated through the GGUS VO Services Support Unit. Through this service, the VO managers can:

  • ask for support while operating their VO
  • ask for clarification of procedures
  • ask for documentation improvements

The VO Service team seats behind the GGUS VO Services Support Unit and is responsible for evaluating VO managers requests and treat them in a timely manner. Because this support unit is fully integrated in GGUS user support workflows, it is possible to reassign requests to other support units and/or involved other experts in the evaluation of a given request. For the previous reasons, the GGUS VO Services Support Unit represents the primordial point of contact for VO managers support. Nevertheless, the VO Services mailing list is also available through vo-services@mailman.egi.eu


Contacts

  • GGUS: VO Services Support Unit
  • Mailing list: vo-services@mailman.egi.eu