Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "VT VAPOR:VAPOR OpsPortal Engage"

From EGIWiki
Jump to navigation Jump to search
 
(24 intermediate revisions by the same user not shown)
Line 1: Line 1:
The Operations Portal has a long experience in terms of... (<span style="background:#ffffaa">je te laisse ajouter le laïus qui va bien sur l'histoire et les buts du portail</span>).
VAPOR has been developed to address VO operation tasks that were hardly covered by existing tools.
Although they are initially targeted to different users (resource centre administrators vs. VO managers), both tools have complementary goals in terms of monitoring and dashboard of resources. Besides, it occurs that their roadmaps are converging towards common goals, in particular regarding the management of user communities and the extension of their scope to cloud resources. Finally, VAPOR has been developed in close collaboration with the Operations Portal team, and relies on the same technologies.
Therefore, it is proposed to extend the scope of both tools to cloud resources while integrating VAPOR more closely into the Operations Portal. Thus, the Operations Portal will become a one-stop tool for resource operations and user community management, either from the resource centre perspective of from the VO perspective.
== User community management ==
== User community management ==


The User Database implemented in the Operations Portal retrieves user information from the VOMS servers. It is proposed to extend this database as a tool dedicated to VO managers, to store user information besides the administrative data available in the VOMS. Three main goals are identified:
The User Database implemented in the Operations Portal retrieves user information from the VOMS servers. It is proposed to extend this database to be a one-stop repository for (i) information about grid and cloud users including users behind a robot certificate, and (ii) the management of user registration life-cycle.
* Manage and follow up on users registration life-cycle: registration (VO membership), group membership, membership expiration. The life cycle workflow integrates interactions with third party services such as the VOMS, file catalog, EGI Applications Database.
This user community management solution is dedicated primarily to VO managers. (<span style="background:#ffffaa">à voir de votre côté à qui vous dédiez d'abord ce type de service?</span>)
* Track information about users "hidden behind" a robot certificate, in order to have a realistic idea of the number of actual users in a VO.
* (Track information about scientific publications to encourage users to acknowledge the usage of EGI resources).
 
=== Users database ===


The User Database should store a least the following information:
The User Database is intended to store user information besides the administrative data available in the VOMS. The following information is envisaged:
* Administrative data (DN, email, affiliation, membership duration...).
* Administrative data (DN, email, affiliation, VO membership, group membership) in sync with the VOMS database.
* Free text field: VO administrators must be able to add free text information regarding user's research field, scientific collaborations, etc.  
* Research discipline classification referring to the [https://wiki.egi.eu/wiki/VT_Scientific_Discipline_Classification VT Scientific Discipline Classification].
* Research discipline classification referring to the [https://wiki.egi.eu/wiki/VT_Scientific_Discipline_Classification VT Scientific Discipline Classification].
* Scientific publications: keep track of published works using the infrastructure.
* Scientific publications: keep track of published works using the infrastructure and encourage users to acknowledge the usage of EGI resources.
* DN of robot certificate (if any): users behind a robot certificate may or may not have their own certificate. If they do not, it is important to be able to register them in the database anyway.
* Robot certificate DN (if any) in order to have a realistic idea of the number of actual users in a VO (may users have their own personal grid certificate or not).
* Scientific application used (linked with the EGI Applications Database)
* Scientific applications used linking to the EGI Applications Database.
* User File Catalog instance (if any) and base directory.
* User File Catalog instance (if any) and base directory.


Several collaborations may be considered there:
As much as possible, the life cycle workflow automates interactions with third party services such as the VOMS, file catalog, EGI Applications Database, mailing list system.
* [http://perun.metacentrum.cz/web/ Perun] user management system for the user life-cycle workflow management.
At registration time, the user is automatically requested to provide details about their activity, affiliation and the applications they use (link to the EGI Applications Database).
* The [https://wiki.egi.eu/wiki/VT_Scientific_Publications_Repository_Implementation_Resources VT Scientific Publications Repository Implementation Resources] to collect scientific publications base on resource usage acknowledgement.
A VO administrator approves or reject pending requests. On approval, the user's root directory is created in the file catalog.
 
After membership expiration, files of expired users are cleaned up after a grace period.
Periodically, robot certificate holders must be asked to enter information about real users in the system: at least a number of users, at most individual data (email, etc.). Exact content of this information is to be detailed with robot users.
 
=== User life cycle management ===


Currently, a new user registers on the VOMS of the VO he/she wishes to join. Then, the VOMS sends an email notification to the VO administrator who may ask the user to provide additional details about his/her activity and affiliation. Then, the VO administrator manually approves the request.<br>
Collaborations may be considered with the
[http://perun.metacentrum.cz/web/ Perun] user management system for the user life-cycle workflow management, and the [https://wiki.egi.eu/wiki/VT_Scientific_Publications_Repository_Implementation_Resources VT Scientific Publications Repository Implementation Resources] and OpenAIRE project to collect scientific publications based on resource usage acknowledgement.


The goal is to keep the VOMS as the service to initiate the subscription of a user.
== Monitoring of infrastructure resources ==
The features below automate the management of subscription requests received from the VOMS:
* '''Registration''': filter emails notifications sent by the VOMS to the VO administrators (no other API seems to be available that would e.g. allow application to register with VOMS notifications).
** Check the validity of the user's DN: no quotes or double-quotes allowed as they are not supported by some services in the infrastructure. In this case, automatically reply and ask the user to require a new certificate.
** Automatically send an email asking the user to provide details about his activity and affiliation, the applications they use (pointer to the EGI Applications Database).
** Display a list of pending subscriptions that the administrator can manually approve or reject. The action of the administrator is reflected on the VOMS server.
** Automatically create the user's root directory in the LCG File Catalog (LFC) with appropriate ACLs. ACLS may be manually relaxed later depending on user's needs.
** Link user with the existing applications that they use in the EGI Applications Database.
** Add user's email address to VO users mailing lists the VO users mailing list.
* '''Membership expiration''': filter email notifications sent by the VOMS, and trigger
** the cleaning up of files of expired users after a grace period (see Data Management features),
** the removal of the user's email address from the VO users mailing list.
* '''Mailing list management''':
** The integration with different mailing list systems should be studied starting with Mailman and GoogleGroups, to keep a mailing list in sync with the current list of users.
** An function must allow to simply export the list of email addresses of the VO users.


''Optional additional features'':
Currently, the Oeprations Portal and VAPOR provide complementary tools and views dedicated to either resource centres or VOs:
* The Operations Portal provides a resource distribution browser and dashboard both dedicated to resource centre administrators (<span style="background:#ffffaa">il faut sans doute ajouter un peu plus de détails</span>).
* VAPOR provides views dedicated to VO managers and support teams:
** The VO resource view provides a consolidated view of resources supporting a VO by crossing GOCDB and BDII data, and a specific view of resource not in production.
** Monitor the number of running vs. waiting jobs at each site with additional statistical reports.
** View of job success/error/time-out rates at each site, compute a white-list of best-performing sites from the VO perspective.
** VO Data Management features perform file catalog-based scan of storage elements (to monitor storage feeling up, handle decommissioning...), and consistency checks (cleanup of dark data/lost files).


* Show a summary of the activity on the VO management mailing list: filter out VOMS email notifications that are already taken care of, and display other messages so that someone deals with them.
It is proposed to extend the scope of both tools to cloud resources while integrating VAPOR more closely into the Operations Portal.
* Collect feedback on the infrastructure, and scientific production (publications). For this purposes different alternatives may be considered:
* Operations Portal (<span style="background:#ffffaa">j'ai copié ça depuis les slides de notre présentation commune, je te laisse compléter/consolider avec le reste</span>):
** Use an existing platform such as Google Schoolar or ResearchGate, and add the publications from the members of the VO after requesting their consent to a specific join profile. This will create H-index factors immediately and it can be easily referenced and shared.  
** Extends the resource distribution browser to cloud resources, give more details (OS , number of cores , capacity), replace some Gstat feature, provide an API.
** Create a database with the entries in different databases and directly link to them. This will be more tedious but less prone to issues due to changes on the interfaces or procedures in the third party tools.
** Extend the dashboard to monitor cloud infrastructures.
* VAPOR:
** Extend the VO resource view to report the status and characteristics of cloud resource providers.
** Monitor the number of running VMs vs. number of VM creation requests at each cloud site.
** Monitor the VM creation success/error/time-out rates at each cloud site, compute a white-list of best-performing cloud sites from the VO perspective.
** Extend VO Data Management features to deal with storage available for a VO at cloud sites, add support for file catalog solutions that will be adopted by the EGI Federated Cloud.

Latest revision as of 11:24, 18 July 2014

The Operations Portal has a long experience in terms of... (je te laisse ajouter le laïus qui va bien sur l'histoire et les buts du portail). VAPOR has been developed to address VO operation tasks that were hardly covered by existing tools.

Although they are initially targeted to different users (resource centre administrators vs. VO managers), both tools have complementary goals in terms of monitoring and dashboard of resources. Besides, it occurs that their roadmaps are converging towards common goals, in particular regarding the management of user communities and the extension of their scope to cloud resources. Finally, VAPOR has been developed in close collaboration with the Operations Portal team, and relies on the same technologies.

Therefore, it is proposed to extend the scope of both tools to cloud resources while integrating VAPOR more closely into the Operations Portal. Thus, the Operations Portal will become a one-stop tool for resource operations and user community management, either from the resource centre perspective of from the VO perspective.

User community management

The User Database implemented in the Operations Portal retrieves user information from the VOMS servers. It is proposed to extend this database to be a one-stop repository for (i) information about grid and cloud users including users behind a robot certificate, and (ii) the management of user registration life-cycle. This user community management solution is dedicated primarily to VO managers. (à voir de votre côté à qui vous dédiez d'abord ce type de service?)

The User Database is intended to store user information besides the administrative data available in the VOMS. The following information is envisaged:

  • Administrative data (DN, email, affiliation, VO membership, group membership) in sync with the VOMS database.
  • Research discipline classification referring to the VT Scientific Discipline Classification.
  • Scientific publications: keep track of published works using the infrastructure and encourage users to acknowledge the usage of EGI resources.
  • Robot certificate DN (if any) in order to have a realistic idea of the number of actual users in a VO (may users have their own personal grid certificate or not).
  • Scientific applications used linking to the EGI Applications Database.
  • User File Catalog instance (if any) and base directory.

As much as possible, the life cycle workflow automates interactions with third party services such as the VOMS, file catalog, EGI Applications Database, mailing list system. At registration time, the user is automatically requested to provide details about their activity, affiliation and the applications they use (link to the EGI Applications Database). A VO administrator approves or reject pending requests. On approval, the user's root directory is created in the file catalog. After membership expiration, files of expired users are cleaned up after a grace period.

Collaborations may be considered with the Perun user management system for the user life-cycle workflow management, and the VT Scientific Publications Repository Implementation Resources and OpenAIRE project to collect scientific publications based on resource usage acknowledgement.

Monitoring of infrastructure resources

Currently, the Oeprations Portal and VAPOR provide complementary tools and views dedicated to either resource centres or VOs:

  • The Operations Portal provides a resource distribution browser and dashboard both dedicated to resource centre administrators (il faut sans doute ajouter un peu plus de détails).
  • VAPOR provides views dedicated to VO managers and support teams:
    • The VO resource view provides a consolidated view of resources supporting a VO by crossing GOCDB and BDII data, and a specific view of resource not in production.
    • Monitor the number of running vs. waiting jobs at each site with additional statistical reports.
    • View of job success/error/time-out rates at each site, compute a white-list of best-performing sites from the VO perspective.
    • VO Data Management features perform file catalog-based scan of storage elements (to monitor storage feeling up, handle decommissioning...), and consistency checks (cleanup of dark data/lost files).

It is proposed to extend the scope of both tools to cloud resources while integrating VAPOR more closely into the Operations Portal.

  • Operations Portal (j'ai copié ça depuis les slides de notre présentation commune, je te laisse compléter/consolider avec le reste):
    • Extends the resource distribution browser to cloud resources, give more details (OS , number of cores , capacity), replace some Gstat feature, provide an API.
    • Extend the dashboard to monitor cloud infrastructures.
  • VAPOR:
    • Extend the VO resource view to report the status and characteristics of cloud resource providers.
    • Monitor the number of running VMs vs. number of VM creation requests at each cloud site.
    • Monitor the VM creation success/error/time-out rates at each cloud site, compute a white-list of best-performing cloud sites from the VO perspective.
    • Extend VO Data Management features to deal with storage available for a VO at cloud sites, add support for file catalog solutions that will be adopted by the EGI Federated Cloud.
Retrieved from "https://wiki.egi.eu/w/index.php?title=VT_VAPOR:VAPOR_OpsPortal_Engage&oldid=68614"