Difference between revisions of "EGI-XSEDE:Collaboration"

From EGIWiki
Jump to: navigation, search
(Administrative)
 
(10 intermediate revisions by 3 users not shown)
Line 14: Line 14:
 
== Administrative  ==
 
== Administrative  ==
  
*Mailinglist:&nbsp; [mailto:egi-tis@psc.edu egi-tis@psc.edu] (managed by Sergiu Sanielevici <sergiu@psc.edu>)
+
*Mailinglist:&nbsp; [mailto:egi-xsede@xsede.org egi-xsede@xsede.org] (managed by Victor Hazlewood vhazlewo@utk.edu)  
*Meetings: [https://indico.egi.eu/indico/conferenceTimeTable.py?confId=1222 EGI-CF 2013], Telcon, [https://www.xsede.org/web/xsede13/schedule XSEDE13-BoF], [https://indico.egi.eu/indico/conferenceTimeTable.py?confId=1417#20130916 EGI TF 2013]<br>
+
*Meetings:  
 +
**[https://indico.egi.eu/indico/conferenceTimeTable.py?confId=1222 EGI-CF 2013]  
 +
**Telcon  
 +
**[https://www.xsede.org/web/xsede13/schedule XSEDE13-BoF]  
 +
**[https://indico.egi.eu/indico/conferenceTimeTable.py?confId=1417#20130916 EGI TF 2013] + [https://wiki.egi.eu/w/images/7/7f/17.09_EGI_XSEDE_OSG_meeting.doc minutes (doc])
 +
**[https://indico.egi.eu/indico/contributionDisplay.py?contribId=179&confId=1994 EGI CF 2014]
  
 
== Goal  ==
 
== Goal  ==
  
 
The goal of this collaboration is to identify and exchange best practices and solutions between the XSEDE and EGI e-infrastructures so they can operate more efficiently to serve scientists in the U.S. and Europe. The collaboration is focussed on four areas of work:  
 
The goal of this collaboration is to identify and exchange best practices and solutions between the XSEDE and EGI e-infrastructures so they can operate more efficiently to serve scientists in the U.S. and Europe. The collaboration is focussed on four areas of work:  
* Operation of e-infrastructure services
+
 
* Cloud services for science and education
+
*Operation of e-infrastructure services  
* Champions to engage with new users
+
*Cloud services for science and education  
* User support and joint use cases
+
*Champions to engage with new users  
 +
*User support and joint use cases
 +
 
 
Each area has a named contact point from EGI and from XSEDE. Further details about the work under these four areas are provided below.  
 
Each area has a named contact point from EGI and from XSEDE. Further details about the work under these four areas are provided below.  
  
 
=== Background information on XSEDE  ===
 
=== Background information on XSEDE  ===
The [https://www.xsede.org Extreme Science and Engineering Discovery Environment (XSEDE)] is the most advanced, powerful, and robust collection of integrated advanced digital resources and services in the world. It is a single virtual system that scientists can use to interactively share computing resources, data, and expertise. XSEDE is a five-year, $121-million project supported by the National Science Foundation. It replaces and expands on the NSF TeraGrid project.
+
 
 +
The [https://www.xsede.org Extreme Science and Engineering Discovery Environment (XSEDE)] is the most advanced, powerful, and robust collection of integrated advanced digital resources and services in the world. It is a single virtual system that scientists can use to interactively share computing resources, data, and expertise. XSEDE is a five-year, $121-million project supported by the National Science Foundation. It replaces and expands on the NSF TeraGrid project.  
  
 
=== Background information on EGI  ===
 
=== Background information on EGI  ===
The [http://www.egi.eu European Grid Infrastructure (EGI)] delivers integrated computing services to European researchers, driving innovation and enabling new solutions to answer the big questions of tomorrow. EGI is a federation of over 340 resource centres, set up to provide computing services and resources to European researchers and their international collaborators. EGI supports research collaborations of all sizes: from the large teams behind the Large Hadron Collider at CERN and Research Infrastructures in the ESFRI roadmap, to the individuals and small research groups that equally contribute to innovation in Europe.
 
  
<br>
+
The [http://www.egi.eu European Grid Infrastructure (EGI)] delivers integrated computing services to European researchers, driving innovation and enabling new solutions to answer the big questions of tomorrow. EGI is a federation of over 340 resource centres, set up to provide computing services and resources to European researchers and their international collaborators. EGI supports research collaborations of all sizes: from the large teams behind the Large Hadron Collider at CERN and Research Infrastructures in the ESFRI roadmap, to the individuals and small research groups that equally contribute to innovation in Europe.
 +
 
 +
<br>  
  
 
== Area 1: Operations  ==
 
== Area 1: Operations  ==
  
*EGI: Tiziana Ferrari <tiziana.ferrari@egi.eu>, Malgorzata Krakowian <malgorzata.krakowian@egi.eu>
+
*EGI: Malgorzata Krakowian &lt;malgorzata.krakowian@egi.eu&gt;
*XSEDE: Victor Hazlewood <victor@utk.edu>
+
*XSEDE: Victor Hazlewood &lt;victor@utk.edu&gt;
 +
 
 +
Goals:
 +
 
 +
#Organisational benchmarking - compare operational processes and services – so we can identify the good practices and learn these from each other
 +
#Helpdesk and ticket procedures (e.g. Escalation processes)
 +
#Resource monitoring
 +
#Use the multi-infrastructure use cases to decide what should be changed in the infrastructures and on which side (Compchem and WeNMR)
 +
#Integration of helpdesk and accounting to support communities that in the future will jointly use XSEDE and involve OSG for those communities like wenmr that will also consume OSG resources<br>
 +
 
 +
=== Actions<br>  ===
 +
 
 +
#Accounting (17.09.2013) on'''Rob Quick''': OSG will share with XSEDE information how they publish accounting data to EGI
 +
#Authentication (17.09.2013) on '''TBD''': identify the IGTF CAs that XSEDE needs to authorize
 +
#Authentication (17.09.2013) on '''TBD '''(XSEDE): get the XSEDE Operations Security team to add the identified CAs to the XSEDE accepted CAs list
 +
#Helpdesk on '''Małgorzata Krakowian''':&nbsp;organize a meeting between EGI&nbsp;helpdesk and XSEDE helpdesk to discusse the integration.
 +
 
 +
=== Documentation  ===
 +
 
 +
==== Helpdesk  ====
 +
 
 +
*The EGI Helpdesk [https://ggus.eu/pages/home.php GGUS]. It is based on Remedy, authentication of users is based on X.509 certificates, access through Shibboleth is being implemented. All users with a valid certificate released by a IGTF CA can have read access to all tickets. GGUS already supports an interface to RT, as various National Grid Initiatives have RT as local helpdesk system. [https://wiki.egi.eu/wiki/Category:FAQ_Interfaces_%28GGUS%29 GGUS Interfaces]
 +
 
 +
*The [https://ggus.eu/report/report_view.php GGUS Report Generator] is the system we are using to collect statistics about usage, distribution of tickets, time to respond and solve tickets etc.
 +
 
 +
*Plenty of GGUS documentation is available on-line: [https://ggus.eu/pages/docu.php GGUS Documentation]
 +
 
 +
*Escalation procedures. Various procedures are in place depending on the type of issue to be escalated. A few examples:
 +
**[https://wiki.egi.eu/wiki/FAQ_GGUS-Waiting-For-PT-Process Handling of tickets waiting for response from the supporters]
 +
**[https://wiki.egi.eu/wiki/FAQ_GGUS-Waiting-For-Submitter-Process Handling of tickets waiting for response from the submitter]
 +
 
 +
==== Accounting  ====
  
Goals:
+
The EGI accounting infrastructure is distributed with a central accounting database gathering usage records either directly from each individual resource centres or from national databases. The central accounting database is based on APEL. Accounting records are being collected for computing jobs that are successfully DONE. Storage and cloud accounting will be rolled to production in 2013 and are being tested.
#Organisational benchmarking - compare operational processes and services – so we can identify the good practices and learn these from each other
 
#Helpdesk and ticket procedures (e.g. Escalation processes)
 
#Resource monitoring
 
#Use the multi-infrastructure use cases to decide what should be changed in the infrastructures and on which side (Compchem and WeNMR)
 
#Integration of helpdesk and accounting to support communities that in the future will jointly use XSEDE and involve OSG for those communities like wenmr that will also consume OSG resources
 
  
 +
*[https://accounting.egi.eu/ EGI Accounting Portal]
 +
*[https://wiki.egi.eu/wiki/APEL APEL documentation]
  
===Documentation===
+
==== Monitoring  ====
  
====Helpdesk ====
+
EGI monitoring is distributed and is based on Service Availability Monitoring ([[SAM|SAM]]). Monitoring data is centrally gathered for access to historical information and for the computation of performance indicators (availability and reliability). SAM is installed by individual National Grid Initiatives, monitoring data is exchanged through messaging (ActiveMQ).  
* The EGI Helpdesk [https://ggus.eu/pages/home.php GGUS]. It is based on Remedy, authentication of users is based on X.509 certificates, access through Shibboleth is being implemented. All users with a valid certificate released by a IGTF CA can have read access to all tickets. GGUS already supports an interface to RT, as various National Grid Initiatives have RT as local helpdesk system. [https://wiki.egi.eu/wiki/Category:FAQ_Interfaces_%28GGUS%29 GGUS Interfaces]
 
  
* The [https://ggus.eu/report/report_view.php GGUS Report Generator] is the system we are using to collect statistics about usage, distribution of tickets, time to respond and solve tickets etc.  
+
*EGI Monitoring information can be consulted via [https://grid-monitoring.egi.eu/myegi MyEGI].
  
* Plenty of GGUS documentation is available on-line: [https://ggus.eu/pages/docu.php GGUS Documentation]
+
==== Procedures and procedures  ====
  
* Escalation procedures. Various procedures are in place depending on the type of issue to be escalated. A few examples:
+
*[[Operations Procedures|Operations Procedures]]
** [https://wiki.egi.eu/wiki/FAQ_GGUS-Waiting-For-PT-Process Handling of tickets waiting for response from the supporters]
+
*[[Operations Procedures#Security|Security Procedures]]  
** [https://wiki.egi.eu/wiki/FAQ_GGUS-Waiting-For-Submitter-Process Handling of tickets waiting for response from the submitter]
+
*[[SPG:Documents|Security policies]]
  
====Accounting ====
+
<br>
The EGI accounting infrastructure is distributed with a central accounting database gathering usage records either directly from each individual resource centres or from national databases. The central accounting database is based on APEL.
 
Accounting records are being collected for computing jobs that are successfully DONE. Storage and cloud accounting will be rolled to production in 2013 and are being tested.
 
* [https://accounting.egi.eu/ EGI Accounting Portal]
 
* [https://wiki.egi.eu/wiki/APEL APEL documentation]
 
  
====Monitoring====
+
==== Resource allocation  ====
EGI monitoring is distributed and is based on Service Availability Monitoring ([[SAM| SAM]]). Monitoring data is centrally gathered for access to historical information and for the computation of performance indicators (availability and reliability).
 
SAM is installed by individual National Grid Initiatives, monitoring data is exchanged through messaging (ActiveMQ).
 
  
* EGI Monitoring information can be consulted via [https://grid-monitoring.egi.eu/myegi MyEGI].
+
===== XSEDE  =====
  
==== Procedures and procedures ====
+
*Contacts: Ken Hackworth, XSEDE Allocations Manager; Dave Hart, POPS
* [[Operations_Procedures |Operations Procedures]]
 
* [[Operations_Procedures#Security |Security Procedures]]
 
* [[SPG:Documents |Security policies]]
 
  
 +
XSEDE's current allocations system, called "[https://portal.xsede.org/submit-request POPS]", already supports allocations of storage resources. In fact, POPS is general enough that it will allow to specify any kind of resource (and associated "billing unit") for allocations.
  
==== Resource allocation ====
+
"POPS 2.0", which is a re-engineering of the POPS system to disentangle it from some legacy decisions, re-work legacy code that dates back to the late 1990s (along with 15+ years of incremental changes), and better integrate it with other components of the XSEDE infrastructure. As part of that process we are designing POPS as "allocations software as a service" such that non-XSEDE client organizations could spin up their own allocations process within the XSEDE service, with minimal effort/investment for the client organization. We're still fairly early in the design and implementation stage, but if EGI might be interested in being a 'customer' of this XSEDE service, we should certainly talk.
  
===== XSEDE =====
+
*[https://www.xsede.org/web/guest/su-converter Service Unit]
* Contacts: Ken Hackworth, XSEDE Allocations Manager; Dave Hart, POPS
+
*[https://www.xsede.org/allocations XSEDE allocation documentation - procedure]
  
XSEDE's current allocations system, called "[https://portal.xsede.org/submit-request POPS]", already supports allocations of storage resources. In fact, POPS is general enough that it will allow to specify any kind of resource (and associated "billing unit") for allocations.
+
===== EGI  =====
  
"POPS 2.0", which is a re-engineering of the POPS system to disentangle it from some legacy decisions, re-work legacy code that dates back to the late 1990s (along with 15+ years of incremental changes), and better integrate it with other components of the XSEDE infrastructure.
+
*Tiziana Ferrari, Malgorzata Krakowian/EGI.eu
As part of that process we are designing POPS as "allocations software as a service" such that non-XSEDE client organizations could spin up their own allocations process within the XSEDE service, with minimal effort/investment for the client organization.
+
*Resources:
We're still fairly early in the design and implementation stage, but if EGI might be interested in being a 'customer' of this XSEDE service, we should certainly talk.
+
**[[Resource Allocation Task Force|Resource Allocation task force]]
 +
**Resource Allocation [[VT Resource Allocation Tools|tool development]], Leader: Tomasz Szepieniec/CYFRONET
  
* [https://www.xsede.org/web/guest/su-converter Service Unit]
+
<br>
* [https://www.xsede.org/allocations XSEDE allocation documentation - procedure]
 
  
===== EGI =====
+
== Area 2: Cloud  ==
* Tiziana Ferrari, Malgorzata Krakowian/EGI.eu
 
* Resources:
 
** [[Resource_Allocation_Task_Force|Resource Allocation task force]]
 
** Resource Allocation [[VT_Resource_Allocation_Tools| tool development]], Leader: Tomasz Szepieniec/CYFRONET
 
  
== Area 2: Cloud ==
+
*EGI: David Wallom &lt;david.wallom@oerc.ox.ac.uk&gt;
 +
*XSEDE: David Lifka &lt;lifka@cac.cornell.edu&gt;
  
*EGI: David Wallom <david.wallom@oerc.ox.ac.uk>
+
Goals:  
*XSEDE: David Lifka <lifka@cac.cornell.edu>
 
  
Goals:
+
#VMI preparation – collaboration on creating endorsed images of common software  
#VMI preparation – collaboration on creating endorsed images of common software
+
#Identify questions that the technical support teams will have to be able to answer  
#Identify questions that the technical support teams will have to be able to answer
 
 
#Hybrid cloud setup – internal cloud that is kept busy, outburst some load to external clouds when needed.
 
#Hybrid cloud setup – internal cloud that is kept busy, outburst some load to external clouds when needed.
  
 +
<br>
  
== Area 3: Champions ==
+
== Area 3: Champions ==
  
*EGI: Catherine Gater <catherine.gater@egi.eu>
+
*EGI: Sara Coelho &lt;sara.coelho@egi.eu&gt;
*XSEDE: Kay Hunt <kay@purdue.edu>
+
*XSEDE: Kay Hunt &lt;kay@purdue.edu&gt;
  
 +
== Area 4: User support  ==
  
== Area 4: User support ==
+
*EGI: Gergely Sipos &lt;gergely.sipos@egi.eu&gt;
 +
*XSEDE: Sergiu Sanielevici &lt;sergiu@psc.edu&gt;, Suresh Marru &lt;smarru@iu.edu&gt;
  
*EGI: Gergely Sipos <gergely.sipos@egi.eu>
+
Goals:  
*XSEDE: Sergiu Sanielevici <sergiu@psc.edu>, Suresh  Marru <smarru@iu.edu>
 
  
Goals:
+
#Support the implementation of the use cases that have been submitted to the [http://www.egi.eu/community/xsede-egi_cues.html 'Collaborative Use Examples' call]  
# Support the implementation of the use cases that have been submitted to the [http://www.egi.eu/community/xsede-egi_cues.html 'Collaborative Use Examples' call]
+
##[[CMMST-XSEDE-EGI|Computational Chemistry use case]] (CMMST): See related gateway activity below.
## [[CMMST-XSEDE-EGI | Computational Chemistry use case]]
+
##WeNRM use case: Active use of OSG by WeNMR achieved. WeNMR put the activities with XSEDE on hold.
## WeNRM use case
+
#Identify science gateways that would benefit from resources from XSEDE and EGI, and facilitate such integration activities.  
# Identify science gateways that would benefit from resources from XSEDE and EGI, and facilitate such integration activities.
+
##New gateway for CMMST based on SCI-BUS technology. Development of a setup is ongoing. Technical contacts: Peter Kacsuk (SCI-BUS), Andrew Grimshaw (XSEDE), Antonio Lagana (CMMST)
# Exchange information, best practiices and tools from the user support and technical outreach areas.
+
#Exchange information, best practices and tools from the user support and technical outreach areas.  
 +
##EGI Applications Database: http://appdb.egi.eu

Latest revision as of 14:38, 2 June 2014

Welcome to the page of the EGI-XSEDE collaboration


Administrative

Goal

The goal of this collaboration is to identify and exchange best practices and solutions between the XSEDE and EGI e-infrastructures so they can operate more efficiently to serve scientists in the U.S. and Europe. The collaboration is focussed on four areas of work:

  • Operation of e-infrastructure services
  • Cloud services for science and education
  • Champions to engage with new users
  • User support and joint use cases

Each area has a named contact point from EGI and from XSEDE. Further details about the work under these four areas are provided below.

Background information on XSEDE

The Extreme Science and Engineering Discovery Environment (XSEDE) is the most advanced, powerful, and robust collection of integrated advanced digital resources and services in the world. It is a single virtual system that scientists can use to interactively share computing resources, data, and expertise. XSEDE is a five-year, $121-million project supported by the National Science Foundation. It replaces and expands on the NSF TeraGrid project.

Background information on EGI

The European Grid Infrastructure (EGI) delivers integrated computing services to European researchers, driving innovation and enabling new solutions to answer the big questions of tomorrow. EGI is a federation of over 340 resource centres, set up to provide computing services and resources to European researchers and their international collaborators. EGI supports research collaborations of all sizes: from the large teams behind the Large Hadron Collider at CERN and Research Infrastructures in the ESFRI roadmap, to the individuals and small research groups that equally contribute to innovation in Europe.


Area 1: Operations

  • EGI: Malgorzata Krakowian <malgorzata.krakowian@egi.eu>
  • XSEDE: Victor Hazlewood <victor@utk.edu>

Goals:

  1. Organisational benchmarking - compare operational processes and services – so we can identify the good practices and learn these from each other
  2. Helpdesk and ticket procedures (e.g. Escalation processes)
  3. Resource monitoring
  4. Use the multi-infrastructure use cases to decide what should be changed in the infrastructures and on which side (Compchem and WeNMR)
  5. Integration of helpdesk and accounting to support communities that in the future will jointly use XSEDE and involve OSG for those communities like wenmr that will also consume OSG resources

Actions

  1. Accounting (17.09.2013) onRob Quick: OSG will share with XSEDE information how they publish accounting data to EGI
  2. Authentication (17.09.2013) on TBD: identify the IGTF CAs that XSEDE needs to authorize
  3. Authentication (17.09.2013) on TBD (XSEDE): get the XSEDE Operations Security team to add the identified CAs to the XSEDE accepted CAs list
  4. Helpdesk on Małgorzata Krakowian: organize a meeting between EGI helpdesk and XSEDE helpdesk to discusse the integration.

Documentation

Helpdesk

  • The EGI Helpdesk GGUS. It is based on Remedy, authentication of users is based on X.509 certificates, access through Shibboleth is being implemented. All users with a valid certificate released by a IGTF CA can have read access to all tickets. GGUS already supports an interface to RT, as various National Grid Initiatives have RT as local helpdesk system. GGUS Interfaces
  • The GGUS Report Generator is the system we are using to collect statistics about usage, distribution of tickets, time to respond and solve tickets etc.

Accounting

The EGI accounting infrastructure is distributed with a central accounting database gathering usage records either directly from each individual resource centres or from national databases. The central accounting database is based on APEL. Accounting records are being collected for computing jobs that are successfully DONE. Storage and cloud accounting will be rolled to production in 2013 and are being tested.

Monitoring

EGI monitoring is distributed and is based on Service Availability Monitoring (SAM). Monitoring data is centrally gathered for access to historical information and for the computation of performance indicators (availability and reliability). SAM is installed by individual National Grid Initiatives, monitoring data is exchanged through messaging (ActiveMQ).

  • EGI Monitoring information can be consulted via MyEGI.

Procedures and procedures


Resource allocation

XSEDE
  • Contacts: Ken Hackworth, XSEDE Allocations Manager; Dave Hart, POPS

XSEDE's current allocations system, called "POPS", already supports allocations of storage resources. In fact, POPS is general enough that it will allow to specify any kind of resource (and associated "billing unit") for allocations.

"POPS 2.0", which is a re-engineering of the POPS system to disentangle it from some legacy decisions, re-work legacy code that dates back to the late 1990s (along with 15+ years of incremental changes), and better integrate it with other components of the XSEDE infrastructure. As part of that process we are designing POPS as "allocations software as a service" such that non-XSEDE client organizations could spin up their own allocations process within the XSEDE service, with minimal effort/investment for the client organization. We're still fairly early in the design and implementation stage, but if EGI might be interested in being a 'customer' of this XSEDE service, we should certainly talk.

EGI


Area 2: Cloud

  • EGI: David Wallom <david.wallom@oerc.ox.ac.uk>
  • XSEDE: David Lifka <lifka@cac.cornell.edu>

Goals:

  1. VMI preparation – collaboration on creating endorsed images of common software
  2. Identify questions that the technical support teams will have to be able to answer
  3. Hybrid cloud setup – internal cloud that is kept busy, outburst some load to external clouds when needed.


Area 3: Champions

  • EGI: Sara Coelho <sara.coelho@egi.eu>
  • XSEDE: Kay Hunt <kay@purdue.edu>

Area 4: User support

  • EGI: Gergely Sipos <gergely.sipos@egi.eu>
  • XSEDE: Sergiu Sanielevici <sergiu@psc.edu>, Suresh Marru <smarru@iu.edu>

Goals:

  1. Support the implementation of the use cases that have been submitted to the 'Collaborative Use Examples' call
    1. Computational Chemistry use case (CMMST): See related gateway activity below.
    2. WeNRM use case: Active use of OSG by WeNMR achieved. WeNMR put the activities with XSEDE on hold.
  2. Identify science gateways that would benefit from resources from XSEDE and EGI, and facilitate such integration activities.
    1. New gateway for CMMST based on SCI-BUS technology. Development of a setup is ongoing. Technical contacts: Peter Kacsuk (SCI-BUS), Andrew Grimshaw (XSEDE), Antonio Lagana (CMMST)
  3. Exchange information, best practices and tools from the user support and technical outreach areas.
    1. EGI Applications Database: http://appdb.egi.eu