Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "EGI-XSEDE:Collaboration"

From EGIWiki
Jump to navigation Jump to search
 
(One intermediate revision by the same user not shown)
Line 14: Line 14:
== Administrative  ==
== Administrative  ==


*Mailinglist:  [mailto:egi-xsede@xsede.org egi-xsede@xsede.org] (managed by Victor Hazlewood vhazlewo@utk.edu)
*Mailinglist:  [mailto:egi-xsede@xsede.org egi-xsede@xsede.org] (managed by Victor Hazlewood vhazlewo@utk.edu)  
*Meetings: [https://indico.egi.eu/indico/conferenceTimeTable.py?confId=1222 EGI-CF 2013], Telcon, [https://www.xsede.org/web/xsede13/schedule XSEDE13-BoF], [https://indico.egi.eu/indico/conferenceTimeTable.py?confId=1417#20130916 EGI TF 2013] + [https://wiki.egi.eu/w/images/7/7f/17.09_EGI_XSEDE_OSG_meeting.doc minutes (doc)], <br> [https://indico.egi.eu/indico/contributionDisplay.py?contribId=179&confId=1994 EGI CF 2014]
*Meetings:  
**[https://indico.egi.eu/indico/conferenceTimeTable.py?confId=1222 EGI-CF 2013]  
**Telcon  
**[https://www.xsede.org/web/xsede13/schedule XSEDE13-BoF]  
**[https://indico.egi.eu/indico/conferenceTimeTable.py?confId=1417#20130916 EGI TF 2013] + [https://wiki.egi.eu/w/images/7/7f/17.09_EGI_XSEDE_OSG_meeting.doc minutes (doc])  
**[https://indico.egi.eu/indico/contributionDisplay.py?contribId=179&confId=1994 EGI CF 2014]


== Goal  ==
== Goal  ==


The goal of this collaboration is to identify and exchange best practices and solutions between the XSEDE and EGI e-infrastructures so they can operate more efficiently to serve scientists in the U.S. and Europe. The collaboration is focussed on four areas of work:  
The goal of this collaboration is to identify and exchange best practices and solutions between the XSEDE and EGI e-infrastructures so they can operate more efficiently to serve scientists in the U.S. and Europe. The collaboration is focussed on four areas of work:  
* Operation of e-infrastructure services
 
* Cloud services for science and education
*Operation of e-infrastructure services  
* Champions to engage with new users
*Cloud services for science and education  
* User support and joint use cases
*Champions to engage with new users  
*User support and joint use cases
 
Each area has a named contact point from EGI and from XSEDE. Further details about the work under these four areas are provided below.  
Each area has a named contact point from EGI and from XSEDE. Further details about the work under these four areas are provided below.  


=== Background information on XSEDE  ===
=== Background information on XSEDE  ===
The [https://www.xsede.org Extreme Science and Engineering Discovery Environment (XSEDE)] is the most advanced, powerful, and robust collection of integrated advanced digital resources and services in the world. It is a single virtual system that scientists can use to interactively share computing resources, data, and expertise. XSEDE is a five-year, $121-million project supported by the National Science Foundation. It replaces and expands on the NSF TeraGrid project.
 
The [https://www.xsede.org Extreme Science and Engineering Discovery Environment (XSEDE)] is the most advanced, powerful, and robust collection of integrated advanced digital resources and services in the world. It is a single virtual system that scientists can use to interactively share computing resources, data, and expertise. XSEDE is a five-year, $121-million project supported by the National Science Foundation. It replaces and expands on the NSF TeraGrid project.  


=== Background information on EGI  ===
=== Background information on EGI  ===
The [http://www.egi.eu European Grid Infrastructure (EGI)] delivers integrated computing services to European researchers, driving innovation and enabling new solutions to answer the big questions of tomorrow. EGI is a federation of over 340 resource centres, set up to provide computing services and resources to European researchers and their international collaborators. EGI supports research collaborations of all sizes: from the large teams behind the Large Hadron Collider at CERN and Research Infrastructures in the ESFRI roadmap, to the individuals and small research groups that equally contribute to innovation in Europe.


<br>
The [http://www.egi.eu European Grid Infrastructure (EGI)] delivers integrated computing services to European researchers, driving innovation and enabling new solutions to answer the big questions of tomorrow. EGI is a federation of over 340 resource centres, set up to provide computing services and resources to European researchers and their international collaborators. EGI supports research collaborations of all sizes: from the large teams behind the Large Hadron Collider at CERN and Research Infrastructures in the ESFRI roadmap, to the individuals and small research groups that equally contribute to innovation in Europe.
 
<br>  


== Area 1: Operations  ==
== Area 1: Operations  ==


*EGI: Tiziana Ferrari &lt;tiziana.ferrari@egi.eu&gt;, Malgorzata Krakowian &lt;malgorzata.krakowian@egi.eu&gt;  
*EGI: Malgorzata Krakowian &lt;malgorzata.krakowian@egi.eu&gt;  
*XSEDE: Victor Hazlewood &lt;victor@utk.edu&gt;
*XSEDE: Victor Hazlewood &lt;victor@utk.edu&gt;


Line 45: Line 54:
#Resource monitoring  
#Resource monitoring  
#Use the multi-infrastructure use cases to decide what should be changed in the infrastructures and on which side (Compchem and WeNMR)  
#Use the multi-infrastructure use cases to decide what should be changed in the infrastructures and on which side (Compchem and WeNMR)  
#Integration of helpdesk and accounting to support communities that in the future will jointly use XSEDE and involve OSG for those communities like wenmr that will also consume OSG resources
#Integration of helpdesk and accounting to support communities that in the future will jointly use XSEDE and involve OSG for those communities like wenmr that will also consume OSG resources<br>


=== Actions<br>  ===
=== Actions<br>  ===


#Accounting (17.09.2013)on Rob Quick: OSG will share with XSEDE information how they publish accounting data to EGI  
#Accounting (17.09.2013) on'''Rob Quick''': OSG will share with XSEDE information how they publish accounting data to EGI  
#[DONE] Support (17.09.2013)on Małgorzata Krakowian: Send EGI Helpdesk team contact to Victor Hazlewood
#Authentication (17.09.2013) on '''TBD''': identify the IGTF CAs that XSEDE needs to authorize  
#Support (17.09.2013)on Alessandro Costantini: review WeNMR ticketing workflow if it meets Compchem requirements
#Authentication (17.09.2013) on '''TBD '''(XSEDE): get the XSEDE Operations Security team to add the identified CAs to the XSEDE accepted CAs list  
#Support (17.09.2013)on Alessandro Costantini and Marco Verlato: Describe ticketing use case (tools + procedures) which are now in place
#Helpdesk on '''Małgorzata Krakowian''':&nbsp;organize a meeting between EGI&nbsp;helpdesk and XSEDE helpdesk to discusse the integration.
#Authentication (17.09.2013) on TBD: identify the IGTF CAs that XSEDE needs to authorize  
#Authentication (17.09.2013) on TBD (XSEDE): get the XSEDE Operations Security team to add the identified CAs to the XSEDE accepted CAs list


=== Documentation ===
=== Documentation ===


==== Helpdesk  ====
==== Helpdesk  ====
Line 77: Line 84:
*[https://wiki.egi.eu/wiki/APEL APEL documentation]
*[https://wiki.egi.eu/wiki/APEL APEL documentation]


==== Monitoring ====
==== Monitoring ====


EGI monitoring is distributed and is based on Service Availability Monitoring ([[SAM|SAM]]). Monitoring data is centrally gathered for access to historical information and for the computation of performance indicators (availability and reliability). SAM is installed by individual National Grid Initiatives, monitoring data is exchanged through messaging (ActiveMQ).  
EGI monitoring is distributed and is based on Service Availability Monitoring ([[SAM|SAM]]). Monitoring data is centrally gathered for access to historical information and for the computation of performance indicators (availability and reliability). SAM is installed by individual National Grid Initiatives, monitoring data is exchanged through messaging (ActiveMQ).  
Line 111: Line 118:
**Resource Allocation [[VT Resource Allocation Tools|tool development]], Leader: Tomasz Szepieniec/CYFRONET
**Resource Allocation [[VT Resource Allocation Tools|tool development]], Leader: Tomasz Szepieniec/CYFRONET


<br>
<br>  


== Area 2: Cloud ==
== Area 2: Cloud ==


*EGI: David Wallom <david.wallom@oerc.ox.ac.uk>
*EGI: David Wallom &lt;david.wallom@oerc.ox.ac.uk&gt;
*XSEDE: David Lifka <lifka@cac.cornell.edu>
*XSEDE: David Lifka &lt;lifka@cac.cornell.edu&gt;


Goals:
Goals:  
#VMI preparation – collaboration on creating endorsed images of common software
 
#Identify questions that the technical support teams will have to be able to answer
#VMI preparation – collaboration on creating endorsed images of common software  
#Identify questions that the technical support teams will have to be able to answer  
#Hybrid cloud setup – internal cloud that is kept busy, outburst some load to external clouds when needed.
#Hybrid cloud setup – internal cloud that is kept busy, outburst some load to external clouds when needed.


<br>


== Area 3: Champions ==
== Area 3: Champions ==


*EGI: Sara Coelho <sara.coelho@egi.eu>
*EGI: Sara Coelho &lt;sara.coelho@egi.eu&gt;
*XSEDE: Kay Hunt <kay@purdue.edu>
*XSEDE: Kay Hunt &lt;kay@purdue.edu&gt;


== Area 4: User support ==
== Area 4: User support ==


*EGI: Gergely Sipos <gergely.sipos@egi.eu>
*EGI: Gergely Sipos &lt;gergely.sipos@egi.eu&gt;
*XSEDE: Sergiu Sanielevici <sergiu@psc.edu>, Suresh Marru <smarru@iu.edu>
*XSEDE: Sergiu Sanielevici &lt;sergiu@psc.edu&gt;, Suresh Marru &lt;smarru@iu.edu&gt;
 
Goals:


Goals:
#Support the implementation of the use cases that have been submitted to the [http://www.egi.eu/community/xsede-egi_cues.html 'Collaborative Use Examples' call]  
# Support the implementation of the use cases that have been submitted to the [http://www.egi.eu/community/xsede-egi_cues.html 'Collaborative Use Examples' call]
##[[CMMST-XSEDE-EGI|Computational Chemistry use case]] (CMMST): See related gateway activity below.  
## [[CMMST-XSEDE-EGI | Computational Chemistry use case]] (CMMST): See related gateway activity below.
##WeNRM use case: Active use of OSG by WeNMR achieved. WeNMR put the activities with XSEDE on hold.  
## WeNRM use case: Active use of OSG by WeNMR achieved. WeNMR put the activities with XSEDE on hold.  
#Identify science gateways that would benefit from resources from XSEDE and EGI, and facilitate such integration activities.  
# Identify science gateways that would benefit from resources from XSEDE and EGI, and facilitate such integration activities.
##New gateway for CMMST based on SCI-BUS technology. Development of a setup is ongoing. Technical contacts: Peter Kacsuk (SCI-BUS), Andrew Grimshaw (XSEDE), Antonio Lagana (CMMST)  
## New gateway for CMMST based on SCI-BUS technology. Development of a setup is ongoing. Technical contacts: Peter Kacsuk (SCI-BUS), Andrew Grimshaw (XSEDE), Antonio Lagana (CMMST)
#Exchange information, best practices and tools from the user support and technical outreach areas.  
# Exchange information, best practices and tools from the user support and technical outreach areas.
##EGI Applications Database: http://appdb.egi.eu
## EGI Applications Database: http://appdb.egi.eu

Latest revision as of 13:38, 2 June 2014

Welcome to the page of the EGI-XSEDE collaboration


Administrative

Goal

The goal of this collaboration is to identify and exchange best practices and solutions between the XSEDE and EGI e-infrastructures so they can operate more efficiently to serve scientists in the U.S. and Europe. The collaboration is focussed on four areas of work:

  • Operation of e-infrastructure services
  • Cloud services for science and education
  • Champions to engage with new users
  • User support and joint use cases

Each area has a named contact point from EGI and from XSEDE. Further details about the work under these four areas are provided below.

Background information on XSEDE

The Extreme Science and Engineering Discovery Environment (XSEDE) is the most advanced, powerful, and robust collection of integrated advanced digital resources and services in the world. It is a single virtual system that scientists can use to interactively share computing resources, data, and expertise. XSEDE is a five-year, $121-million project supported by the National Science Foundation. It replaces and expands on the NSF TeraGrid project.

Background information on EGI

The European Grid Infrastructure (EGI) delivers integrated computing services to European researchers, driving innovation and enabling new solutions to answer the big questions of tomorrow. EGI is a federation of over 340 resource centres, set up to provide computing services and resources to European researchers and their international collaborators. EGI supports research collaborations of all sizes: from the large teams behind the Large Hadron Collider at CERN and Research Infrastructures in the ESFRI roadmap, to the individuals and small research groups that equally contribute to innovation in Europe.


Area 1: Operations

  • EGI: Malgorzata Krakowian <malgorzata.krakowian@egi.eu>
  • XSEDE: Victor Hazlewood <victor@utk.edu>

Goals:

  1. Organisational benchmarking - compare operational processes and services – so we can identify the good practices and learn these from each other
  2. Helpdesk and ticket procedures (e.g. Escalation processes)
  3. Resource monitoring
  4. Use the multi-infrastructure use cases to decide what should be changed in the infrastructures and on which side (Compchem and WeNMR)
  5. Integration of helpdesk and accounting to support communities that in the future will jointly use XSEDE and involve OSG for those communities like wenmr that will also consume OSG resources

Actions

  1. Accounting (17.09.2013) onRob Quick: OSG will share with XSEDE information how they publish accounting data to EGI
  2. Authentication (17.09.2013) on TBD: identify the IGTF CAs that XSEDE needs to authorize
  3. Authentication (17.09.2013) on TBD (XSEDE): get the XSEDE Operations Security team to add the identified CAs to the XSEDE accepted CAs list
  4. Helpdesk on Małgorzata Krakowian: organize a meeting between EGI helpdesk and XSEDE helpdesk to discusse the integration.

Documentation

Helpdesk

  • The EGI Helpdesk GGUS. It is based on Remedy, authentication of users is based on X.509 certificates, access through Shibboleth is being implemented. All users with a valid certificate released by a IGTF CA can have read access to all tickets. GGUS already supports an interface to RT, as various National Grid Initiatives have RT as local helpdesk system. GGUS Interfaces
  • The GGUS Report Generator is the system we are using to collect statistics about usage, distribution of tickets, time to respond and solve tickets etc.

Accounting

The EGI accounting infrastructure is distributed with a central accounting database gathering usage records either directly from each individual resource centres or from national databases. The central accounting database is based on APEL. Accounting records are being collected for computing jobs that are successfully DONE. Storage and cloud accounting will be rolled to production in 2013 and are being tested.

Monitoring

EGI monitoring is distributed and is based on Service Availability Monitoring (SAM). Monitoring data is centrally gathered for access to historical information and for the computation of performance indicators (availability and reliability). SAM is installed by individual National Grid Initiatives, monitoring data is exchanged through messaging (ActiveMQ).

  • EGI Monitoring information can be consulted via MyEGI.

Procedures and procedures


Resource allocation

XSEDE
  • Contacts: Ken Hackworth, XSEDE Allocations Manager; Dave Hart, POPS

XSEDE's current allocations system, called "POPS", already supports allocations of storage resources. In fact, POPS is general enough that it will allow to specify any kind of resource (and associated "billing unit") for allocations.

"POPS 2.0", which is a re-engineering of the POPS system to disentangle it from some legacy decisions, re-work legacy code that dates back to the late 1990s (along with 15+ years of incremental changes), and better integrate it with other components of the XSEDE infrastructure. As part of that process we are designing POPS as "allocations software as a service" such that non-XSEDE client organizations could spin up their own allocations process within the XSEDE service, with minimal effort/investment for the client organization. We're still fairly early in the design and implementation stage, but if EGI might be interested in being a 'customer' of this XSEDE service, we should certainly talk.

EGI


Area 2: Cloud

  • EGI: David Wallom <david.wallom@oerc.ox.ac.uk>
  • XSEDE: David Lifka <lifka@cac.cornell.edu>

Goals:

  1. VMI preparation – collaboration on creating endorsed images of common software
  2. Identify questions that the technical support teams will have to be able to answer
  3. Hybrid cloud setup – internal cloud that is kept busy, outburst some load to external clouds when needed.


Area 3: Champions

  • EGI: Sara Coelho <sara.coelho@egi.eu>
  • XSEDE: Kay Hunt <kay@purdue.edu>

Area 4: User support

  • EGI: Gergely Sipos <gergely.sipos@egi.eu>
  • XSEDE: Sergiu Sanielevici <sergiu@psc.edu>, Suresh Marru <smarru@iu.edu>

Goals:

  1. Support the implementation of the use cases that have been submitted to the 'Collaborative Use Examples' call
    1. Computational Chemistry use case (CMMST): See related gateway activity below.
    2. WeNRM use case: Active use of OSG by WeNMR achieved. WeNMR put the activities with XSEDE on hold.
  2. Identify science gateways that would benefit from resources from XSEDE and EGI, and facilitate such integration activities.
    1. New gateway for CMMST based on SCI-BUS technology. Development of a setup is ongoing. Technical contacts: Peter Kacsuk (SCI-BUS), Andrew Grimshaw (XSEDE), Antonio Lagana (CMMST)
  3. Exchange information, best practices and tools from the user support and technical outreach areas.
    1. EGI Applications Database: http://appdb.egi.eu