Difference between revisions of "EGI-DRIHM:Collaboration"

From EGIWiki
Jump to: navigation, search
(Task 2: setup of a science gateway and integrate with the 'DRIHM Repository')
(Administrative)
 
(9 intermediate revisions by 2 users not shown)
Line 16: Line 16:
 
*Coordinator: Gergely Sipos, Technical Outreach Manager, EGI.eu (gergely.sipos@egi.eu)
 
*Coordinator: Gergely Sipos, Technical Outreach Manager, EGI.eu (gergely.sipos@egi.eu)
 
*Start Date: 17. May 2013  
 
*Start Date: 17. May 2013  
*End Date: First phase to end by the 2nd DRIHM EC review (end of October)
+
*End Date: End of DRIHM project (31st March 2015)
 
*Mailinglist: egi-drihm @ mailman.egi.eu
 
*Mailinglist: egi-drihm @ mailman.egi.eu
 
*Meetings: [https://indico.egi.eu/indico/categoryDisplay.py?categId=109 Indico page] (slides and minutes from meetings)
 
*Meetings: [https://indico.egi.eu/indico/categoryDisplay.py?categId=109 Indico page] (slides and minutes from meetings)
 +
*Status of ongoing work is described at [http://www.drihm.eu/index.php/ict-status-roadmap DRIHM ICT Status page] (The page requires authentication. Email Christian Straube drihm@nm.ifi.lmu.de to request an account.)
 +
* Read more about the resources that participate in the DRIHM VO in the Testbed section: [[EGI-DRIHM:Testbed]]
  
== Goal ==
+
=== Goal ===
 
+
The goal of the collaboration is to setup a web based science gateway for the hydrometeorogycal community to enable them running pre-defined simulation workflows useing resources from the European Grid Infrastructure. To achieve this goal the collaboration has to integrate varioushydrometeorological models with EGI, setup of a science gateway, develop workflows in the science gateway, develop unique interfaces for the workflows to simpify their use by scientists.  
+
The goal of the EGI-DRIHM collaboration is to facilitate the setup of a web based science gateway for the hydrometeorogy community to enable them running pre-defined simulation workflows on resources from European e-infrastructures, including EGI. To achieve this goal the collaboration has to integrate various hydrometeorology models with e-infrastructures, setup a science gateway, develop and integrated workflows in the science gateway and develop unique interfaces for workflow users.
 
 
=== Details ===
 
* Demonstre the system at the 2nd DRIHM EC project review (Date: End of October)
 
* Demo goal 1: demonstrate that hydrometeorology models can run on EGI
 
* Demo goal 2: demonstrate that data transfer between models works in workflow/pipleline
 
* Demo goal 3: demonstrate the workflows can be hidden behind unique interfaces, simplifying access for scientists
 
* Workflow 1: WRF-NMM --> RainFARM --> DRIFT
 
* Workflow 2: WRF-NMM --> RainFARM --> RIBS
 
* Task 1: Integration of four models with EGI,
 
* Task 2: setup of a science gateway and integrate with the ‘DRIHM Repository’
 
* Task 3: development of workflows
 
* Task 4: development of unique interfaces for the workflows
 
  
=== Background information on the DRIHM project ===
+
=== DRIHM project ===
 
* [http://www.drihm.eu DRIHM] aims for facilitating the access to hydrometeorological data and models, the collaboration between meteorologists, hydrologists, and Earth science experts for accelerated scientific advances in hydrometeorological research (HMR).
 
* [http://www.drihm.eu DRIHM] aims for facilitating the access to hydrometeorological data and models, the collaboration between meteorologists, hydrologists, and Earth science experts for accelerated scientific advances in hydrometeorological research (HMR).
 
* Leader: Antonio Parodi (antonio.parodi @ cimafoundation.org)
 
* Leader: Antonio Parodi (antonio.parodi @ cimafoundation.org)
 
  
 
== Infrastructure ==
 
== Infrastructure ==
 
 
* The collaboration setup a new production VO on EGI: vo.drihm.eu.
 
* The collaboration setup a new production VO on EGI: vo.drihm.eu.
 
** Member list url: https://vomsmania.cnaf.infn.it:8443/voms/drihm.eu/services/VOMSAdmin?method=listMembers
 
** Member list url: https://vomsmania.cnaf.infn.it:8443/voms/drihm.eu/services/VOMSAdmin?method=listMembers
Line 50: Line 38:
 
** vosecurity at drihm.eu: Write to security representatives of the VO
 
** vosecurity at drihm.eu: Write to security representatives of the VO
 
** vomanagers at drihm.eu: Write to VO managers
 
** vomanagers at drihm.eu: Write to VO managers
* In the first phase of the collaboration only gLite CREAM sites of the VO will be used in the first phase. Jobs will be submitted through the WMS.
 
* In a later phase the GRAM5 sites of the VO can be also used through WMS. This requires the debugging of access to GRAM5 sites through the WMS (currently job submission through the WMS works to the Nikhef site, but not to the LRZ cluster). Daniele can work on this with the GRAM5 site admins and with WMS developers.
 
* In a later phase the LRZ supercomputer site from PRACE SuperMuc) can be used to run the WRF model. This requires the installation of GRAM5 onto SuperMuc, and the acceptance of a community account/robot certificate on the site. (To avoid that individual users will have to have certificates and allocations on the site). Matteo to work on this and report back to the email list.
 
* If the SuperMuc site becomes available for WRF, then the tools produced by the EGI-EUDAT-PRACE Pilots can be used in the workflows to transfer data between PRACE and EGI sites. (More info about the pilots: https://wiki.egi.eu/wiki/EGI_EUDAT_PRACE_collaboration.
 
* Read more about the resources that participate in the DRIHM VO in the Testbed section: [[EGI-DRIHM:Testbed]]
 
 
== Task 1 ==
 
=== WRF integration ===
 
* WRF is a parallel code, can run on HPC sites.
 
* WRF requirements for the sites:
 
** MPICH or  MPICH2 or OPENMPI
 
**160-240 processors, but the simulations can be designed also to run on 50-100 processors
 
**about 24 hours of physical simulation time, to an elapsed time of a single run of 144--96 hours (6--4 days).
 
**Temporary disk space (GB): 80 GB for input, output, which corresponds to 24 h physical time, saving the data every 30 minutes of physical time. 28 GB for restart files, saving files every 6 hours of physical time.
 
* Currently there are only two sites in the DRIHM VO where WRF successfully runs (on 40 CPUs), report by Daniele C. (The two sites are UNINA_EGEE and prague_cesnet_lcg2) Daniele to follow up the problems found with the other sites directly with the site managers. Report back to the list when additional sites become available for WRF.
 
 
=== RainFARM, DRIFT, RIBS integration ===
 
* RainFARM, DRIFT, RIBS are sequential codes, have to run in a parameter study style on HTC grid sites
 
* Hardware requirements have to be collected and added to the Wiki (action on Antonio and Fabio, who invite Luis)
 
 
== Task 2: setup of a science gateway and integrate with the 'DRIHM Repository' ==
 
* A gUSE 3.5.6 science gateway is already setup: [http://sci-bus.drihm.eu:8080/liferay-portal-6.1.0/]
 
* The DRIHM Repository has to be integrated with the portal (or with the individual workflows?). The Repository stores up to date executables for the models and the grid sites participating in the VO has to download these prior to model execution.
 
* LMU’s solution to be used as the Repository. (This is based on rsync). Person responsible: Nils from LMU
 
* CVMFS solution may be used later for the Repository, but this is condition of availability of this service on the sites. Tiziana to discuss CVMFS installation with the NGIs, report back to the email list. (It’s very unlikely that CVFMS would be production level on the sites by the DRIHM review.)
 
 
== Task 3: development of workflows ==
 
* Try WRF execution through the gUSE portal as a ‘one job workflow’ (Christian, Daniele C.) Inform us when this is done.
 
* Try RainFARM, DRIFT, RIBS execution through the gUSE portal as a ‘one job workflow’ (Antonio, Fabio, Luis, Daniele C.) Inform us when this is done.
 
* After the models run successfully individually through gUSE, integrate these together into the two target workflows.
 
* gUSE documentations for workflow developers (gUSE Manual Blue Series): http://guse.hu/documentation/how-to
 
 
== Task 4: development of unique interfaces for the workflows ==
 
* Daniele Dagostino already works on the portlets. Consult with Daniele C, Christian (WRF), Antonio, Fabio and Luis (RainFARM, DRIFT, RIBS) about what interfaces their models expect.
 

Latest revision as of 17:07, 26 March 2014

Welcome to the page of the EGI-DRIHM collaboration

EGI DRIHAM collaboration menu: Home Members Infrastructure Roadmap



Administrative

  • Coordinator: Gergely Sipos, Technical Outreach Manager, EGI.eu (gergely.sipos@egi.eu)
  • Start Date: 17. May 2013
  • End Date: End of DRIHM project (31st March 2015)
  • Mailinglist: egi-drihm @ mailman.egi.eu
  • Meetings: Indico page (slides and minutes from meetings)
  • Status of ongoing work is described at DRIHM ICT Status page (The page requires authentication. Email Christian Straube drihm@nm.ifi.lmu.de to request an account.)
  • Read more about the resources that participate in the DRIHM VO in the Testbed section: EGI-DRIHM:Testbed

Goal

The goal of the EGI-DRIHM collaboration is to facilitate the setup of a web based science gateway for the hydrometeorogy community to enable them running pre-defined simulation workflows on resources from European e-infrastructures, including EGI. To achieve this goal the collaboration has to integrate various hydrometeorology models with e-infrastructures, setup a science gateway, develop and integrated workflows in the science gateway and develop unique interfaces for workflow users.

DRIHM project

  • DRIHM aims for facilitating the access to hydrometeorological data and models, the collaboration between meteorologists, hydrologists, and Earth science experts for accelerated scientific advances in hydrometeorological research (HMR).
  • Leader: Antonio Parodi (antonio.parodi @ cimafoundation.org)

Infrastructure