Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

EGI-DRIHM:Collaboration

From EGIWiki
Revision as of 23:51, 24 July 2013 by Sipos (talk | contribs) (→‎Details)
Jump to navigation Jump to search
Welcome to the page of the EGI-DRIHM collaboration

EGI DRIHAM collaboration menu: Home Members Infrastructure Roadmap



Administrative

  • Coordinator: Gergely Sipos, Technical Outreach Manager, EGI.eu (gergely.sipos@egi.eu)
  • Start Date: 17. May 2013
  • End Date: First phase to end by the 2nd DRIHM EC review (end of October)
  • Mailinglist: egi-drihm @ mailman.egi.eu
  • Meetings: Indico page (slides and minutes from meetings)

Goal

The goal of the collaboration is to setup a web based science gateway for the hydrometeorogycal community to enable them running pre-defined simulation workflows useing resources from the European Grid Infrastructure. To achieve this goal the collaboration has to integrate varioushydrometeorological models with EGI, setup of a science gateway, develop workflows in the science gateway, develop unique interfaces for the workflows to simpify their use by scientists.

Details

  • Demonstre the system at the 2nd DRIHM EC project review (Date: End of October)
  • Demo goal 1: demonstrate that hydrometeorology models can run on EGI
  • Demo goal 2: demonstrate that data transfer between models works in workflow/pipleline
  • Demo goal 3: demonstrate the workflows can be hidden behind unique interfaces, simplifying access for scientists
  • Models: WRF-NMM, RainFARM, DRIFT, RIBS
  • Workflow 1: WRF-NMM --> RainFARM --> DRIFT
  • Workflow 2: WRF-NMM --> RainFARM --> RIBS
  • Task 1: Integration of four models with EGI,
  • Task 2: setup of a science gateway and integrate with the ‘DRIHM Repository’
  • Task 3: development of workflows
  • Task 4: development of unique interfaces for the workflows

Background information on the DRIHM project

  • DRIHM aims for facilitating the access to hydrometeorological data and models, the collaboration between meteorologists, hydrologists, and Earth science experts for accelerated scientific advances in hydrometeorological research (HMR).
  • Leader: Antonio Parodi (antonio.parodi @ cimafoundation.org)


Infrastructure

  • The collaboration setup a new production VO on EGI: vo.drihm.eu.
  • In the first phase of the collaboration only gLite CREAM sites of the VO will be used in the first phase. Jobs will be submitted through the WMS.
  • In a later phase the GRAM5 sites of the VO can be also used through WMS. This requires the debugging of access to GRAM5 sites through the WMS (currently job submission through the WMS works to the Nikhef site, but not to the LRZ cluster). Daniele can work on this with the GRAM5 site admins and with WMS developers.
  • In a later phase the LRZ supercomputer site from PRACE SuperMuc) can be used to run the WRF model. This requires the installation of GRAM5 onto SuperMuc, and the acceptance of a community account/robot certificate on the site. (To avoid that individual users will have to have certificates and allocations on the site). Matteo to work on this and report back to the email list.
  • If the SuperMuc site becomes available for WRF, then the tools produced by the EGI-EUDAT-PRACE Pilots can be used in the workflows to transfer data between PRACE and EGI sites. (More info about the pilots: https://wiki.egi.eu/wiki/EGI_EUDAT_PRACE_collaboration.
  • Read more about the resources that participate in the DRIHM VO in the Testbed section: EGI-DRIHM:Testbed

Task 1

WRF integration

  • WRF is a parallel code, can run on HPC sites.
  • WRF requirements for the sites:
    • MPICH or MPICH2 or OPENMPI
    • 160-240 processors, but the simulations can be designed also to run on 50-100 processors
    • about 24 hours of physical simulation time, to an elapsed time of a single run of 144--96 hours (6--4 days).
    • Temporary disk space (GB): 80 GB for input, output, which corresponds to 24 h physical time, saving the data every 30 minutes of physical time. 28 GB for restart files, saving files every 6 hours of physical time.
  • Currently there are only two sites in the DRIHM VO where WRF successfully runs (on 40 CPUs), report by Daniele C. (The two sites are UNINA_EGEE and prague_cesnet_lcg2) Daniele to follow up the problems found with the other sites directly with the site managers. Report back to the list when additional sites become available for WRF.

RainFARM, DRIFT, RIBS integration

  • RainFARM, DRIFT, RIBS are sequential codes, have to run in a parameter study style on HTC grid sites
  • Hardware requirements have to be collected and added to the Wiki (action on Antonio and Fabio, who invite Luis)

Task 2: setup of a science gateway and integrate with the 'DRIHM Repository'

  • A gUSE 3.5.6 science gateway is already setup: http://sci-bus.drihm.eu:8080/liferay-portal-6.1.0/
  • The "DRIHM Repository" stores up to date executables for the models and the grid sites participating in the VO has to download these prior to model execution. The binaries are downloaded during the job preface (based on rsync), which is triggered by the job description. Person responsible: Nils from LMU
  • CVMFS solution may be used later for the Repository, but this is condition of availability of this service on the sites. Tiziana to discuss CVMFS installation with the NGIs, report back to the email list. (It’s very unlikely that CVFMS would be production level on the sites by the DRIHM review.)

Task 3: development of workflows

  • Try WRF execution through the gUSE portal as a ‘one job workflow’ (Christian, Daniele C.) Inform us when this is done.
  • Try RainFARM, DRIFT, RIBS execution through the gUSE portal as a ‘one job workflow’ (Antonio, Fabio, Luis, Daniele C.) Inform us when this is done.
  • After the models run successfully individually through gUSE, integrate these together into the two target workflows.
  • gUSE documentations for workflow developers (gUSE Manual Blue Series): http://guse.hu/documentation/how-to

Task 4: development of unique interfaces for the workflows

  • Daniele Dagostino already works on the portlets. Consult with Daniele C, Christian (WRF), Antonio, Fabio and Luis (RainFARM, DRIFT, RIBS) about what interfaces their models expect.