Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "EGI-DRIHM:Roadmap"

From EGIWiki
Jump to navigation Jump to search
 
(49 intermediate revisions by 6 users not shown)
Line 11: Line 11:


{{EGI-DRIHM_menubar}}
{{EGI-DRIHM_menubar}}
{{TOC_right}}
[[Category:EGI_DRIHM]]


Note well. The detailed description of the computational and storage activity requirements to support DRIHM applications, both the highly parallel and less demanding ones, will be added to the wiki EGI-DRIHM collaboration wiki pages (June 15 - June 30 2013).
'''This page is outdated and not used any more. Progress is tracked on the [http://www.drihm.eu/index.php/ict-status-roadmap DRIHM ICT Status page] (The page requires authentication. Email Christian Straube drihm@nm.ifi.lmu.de to request an account.)'''


= DRIHM VO Support =
*'''June 15 - July 31 2013'''. Add the support for the DRIHM VO in the resources listed in the EGI testbed, including some Storage Elements to store the input and output data of the models and to test the file transfers to local resources (i.e. storage used by a portal).


*'''June 15 - July 31 2013'''. Enable the DRIHM VO in a couple of EGI scheduler service (i.e. gLite-WMS/LB), in order to test the facilities offered by these services (jobs bookeeping, rescheduling, resources matchmaking, Input/Output SandBox handling, etc.).
= Table of actions 2013  =


= Services for DRIHM =
{| class="wikitable" style="text-align: left;"
'''June 15 - June 30 2013'''. Other services needed are: a VOMS server (already available for DRIHM), a file catalogue (i.e. gLite-LFC) and a MyProxy server.
|-
| Nr.
| From
| To
| Involved
| Action
| Done
| Notes and progress
|-
| colspan="7" | '''DRIHM VO'''  
|-
| 1.
| June 15  
| June 30
| NGIs
| Enable DRIHM VO at sites listed in the testbed (computing and storage resources)
| No
| '''Below is the list of sites already supporting DRIHM VO:'''<br>
Serbia: <br> AEGIS01-IPB-SCL<br> <br> Greece:<br> GR-06-IASA<br> GR-10-UOI<br> HG-02-IASA<br> HG-03-AUTH<br> HG-04-CTI-CEID<br> HG-05-FORTH<br> HG-06-EKT<br> HG-08-Okeanos<br> <br> Italy:<br> IGI-BOLOGNA <br> NAPLES(UNINA-EGEE) <br><br> Germany:<br> LRZ<br> <br> Netherlands:<br> NIKHEF-ELPROD<br> <br> Czech Republic:<br> prague_cesnet_lcg2<br><br> Spain:<br> CETA-GRID <br><br> REF: [http://gstat2.grid.sinica.edu.tw/gstat/summary/VO/drihm.eu/ List of sites supporting VO DRIHM.EU] <br><br> '''Below is the list of sites planning to enable DRIHM VO soon:'''<br> Croatia:<br> 3 sites will support<br> <br> Italy:<br> Roma3 INFN<br> <br> Macedonia:<br> MK-03-FINKI
|-
| 2.
| June 15
| June 30
| NGIs
| Enable DRIHM VO in EGI schedulers
| YES
| in order to test the facilities offered by these services (jobs bookeeping, rescheduling, resources matchmaking, Input/Output SandBox handling, etc.).
|-
|
|
|
|
| gLite-WMS/LB
|
|
|-
| colspan="7" | '''DRIHM Services'''
|-
| 3.
| June 15
| June 30
| NGIs &amp; DRIHM
| valign="top" | WMS
|
| <span style="line-height: 19.90625px;">WMS:&nbsp;</span><br><span style="line-height: 19.90625px;">$ lcg-infosites --is lcg-bdii.cern.ch --vo drihm.eu wms</span><br><span style="line-height: 19.90625px;">https://graszode.nikhef.nl:7443/glite_wms_wmproxy_server</span><br><span style="line-height: 19.90625px;">https://gridrb.fe.infn.it:7443/glite_wms_wmproxy_server</span><br><span style="line-height: 19.90625px;">https://prod-wms-01.ct.infn.it:7443/glite_wms_wmproxy_server</span><br><span style="line-height: 19.90625px;">https://prod-wms-02.ct.infn.it:7443/glite_wms_wmproxy_server</span><br><span style="line-height: 19.90625px;">https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server</span><br><span style="line-height: 19.90625px;">https://wms-demo.ipb.ac.rs:7443/glite_wms_wmproxy_server</span><br><span style="line-height: 19.90625px;">https://wms.ipb.ac.rs:7443/glite_wms_wmproxy_server</span><br><span style="line-height: 19.90625px;">https://wms004.cnaf.infn.it:7443/glite_wms_wmproxy_server</span><br><span style="line-height: 19.90625px;">https://wms005.cnaf.infn.it:7443/glite_wms_wmproxy_server</span><br><span style="line-height: 19.90625px;">https://wms014.cnaf.infn.it:7443/glite_wms_wmproxy_server</span><br><span style="line-height: 19.90625px;">https://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server</span><br><span style="line-height: 19.90625px;">https://wms1.grid.cesnet.cz:7443/glite_wms_wmproxy_server</span><br> AND [https://wiki.egi.eu/wiki/Catch_All_Grid_Core_Services Catch All Services]
|-
| 3.
| June 15
| June 30
| NGIs &amp; DRIHM
| MyProxy
|
| VOMS: vomsmania.cnaf.infn.it
|-
| 3.
| June 15
| June 30
| NGIs &amp; DRIHM
| LFC
|
| <span style="line-height: 19.90625px;">LFC:&nbsp;</span><br><span style="line-height: 19.90625px;">$ lcg-infosites --is lcg-bdii.cern.ch --vo drihm.eu lfc</span><br><span style="line-height: 19.90625px;">lfc.ipb.ac.rs</span><br> AND [https://wiki.egi.eu/wiki/Catch_All_Grid_Core_Services Catch All Services]
|-
| 3.
| June 15
| June 30
| NGIs &amp; DRIHM
| VOMS  
|
| <span style="line-height: 19.90625px;">MyProxy: myproxy.hellasgrid.gr</span><br> AND [https://wiki.egi.eu/wiki/Catch_All_Grid_Core_Services Catch All Services]
|-
| colspan="7" | '''Porting'''
|-
| 3.
| June 15
| June 30
| NGIs &amp; DRIHM  
| WRF with MPICH2, GFORTRAN, NetCDF, GRIB2 deployment at NGIs.
|
| Porting of the demanding model (i.e. WRF) to the Grid Environment in collaboration with the EGI/NGIs user support teams for some selected "Critical Cases" and preliminary tests of the Grid submission/data management. (using NGI general purpose VO if DRIHM is not deployed yet)<br />
By Daniele Cesini:<br>
One of the use cases provided by CIMA with WRF on the Naples site (SL6 WNs), using 16 core on 8 differnet nodes with infiniband and OpenMPI, but the performance (*) were not so exiting,<br> The use case was the Genova flooding on 11-04-2011, 24hrs starting from 00:00, only the coarser domain, not the nested one.<br> If someone else is interested in trying this run, jdl, exe and prologue (with reference to the data stored in a Grid SE that should be world readable) are available here:[https://dl.dropboxusercontent.com/u/3347575/wrf-test.tgz%7C wrf-test.tgz]<br> (I didn't installed WRF at the site, everything it's hardcoded, but it can be used for a test with few changes in the script - I can provide more details if needed)<br>


= Porting =
Jerome from France provided the WRF configruation file with needed dependencis by DRIHM.<br>
Porting of the demanding model (i.e. WRF) to the Grid Environment in collaboration with the EGI/NGIs user support teams for some selected "Critical Cases" and preliminary tests of the Grid submission/data management. (using NGI general purpose VO if DRIHM is not deployed yet) '''15 June - 01 July 2013'''


= Define sites to support the demanding jobs =
'''No need to do software deployment as Christian reported that they will fetch all needed compiled libraries from DRIHM repo during the Grid job run time.'''
Identification of the testbed members willing to run demanding jobs, as the meteo models, with the indication of the maximum allowable parallelism degree. '''01 July - 15 August 2013'''


= Job submission testing =
*Test the job submission (direct to CE and via WMS/LB) and data management from a command line user interface for the most demanding models on the DRIHM testbed - Performance evaluation of the infrastructure resources and services. '''01 July - 31 Aug 2013'''


*Porting and preliminary tests for the submission of jobs and data management from a command line user interface of the less demanding applications (hydrologic and hydraulic models) - Performance evaluation of the infrastructure and investigation of the capabilities offered by the virtualization systems available within EGI that can be suitable to run these applications. '''01 July - 31 Aug 2013'''
|-
 
| colspan="7" | '''Identify sites for demanding jobs'''
*Preliminary test for the submission of jobs from the DRIHM sci-bus-based portal to the EGI testbed resources. '''01-30 September 2013'''
|-
 
| 4.
*Incremental setup and tests of the complete suite of DRIHM models on the testbed resources '''1 October 2013 - June 2014''' ?
| July 01
 
| August 15
= Production phase =
| NGIs
'''November 2013 ?  - end of the DRIHM project'''
| List of sites for demanding jobs (indicate maximum number of cores for paralell computing). Identification of the testbed members willing to run demanding jobs, as the meteo models, with the indication of the maximum allowable parallelism degree.
|
|
|-
| colspan="7" | ''Job submission testing'''
|-
| 5.
| July 01
| August 31
| DRIHM
| Testing DRIHM models at NGIs computing and storage resources (job submission of WRF models, data management using command line interface). Test the job submission (direct to CE and via WMS/LB) and data management from a command line user interface for the most demanding models on the DRIHM testbed - Performance evaluation of the infrastructure resources and services.
|
|
|-
| 5.
| July 01
| August 31
| DRIHM
| Porting and preliminary tests for the submission of jobs and data management from a command line user interface of the less demanding applications (hydrologic and hydraulic models) - Performance evaluation of the infrastructure and investigation of the capabilities offered by the virtualization systems available within EGI that can be suitable to run these applications.
|
|
|-
| 6.
| September 01  
| September 30
| DRIHM
| Testing DRIHM models at NGIs computing and storage resources (job submission of WRF models, data management using dedicated SCI-BUS portal). Preliminary test for the submission of jobs from the DRIHM sci-bus-based portal to the EGI testbed resources.
|
|
|}

Latest revision as of 16:12, 26 March 2014

Welcome to the page of the EGI-DRIHM collaboration

EGI DRIHAM collaboration menu: Home Members Infrastructure Roadmap


This page is outdated and not used any more. Progress is tracked on the DRIHM ICT Status page (The page requires authentication. Email Christian Straube drihm@nm.ifi.lmu.de to request an account.)


Table of actions 2013

Nr. From To Involved Action Done Notes and progress
DRIHM VO
1. June 15 June 30 NGIs Enable DRIHM VO at sites listed in the testbed (computing and storage resources) No Below is the list of sites already supporting DRIHM VO:

Serbia:
AEGIS01-IPB-SCL

Greece:
GR-06-IASA
GR-10-UOI
HG-02-IASA
HG-03-AUTH
HG-04-CTI-CEID
HG-05-FORTH
HG-06-EKT
HG-08-Okeanos

Italy:
IGI-BOLOGNA
NAPLES(UNINA-EGEE)

Germany:
LRZ

Netherlands:
NIKHEF-ELPROD

Czech Republic:
prague_cesnet_lcg2

Spain:
CETA-GRID

REF: List of sites supporting VO DRIHM.EU

Below is the list of sites planning to enable DRIHM VO soon:
Croatia:
3 sites will support

Italy:
Roma3 INFN

Macedonia:
MK-03-FINKI

2. June 15 June 30 NGIs Enable DRIHM VO in EGI schedulers YES in order to test the facilities offered by these services (jobs bookeeping, rescheduling, resources matchmaking, Input/Output SandBox handling, etc.).
gLite-WMS/LB
DRIHM Services
3. June 15 June 30 NGIs & DRIHM WMS WMS: 
$ lcg-infosites --is lcg-bdii.cern.ch --vo drihm.eu wms
https://graszode.nikhef.nl:7443/glite_wms_wmproxy_server
https://gridrb.fe.infn.it:7443/glite_wms_wmproxy_server
https://prod-wms-01.ct.infn.it:7443/glite_wms_wmproxy_server
https://prod-wms-02.ct.infn.it:7443/glite_wms_wmproxy_server
https://wms-aegis.ipb.ac.rs:7443/glite_wms_wmproxy_server
https://wms-demo.ipb.ac.rs:7443/glite_wms_wmproxy_server
https://wms.ipb.ac.rs:7443/glite_wms_wmproxy_server
https://wms004.cnaf.infn.it:7443/glite_wms_wmproxy_server
https://wms005.cnaf.infn.it:7443/glite_wms_wmproxy_server
https://wms014.cnaf.infn.it:7443/glite_wms_wmproxy_server
https://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server
https://wms1.grid.cesnet.cz:7443/glite_wms_wmproxy_server
AND Catch All Services
3. June 15 June 30 NGIs & DRIHM MyProxy VOMS: vomsmania.cnaf.infn.it
3. June 15 June 30 NGIs & DRIHM LFC LFC: 
$ lcg-infosites --is lcg-bdii.cern.ch --vo drihm.eu lfc
lfc.ipb.ac.rs
AND Catch All Services
3. June 15 June 30 NGIs & DRIHM VOMS MyProxy: myproxy.hellasgrid.gr
AND Catch All Services
Porting
3. June 15 June 30 NGIs & DRIHM WRF with MPICH2, GFORTRAN, NetCDF, GRIB2 deployment at NGIs. Porting of the demanding model (i.e. WRF) to the Grid Environment in collaboration with the EGI/NGIs user support teams for some selected "Critical Cases" and preliminary tests of the Grid submission/data management. (using NGI general purpose VO if DRIHM is not deployed yet)

By Daniele Cesini:
One of the use cases provided by CIMA with WRF on the Naples site (SL6 WNs), using 16 core on 8 differnet nodes with infiniband and OpenMPI, but the performance (*) were not so exiting,
The use case was the Genova flooding on 11-04-2011, 24hrs starting from 00:00, only the coarser domain, not the nested one.
If someone else is interested in trying this run, jdl, exe and prologue (with reference to the data stored in a Grid SE that should be world readable) are available here:wrf-test.tgz
(I didn't installed WRF at the site, everything it's hardcoded, but it can be used for a test with few changes in the script - I can provide more details if needed)

Jerome from France provided the WRF configruation file with needed dependencis by DRIHM.

No need to do software deployment as Christian reported that they will fetch all needed compiled libraries from DRIHM repo during the Grid job run time.


Identify sites for demanding jobs
4. July 01 August 15 NGIs List of sites for demanding jobs (indicate maximum number of cores for paralell computing). Identification of the testbed members willing to run demanding jobs, as the meteo models, with the indication of the maximum allowable parallelism degree.
Job submission testing'
5. July 01 August 31 DRIHM Testing DRIHM models at NGIs computing and storage resources (job submission of WRF models, data management using command line interface). Test the job submission (direct to CE and via WMS/LB) and data management from a command line user interface for the most demanding models on the DRIHM testbed - Performance evaluation of the infrastructure resources and services.
5. July 01 August 31 DRIHM Porting and preliminary tests for the submission of jobs and data management from a command line user interface of the less demanding applications (hydrologic and hydraulic models) - Performance evaluation of the infrastructure and investigation of the capabilities offered by the virtualization systems available within EGI that can be suitable to run these applications.
6. September 01 September 30 DRIHM Testing DRIHM models at NGIs computing and storage resources (job submission of WRF models, data management using dedicated SCI-BUS portal). Preliminary test for the submission of jobs from the DRIHM sci-bus-based portal to the EGI testbed resources.