Difference between revisions of "EGI ENVRI"
Line 23: | Line 23: | ||
A study case was set up to identify existing services and solutions from EGI that could address the data pre-processing, post-processing, publishing needs of these two ESFRI projects. The outcome of the pilot is expected to be directly applicable to EISCAT_3D and EURO-ARGO, and indirectly by other ESFRIs of ENVRI. In cooperation with EISCAT-3D and EURO-ARGO representatives in ENVRI, EGI.eu will try to find best suitable solutions for data pre-processing of primary data and post-processing toward publishing. <br> | A study case was set up to identify existing services and solutions from EGI that could address the data pre-processing, post-processing, publishing needs of these two ESFRI projects. The outcome of the pilot is expected to be directly applicable to EISCAT_3D and EURO-ARGO, and indirectly by other ESFRIs of ENVRI. In cooperation with EISCAT-3D and EURO-ARGO representatives in ENVRI, EGI.eu will try to find best suitable solutions for data pre-processing of primary data and post-processing toward publishing. <br> | ||
== <span id=" | == <span id="Members" class="mw-headline">Members </span> == | ||
'''EGI.eu'''<br> | '''EGI.eu'''<br> | ||
Line 41: | Line 37: | ||
'''EURO-ARGO: '''<br> | '''EURO-ARGO: '''<br> | ||
*Thierry Carval Thierry.Carval [at] ifremer.fr | *Thierry Carval Thierry.Carval [at] ifremer.fr | ||
*Jerome Detoc Jerome.Detoc [at] ifremer.fr (data management, big data) | *Jerome Detoc Jerome.Detoc [at] ifremer.fr (data management, big data) | ||
<br> | '''ENVRI'''''':''' | ||
*Yin Chen ChenY58 [at] cardiff.ac.uk<br> | |||
== <span class="mw-headline | == <span id="Resources" class="mw-headline">Resources </span><br> == | ||
*[https://drive.google.com/a/egi.eu/?pli=1#folders/0Bzrt5PnQFpWocjd1V2ZHeXJUamM Google doc for study case (internal for EGI)] | *[https://drive.google.com/a/egi.eu/?pli=1#folders/0Bzrt5PnQFpWocjd1V2ZHeXJUamM Google doc for study case (internal for EGI)] | ||
Line 59: | Line 57: | ||
*[http://envri.eu/group/envri/wiki/-/wiki/Main/Euro-Argo Euro-Argo description ] | *[http://envri.eu/group/envri/wiki/-/wiki/Main/Euro-Argo Euro-Argo description ] | ||
*[http://www.argodatamgt.org/ Ero-Argo data management - ftp server] | *[http://www.argodatamgt.org/ Ero-Argo data management - ftp server] | ||
*[http://envri.eu/group/envri/wiki/-/wiki/Main/EISCAT_3D EISCAT-3D description ] | *[http://envri.eu/group/envri/wiki/-/wiki/Main/EISCAT_3D EISCAT-3D description ] | ||
*[http://envri.eu/group/envri/wiki/-/wiki/Main/Analyse%20Common%20Requirements%20for%20Data%20Processing Analyse Common Requirements for Data Processing ] | *[http://envri.eu/group/envri/wiki/-/wiki/Main/Analyse%20Common%20Requirements%20for%20Data%20Processing Analyse Common Requirements for Data Processing ] | ||
*[http://envri.eu/group/envri/wiki/-/wiki/Main/Iceland%20Volcano%20Use%20Case Iceland Volcano Study Case ] | *[http://envri.eu/group/envri/wiki/-/wiki/Main/Iceland%20Volcano%20Use%20Case Iceland Volcano Study Case ] | ||
== <span | == <span class="mw-headline" id="Progress">Progress </span><br> == | ||
=== EISCAT-3D === | === EISCAT-3D === | ||
A short document will be prepared that outlines the scope of the EISCAT_3D - EGI collaboration. The document would be used by EGI and by EISCAT_3D to invite contributors into the collaboration (NGIs, institutes) | A short document will be prepared that outlines the scope of the EISCAT_3D - EGI collaboration. The document would be used by EGI and by EISCAT_3D to invite contributors into the collaboration (NGIs, institutes) | ||
Line 80: | Line 78: | ||
2. registering the data into EGI file catalogues and metadata services | 2. registering the data into EGI file catalogues and metadata services | ||
3. studying, identifying and - if feasible - implementing data mirroring and indexing strategies that would suit to the envisaged processing strategies of EISCAT_3D. The studies could cover areas such as use of Hadoop or other big-data technologies on EGI. | 3. studying, identifying and - if feasible - implementing data mirroring and indexing strategies that would suit to the envisaged processing strategies of EISCAT_3D. The studies could cover areas such as use of Hadoop or other big-data technologies on EGI. | ||
=== Euro-ARGO === | |||
[[Category:EGI_ENVRI]] | [[Category:EGI_ENVRI]] |
Revision as of 14:23, 17 May 2013
|
General Project Information
- Leader: Malgorzata Krakowian (malgorzata.krakowian @ egi.eu)
- Start Date: 1.02.2013
- End Date:
- Meetings: Indico page (slides and minutes from meetings)
Motivation
A study case was set up to identify existing services and solutions from EGI that could address the data pre-processing, post-processing, publishing needs of these two ESFRI projects. The outcome of the pilot is expected to be directly applicable to EISCAT_3D and EURO-ARGO, and indirectly by other ESFRIs of ENVRI. In cooperation with EISCAT-3D and EURO-ARGO representatives in ENVRI, EGI.eu will try to find best suitable solutions for data pre-processing of primary data and post-processing toward publishing.
Members
EGI.eu
- (Technical Outreach Manager) Gergely Sipos gergely.sipos [at] egi.eu
- (User Community Support Officer) Nuno Ferreira nuno.ferreir [at] egi.eu
EISCAT-3D:
- Dr. Ingemar Haggstrom Ingemar.Haggstrom [at] eiscat.se
- Dr Ingrid Mann ingrid.mann [at] eiscat.se
EURO-ARGO:
- Thierry Carval Thierry.Carval [at] ifremer.fr
- Jerome Detoc Jerome.Detoc [at] ifremer.fr (data management, big data)
'ENVRI':
- Yin Chen ChenY58 [at] cardiff.ac.uk
Resources
- Google doc for study case (internal for EGI)
- Euro-Argo architecture (image file)
- EISCAT-3D architecture (image file)
External tools and presentations (could be useful)
- The IT Challenges For Research Infrastructures In Physics presentation (CRISP ESFRI cluster project)
ENVRI wiki
- Euro-Argo description
- Ero-Argo data management - ftp server
- EISCAT-3D description
- Analyse Common Requirements for Data Processing
- Iceland Volcano Study Case
Progress
EISCAT-3D
A short document will be prepared that outlines the scope of the EISCAT_3D - EGI collaboration. The document would be used by EGI and by EISCAT_3D to invite contributors into the collaboration (NGIs, institutes)
Key points in the document:
- EISCAT archive includes 60TB of correlated data collected during 1981-2013. There are interfaces to interact with the data but the data is not searchable.
- The collaboration would make the archive searchable by:
1. moving the data to EGI storages (grid or cloud to be decided later)
2. registering the data into EGI file catalogues and metadata services
3. studying, identifying and - if feasible - implementing data mirroring and indexing strategies that would suit to the envisaged processing strategies of EISCAT_3D. The studies could cover areas such as use of Hadoop or other big-data technologies on EGI.