The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.
Difference between revisions of "EGI ENVRI"
| Line 90: | Line 90: | ||
A document is emerging from the collaboration and it outlines a project that would take the first steps towards defining the EISCAT_3D big data strategy. The document is a working draft and is available at [ Towards a Big Data Strategy for EISCAT-3D document] (external link). It is updated on a regular basis. | A document is emerging from the collaboration and it outlines a project that would take the first steps towards defining the EISCAT_3D big data strategy. The document is a working draft and is available at [ Towards a Big Data Strategy for EISCAT-3D document] (external link). It is updated on a regular basis. | ||
<br> | <br> | ||
Following questionnaires are used to collect requirements from EISCAT data managers and scientists: | |||
For scientists: | For scientists: | ||
Revision as of 14:02, 16 August 2013
|
General Project Information
- Leader: Malgorzata Krakowian (malgorzata.krakowian @ egi.eu)
- Start Date: 1.02.2013
- Meetings: Indico page (slides and minutes from meetings)
Mailing lists:
- envri-eiscat3d @ mailman.egi.eu - for EISCAT-3D study case
Motivation
A study case was set up to identify existing services and solutions from EGI that could address the data pre-processing, post-processing, publishing needs of these two ESFRI projects. The outcome of the pilot is expected to be directly applicable to EISCAT_3D and EURO-ARGO, and indirectly by other ESFRIs of ENVRI. In cooperation with EISCAT-3D and EURO-ARGO representatives in ENVRI, EGI.eu will try to find best suitable solutions for data pre-processing of primary data and post-processing toward publishing.
Members
- (Study case leader & EGI Operations Officer) Malgorzata Krakowian malgorzata.krakowian [at] egi.eu
- (Technical Outreach Manager) Gergely Sipos gergely.sipos [at] egi.eu
- (User Community Support Officer) Nuno Ferreira nuno.ferreir [at] egi.eu
- (EGI Chief Operations Officer) Tiziana Ferrari
- Dr. Ingemar Haggstrom Ingemar.Haggstrom [at] eiscat.se
EURO-ARGO:
- Thierry Carval Thierry.Carval [at] ifremer.fr
- Jerome Detoc Jerome.Detoc [at] ifremer.fr (data management, big data)
- Yin Chen ChenY58 [at] cardiff.ac.uk
- Malcolm Atkinson
- Alex Hardisty
- Yannick Legré
- Paul Martin
- Alun Preece
- Ari Lukkarinen ari.lukkarinen [at] csc.fi
- Antti Pursula
- Ville Savolainen
Resources
- Google doc for study case (internal for EGI)
- Euro-Argo architecture (image file)
- EISCAT-3D architecture (image file)
External tools and presentations (could be useful)
- The IT Challenges For Research Infrastructures In Physics presentation (CRISP ESFRI cluster project)
ENVRI wiki
- Euro-Argo description
- Ero-Argo data management - ftp server
- EISCAT-3D description
- Analyse Common Requirements for Data Processing
- Iceland Volcano Study Case
Progress
EISCAT-3D
The design of the next generation incoherent scatter radar system, EISCAT_3D, opens up opportunities for physicists to explore many new research fields. On the other hand, it also introduces significant challenges in handling large-scale experimental data which will be massively generated at great speeds and volumes. This challenge is typically referred to as a big data problem and requires solutions from beyond the capabilities of conventional database technologies. To identify existing services and new services that can tackle the EISCAT_3D big data challenge, a collaboration has been formed in February 2013 among EISCAT_3D, EGI and the EUDAT infrastructures under the ENVRI project.
'Towards a Big Data Strategy for EISCAT-3D' document
A document is emerging from the collaboration and it outlines a project that would take the first steps towards defining the EISCAT_3D big data strategy. The document is a working draft and is available at [ Towards a Big Data Strategy for EISCAT-3D document] (external link). It is updated on a regular basis.
Following questionnaires are used to collect requirements from EISCAT data managers and scientists:
For scientists: https://www.surveymonkey.com/s/ENVRI-EISCAT_Scientists
For data managers: https://www.surveymonkey.com/s/ENVRI-EISCAT_Data_managers
- 'Towards a Big Data Strategy for EISCAT-3D' presentation (pdf) during 16th EISCAT International Symposium 2013
Euro-ARGO
A short document will be prepared that outlines the scope of the EURO-ARGO - EGI collaboration. The document would be used by EGI and by EURO-ARGO to invite contributors into the collaboration (NGIs, institutes).
Key points in the document:
- EURO-ARGO architecture and interfaces to interact with the data.
- The collaboration would make the archive searchable by
- moving the data to EGI storages (grid or cloud?)
- registering the data into EGI file catalogues and metadata services
- Advertisement of the data to potentially interested EGI user communities
