Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

EGI ENVRI

From EGIWiki
Revision as of 12:32, 13 August 2014 by Krakow (talk | contribs)
Jump to navigation Jump to search
Welcome to the page of the EGI ENVRI Study Case page



General Project Information

  • Leader: Malgorzata Krakowian (malgorzata.krakowian @ egi.eu)
  • Start Date: 1.02.2013
  • End Date: 1.08.2014
  • Meetings: Indico page (slides and minutes from meetings)
  • Official webpage


Motivation

A study case was set up to identify existing services and solutions from EGI and EUDAT that could address the data pre-processing, post-processing, publishing needs of these two ESFRI projects. The outcome of the pilot is expected to be directly applicable to EISCAT_3D, and indirectly by other ESFRIs of ENVRI.  In cooperation with EISCAT-3D representatives in ENVRI, EGI.eu will try to find best suitable solutions for data pre-processing of primary data and post-processing toward publishing.

Outcome

The design of the next generation incoherent scatter radar system, EISCAT_3D, opens up opportunities for physicists to explore many new research fields. On the other hand, it also introduces significant challenges in handling large-scale experimental data which will be massively generated at great speeds and volumes. This challenge is typically referred to as a big data problem and requires solutions from beyond the capabilities of conventional database technologies. To identify existing services and new services that can tackle the EISCAT_3D big data challenge, a collaboration has been formed in February 2013 among EISCAT_3D, EGI and the EUDAT infrastructures under the ENVRI project.


Phase 1 Proof of concept architecture draft

A 'Towards a Big Data Strategy for EISCAT-3D' document is emerging from the collaboration and it outlines a project that would take the first steps towards defining the EISCAT_3D big data strategy. 

EISCAT-3d.png

Phase 2 Requirements gathering

Following questionnaires has been used to collect requirements from EISCAT data managers and scientists:

For scientists: https://www.surveymonkey.com/s/ENVRI-EISCAT_Scientists

For data managers: https://www.surveymonkey.com/s/ENVRI-EISCAT_Data_managers

Responses: https://documents.egi.eu/public/ShowDocument?docid=1983

Phase 3 Prototype system based on technologies and resources of EGI and EUDAT

Development and pilot deployment of OSGC - OpenSource Geospatial Catalogue

Presentation: EGI OpenSearch Catalogue Appliances for EISCAT 3D


OSGC is an Open Source implementation of an OpenSearch GeoSpatial Catalogue compliant to OGC 10-32r3 specification, developed by EGI.eu (http://www.egi.eu/) under the ENVRI (http://envri.eu/) project.

OSGC provides a catalogue engine built on top of a PostgreSQL+Postgis database, which exposes a cusmizable OpenSearch interface. Most of the application configuration can be set from the Admin web interface, while Data Administrators have a separated Dropbox interface, which ease the management of the catalog and the data storage, and a Data Gateway interface, which controls access to data and produces data access statistics.

A stand-alone client web interface, written in HTML5 and Javascript, can be used to query this catalog and other compliant OpenSearch catalogs.

Features

  • OpenSearch catalogue engine with customizable output formats, products metadata, query schema, input formats (for ingestion).
  • Web admin interface (to offer the catalog as a Platform-As-A-Service on the Cloud).
  • Dropbox, to automatically extract metadata, register it into the catalogue and optionally push the data file into Cloud or other connected storage.
  • Data Gateway interface, to control access to data, produce data access statistics and bridge non-http protocols
  • OpenSearch web client interface, with the possibility to execute it remotely or as a standalone application (for integration into Cloud Virtual Laboratories PaaS services) and cumulative download (with shop-chart functionality).

Members

EGI.eu

  • (Study case leader & EGI Operations Officer) Malgorzata Krakowian malgorzata.krakowian [at] egi.eu
  • (Technical Outreach Manager) Gergely Sipos gergely.sipos [at] egi.eu
  • (User Community Support Officer) Nuno Ferreira nuno.ferreir [at] egi.eu
  • (EGI Chief Operations Officer) Tiziana Ferrari

EISCAT-3D:

  • Dr. Ingemar Haggstrom Ingemar.Haggstrom [at] eiscat.se

ENVRI:

  • Yin Chen ChenY58 [at] cardiff.ac.uk
  • Malcolm Atkinson
  • Alex Hardisty
  • Yannick Legré
  • Paul Martin
  • Alun Preece

EUDAT/CSC:

  • Ari Lukkarinen ari.lukkarinen [at] csc.fi
  • Antti Pursula
  • Ville Savolainen

Resources

External tools and presentations (could be useful)

ENVRI wiki