Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Competence centre EISCAT 3D"

From EGIWiki
Jump to navigation Jump to search
Line 198: Line 198:


== Resources ==
== Resources ==
* Source code and deployment documentation are available [http://sourceforge.net/projects/osgcat/files/?source=navbar here]
* Source code and deployment documentation are available in sourceforge: [http://sourceforge.net/projects/osgcat/files/?source=navbar here]
* Testing instance:
* Testing instance:
** [https://155.210.198.184/admin/ Admin Panel]
** [https://155.210.198.184/admin/ Admin Panel]

Revision as of 08:28, 7 April 2015

EGI-Engage Competence centres: Main page ELIXIR BBMRI MoBrain DARIAH LifeWatch EISCAT_3D EPOS Disaster Mitigation | EGI-Engage Knowledge Commons




CC Coordinator: Ingemar Häggström

CC members' list: cc-eiscat3d AT mailman.egi.eu


Type of Competence Centre: Science-oriented

Target user communities: EISCAT_3D users

List of organizations representing the user communities: EISCAT Scientific Association

Duration of the CC: 30 M

Starting at Project Month: 1 M

Ending at Project Month: 30 M



The design of the next generation incoherent scatter radar system, EISCAT_3D introduces significant challenges in handling large-scale experimental data which will be massively generated at great speeds and volumes. The CC will build an e-Infrastructure to meet the requirements of the EISCAT_3D data system, will support the EISCAT science community in their acquisition, curation, access to and processing of the data, and will train data scientists who can explore new approaches to solve problems via new data-centric way of conceptualising, organising and carrying out research activities.

Objectives

  1. To build common e-Infrastructure to meet the requirements of a big scientific data system such as EISCAT_3D data system
  2. To demonstrate the developed e-Infrastructure can support the EISCAT science community in their acquisition, curation, access to and processing of the data
  3. To train data scientists who can explore new approaches to solve problems via new data-centric way of conceptualising, organising and carrying out research activities, which will lead to new discoveries and significant scientific breakthroughs.


Tasks

Task 1: User Support and Training

(M15-M30) Effort: 3 PM

Objectives: Disseminate the possibilities for users with the portal.
Arrange training events co-located with other activities for EISCAT users. The yearly radar-school and the yearly EISCAT_3D user meeting are suitable for co-location of training events. Two training events are planned. The first focuses on the basic functionality of the portal and is held after the portal is available on a production platform. The second training event focuses on the new features implemented in the mini-projects (Task 3 and 4). Sigma is the lead partner, will provide the training and the training material. EISCAT will contribute with relevant use-cases for the training.

Task 2: Deploy the portal as a production system

(M1-M12) Effort: 6 PM

Objectives: Deploy the pilot as a production system.

Develop the pilot into a portal that is of production quality. Include all of the existing EISCAT data in the archive. Sigma leads this project, and provides the development effort. EISCAT provides the data, and testing of the portal.

Task 3: Basic reanalysis within the portal

(M13-M24) Effort: 6 PM

Objectives: Enabling basic reanalysis within the portal.

Basic reanalysis within the portal with user-specified constraints. Instead of downloading large sets of data, it is desirable to be able to do some basic reanalysis of the low levels of data with other constraints than were used in the standard analysis. Basic constraints are defining the volumes of space and the time intervals to integrate. In total there are about 100 analysis parameters to set, and the portal should identify them and provide tools to change them.

EISCAT leads this task providing specifications for the new capabilities. CSC provides the development effort to integrate these into the portal.

Task 4: Use level 3 data as metadata

(M8-M30) Effort: 12PM

Objectives: Enable the use of level 3 data of EISCAT as metadata.

Setup the level 3 data of EISCAT to complement the radar metadata. The level 3 data of EISCAT are the derived ionospheric physical parameters like densities, temperatures and drifts at selected volumes of space. The data is currently stored in a separate database, Madrigal, with no connection to the lower levels of the data. This task should complement the radar metadata, parameters of the radar hardware, with the physical parameters. Currently the only link between the data sets are the time stamps, but it's desirable to use a set of identifiers to more clearly follow how the different levels of data have been formed, exactly which set of level 1 data were used to derive the level 2 and finally level 3 data. Expand the portal of the pilot to be able to search the data based on the expanded metadata set, and allow download of selected levels of data. Investigate further how the access rights of the data should be followed.

EISCAT leads this complex task, specifying how to implement these advanced features. SNIC and CSC provides the development and testing resources.

Task 5: Exploitation

(M13-M30) Effort: 2PM

Objectives: Maintain the portal on a production platform

Deploy, maintain and update the production version of the portal. SNIC leads this task providing suitable resources for operating the portal. EISCAT provides new data to be added to archive.

Deliverables and milestones

The following gives an overview of deliverables/Milestones scheduled

Code Title Delivery PM Delivery CM Delivered date Status
M1.1 One half-day training event co-located with the 2016 radar school. An event to train the users in the basic use of the portal. M19



M1.2 One half-day training event co-located with the 2017 EISCAT_3D user meeting. An event to train users in the more advanced, newly developed features of the portal. M29



D1.1 Report on user satisfaction survey. M30



M2.1 Production version of the portal. M11



D2.1 Documentation of the portal. M12



M3.1 Release of portal with basic reanalysis implemented. M23


D3.1 Documentation for the upgraded portal. M24


D4.1 Define identifiers for the different levels of data and the connection between them. M13


M4.1 First version of portal ready for tests by selected users. M20


M4.2 Second version ready after user feedbacks and ready for production. M28


D4.2 Documentation of the new enhanced portal. M30


M5.1 Production version of portal deployed on a production platform. M13



ENVRI Pilot - OpenSource Geospatial Catalogue

OSGC is an Open Source implementation of an OpenSearch GeoSpatial Catalogue compliant to OGC 10-32r3 specification, developed by EGI.eu under the ENVRI project.

OSGC provides a catalogue engine built on top of a PostgreSQL+Postgis database, which exposes a cusmizable OpenSearch interface. Most of the application configuration can be set from the Admin web interface, while Data Administrators have a separated Dropbox interface, which ease the management of the catalog and the data storage, and a Data Gateway interface, which controls access to data and produces data access statistics.

A stand-alone client web interface, written in HTML5 and Javascript, can be used to query this catalog and other compliant OpenSearch catalogs.

Features

  • OpenSearch catalogue engine with customizable output formats, products metadata, query schema, input formats (for data injection).
  • Web admin interface (to offer the catalogue as a Platform-As-A-Service on the Cloud).
  • 'Dropbox feature', to automatically extract metadata, register it into the catalogue and optionally push the data file into Cloud or other connected storage.
  • Data Gateway interface, to control access to data, produce data access statistics and bridge non-http protocols.
  • OpenSearch web client interface, with the possibility to execute it remotely or as a standalone application (for integration into Cloud Virtual Laboratories PaaS services) and cumulative download (with shop-chart functionality).

Known limitations and next steps to evolve the system into a production version

  • Test system scalability
  • Develop pre-processing service to extract further metadata from the data
  • Develop visualisation service
  • Develop Near Real Time tool to import data automatically from receiving stations
  • Evolve admin panel based on user requirements
  • Evolve client panel based on user requirements

Resources

Resources


Demo instance:

- Admin panel: https://155.210.198.184/admin/
- User panel: https://155.210.198.184/client/