Competence centre DARIAH

From EGIWiki
(Redirected from CC-DARIAH)
Jump to: navigation, search
EGI-Engage Competence centres: Main page ELIXIR BBMRI MoBrain DARIAH LifeWatch EISCAT_3D EPOS Disaster Mitigation | EGI-Engage Knowledge Commons


Contents

DARIAH.png


About

The DARIAH Competence Center (CC) is a virtual research group, consisting of computer scientists and the researchers coming from the Arts and Humanities. Its mission is to broadening the usage of the advanced research infrastructures and technologies, such as cloud-oriented services, computational and storage resources in the domain of the Arts and Humanities research. The main activities of the DARIAH CC are:

  1. Bringing together relevant stakeholders from the EGI and DARIAH communities who help DARIAH members to better understand their research infrastructure needs, and who operate as a ‘knowledge hub’ concerning the use of e-Infrastructures in the domain. >> Read about the CC members.
  2. Providing customised e-Infrastructure services for Arts and Humanities, by extending generic EGI e-Infrastructure services according to the unique needs of Arts and Humanities users. >> Try the e-infrastructure services
  3. Creating demonstrator applications from the Arts and Humanities domain that use e-Infrastructure services. The demonstrators display state-of-the-art capabilities of what's available in digital science with the use of EGI. >> Learn about and try our demonstrators
  4. Raising awareness within the Arts and Humanities researcher community teams about the benefits of e-Infrastructure technologies. The CC is organising outreach, training and workshop events, and contributes to high-impact events in the field. >> Check our event calendar

THE REST OF THE PAGE, INCLUDING THE REFERENCED SERVICES, IS UNDER FINALISATION


Participants

Ruđer Bošković Institute (RBI), Croatia (Leader institute) Davor Davidović (CC coordinator), Karolj Skala
Hungarian Academy of Science, Institute for Computer Science and Control (MTA SZTAKI), Hungary Zoltán Farkas (CC technical coordinator)
Italian National Institute of Nuclear Physics (INFN), Italy

Roberto Barbera,

Antonio Calanducci

Gesellschaft für wissenschaftliche Datenverarbeitung mbH (GWDG), Germany

Philipp Wieder,

Tibor Kalman

Data Archiving and Networked Services (DANS), Netherlands Rene van Horik
Austrian Academy of Science (AAS), Austria Eveline Wandl-Vogt


Generic services for Digital Arts and Humanities

This section describes the services developed and offered by the DARIAH Competence Centre to the researchers coming from the Arts and Humanities research domain.


Cloud-based compute and storage infrastructure

The most basic service provided by the DARIAH-CC is a federated 'Infrastructure as a Service' (IaaS) cloud. The system is open for any researcher and research project or institute from the Digital Arts and Humanities domain. The infrastructure is based on the EGI Federated Cloud technology and currently connects OpenStack clouds from two providers: INFN-Catania and INFN-Bari. Three more sites are expected to join later in 2016 (GWDG, IRB, SZTAKI). The participating clouds are joint into a virtually single system through an identity federation that enables researchers and their applications to access the different clouds with a single identity and with single-sign on. Access to the infrastructure is controlled by the Competence Centre.

The infrastructure is offered as a scalable system for data and compute intensive applications. These applications can be deployed on the DARIAH-CC cloud in the form of Virtual Machine images (VMIs). Existing VMIs can be browsed, and new VMIs can be deployed on the DARIAH-CC cloud through the AppDB Cloud Marketplace portal. The distinct clouds of the DARIAH-CC cloud federation can all be accessed through programming APIs, command line tools and the high level gateways and applications offered by the DARIAH-CC.

The list of resources provided by the DARIAH Competence Centre is available here

It is our pleasure to inform you that the EGI.eu and DARIAH Competence Centre have signed an agreement on providing the EGI Cloud resources for the DARIAH community.


DARIAH Science Gateway

The gateway provides various web-based applications and services for the Digital Humanities researchers, institutes and communities:

To access DARIAH Science Gateway, please click here.

Short tutorial on how to use DARIAH Science Gateway and the simple Cloud Access service [1]

EGI Training Infrastructure

The EGI training infrastructure is an 'Infrastructure as a Service' type cloud system, that is open for face-to-face and online training tutorials and schools for the EGI community and their partners. The infrastructure can basically support two types of tutorials:

  1. Events that demonstrates services of the EGI federated cloud. In this operational mode the infrastructure can accommodate courses that focus on the usage of the EGI cloud services themselves. Such courses typically target programmers or other technical members of scientific communities or projects.
  2. Training courses about scientific software and/or scientific services. In this operational mode the representatives of the community (the trainers) deploy custom Virtual Machine images on the training infrastructure before the training, and these images offer the training environment for the students. Because of the cloud-based operational model the students can have dedicated training environments, and the community can benefit from the easy deployment, predictability and repeatability of courses.

Do you want to use the training infrastructure for an event? Please email support@egi.eu. Further information and sample exercises about the training infrastructure are available at https://wiki.egi.eu/wiki/Training_infrastructure.

Services for developers and service providers in Digital Humanities

gLibrary

gLibrary (v2.0) is a framework, designed and developed by the Division of Catania of the Italian National Institute of Nuclear Physics (INFN), that offers both access to existing (both closed and open) data repositories and the creation of new ones via a simple REST API.

A repository in the gLibrary lingo is a virtual container of one or more data collections. A collection provides access to a relational DB table or to a non-relational (NoSQL) DB collection. Currently, gLibrary supports MySQL, PostgreSQL, Oracle and MongoDB.
Each repository can group together one of more collections, providing a virtual and uniform interface to data tables coming from different databases that could be potentially of different types (for example one collection provides access to a PostGreSQL table and another to a MongoDB collection). JSON is used as input and output data format.
Once collections are imported or created from scratch, the gLibrary REST APIs can be used to retrieve, create, update and delete collection’s records, that in gLibrary lingo are called items. Moreover, a powerful hierarchical ("à la iTunes") filtering system is available to make queries on collections. 
Each item can have one or more attachment, that we call replica. Replicas can be stored on Grid Storage Elements (based on the Disk Pool Manager software) or in Cloud Storage (OpenStack Swift is currently supported).
Relations between two collections of the same repository can be created, if foreign keys are properly assigned. Currently we support one-to-many relations.

gLibrary is intended to the researchers, scholars and research groups in the domain of the Arts and Humanities as well as other disciplines, that wish to organize their digital assets and the accompanying metadata on distributed storage systems based on different technologies. The system provides to non-IT experts an easy-to-use service that features browsing, searching, downloading and replicating of digital objects.

The code of gLibrary 2.0 is available on GitHub at this page and it is released under the Apache 2.0 license. The documentation is available on ReadTheDocs at this page and some training material is available at the following links: presentation, tutorial, video lecture, video tutorial.

In the context of the EGI DARIAH Competence Centre, gLibrary has been used to build, with data provided by the Austrian Academy of Sciences, a demonstrative repository of Bavarian dialects within the Austrian-Hungarian monarchy. To access the repository, called DBO@Cloud, follow this link.

CDSTAR

The Common Data Storage ARchitecture (CDSTAR) is a customizable object storage solution for science and research. CDSTAR addresses the specific requirements of research data management according to the good scientific practice. The system integrates the ability of storing metadata along the research data in a flexible metadata schema that can be tailored for the specific use in different scientific disciplines. Additionally, the data objects that are stored in CDSTAR can be registered automatically at the EPIC Persistent Identifier (PID) service. The EPIC service gives data sets a unique, globally resolvable identifier as an additional abstraction layer that allows citing data sets in scientific publications. A role-based security concept is also integrated into CDSTAR, which allows the protection of data sets with an individual set of permissions and rights for each user. Additionally, CDSTAR is capable to use SAML infrastructures such as the one provided by DARIAH in order to verify data access. This allows to use the data permissions for the infrastructure at a central data point provided by DARIAH and to use the DARIAH access credentials.

CDSTAR offers the following features:

CDSTAR is developed by EGI DARIAH-CC partner GWDG as Open Source Software under the Apache2 Licence. CDSTAR is delivered as a container (Docker/LCX) or can be deployed from source. CDSTAR is the basis for many application-specific research infrastructures, e.g. in DH, medical research and many others. CDSTAR is offered to the DARIAH researchers as individual service or to DARIAH service providers and applications developers as a software product that can be extended for individual use cases.

As part of the DARIAH-CC efforts we offer a test installation for testing purposes. In case you want your own installation, please contact: dariah AT gwdg DOT de.

Links:

Demonstrator applications

Semantic Search Engine

The e-Infrastructure Knowledge Base, originally developed in the context of the CHAIN-REDS project and now maintained by the Sci-GaIA project is one of the largest existing e-Infrastructure related digital information systems. It currently contains information, gathered both from dedicated surveys and other web and documental sources, for largely more than half of the countries in the world. Information is presented to visitors through geographic maps and tables that show and list e-Infrastructure, sites, services, and applications. In addition, the Knowledge Base includes information Open Access Document Repositories (OADRs) Data [http://www.sci-gaia.eu/knowledge-base/data-repositories-map/ Repositories (DRs) and Open Educational Repositories (OERs) in both geo- and table-view. Many of the OADRs, DRs and OERs contain material related to Cultural Heritage.

On top of the e-Infrastructure Knowledge Base, the Division of Catania of the Italian National Institute of Nuclear Physics (INFN) has developed a Semantic Search Engine (SSE) to semantically enrich the metadata of the more than 30 million resources included in the Knowledge Base and allow the discovery of new correlations about document and data and, ultimately, the creation of new knowledge. In this sense, the SSE goes way beyond normal Google/Google Scholar searches. The queries to the Semantic Search Engine can be made in more than 110 languages (including those not based on the Latin alphabet) and the results are ranked according to the latest issue of RankingWeb of Repositories. Moreover, they are connected, whenever the information is available, to Google Scholar and Altmetric in order to provide users with additional information about versions and citations of a given resource found by the query. Last but not least, the SSE exploits LodLive API to allow users navigate/explore the Linked Data graph of each resource found by a search.

The SSE may be very useful for the Cultural Heritage community in the following respects:

In order to use the SSE, click here.

Parallel Semantic Search Engine

The Parallel Semantic Search Engine (PSSE) service has been developed by the Division of Catania of the Italian National Institute of Nuclear Physics (INFN) as an extension of the Semantic Search Engine (SSE). The PSSE allows parallel searches across different online repositories of open & linked data. Using the PSSE, users can search and semantically correlate contents in geographically distributed digital repositories across several different domains. The service is currently configured to search digital contents in the e-Infrastructure Knowledge Base, in OpenAgris, in Europeana, in Cultura Italia, in Isidore, in PubMed and in DBpedia but others repositories can be easily added if need arises.

In order to use the PSSE, click here.

DBO@Cloud - The Virtual Dialect Dictionary

DBO@Cloud is a Cloud-based repository presenting the work of a 100+ years old collection of Bavarian dialects within the Austrian-Hungarian monarchy from the beginning of German language to nowadays. The datasets are provided by the Austrian Academy of Science. The service is based on gLibrary and demonstrates the functionalities of the framework, especially its hierarchical, "à la iTunes" searching functionality.

To access the repository click here.

This short video demonstrates the functionalities of the DBO@Cloud repository and describes how to browse and manage collections.

Multi-Source Distributed Real-Time Search and Information Retrieval Application (SIR)

The purpose of the Multi-Source Distributed Real-Time Search and Information Retrieval Application (SIR) is to offer big data search and analytics capabilities for large data sets. A combination of configurable pipelines and an interface for manual interaction will allow to implement workflows that benefit from state-of-the art technologies like elastic search or Hadoop.

As a first use case for SIR, the DARIAH-CC implemented an Optical Character Recognition (OCR) post-processing pipeline for data masking. It can help users to automatically find and mask terms, e.g. in scenarios where documents have to be anonymised. The service is currently only available for German texts.

Links:

The service is currently under development.

Events

Past events

  1. Interdisciplinary Conference on Digital Cultural Heritage - DCH2017 [2], 30 August - 1 September 2017, Berlin, Germany (presentation)
  2. Methods in Dialectology XVI [3], 7-11 August 2017, Tokyo, Japan (presentation [4], paper [5])
  3. Workshop "Perspectives of Digital Humanities in the European Open Science Cloud" [6], 12 July 2017, Ruđer Bošković Institute, Zagreb, Croatia
    1. presentation (in Croatian) [7],
    2. hands-on (in Croatian) [8],
    3. photos [9]
  4. DPASSH Conference [10], 14-15 June 2017, Brighton, UK (abstract "Preserving and Managing Digital Arts and Humanities Data in the Open Science Cloud" [11], presentation [12])
  5. EGI Conference & Indigo Summit [13], 9-12 May 2017, Catania, Italy (presentation [14])
  6. 2nd DARIAH Annual meeting [15], 26-27 April 2017, Berlin, Germany (poster [16])
  7. Global Digital Humanities Symposium [17], 16-17 March 2017, Michigan, USA (presentation: "Harvesting cultural knowledge from a traditional non-standard language resource: discovering digital access routes")
  8. 6th ENel COST meeting and workshop, 24-25 February 2017, Budapest, Hungary (presentation "Towards a Collaborative Cultural Lexicography Portal: the DARIAH Competence Centre" [18], abstract [19])
  9. Bayerisch-österreichische Dialektologentagung [20], 28 September - 1 October 2016, Erlangen, Germany (presentation "Cloud Computing, Data Analytics and Open Science: Ein neues Paradigma für die bairische Dialektforschung am Beispiel der Datenbank der bairischen Mundarten in Österreich (DBÖ)")
  10. EURALEX Conference, 6-10 September 2016, Tbilisi, Georgia (workshop, paper [21] and presentation)
  11. DARIAH Annual Meeting [22], Ghent, Belgium, 10-12 October 2016
    1. lightning talk [23],
    2. Cloud infrastructure Working group presentation [24],
    3. Movie clip [25])
  12. DI4R 2016 [26], 28-30 September 2016, Krakow, Poland (abstract [27], presentation [28])
  13. Digital Humanities 2016 [29], 11-16 July 2016, Krakow, Poland (poster [30], abstract [31])
  14. e-Infrastructure Days at University Computing Centre, 23-25 May 2016, Zagreb, Croatia (presentation [32] in Croatian)
  15. Interview "DARIAH and EGI: Agreement on Cloud-Computing Services Signed" [33], 10 April 2016, DARIAH Newsletter [34]
  16. EGI Conference, 6-8 April 2016, Amsterdam, Netherlands (presentation [35])
  17. EGI Community Forum, 9-13 Nov 2015, Bari, Italy
    1. presentation "The SADE mini-project of the EGI DARIAH Competence Centre" [36]
    2. presentation "DARIAH requirements and roadmap in EGI" [37]
  18. JaDH 2015 (The Japanese Association for Digital Humanities annual meeting), 1-3 Sep 2015, Kyoto, Japan, ([38])
  19. 12th ESWC2015 conference, 31st May-4th June 2015, Portorož, Slovenia (presentation [39], poster [40], abstract [41])
  20. 38th MIPRO conference, 25-29 May 2015, Opatija, Croatia (poster [42])
  21. DARIAH VCC meeting, 22-23 April, 2015, Ljubljana, Slovenia (CC kickoff meeting [43])

Upcoming events

None at the moment

CC project background information

CC mailing list: cc-dariah AT mailman.egi.eu

Type of Competence Centre: Science-oriented

Target user communities: Digital Arts, Humanities, and Social Sciences

List of organizations representing the user communities: DARIAH-EU

Duration of the CC: 30 M

Starting at Project Month: 1 M

Ending at Project Month: 30 M

Tasks

Task 1: User Support and Training

Duration: M6-M30, Leader: RBI

Objectives: Raise awareness of Arts and Humanities (A&H) researchers about the necessity of digital research and to qualify them to work with EGI DARIAH CC services

Task 2: DARIAH eScience Gateway on EGI

Duration: M1-M24, Leader: SZTAKI

This task develops a DARIAH eScience Gateway that provides a user friendly web-based environment capable of exploiting EGI cloud and grid resources transparently with the help of WS-PGRADE/gUSE and its customization methodologies (Remote API or Application Specific Module). The specific objectives of this task are:

The work of this task can basically split into five different topics: requirements collection, application interface adaptation and implementation, testing, documentation and eScience gateway operation. WS-PGRADE/gUSE offers different approaches for integrating EGI resources into an application:

Task 3: Storing and Accessing DARIAH contents on EGI (SADE)

Duration: M1-M12, Leader: INFN

The overall goal of this mini-project is to create a digital repository of DARIAH contents using gLibrary, the framework developed by the Italian National Institute of Nuclear Physics (INFN) to create and manage archives of digital assets (data and metadata) on local, Grid and Cloud storage resources. The digital repository will be created taking into account the requirements of the DARIAH end-users.

For this specific mini-project the data-sets will be provided by the Austrian Academy of Sciences (AAS), one of the leading Austrian research institutions with a very long-running experience and interest in the Arts and Humanities domain. The AAS datasets represent the work on a 100+ years old collection on Bavarian dialects within the Austrian-Hungarian monarchy from the beginnings of German language to nowadays. Several data types are taken into account: text, multimedia (images, audio files etc.), URIs; as well as primary collection data, interpreted data, secondary background data and geo-data with different license opportunities.

An extract is available at the website of the Database of Bavarian dialects in Austria [44] electronically mapped.

The AAS datasets will be orchestrated by the INFN gLibrary Digital Repository System whose high-level overview is shown in the following figure:
GLibrary.png

The repositories will be exposed to end-user through two channels:

  1. As a (series of) portlet(s) integrated both in one of the already existing Science Gateways implemented with the Catania Science Gateway Framework [52] and in the Science Gateway developed by the lighthouse project;
  2. As native apps for mobile appliances based on Android and iOS operating systems and downloadable from the official App Stores. The mobile apps will be coded using a cross-platform development environment so that other mobile operating systems could be supported, if needed. Furthermore, the apps could exploit geo-localisation services available on smartphones and tablets to find “near” contents.

Task 4: Multi-Source Distributed Real-Time Search and Information Retrieval (SIR)

Duration: M1-M12, Leader: GWDG

The Multi-Source Distributed Real-Time Search and Information Retrieval (SIR) pathfinding mini-project investigates and implements the possibility of using distributed real-time search engines built on-top of a big-data search and analytics platform to offer A&H users information retrieval and search functionality on heterogeneous data sets, commonly found in the domain of A&H, similar to industry-like platforms such as Google Search or commercial data analytic dashboards. The multi-source distributed real-time search and analytics aims on bringing a next-generation search and data retrieval platform sourced by different systems and heterogeneous data to the users from the Arts and Humanities community. For this reasons, a data hub is implemented based on big data stream and batch processing techniques. The data hub is the foundation for sourcing the distributed source engine based on elastic search that is driven by a software called Common Data Storage Architecture (CDSTAR). On top of this stack, as an interface for the user, a portlet for the DARIAH infrastructure is created that uses the REST-interface of CDSTAR for interacting with the big-data-enabled search and analytics stack. The portlet will form the user interface for searching and browsing data, and displaying the search results by lists, interactive graphics or dash boards that allow a refinement of data retrieval queries.

Task 5: Exploitation

Duration: M7-M30, Leader: DANS

Objectives: Ensure a successful transfer of the mini-projects’ results to its targeted user community and researchers and to increase the applicability and impact of the mini-projects on research conducted.

DARIAH EU aims to develop and maintain an eInfrastructure that supports of A&H research practices. EGI DARIAH CC will foster this aim by promoting the exploitation of EGI Grid and Cloud infrastructure to DARIAH user community. Systematic plan of approaching towards the DARIAH user community and beyond must take place by applying and integrating national roadmaps that rely on different eInfrastructure and eScience technologies. In this regard, exploitation activities will include horizontal and vertical user support. Horizontal support will be based on the dissemination of knowledge and skills and vertical by establishing of new technological solutions and environments. The grounds for this vertical component will be placed from the mini-project activities and used for designing a roadmap for sustainable A&H community involvement in EGI.


Milestones (M) and Deliverables (D)

The following gives an overview of deliverables/milestones scheduled. The 'Scope' of the deliverable defines the applicability and visibility of a deliverable. The deliverables marked as 'internal' are internal deliverables for EGI DARIAH CC and if 'external' then it is a deliverable of the EGI-Engage project that is presentable to the EC.

Code Title Lead task Lead participant Type Scope (Internal=Not sent to the EC; External=Sent to EC) Delivery PM Delivery CM
D1.1
User support and training plan
T1 RBI
R
Internal
M09
11.2015
D1.2a Report on dissemination and training activities
T1 RBI
R
Internal
M18
08.2016
D1.2b
Report on dissemination and training activities
T1 RBI
R
Internal
M30
08.2017
D2.1 Requirements collection
T2,T3,T4 RBI
R Internal M04
06.2015
D2.2 Initial technical concept and integration plan
T2 SZTAKI
R Internal M03
05.2015
D2.3a Mini-projects progress report
T2,T3,T4 GWDG
R Internal M18
08.2016
D2.3b Mini-projects progress report
T2,T3,T4 INFN
R Internal M30
08.2017
D5.1 Sustainability and user involvement plan DRAFT
T5 DANS
R Internal M12
02.2016
D6.2
Data repository for DARIAH [53] SA2.6 DARIAH INFN
OTHER
External M11
01.2016
D6.5
Final version of Multi-Source Distributed Real-Time Search and Information Retrieval application [54]
SA2.6 DARIAH GWDG
OTHER
External
M12
02.2016
D6.17
Production level gateway for Arts and Humanities [55]
SA2.6 DARIAH SZTAKI
OTHER
External
M24
02.2017
M2.1
Prototype version of the EGI-enabled application
T2


M12
02.2016
M3.1
First version of the repository in place
T3


M06
08.2015

Outputs

Deliverables

See the list of deliverables in the table above, together with links to the already produced deliverables.

Virtual organisation

The Competence Centre has established and operates a Virtual Organisation in EGI to serve users from arts and humanities. Further information is available at: EGI Virtual Organisation for arts and humanities: vo.dariah.eu

Science gateways

The competence centre is in the process of setting up three science gateway environments to provide user friendly environments to carry out data and compute intensive simulations in arts and humanities.

Applications

Survey

The Competence Centre has launched the survey to collect information from the digital arts and humanities communities about their experiences, expectations and needs concerning e-Infrastructures. The responses collected through this survey will help the EGI DARIAH Competence Centre select, deploy and sustainably operate appropriate online tools, training and user support services for digital arts and humanities groups within and beyond Europe.

Dissemination activities

List of publications

The list of publications, presentations and posters is available on google docs (access permission required)

Useful links

Available resources in the DARIAH CC

Current available resources in the Competence Centre

Type Name Description
Cloud and storage

INFN Catania 

INFN-CATANIA-STACK site capacity:

  • Number of Virtual CPU cores: 20 
  • Memory: 50GB
  • Scratch/ephemeral storage: 1TB
  • Public IP addresses: 10
  • Access type: Opportunistic
  • Middleware: Openstack
INFN Bari

INFN-BARI site capacity:

  • Number of Virtual CPU cores: 10 
  • Memory: 20GB
  • Scratch/ephemeral storage: 1TB
  • Max number of instances: 7VM
  • Public IP address: 7
  • Access type: Opportunistic
  • Middleware: Openstack
Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Print/export