Difference between revisions of "EGI-Engage:PY2 review"
|(One intermediate revision by one other user not shown)|
|Line 197:||Line 197:|
TSA6.10: Disaster Mitigation:
TSA6.10: Disaster Mitigation:
The Disaster Mitigation Competence Centre (DMCC) of the EGI-Engage project designs, develops and makes available customised IT services for the support of climate and disaster mitigation research, During PY2 the collaboration implemented three scientific case study demonstrators that use of the WRF and iCOMCOT simulation portals that had been published online by the same group earlier in PY2. The demonstrators use e-infrastructure resources from the Asia-Pacific regional infrastructure of EGI. The case studies advance our understanding of three disastrous events that affected Asia recently: typhoon Haiyan in the Philippines in 2013, extreme floods in Malaysia in 2014-15, and forest fires in Southeast Asia in 2015. During PY3 the demonstrators will be used to broaden the e-infrastructure engagement between Europe and Asia, and to establish a partnership that can last beyond the project.
The Disaster Mitigation Competence Centre (DMCC) of the EGI-Engage project designs, develops and makes available customised IT services for the support of climate and disaster mitigation research, During PY2 the collaboration implemented three scientific case study demonstrators that use of the WRF and iCOMCOT simulation portals that had been published online by the same group earlier in PY2. The demonstrators use e-infrastructure resources from the Asia-Pacific regional infrastructure of EGI. The case studies advance our understanding of three disastrous events that affected Asia recently: typhoon Haiyan in the Philippines in 2013, extreme floods in Malaysia in 2014-15, and forest fires in Southeast Asia in 2015. During PY3 the demonstrators will be used to broaden the e-infrastructure engagement between Europe and Asia, and to establish a partnership that can last beyond the project
Latest revision as of 15:51, 24 April 2017
|EGI-Engage project:||Main page||WP1(NA1)||WP3(JRA1)||WP5(SA1)||PMB||Deliverables and Milestones||Quality Plan||Risk Plan||Data Plan|
|WP2(NA2)||WP4(JRA2)||WP6(SA2)||AMB||Software and services||Metrics||Project Office||Procedures|
- ISO 9k and 20k certification
- 2 risk registry reviews conducted
- New website
- EGI service catalogue brochure
- Events: supported DI4R execution and EGI Conference 2017 preparation
- Task 2.2 (Sergio/Sy)
- Integrated Management System implemented + ISO9000/ISO2000 Certification
- Pay for use in production (5 total providers signed LoI - 2)
- 2 new pay-for-use providers having sent formal Letter of Intent
- Request via wiki + email in place
- Request via the EGI Marketplace in progress
- Service portfolio approved / Catalogue published
- Procurement study
- Task 2.3 (Sy/Roberta)
- MoUs: CloudSME, Terradue
- Market Analysis
- Marine Fisheries Market Analysis
- As follow-up, there is not a defined strategy through BlueBridge; structured business model for collaboration between EGI, BlueBridge and early adopters
- Agriculture Market Analysis
- Thanks to the inclusion of agriculture (GRNET/AgroKnow) within EGI-Engage, EGI was invited into AGINFRA+ project consortium, to which collaboration is continuing https://www.egi.eu/about/newsletters/aginfra-food-and-agriculture-research/
- Success stories:
- IBM Research through PSNC (paid)
- Pilot on hybrid cloud for exploiting Copernicus data
- NUMECA SME: training support with EGI FedCloud
- UberCloud Case Study
- Expanded the network of relationships with key stakeholders
- EO sector (e.g. DG Grow, large companies, SMEs)
- 30 business pilots submissions to EINFRA-12 (at least 6 resulted from NGIs local engagement thanks to EGI-Engage SME engagement activities)
- 145 Industry contacts in DB
- Technical enhancement on CheckIn service
- Community and operations tools integration
- Service order management process
- Integration with LTOS platform and e-Grant
- PrestaShop Marketplace prototype
- OpenIRIS prototype
- Technology assessment
- Service order management process
- New accounting portal
- New Cloud Usage Record
- Data Accounting prototype
- GOCDB, ARGO, Ops Portal new releases
- New messaging infrastructure
- Activities re-organised according to the outcome of JRA1.2
- Release of first prototype of Open Data Platform with support for open data management, DOI and PID support and OAI-PMH interfaces for open data metadata publishing
- Establishment of EGI DataHub service based on Open Data Platform serving as a repository for reference data sets, currently mirroring a subset of Sentinel-2 data set
- OCCI 1.2 released for OpenStack and OpenNebula, including support for snapshots and resizing of images.
- Design and first implementation of new image distribution method, providing better integration with CMFs.
- gCube/D4Science integrated into the EGI Federated Cloud, enabling the use of EGI resources for users of the VREs
- CANFAR integration ready to accept credentials from Europe and Canada, VOSpace prototype available.
- EGI/EUDAT use cases showing the combined use of both infrastructures started and first pilots deployed.
- Establish GPGPU specific VO acc-comp.egi.eu for accelerated computing, create VO-specific image list with pre-installed GPGPU software
- Support for GPGPU on OpenNebula (site deployment and certification, upgrade of rOCCI server to support GPGPU in resource template)
- Deployment and experimentation with support GPGPU via nova-lxd (LXC container instead of KVM hypervisor)
- Operational integration of new services such as WebDAV, DataHub and CheckIn
- Harmonization of cloud and HTC service provisioning
- New SLAs
- Security: threat risk assessment follow up and mitigations
- Strengthen of the site security through new procedures and security trainings
- Security: international collaboations
- Support of iMarine and ESA TEP use cases
- Long tail of science
- Reusable content (reference material) on cloud CMDline & API; Specialised services in the cloud (Docker, Hadoop); DataHub; Security role-play
- Delivered 15 training events (5 webinars, 7 f2f, 1 remote EGI tutorial). These reached approx 300 participants in total (over 20/event on average).
Technical support for new communities (SA2.2):
- Most significant new communities: EMSOdev, ICOS, Geohazard Exploitation Platform, BILS
- 10 new SLAs were arranged (See respective KPI)
- Finalised and released the Application platform for the Long-tail of science
- Pilot use of EGI cloud and storage services by EMSO, ICOS, Euro-Argo
- Delivered two 'Design your e-infrastructure' events co-located with EGI Forums. ICOS, Euro-Argo, ELItrans, ERIC CLL are actively supported since then.) The workshop format is now adopted by the Dutch NGI for national community support
- ISO-certification for Opportunity prioritization and CRM process (incl. regular interviews with customers)
Competence centres (SA2.3-10):
- Infrastructure integration concluded for early adopters: ELIXIR, DARIAH, BBMRI. They entered into application development and external promotion phase.
- Application demonstrators have been setup by EPOS and DMCC
- Active exploitation of services with uptake of new services: MoBrain (GPGPU), LifeWatch (cloud)
DETAILED VIEW ON CCs:
- We established an ELIXIR VO which can be accessed by the users with their ELIXIR accounts (no need for X509 certificates!). CESNET, EBI and CNRS clouds are federated in the VO.
- We simplified the way how OpenStack clouds can federate into EGI VOs.
- We piloted Terraform orchestrator integration with OCCI and OpenStack Nova on the VO.
- Life science se cases started integrating with the ELIXIR VO clouds: Metagenomics from CSC, Insyght Comparative Genomics from CNRS, PhenoMeNal from EBI.
- We started an infrastructure compatibility study with the JetStream cloud system in the US (to see whether life science communities could exchange of applications between Europe and the US).
- Developed an EGI-specific extension of the BioBankCloud platform
- Integrated with SAML-based authentication (practical implementation tested with Shibboleth) for user authentication - note that BiobankCloud combines this with affiliation to projects so that the actual access control is based on person_project level.
- Deployed at Karolinska Instituet (BBMRI.se): 2PB of storage, 28x48 cores=1344 cores, 7.1TB of RAM, 4x10Gb network connections
- Preparation of deployment at Masaryk Memorial Cancer Institute (BBMRI_CZ).
- Identified biobank workflows:
- genomics (DNA/RNA) pipelines and proteomics pipelines, with primary interests by BBMRI-NL and BBMRI_CZ respectively (these are common pipelines typical for many biobanks and BBMRI-ERIC national nodes).
- BBMRI_CZ has tested switching from GUI tools to command-line tools capable of integration into the automated workflows.
- Setup collaboration between SurfSARA (EUDAT representative) and BBMRI-ERIC to start assigning ePIC Peristent Identifiers (PIDs). Contract signed by BBMRI-ERIC at the end of March 2017.
- BBMRI-ERIC works on developing policy for assigning PIDs.
- Ported AMBER, DisVis and PowerFit to GPGPUs, then integrated into the respective WeNMR portals:
- The portals are in production with a growing user community and making use of INDIGO-Datacloud solutions.
- Created a Virtualised Appliance from the Scipion framework (to obtain 3D maps of macromolecular complexes).
- This image is being used internally within the CNB-CSIC on multiples projects, including Pharma projects, and it is been made widely known through workshops, with the most recent one the cryo-EM workshop at ISGC2017 where also DisVis, PowerFit and AMBER were demonstrated..
- The VM is available from the EGI appdb: https://appdb.egi.eu/store/vappliance/scipion.v1.0
- Contributed to the development of a solution enabling GPU support in CREAM-CE
- Science gateway was opened for the community, integrating the following applications:
- Semantic Search Engine
- Cloud access
- Workflow development and management
- Data transfer
- Gateway integrated with EduGain and DARIAH IdP
- SLA agreement signed between DARIAH CC and 2 FedCloud providers (INFN-Catania, and INFN-Bari)
- Promotion of the services officially started on DARIAH Annual meeting, 10-12 Oct 2016 in Gent, Belgium
- Other presentations in community:
- EURALEX Conference, 6-10 September 2016, Tbilisi, Georgia (workshop, oral presentation)
- DI4R 2016, 28-30 September 2016, Krakow, Poland (presentation)
- Digital Humanities 2016, 11-16 July 2016, Krakow, Poland (poster)
- e-Infrastructure Days at University Computing Centre, 23-25 May 2016, Zagreb, Croatia (presentation)
- EGI Conference, 6-8 April 2016, Amsterdam, Netherlands (presentation)
- Demonstrators and applications
- Paper published:
- Wandl-Vogt, Eveline; Barbera, Roberto; La Rocca, Giuseppe; Calanducci, Antonio; Carrubba, Carla; Inserra, Giuseppina; Kalman, Tibor; Sipos, Gergely; Farkas, Zoltan; Davidović, Davor. Furthering the Exploration of Language Diversity and Pan-European Culture: The DARIAH-CC Science Gateway for Lexicographers. Proceedings of the XVII EURALEX International Congress - Lexicography and Linguistic Diversity, 2016, pp. 292-298
- User feedback
- Reports from Google analytics (1st March 2017 - 30 March 2017): 102 users (18 registered), 365 sessions, 10 countries
- Other presentations in community:
- The VRE Marine in operation for last year, and being extended to non-marine topics (terrestrial VRE) - Daily usage by the researchers at VLIZ and HCMR centres to assist in the collection and processing of the observations providing the monitoring measurements of the different marine areas in Europe
- GBIF.es and GBIF.pt, including data support is operating on FedCloud resources in full production (alongside other local resources) - Currently approx 10.000 users.
- A new Citizen Science Platform, Natusfera, operating on FedCloud resources. The automatic pattern recognition tool for species being integrated, reaching outstanding levels thanks to deep learning on GPUs - Approx 1000 users with 5000 images in the last 12 months.
- New features have been added to the EISCAT_3D portal
- Data access based on EISCAT data access policy
- Data downloading refined
- Submitting files for processing
- Data Model for EISCAT_3D adopted to the present EISCAT data for evaluation of Dirac
- Early adoption within the community
- Step-by-step users guide is prepared: https://www.eiscat.se/wp-content/uploads/2016/03/DIRAC-instruction.pdf
- Active tests for debugging the DIRAC portal
- Internal feedback from EISCAT HQ staff
- one tester at the Swedish Institute of Space Physics
- Discussion to add two more from NIPR in Japan and SGO in Finland
- Presentations in community:
- EISCAT_3D user meeting and Optical meeting 2016
- Smaller meetings: SGO observatory days 2017, NEIC user group, EISCAT Scientific Committe
- 3 infrastructure assessment pilots have been reached by the end of PY2, with the use of cloud and AAI services from EGI:
- AAI (Accessing EPOS Distributed Integrated Core Services via EGI CheckIn). The setup is planned to be integrated with the EPOS ICS-C science gateway by fall 2017.
- Cloud computing (Earthquake simulation on EGI cloud resources from the VERCE gateway). During the rest of the project a Volcanological research applications will be developed in the VERCE framework, with the scope of scaling all the generic components to a larger user-base and going towards the characterization of the EPOS Computational EarthScience strategy.
- Connecting the DInSAR processing chain with EGI Cloud compute and storage services with the use of Terradue's Geohazard Thematic Exploitation Platform. Members of the EPOS community will test and validate the service during PY3. In addition, the integration of the Geohazard Exploitation Platform and EGI Federated Cloud will be used to run new Satellite Data pilots driven by emerging use cases from the EPOS community.
TSA6.10: Disaster Mitigation:
- The Disaster Mitigation Competence Centre (DMCC) of the EGI-Engage project designs, develops and makes available customised IT services for the support of climate and disaster mitigation research, During PY2 the collaboration implemented three scientific case study demonstrators that use of the WRF and iCOMCOT simulation portals that had been published online by the same group earlier in PY2. The demonstrators use e-infrastructure resources from the Asia-Pacific regional infrastructure of EGI. The case studies advance our understanding of three disastrous events that affected Asia recently: typhoon Haiyan in the Philippines in 2013, extreme floods in Malaysia in 2014-15, and forest fires in Southeast Asia in 2015. During PY3 the demonstrators will be used to broaden the e-infrastructure engagement between Europe and Asia, and to establish a partnership that can last beyond the project.
- Project-wide issues to report:
- Difficult exploitation of ICT services due to scattered RI community and lack of coordinated exploitation of services at the level of RI management.
- Some life Science communities are using cloud services (public/commercial) locally (for local user community). Federation of data/applications seems to be more interesting, using local compute infrastructures according to the locally applicable access policies. RI access policies need to be sorted out (which user communities from which suppliers).
- Biobank cloud service deployed in SE. Now under deployment in CZ. Status of deployment needs to be investigated.