Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "ELIXIR Virtual Organisation"

From EGIWiki
Jump to navigation Jump to search
 
(44 intermediate revisions by 4 users not shown)
Line 9: Line 9:


The VO is currently open for the application piloting activities of the ELIXIR Competence Centre of the EGI-Engage H2020 project. Further information about the Competence Centre is available at https://wiki.egi.eu/wiki/CC-ELIXIR. Access to the VO resources is restricted to those working on the Competence Centre activities.
The VO is currently open for the application piloting activities of the ELIXIR Competence Centre of the EGI-Engage H2020 project. Further information about the Competence Centre is available at https://wiki.egi.eu/wiki/CC-ELIXIR. Access to the VO resources is restricted to those working on the Competence Centre activities.
=== Cloud resources in the VO ===
* CESNET-MetaCloud: OpenNebula cloud;  Local site admin contact: cloud@metacentrum.cz
* IN2P3-IRES: OpenStack cloud;  Local site admin contact: grid.admin@iphc.cnrs.fr
* EMBL-EBI: OpenStack cloud: Local site admin contact: Charles Short (<cems@ebi.ac.uk>)
Under finalisation: GRNET
Under discussion: CSC, SURFsara
=== VO Managers ===
* Steven Newhouse (EMBL-EBI)
* Miroslav Ruda (CESNET)
=== VO ID card in the EGI Operations Portal ===
* http://operations-portal.egi.eu/vo/view/voname/vo.elixir-europe.org


== Acceptable use policy ==
== Acceptable use policy ==
Line 14: Line 32:
This Acceptable Use Policy applies to all members of the "vo.elixir-europe.org" Virtual Organisation, hereafter referred to as the VO, and the resources that members are able to access through the VO mechanism. Members of the EGI-Engage Competence Centre (https://wiki.egi.eu/wiki/CC-ELIXIR) owns and gives authority to this policy. This VO contributes to the ELIXIR effort of building a sustainable European infrastructure for biological information, supporting life science research and its translation to medicine, agriculture, bioindustries and society.  
This Acceptable Use Policy applies to all members of the "vo.elixir-europe.org" Virtual Organisation, hereafter referred to as the VO, and the resources that members are able to access through the VO mechanism. Members of the EGI-Engage Competence Centre (https://wiki.egi.eu/wiki/CC-ELIXIR) owns and gives authority to this policy. This VO contributes to the ELIXIR effort of building a sustainable European infrastructure for biological information, supporting life science research and its translation to medicine, agriculture, bioindustries and society.  


All VO members (users, managers, infrastructure providers) agree to be bound by this Acceptable Use Policy and to use the resources within the VO only in the furtherance of the stated goal of the VO. The VO includes core services from the EGI e-infrastructure, thus other relevant EGI Policies and Procedures also apply to certain VO member groups (to regular members, to VO managers or to infrastructure providers). See these policies at http://www.egi.eu/about/policy/policies_procedures.html.
All VO members (users, managers, infrastructure providers) agree to be bound by this Acceptable Use Policy and to use the resources within the VO only in the furtherance of the stated goal of the VO. By registering in the VO as a user you shall be deemed to accept these conditions of use:
# You shall only use the VO services to perform work, or transmit or store data consistent with the stated goals, policies and conditions of use as defined by the body or bodies granting you access.
# You shall not use the VO for any unlawful purpose and not (attempt to) breach or circumvent any administrative or security controls.
# You shall respect intellectual property and confidentiality agreements.
# You shall protect your access credentials (e.g. passwords or private keys).
# You shall immediately report any known or suspected security breach or misuse of the VO or access credentials to abuse@egi.eu and to the relevant credential issuing authorities. (aai-contact@elixir-europe.org for ELIXIR accounts)
# You must notify the Registrar of any changes to your Registration Information.
# Use of the VO is at your own risk. There is no guarantee that the VO will be available at any time or that it will suit any purpose.
# Logged information, including information provided by you for registration purposes, is used for administrative, operational, accounting, monitoring and security purposes only. This information may be disclosed, via secured mechanisms, only for the same purposes and only as far as necessary to other organisations cooperating with the VO. Although efforts are made to maintain confidentiality, no guarantees are given.
# The access-granting bodies and Resource Providers are entitled to regulate, suspend or terminate your access, within their domain of authority, and you shall immediately comply with their instructions.
# You are liable for the consequences of you violating any of these conditions of use.
# The VO includes core services from the EGI e-infrastructure, thus other relevant EGI Policies and Procedures also apply to certain VO member groups. See these policies at http://www.egi.eu/about/policy/policies_procedures.html. (Policies tagged 'Users' apply to VO members, Policies tagged 'Infrastructure' apply to cloud providers)
 
== Instruction for users ==
=== How to register ===
 
# Apply for VO membership at https://perun.elixir-czech.cz/registrar/?vo=elixir&group=EGI:vo.elixir-europe.org. You will be asked to create an account in ELIXIR, this is part of the registration process to the VO vo.elixir-europe.org.
# VO membership is received and evaluated by the VO Managers. Membership is currently restricted to those working in the ELIXIR Competence Centre application/service porting activities. Membership is expected to be broadened for other ELIXIR partners during 2017.
# You receive a notification email about the approval/rejection of your VO membership request.
 
=== How to use IaaS  clouds ===
 
==== GUI: AppDB VMOps Dashboard ====
 
The [https://appdb.egi.eu EGI Application Database (AppDB)] has recently evolved its functionalities from its catalogue of applications and virtual machines (VMs) to include a Graphical User Interface (GUI) to perform VM management operations on the distributed infrastructure.
 
Follow the [[Federated_Cloud_AppDB_VMOps_Dashboard| VMOps Dashboard guide]] to get more information about its usage. You should be able to access it with login in via EGI CheckIn and selecting ELIXIR as identity provider.
 
 
==== API and CLI ====
===== Authentication =====


By registering in the VO as a user you shall be deemed to accept these conditions of use:
Resources available on the ELIXIR VO are accessed using X.509 proxy certificates with VOMS extensions (claims stating the membership of the user to the VO). You can use your ELIXIR identity to get one valid proxy by interacting with the CILogin service, currently there are two methods:
1. You shall only use the VO services to perform work, or transmit or store data consistent with the stated goals, policies and conditions of use as defined by the body or bodies granting you access.
* Creating a client for the CILogon service, check the example at https://github.com/elixirhub/elixir-aai/tree/master/cilogon
* Using CILogon/RCAuth.eu certs on [https://elixir-cilogon-mp.grid.cesnet.cz/vo-portal/ demo VO portal]


2. You shall not use the VO for any unlawful purpose and not (attempt to) breach or circumvent any administrative or security controls.
Once you have a proxy, you need to add the VOMS extensions. This can be done with voms-proxy-init with the <code>--noregen</code> option, for example (this copies first the proxy to the default location so you don't overwrite you original one):
<pre>
cp your_proxy /tmp/x509up_u$(id -u)
chmod 600 /tmp/x509up_u$(id -u)
voms-proxy-init --noregen --rfc --voms vo.elixir-europe.org
</pre>


3. You shall respect intellectual property and confidentiality agreements.
=====OCCI and OpenStack access=====
4. You shall protect your access credentials (e.g. passwords or private keys).
5. You shall immediately report any known or suspected security breach or misuse of the VO or access credentials to abuse@egi.eu and to the relevant credential issuing authorities. (aai-contact@elixir-europe.org for ELIXIR accounts)
6. You must notify the Registrar of any changes to your Registration Information.
7. Use of the VO is at your own risk. There is no guarantee that the VO will be available at any time or that it will suit any purpose.
8. Logged information, including information provided by you for registration purposes, is used for administrative, operational, accounting, monitoring and security purposes only. This information may be disclosed, via secured mechanisms, only for the same purposes and only as far as necessary to other organisations cooperating with the VO. Although efforts are made to maintain confidentiality, no guarantees are given.
9. The access-granting bodies and Resource Providers are entitled to regulate, suspend or terminate your access, within their domain of authority, and you shall immediately comply with their instructions.
10. You are liable for the consequences of you violating any of these conditions of use.


== How to register ==
IaaS cloud resources can expose two types of interfaces towards users (one or the other or both - depending on the cloud provider):
* '''Open Standard interfaces''': OCCI ([http://occi-wg.org/ Open Cloud Computing Interface]) to manage compute, blocks storage and network resources. This interface set are currently exposed by all of the OpenNebula and Synnefo cloud providers, and some of the OpenStack providers.
* '''OpenStack interfaces''': The native OpenStack interfaces (with X.509 authentication). These interfaces are currently exposed by the OpenStack-based EGI cloud providers.
The user can interact with IaaS cloud resources via programming APIs and command line interfaces. Web dashboard access and Ansible orchestrator access are currently under development. The different access modes are summarized in the following table:


You can become member in the VO with an ELIXIR AAI account. (You can create an account here: https://www.elixir-europe.org/intranet.)
{| cellspacing="5" cellpadding="5" border="0" class="wikitable"
Application link to the VO: https://perun.elixir-czech.cz/fed/registrar/?vo=vo.elixir-europe.org.  
|-
|
| '''Open Standards interface'''
| '''OpenStack interface'''
|-
| '''API level access'''
| [[Federated_Cloud_APIs_and_SDKs#API|OCCI]]
| [[Federated_Cloud_APIs_and_SDKs#API_2|OpenStack Compute & Openstack Object Storage]]
|-
| '''Command Line access'''
| [[Federated_Cloud_APIs_and_SDKs#CLI|rOCCI-cli]]
| [[Federated_Cloud_APIs_and_SDKs#CLI_2|OpenStack CLI with VOMS authentication plugin]]
|-
| '''Web dashboard access'''
| [[Federated_Cloud_AppDB_VMOps_Dashboard |AppDB VMOps Dashboard]]
| OpenStack Horizon (in tests)
|-
| '''Orchestrator access'''
|
Known options:
* OCCOPUS: http://occopus.lpds.sztaki.hu/
* Infrastructure Manager: http://www.grycap.upv.es/im/index.php
|
Known options:
* Terraform (See below)
|}


VO membership is managed by the PERUN service at CESNET, and controlled by the VO Managers. Membership is currently restricted to members and partners of the ELIXIR Competence Centre of EGI-Engage. Membership is expected to be opened up for other ELIXIR partners in 2017.
Check out these [https://documents.egi.eu/document/2916 tutorial slides] for a practical overview on how to use the IaaS resources using the rOCCI-cli. The slides cover the following topics:
* Using pre-defined Virtual Machine images
* Customised VM deployments (contextualisation)
* Docker containers in the EGI cloud
* Preparing your own VM image (with Packer)


== VO Manager ==
=====Terraform orchestrator=====
 
====== Native OpenStack with EGI-OpenStack-Terraform ======
 
EGI provides a [https://github.com/enolfc/egi-openstack-terraform Terraform provider plugin] that  extends the builtin OpenStack provider of OpenStack with support for EGI AAI. Documentation on how to install and use is avaiable at [[Federated_Cloud_IaaS_Orchestration#Terraform|Federated Cloud IaaS Orchestration page]]
 
====== Native OpenStack with tokens ======
Terraform has OpenStack support out of the box, however it does not support the X.509 based authentication of EGI. Instead you can use [https://www.terraform.io/docs/providers/openstack/index.html#token token based authentication]. '''Tokens normally have a lifetime of 1 hour, if your deployment last longer you should check the [[Federated_Cloud_IaaS_Orchestration#Terraform|EGI OpenStack Terraform plugin]]''' . For obtaining such token, you can follow these steps:
 
* Install the OpenStack CLI, there are several ways of getting this, but an easy way is just using pip, you can use a virtualenv to ensure your regular python environment is intact:
<pre>
$ pip install python-openstackclient
</pre>
* Install the VOMS authorisation plugin
<pre>
$ pip install openstack-voms-auth-type
</pre>
* Get a list of projects where you are allowed (in this example, using the EBI access point):
<pre>
$ openstack --os-auth-url https://extcloud04.ebi.ac.uk:5000/v2.0 --os-auth-type v2voms \
            --os-x509-user-proxy /tmp/x509up_u1000 project list
+----------------------------------+--------+
| ID                              | Name  |
+----------------------------------+--------+
| e99a879a2d9e4b01b9152637c7bde4cb | elixir |
+----------------------------------+--------+
</pre>
 
In case of getting SSL errors, check the [[Federated_Cloud_APIs_and_SDKs#CA_Certificates|CA Certificates information for OpenStack CLI]]
 
* With the project you can get a token:
 
<pre>
$ export OS_AUTH_TOKEN=$(openstack --os-auth-url https://extcloud04.ebi.ac.uk:5000/v2.0 \
                                  --os-auth-type v2voms \
                                  --os-x509-user-proxy /tmp/x509up_u1000 \
                                  --os-project-id e99a879a2d9e4b01b9152637c7bde4cb \
                                  token issue -c id -f value)
 
</pre>
 
The <code>OS_AUTH_TOKEN</code> variable will be used by Terraform if available in the environment so you don't have to include in your <code>.tf</code> file.
 
====== Terraform with OCCI ======
 
There is a OCCI plugin for Terraform developed by CESNET and available at GitHub: https://github.com/cduongt/terraform/tree/occi.
 
Installation of this plugin requires compilation, check the [https://github.com/cduongt/terraform/blob/occi/README.md README file] for specific information. The plugin allows to manage VMs at OCCI endpoints


* Steven Newhouse (EMBL-EBI)
<!--
<!--
=== Guides for VO managers ===
=== Guides for VO managers ===
* To list, approve, add, remove VO members: https://voms1.egee.cesnet.cz:8443/voms/vo.elixir-europe.org/user/search.action
* To list, approve, add, remove VO members: https://perun.elixir-czech.cz/fed/gui/#grp/detail?id=9810&active=1;
 
== User support ==
 
There is a generic e-mail list provided for any topic related to the VO (operations, user support, VO users, security): [cc-elixir@mailman.egi.eu cc-elixir@mailman.egi.eu]. Please contact the VO through this e-mail address.
 
 


=== VOMS information ===
=== VOMS information ===
Line 58: Line 190:
-->
-->


== Resources in the VO ==
== Joining the VO as IaaS cloud provider ==


* CESNET cloud
The VO welcomes further IaaS cloud providers. D6.10 deliverable of the ELIXIR Competence Centre provides guidance for cloud providers on how can an IaaS cloud federate into the VO: https://documents.egi.eu/document/2841. Technology currently exist to federate OpenStack, OpenNebula and Synnefo cloud management framework based cloud sites.
* GRNET cloud
* EMBL-EBI cloud
* Under discussion: CSC cloud; SURFsara cloud; CNRS cloud


== How to join with new resources to the VO ==
Please write to <cc-elixir@mailman.egi.eu> to express your interest in joining the VO as a cloud provider.
 
The D6.10 deliverable of the EGI-Engage ELIXIR Competence Centre provides guidance for cloud providers on how to federate their site into the Virtual Organisation: https://documents.egi.eu/document/2841. Fur further support email <cc-elixir@mailman.egi.eu>.
 
* VO ID card in the EGI Operations Portal: http://operations-portal.egi.eu/vo/view/voname/vo.elixir-europe.org
 
== Support ==
 
There is a generic e-mail list provided for any topic related to the VO (operations, user support, VO users, security): [cc-elixir@mailman.egi.eu cc-elixir@mailman.egi.eu]. Please contact the VO through this e-mail address.

Latest revision as of 11:48, 3 July 2017

EGI-Engage Competence centres: Main page ELIXIR BBMRI MoBrain DARIAH LifeWatch EISCAT_3D EPOS Disaster Mitigation | EGI-Engage Knowledge Commons


Introduction

This Virtual Organisation (VO) contributes to the ELIXIR effort of building a sustainable European infrastructure for biological information, supporting life science research and its translation to medicine, agriculture, bioindustries and society. The Virtual Organisation constitutes the backbone of the ELIXIR Compute Platform and federates cloud compute and storage resources from ELIXIR and EGI providers.

The VO is currently open for the application piloting activities of the ELIXIR Competence Centre of the EGI-Engage H2020 project. Further information about the Competence Centre is available at https://wiki.egi.eu/wiki/CC-ELIXIR. Access to the VO resources is restricted to those working on the Competence Centre activities.

Cloud resources in the VO

  • CESNET-MetaCloud: OpenNebula cloud; Local site admin contact: cloud@metacentrum.cz
  • IN2P3-IRES: OpenStack cloud; Local site admin contact: grid.admin@iphc.cnrs.fr
  • EMBL-EBI: OpenStack cloud: Local site admin contact: Charles Short (<cems@ebi.ac.uk>)

Under finalisation: GRNET Under discussion: CSC, SURFsara

VO Managers

  • Steven Newhouse (EMBL-EBI)
  • Miroslav Ruda (CESNET)

VO ID card in the EGI Operations Portal

Acceptable use policy

This Acceptable Use Policy applies to all members of the "vo.elixir-europe.org" Virtual Organisation, hereafter referred to as the VO, and the resources that members are able to access through the VO mechanism. Members of the EGI-Engage Competence Centre (https://wiki.egi.eu/wiki/CC-ELIXIR) owns and gives authority to this policy. This VO contributes to the ELIXIR effort of building a sustainable European infrastructure for biological information, supporting life science research and its translation to medicine, agriculture, bioindustries and society.

All VO members (users, managers, infrastructure providers) agree to be bound by this Acceptable Use Policy and to use the resources within the VO only in the furtherance of the stated goal of the VO. By registering in the VO as a user you shall be deemed to accept these conditions of use:

  1. You shall only use the VO services to perform work, or transmit or store data consistent with the stated goals, policies and conditions of use as defined by the body or bodies granting you access.
  2. You shall not use the VO for any unlawful purpose and not (attempt to) breach or circumvent any administrative or security controls.
  3. You shall respect intellectual property and confidentiality agreements.
  4. You shall protect your access credentials (e.g. passwords or private keys).
  5. You shall immediately report any known or suspected security breach or misuse of the VO or access credentials to abuse@egi.eu and to the relevant credential issuing authorities. (aai-contact@elixir-europe.org for ELIXIR accounts)
  6. You must notify the Registrar of any changes to your Registration Information.
  7. Use of the VO is at your own risk. There is no guarantee that the VO will be available at any time or that it will suit any purpose.
  8. Logged information, including information provided by you for registration purposes, is used for administrative, operational, accounting, monitoring and security purposes only. This information may be disclosed, via secured mechanisms, only for the same purposes and only as far as necessary to other organisations cooperating with the VO. Although efforts are made to maintain confidentiality, no guarantees are given.
  9. The access-granting bodies and Resource Providers are entitled to regulate, suspend or terminate your access, within their domain of authority, and you shall immediately comply with their instructions.
  10. You are liable for the consequences of you violating any of these conditions of use.
  11. The VO includes core services from the EGI e-infrastructure, thus other relevant EGI Policies and Procedures also apply to certain VO member groups. See these policies at http://www.egi.eu/about/policy/policies_procedures.html. (Policies tagged 'Users' apply to VO members, Policies tagged 'Infrastructure' apply to cloud providers)

Instruction for users

How to register

  1. Apply for VO membership at https://perun.elixir-czech.cz/registrar/?vo=elixir&group=EGI:vo.elixir-europe.org. You will be asked to create an account in ELIXIR, this is part of the registration process to the VO vo.elixir-europe.org.
  2. VO membership is received and evaluated by the VO Managers. Membership is currently restricted to those working in the ELIXIR Competence Centre application/service porting activities. Membership is expected to be broadened for other ELIXIR partners during 2017.
  3. You receive a notification email about the approval/rejection of your VO membership request.

How to use IaaS clouds

GUI: AppDB VMOps Dashboard

The EGI Application Database (AppDB) has recently evolved its functionalities from its catalogue of applications and virtual machines (VMs) to include a Graphical User Interface (GUI) to perform VM management operations on the distributed infrastructure.

Follow the VMOps Dashboard guide to get more information about its usage. You should be able to access it with login in via EGI CheckIn and selecting ELIXIR as identity provider.


API and CLI

Authentication

Resources available on the ELIXIR VO are accessed using X.509 proxy certificates with VOMS extensions (claims stating the membership of the user to the VO). You can use your ELIXIR identity to get one valid proxy by interacting with the CILogin service, currently there are two methods:

Once you have a proxy, you need to add the VOMS extensions. This can be done with voms-proxy-init with the --noregen option, for example (this copies first the proxy to the default location so you don't overwrite you original one):

cp your_proxy /tmp/x509up_u$(id -u)
chmod 600 /tmp/x509up_u$(id -u)
voms-proxy-init --noregen --rfc --voms vo.elixir-europe.org
OCCI and OpenStack access

IaaS cloud resources can expose two types of interfaces towards users (one or the other or both - depending on the cloud provider):

  • Open Standard interfaces: OCCI (Open Cloud Computing Interface) to manage compute, blocks storage and network resources. This interface set are currently exposed by all of the OpenNebula and Synnefo cloud providers, and some of the OpenStack providers.
  • OpenStack interfaces: The native OpenStack interfaces (with X.509 authentication). These interfaces are currently exposed by the OpenStack-based EGI cloud providers.

The user can interact with IaaS cloud resources via programming APIs and command line interfaces. Web dashboard access and Ansible orchestrator access are currently under development. The different access modes are summarized in the following table:

Open Standards interface OpenStack interface
API level access OCCI OpenStack Compute & Openstack Object Storage
Command Line access rOCCI-cli OpenStack CLI with VOMS authentication plugin
Web dashboard access AppDB VMOps Dashboard OpenStack Horizon (in tests)
Orchestrator access

Known options:

Known options:

  • Terraform (See below)

Check out these tutorial slides for a practical overview on how to use the IaaS resources using the rOCCI-cli. The slides cover the following topics:

  • Using pre-defined Virtual Machine images
  • Customised VM deployments (contextualisation)
  • Docker containers in the EGI cloud
  • Preparing your own VM image (with Packer)
Terraform orchestrator
Native OpenStack with EGI-OpenStack-Terraform

EGI provides a Terraform provider plugin that extends the builtin OpenStack provider of OpenStack with support for EGI AAI. Documentation on how to install and use is avaiable at Federated Cloud IaaS Orchestration page

Native OpenStack with tokens

Terraform has OpenStack support out of the box, however it does not support the X.509 based authentication of EGI. Instead you can use token based authentication. Tokens normally have a lifetime of 1 hour, if your deployment last longer you should check the EGI OpenStack Terraform plugin . For obtaining such token, you can follow these steps:

  • Install the OpenStack CLI, there are several ways of getting this, but an easy way is just using pip, you can use a virtualenv to ensure your regular python environment is intact:
$ pip install python-openstackclient
  • Install the VOMS authorisation plugin
$ pip install openstack-voms-auth-type
  • Get a list of projects where you are allowed (in this example, using the EBI access point):
$ openstack --os-auth-url https://extcloud04.ebi.ac.uk:5000/v2.0 --os-auth-type v2voms \
            --os-x509-user-proxy /tmp/x509up_u1000 project list
+----------------------------------+--------+
| ID                               | Name   |
+----------------------------------+--------+
| e99a879a2d9e4b01b9152637c7bde4cb | elixir |
+----------------------------------+--------+ 

In case of getting SSL errors, check the CA Certificates information for OpenStack CLI

  • With the project you can get a token:
$ export OS_AUTH_TOKEN=$(openstack --os-auth-url https://extcloud04.ebi.ac.uk:5000/v2.0 \
                                   --os-auth-type v2voms \
                                   --os-x509-user-proxy /tmp/x509up_u1000 \
                                   --os-project-id e99a879a2d9e4b01b9152637c7bde4cb \
                                   token issue -c id -f value)

The OS_AUTH_TOKEN variable will be used by Terraform if available in the environment so you don't have to include in your .tf file.

Terraform with OCCI

There is a OCCI plugin for Terraform developed by CESNET and available at GitHub: https://github.com/cduongt/terraform/tree/occi.

Installation of this plugin requires compilation, check the README file for specific information. The plugin allows to manage VMs at OCCI endpoints


Joining the VO as IaaS cloud provider

The VO welcomes further IaaS cloud providers. D6.10 deliverable of the ELIXIR Competence Centre provides guidance for cloud providers on how can an IaaS cloud federate into the VO: https://documents.egi.eu/document/2841. Technology currently exist to federate OpenStack, OpenNebula and Synnefo cloud management framework based cloud sites.

Please write to <cc-elixir@mailman.egi.eu> to express your interest in joining the VO as a cloud provider.