Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Federated Cloud OpenNebula guide"

From EGIWiki
Jump to navigation Jump to search
(Created page with "{{Fedcloud_Menu}} {{TOC_right}} Category:Federated_Cloud EGI Federated Cloud Site based on OpenNebula is an ordinary OpenNebula installation with some EGI-specific integr...")
 
(Redirected page to MAN10)
 
Line 1: Line 1:
{{Fedcloud_Menu}} {{TOC_right}}
#REDIRECT [[MAN10]]
 
[[Category:Federated_Cloud]]
 
 
EGI Federated Cloud Site based on OpenNebula is an ordinary OpenNebula installation with some EGI-specific integration components. There are no additional requirements placed on internal site architecture.
Follow [http://opennebula.org/documentation/ OpenNebula Documentation] if you need advice on how to install and configure OpenNebula itself.
 
'''The following OpenNebula versions are supported:'''
* OpenNebula v5.2.x
* OpenNebula v5.4.x
 
'''Integration Prerequisites:'''
* Working OpenNebula installation.
* Valid IGTF-trusted host certificates for selected hosts.
 
'''Please consider that:'''
* '''CDMI''' storage endpoints are currently '''not supported''' for OpenNebula-based sites.
* OpenNebula GUI integration is '''not''' supported.
 
[[File:OpenNebulaSite.png]]
 
The following '''components''' must be installed:
* '''rOCCI-server''' -- provides a standard virtual machine management interface. Configuration is [[#EGI_Virtual_Machine_Management|explained bellow]].
* '''keystorm''' -- serves federated authentication and authorization. Configuration is [[#EGI_User_Management|explained bellow]].
* '''cloudkeeper''' and '''cloudkeeper-one''', synchronize site with appliances from [https://appdb.egi.eu/browse/cloud EGI AppDB] Configuration is [[#EGI_Image_Management|explained bellow]].
* '''oneacct-export''' and '''apel-ssm''' -- collect accounting and publishe it into EGI's accounting database. Configuration is [[#EGI_Accounting|explained bellow]].
* '''cloud-info-provider''' and '''BDII''', register site in the EGI Information System. Configuration is [[#EGI_Information_System|explained bellow]].
 
The following '''ports''' must be open to allow access to an OpenNebula-based FedCloud sites:
 
{| class="wikitable" style="margin: auto; margin-top: 30px; margin-bottom: 30px;"
|+ Open Ports for OpenNebula and other components
! style="width: 90px;" | Port
! style="width: 110px;" | Application
! style="width: 430px;" | Host
! style="width: 250px;" | Note
|-
|'''2633'''/TCP
|'''OpenNebula'''/XML-RPC
|'''OpenNebula'''
|Communication between integration components and OpenNebula.
|-
|'''2170'''/TCP
|'''BDII'''/LDAP
|'''cloud-info-provider'''/'''BDII'''
|EGI Service Discovery/Information System
|-
|'''11443'''/TCP
|'''OCCI'''/HTTPS
|'''rOCCI-server'''
|EGI Virtual Machine Management
|-
|'''5000'''/TCP
|'''keystorm'''/HTTPS
|'''keystorm'''
|EGI User Management
|-
|'''50505'''/TCP
|'''cloudkeeper'''/HTTP
|'''cloudkeeper'''
|EGI Image Management, needs to be accessible from '''cloudkeeper-one''' node only
|-
|'''50051'''/TCP
|'''cloudkeeper-one'''/gRPC
|'''cloudkeeper-one'''
|EGI Image Management, needs to be accessible from '''cloudkeeper''' node only
|}
 
There are no additional requirements for '''OpenNebula''' hosts used to run virtual machines.
 
This is an overview of '''service accounts''' used in an OpenNebula-based site. The names are default and can be changed if required.
 
{| class="wikitable" style="margin: auto; margin-top: 30px; margin-bottom: 30px;"
|+ Service Accounts
! style="width: 90px;" | Type
! style="width: 110px;" | Account name
! style="width: 180px;" | Host
! style="width: 500px;" | Use
|-
|rowspan="6"|System accounts
|<code>rocci</code>
|rOCCI-server
|Service account for '''rOCCI-server'''. It is only a service account, no access required.
|-
|<code>keystorm</code>
|keystorm
|Service account for '''keystorm'''. It is only a service account, no access required.
|-
|<code>apel</code>
|oneacct-export/APEL
|Service account for '''oneacct-export/APEL'''. Just a service account, no access required.
|-
|<code>openldap</code>
|cloud-info-provider/BDII
|Service account for '''cloud-info-provider/BDII'''. Just a service account, no access required.
|-
|<code>cloudkeeper</code>
|cloudkeeper
|Service account for '''cloudkeeper'''. Just a service account, no access required.
|-
|<code>cloudkeeper-one</code>
|cloudkeeper-one
|Service account for '''cloudkeeper-one'''. Just a service account, no access required.
|}
 
== EGI Virtual Machine Management ==
=== Prerequisites ===
Enable EPEL and install the following packages prior to installation:
  yum install -y epel-release wget
 
=== Installation ===
rOCCI-server is distributed as package for multiple Linux distributions which is available in AppDB. This guide will expect CentOS 7 distribution but installation on any other supported distribution is very similar.
 
* Register <code>rOCCI-server</code> repositories
  wget http://repository.egi.eu/community/software/rocci.server/2.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/rocci-server.repo
* Install package
  yum install -y occi-server
 
=== Configuration ===
* Make rOCCI-server listen on a public interface
<pre>
mkdir -p /etc/systemd/system/occi-server.socket.d
cat > /etc/systemd/system/occi-server.socket.d/override.conf <<EOS
[Socket]
# lines below are NOT duplicated by mistake
ListenStream=
ListenStream=0.0.0.0:11443
EOS
</pre>
<pre>
sed -i 's/HOST=127.0.0.1/HOST=0.0.0.0/g' /etc/occi-server/variables
</pre>
* Uncomment and configure optional parameters in ''/etc/occi-server/variables''
<pre>
export HOST_CERT=/path/to/cert                                    # host certificate readable by the rocci user
export HOST_KEY=/path/to/key                                      # host key readable by the rocci user
</pre>
<pre>
export ROCCI_SERVER_KEYSTONE_URI=https://localhost:5000/          # URL pointing to keystorm installation
</pre>
<pre>
export ROCCI_SERVER_OPENNEBULA_ENDPOINT=http://localhost:2633/RPC2 # URL pointing to OpenNebula installation
</pre>
<pre>
export ROCCI_SERVER_ENCRYPTION_TOKEN_CIPHER=                      # crypto options MUST MATCH keystorm's crypto options, see /etc/keystorm/variables
export ROCCI_SERVER_ENCRYPTION_TOKEN_KEY=                          # crypto options MUST MATCH keystorm's crypto options, see /etc/keystorm/variables
export ROCCI_SERVER_ENCRYPTION_TOKEN_IV=                          # crypto options MUST MATCH keystorm's crypto options, see /etc/keystorm/variables
</pre>
* Enable and start the service
<pre>
systemctl enable occi-server
systemctl start occi-server
</pre>
 
=== Runtime ===
* Import resource templates to OpenNebula
<pre>
/opt/occi-server/bin/oneresource create --endpoint http://one.example.org:2633/RPC2 # --username PRIVILEGED_USER --password PASSWD
# re-run with `--resources /opt/occi-server/embedded/app/rOCCI-server/lib/resources/gpu/` to enable GPU resource templates
</pre>
* In OpenNebula, set flags for groups by adding attributes:
<pre>
DEFAULT_CLUSTER_ID="0"              # Default cluster for this group
DEFAULT_CONNECTIVITY="public"      # Default connectivity for this group: public|nat|private
</pre>
* In OpenNebula, set network type on networks used via OCCI by adding an attribute:
<pre>
NETWORK_TYPE="public"              # Supported types: public|nat|private
</pre>
* In OpenNebula, set flag for networks that should be treated as public IP pools (for IP reservations) by adding an attribute:
<pre>
FLOATING_IP_POOL="yes"
</pre>
* In OpenNebula, set additional network attributes:
<pre>
NETWORK_ADDRESS=""                  # e.g., "172.16.100.0"
NETWORK_MASK=""                    # e.g., "255.255.255.0"
GATEWAY=""                          # e.g., "172.16.100.1"
</pre>
 
=== Migration from v1 to v2 ===
In order to migrate from rOCCI-server v1 with Perun-managed user accounts, perform the following steps.
 
==== Preparation ====
* Disconnect direct propagation (slave scripts)
* Remove all user accounts that do not have any resource allocations
 
==== Migration ====
* Merge multiple single-group accounts into one account with multiple groups
<pre>
Single-group accounts owned by the same person can be identified as having:
 
* `NAME` following the naming convention $VONAME_$ID where the same user always has the same $ID
* `TEMPLATE/X509_DN` where the same user always has the same DN
</pre>
<pre>
Name of the merged user MUST be a SHA256 digest of the `TEMPLATE/X509_DN` attribute value.
 
In ruby, SHA256 digest can be generated as:
 
require 'digest'
Digest::SHA256.hexdigest 'DN_STRING_HERE'
</pre>
* Manually add user attributes
<pre>
For each user, add the following attributes:
 
* TEMPLATE/ID
* TEMPLATE/NAME
* TEMPLATE/IDENTITY
* TEMPLATE/AUTHENTICATION
 
Where
 
* `TEMPLATE/ID` is a SHA256 digest of the `TEMPLATE/X509_DN` attribute value
* `TEMPLATE/IDENTITY` and `TEMPLATE/NAME` contain the old `TEMPLATE/X509_DN` value
* `TEMPLATE/AUTHENTICATION` is a static value 'voms'
</pre>
* ''chown'' all user-owned resources to the new user
 
== EGI User Management ==
=== Prerequisites ===
Enable EPEL and install the following packages prior to installation:
  yum install -y epel-release wget
 
=== Installation ===
keystorm is distributed as package for multiple Linux distributions which is available in AppDB. This guide will expect CentOS 7 distribution but installation on any other supported distribution is very similar.
 
* Register <code>keystorm</code> repositories
  wget http://repository.egi.eu/community/software/keystorm/1.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/keystorm.repo
* Install package
  yum install -y keystorm
 
=== Configuration ===
* Uncomment and configure optional parameters in ''/etc/keystorm/variables''
<pre>
export KEYSTORM_OPENNEBULA_ENDPOINT=http://localhost:2633/RPC2    # URL pointing to OpenNebula installation
export KEYSTORM_OPENNEBULA_SECRET=oneadmin:opennebula              # Privileged OpenNebula credentials (with user and group management permissions)
</pre>
* Enable and start the service
<pre>
systemctl enable keystorm
systemctl start keystorm
</pre>
* Configure Apache2/httpd
<pre>
# on Ubuntu/Debian only
a2enmod ssl && \
  a2enmod headers && \
  a2enmod proxy && \
  a2enmod proxy_http && \
  a2enmod remoteip && \
  a2enmod auth_openidc && \
  a2enmod zgridsite
</pre>
<pre>
# make sure the following files exist
SSLCertificateFile /etc/grid-security/hostcert.pem
SSLCertificateKeyFile /etc/grid-security/hostkey.pem
 
# make sure the following directory exists
SSLCACertificatePath /etc/grid-security/certificates
</pre>
* Enable and start Apache2/httpd
<pre>
# on Ubuntu/Debian only
systemctl enable apache2
systemctl restart apache2
</pre>
<pre>
# on CentOS/SL only
systemctl enable httpd
systemctl start httpd
</pre>
* Enable support for EGI VOs via VOMS: [[HOWTO16 | VOMS configuration]]
* Enable support for EGI VOs via OIDC: ''TBD''
 
=== Runtime ===
* In OpenNebula, create empty groups for ''fedcloud.egi.eu'', ''ops'', and ''dteam'' with group attribute:
<pre>
KEYSTORM="YES"                  # Allow keystorm to manage membership for this group
</pre>
 
== EGI Accounting ==
 
=== Prerequisites ===
<code>oneacct-export</code> uses '''Secure Stomp Messenger''' to send accounting records to the central repository. Please, refer to <code>ssm</code> documentation for [https://github.com/apel/ssm installation instructions]. By default, accounting records are placed in <code>/var/spool/apel/outgoing/00000000</code>. You '''have to''' configure and run <code>ssmsend</code> periodically, this is not handled by oneacct-export.
 
Enable EPEL and install the following packages prior to oneacct-export installation:
  yum install -y epel-release wget
 
=== Installation ===
 
oneacct-export is distributed as package for multiple Linux distributions which is available in AppDB. This guide will expect CentOS 7 distribution but installation on any other supported distribution is very similar.
 
* Register <code>oneacct-export</code> repositories
  wget http://repository.egi.eu/community/software/oneacct.export/0.4.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/oneacct-export.repo
* Install package
  yum install -y oneacct-export
 
=== Configuration ===
* Edit <code>/etc/oneacct-export/conf.yml</code>
 
  apel:
    site_name: Undefined                    # Usually a short provider name, e.g. CESNET
    cloud_type: OpenNebula                  # CMF type, only OpenNebula is supported
    endpoint: https://localhost.edu:11443/ # Public URL of your OCCI endpoint
 
  xml_rpc:
    secret: oneadmin:opennebula            # OpenNebula credentials, privileged
    endpoint: http://localhost:2633/RPC2 # OpenNebula XML RPC endpoint
 
* Add the following lines to <code>/etc/one/oned.conf</code> and restart OpenNebula
  INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_AD_MPURI"
  INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_DC_IDENTIFIER"
  INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_IL_DC_IDENTIFIER"
  INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_SL_CHECKSUM_SHA512"
  INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_HV_VERSION"
 
* Set benchmark values on CLUSTERs (applies to all hosts in the cluster) or HOSTs (only for that host) in OpenNebula
  BENCHMARK_TYPE  = "HEP-SPEC06" # benchmark type
  BENCHMARK_VALUE = "84.46"      # represents a per-core measured value of said benchmark
 
* Use <code>/etc/oneacct-export/groups.include</code> or <code>/etc/oneacct-export/groups.exclude</code> to control which information gets exported. Specify one group name per line.
 
=== Usage ===
* Enable and register service 'redis'
  service redis start
  chkconfig redis on
 
* Enable and register service 'oneacct-export-sidekiq'
  service oneacct-export-sidekiq start
  chkconfig oneacct-export-sidekiq on
 
* Perform the first export manually
  # This process may take a long time, consider using '''tmux''' or '''screen'''
  sudo -u apel /usr/bin/oneacct-export-cron --all
 
* Enable and register service 'oneacct-export-cron'
  service oneacct-export-cron start
  chkconfig oneacct-export-cron on
 
This service registers a cron job which will run oneacct-export every 2 hours.
 
== EGI Information System ==
 
Sites must publish information to EGI information system which is based on BDII. There is a common [https://github.com/EGI-FCTF/cloud-bdii-provider bdii provider] for all cloud management frameworks. Information on installation and configuration is available in the cloud-bdii-provider [https://github.com/EGI-FCTF/cloud-bdii-provider/blob/master/README.md README.md] and in the [[Fedclouds BDII instructions]], there is a [[Fedclouds_BDII_instructions#OpenNebula_.2B_rOCCI|specific section with OpenNebula details]].
 
== EGI VM Image Management ==
[https://github.com/the-cloudkeeper-project/cloudkeeper cloudkeeper] and [https://github.com/the-cloudkeeper-project/cloudkeeper-one cloudkeeper-one] are tools used to ensure synchronization of virtual appliances with an [https://opennebula.org/ OpenNebula]-based cloud.
 
[[File:Cloudkeeper-setup.png]]
 
=== Prerequisites ===
<code>cloudkeeper</code> uses VO-wide image lists provided by AppDB to synchronize virtual appliances to clouds. In order to use VO-wide image lists you need to have a valid access token to AppDB. Check [https://wiki.appdb.egi.eu/main:faq:how_to_get_access_to_vo-wide_image_lists how to access to VO-wide image lists] and [https://wiki.appdb.egi.eu/main:faq:how_to_subscribe_to_a_private_image_list_using_the_vmcatcher how to subscribe to a private image list] documentation for more information.
 
* Install recent <code>qemu-img</code> and <code>wget</code>
  yum install -y centos-release-qemu-ev wget sudo
 
=== Installation ===
Both <code>cloudkeeper</code> and <code>cloudkeeper-one</code> are distributed as packages for multiple Linux distributions which are available in AppDB. This guide will expect CentOS 7 distribution but installation on any other supported distribution is very similar.
* Register <code>cloudkeeper</code> and <code>cloudkeeper-one</code> repositories
wget http://repository.egi.eu/community/software/cloudkeeper/1.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/cloudkeeper.repo
wget http://repository.egi.eu/community/software/cloudkeeper.one/1.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/cloudkeeper-one.repo
 
* Install <code>cloudkeeper</code> and <code>cloudkeeper-one</code>
yum install -y cloudkeeper cloudkeeper-one
 
=== <code>cloudkeeper</code> configuration ===
<code>cloudkeeper</code> configuration file can be found in <code>/etc/cloudkeeper/cloudkeeper.yml</code>.
 
;image-lists
: URLs of image lists containing appliances which you want to synchronize to your cloud. Must contain authentication token.
  image-lists: # List of image lists to sync against
    - https://APPDB_TOKEN:x-oauth-basic@vmcaster.appdb.egi.eu/store/vo/somevo/image.list
    - https://APPDB_TOKEN:x-oauth-basic@vmcaster.appdb.egi.eu/store/vo/othervo/image.list
;authentication
: Says whether <code>cloudkeeper</code> and <code>cloudkeeper-one</code> will communicate securely via TLS. This requires options <code>certificate</code>, <code>key</code> and <code>backend->certificate</code> to be properly set.
;image-dir
: Directory where images will be downloaded and converted before uploading to OpenNebula. Directory is cleaned after each appliance registration/update nonetheless, it should provide sufficient free space (some runs may require up to 200GB of free space).
;remote-mode
: Says whether to serve downloaded images via web server or to copy them locally. Should be <code>true</code> especially if OpenNebula is running on different machine than <code>cloudkeeper</code> and <code>cloudkeeper-one</code>.
;nginx->ip-address
: IP address on which NGINX will serve images in remote mode. This address MUST be accessible from the machine hosting <code>cloudkeeper-one</code> and your OpenNebula installation.
;formats
: List of image formats images can be converted to and are supported by the cloud.
 
=== <code>cloudkeeper-one</code> configuration ===
<code>cloudkeeper-one</code> configuration file can be found in <code>/etc/cloudkeeper-one/cloudkeeper-one.yml</code>.
 
;authentication
: Says whether <code>cloudkeeper</code> and <code>cloudkeeper-one</code> will communicate securely via TLS. This requires options <code>certificate</code>, <code>key</code> and <code>core->certificate</code> to be properly set.
;appliances->tmp-dir
: Directory images will be copied to before registration in OpenNebula when in non-remote mode.
;appliances->template-dir
: Directory for ERB-enabled templates of OpenNebula images and templates used for registration. More information in the next section.
;opennebula->datastores
: List of OpenNebula datastores images are uploaded to.
;opennebula->allow-remote-source
: Allows OpenNebula to directly download images in remote mode.
 
=== Templates configuration ===
The directory specified by option <code>appliances->template-dir</code> contains templates for OpenNebula images and templates in files <code>image.erb</code> and <code>template.erb</code>. These files can be customized to register images and templates according to your needs. Files are using standard ERB templating mechanism. By default, these files can be found in <code>/etc/cloudkeeper-one/templates/</code>.
 
* <code>image.erb</code> available variables:
;name
: Name, under which will the image be registered
;appliance
: Appliance object. Contains following attributes: <code>identifier</code>, <code>title</code>, <code>description</code>, <code>mpuri</code>, <code>group</code>, <code>ram</code>, <code>core</code>, <code>version</code>, <code>architecture</code>, <code>operating_system</code>, <code>vo</code>, <code>expiration_date</code>, <code>image_list_identifier</code>, <code>attributes</code>.
;image
: Image object. Contains following attributes: <code>format</code>, <code>uri</code>, <code>checksum</code>, <code>size</code>
 
* <code>template.erb</code> available variables:
;name
: Name, under which will the template be registered
;image_id
: ID of the previously registered image (same appliance)
;appliance
: Appliance object. Same as for <code>image.erb</code>
;image
: Image object. Same as for <code>image.erb</code>
 
 
'''For compatibility with other integration components, add the following lines to <code>image.rb</code>:'''
VMCATCHER_EVENT_AD_MPURI="<%= appliance.mpuri %>"
VMCATCHER_EVENT_HV_VERSION="<%= appliance.version %>"
VMCATCHER_EVENT_DC_DESCRIPTION="<%= appliance.description %>"
VMCATCHER_EVENT_DC_TITLE="<%= appliance.title %>"
 
=== Usage ===
* Start and enable <code>cloudkeeper-one</code> service
systemctl enable cloudkeeper-one
systemctl start cloudkeeper-one
<code>cloudkeeper-one</code> will be now listening for communication from <code>cloudkeeper</code>.
 
* Perform the first synchronization manually
# This MAY take a long time, keep checking for successful exit with `systemctl status cloudkeeper`
systemctl start cloudkeeper
 
* Start and enable systemd timer for <code>cloudkeeper</code>
systemctl enable cloudkeeper.timer
systemctl start cloudkeeper.timer
This service registers a systemd timer which will run <code>cloudkeeper</code> approx. every 2 hours.

Latest revision as of 11:34, 10 September 2018

Redirect to: