MAN10

From EGIWiki
Jump to: navigation, search
Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home Manuals Procedures Training Other Contact For: VO managers Administrators


Contents


Title Cloud Resource Centre Installation Manual
Document link https://wiki.egi.eu/wiki/MAN10
Last modified 19 May 2017
Policy Group Acronym OMB
Policy Group Name Operations Management Board
Contact Group operations-support@mailman.egi.eu
Document Status DRAFT
Approved Date
Procedure Statement This manual provides information on how to set up a Resource Centre providing cloud resources in the EGI infrastructure.



Common prerequirements and documentation

General minimal requirements are:

In order to configure Virtual Organisations and private image lists, please consider the following guides to:

Integrating OpenStack

Integration with FedCloud requires a working OpenStack installation as a prerequirement (see http://docs.openstack.org/ for details). There are packages ready to use for most distributions (check for example RDO for RedHat based distributions).

OpenStack integration with FedCloud is known to work with the following versions of OpenStack:

See http://releases.openstack.org/ for more details on the OpenStack releases.

Integration components

Which components must be installed and configured depends on the services the RC wants to provide.

Openstack-fedcloud.png

As you can see from the schema above, the integration is performed installing some EGI extensions on top of the OpenStack components.

EGI User Management/AAI (Keystone-VOMS)

Every FedCloud site must support authentication of users with X.509 certificates with VOMS extensions. The Keystone-VOMS extension enables this kind of authentication on Keystone.

Documentation on the installation is available on https://keystone-voms.readthedocs.org/

Notes:

mysql> use keystone;
mysql> update endpoint set url="https://<keystone-host>:5000/v2.0" where url="http://<keystone-host>:5000/v2.0";
mysql> update endpoint set url="https://<keystone-host>:35357/v2.0" where url="http://<keystone-host>:35357/v2.0";

EGI Virtual Machine Management Interface

EGI currently operates two realms: the Open Standards Realm and the OpenStack Realm. Both are completely integrated with the EGI federation services but they expose different interfaces to offer IaaS capabilities to the users. The Open Standards Realm uses OCCI standard (supported by providers with OpenNebula, OpenStack and Synnefo cloud management frameworks), while the OpenStack Realm uses the OpenStack native Nova API (support limited to OpenStack providers).

You can provide your service in one or both of the realms. For the OpenStack Realm, you just need to declare your endpoint in GOCDB as described below. For the Open Standards realm you will need to deploy an additional service for providing OCCI access.

ooi is the recommended software to provide OCCI for OpenStack (from Juno onwards). Installation and configuration of ooi is available at ooi documentation. Packages for several distributions can be found at ooi entry at EGI's AppDB (recommended version is 0.2.0)

For older OpenStack releases OCCI-OS can be used. Follow the README.md file in the github repo for instructions on installation and configuration. Be sure to select the branch (e.g. stable/icehouse) corresponding to your OpenStack deployment.

Once the OCCI interface is installed, you should register it on your installation (adapt the region and URL to your deployment):

$ openstack service create --name occi --description "OCCI Interface" occi
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | OCCI Interface                   |
| enabled     | True                             |
| id          | 6dfd6a56c9a6456b84e8c86038e58f56 |
| name        | occi                             |
| type        | occi                             |
+-------------+----------------------------------+

$ openstack endpoint create --region RegionOne occi --publicurl http://172.16.4.70:8787/occi1.1

+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |           OCCI service           |
|      id     | 8e6de5d0d7624584bed6bec9bef7c9e0 |
|     name    |             occi_api             |
|     type    |               occi               |
+-------------+----------------------------------+

Integration with EGI FedCloud Appliance

Check the Federated Cloud OpenStack Appliance manual for information on how to perform the integration with the appliance.

Integration with individual components

EGI Accounting (cASO/SSM)

Every cloud RC should publish utilization data to the EGI accounting database. You will need to install cASO, a pluggable extractor of Cloud Accounting Usage Records from OpenStack.

Documentation on how to install and configure cASO is available at https://caso.readthedocs.org/en/latest/

In order to send the records to the accounting database, you will also need to configure SSM, whose documentation can be found at https://github.com/apel/ssm

EGI Information System (BDII)

Sites must publish information to EGI information system which is based on BDII. The BDII can be installed easily directly from the distribution repository, the package is usually named "bdii".

There is a common cloud information provider for all cloud management frameworks that collects the information from the used CMF and send them to the aforementioned BDII. It can be installed on the same machine as the BDII or on another machine. The installation and configuration guide for the cloud information provider can be found in the following Fedclouds BDII instructions; more detailed installation and configuration instructions are available at https://github.com/EGI-FCTF/cloud-bdii-provider

EGI Image Management (Cloudkeeper)

Sites in FedCloud offering VM management capability must give access to VO-endorsed VM images. This functionality is provided by the conjugation of two tools :

Cloudkeeper-openstack.png

Prerequisites

cloudkeeper uses VO-wide image lists provided by AppDB to synchronize virtual appliances to clouds. In order to use VO-wide image lists you need to have a valid access token to AppDB. Check how to access to VO-wide image lists and how to subscribe to a private image list documentation for more information.


Cloudkeeper

Cloudkeeper is distributed as packages for multiple Linux distributions which are available in AppDB. This guide will expect CentOS 7 distribution but installation on any other supported distribution is very similar.

wget http://repository.egi.eu/community/software/cloudkeeper/1.5.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/cloudkeeper.repo
yum install -y cloudkeeper

cloudkeeper configuration file can be found in /etc/cloudkeeper/cloudkeeper.yml.

image-lists
URLs of image lists containing appliances which you want to synchronize to your cloud. Must contain authentication token.
  image-lists: # List of image lists to sync against
    - https://APPDB_TOKEN:x-oauth-basic@vmcaster.appdb.egi.eu/store/vo/somevo/image.list
    - https://APPDB_TOKEN:x-oauth-basic@vmcaster.appdb.egi.eu/store/vo/othervo/image.list
image-dir
Directory where images will be downloaded and converted before uploading to OpenStack. Directory is cleaned after each appliance registration/update nonetheless, it should provide sufficient free space (some runs may require up to 200GB of free space).
remote-mode
Says whether to serve downloaded images via web server or to copy them locally. Should be true especially if OpenStack is running on different machine than cloudkeeper and cloudkeeper-os.
nginx->ip-address
IP address on which NGINX will serve images in remote mode. This address MUST be accessible from the machine hosting cloudkeeper-os and your OpenStack installation.
formats
List of image formats images can be converted to and are supported by the cloud.


Cloudkeeper-OS

Cloudkeeper-os is distributed as packages for RHEL 7 and its derivatives (CentOS 7, Scientific Linux 7). In case you are using other Linux distributions, the software can be easily installed using the source code as described in the README file. To install Cloudkeeper-OS on CentOS 7, follow this procedure:

wget http://grand-est.fr/resources/software/cloudkeeper-os/repofiles/centos7/cloudkeeper-os.repo
yum install -y cloudkeeper-os


The cloudkeeper-os.conf configuration file has several sections and has a description for each option. Most of the options have default values. You should check at least the following parameters in the keystone_authtoken section:

For example:

[keystone_authtoken]
username = cloudkeeper
password = cloudkeeper
auth_url = http://controller:5000/v3

In addition, you have to edit the voms.json JSON file to map correctly the VO and the OpenStack project's name. Note that you can use the same JSON file as for the keystone-voms component, by setting the mapping_file parameter with the right path in the cloudkeeper-os.conf file.

Note that the user defined by the username parameter in the cloudkeeper-os.conf file should have the right to manage the images for all the project defined in the voms.json file.

To take into account the modifications, do not forget to restart the cloudkeeper-os service.


Usage

service cloudkeeper-os start
chkconfig cloudkeeper-os on

cloudkeeper-os will be now listening for communication from cloudkeeper.

# This process may take a long time, consider using tmux or screen
sudo -u cloudkeeper /opt/cloudkeeper/bin/cloudkeeper
service cloudkeeper-cron start
chkconfig cloudkeeper-cron on

This service registers a cron job which will run cloudkeeper every 2 hours.

Post-installation

After the installation of all the needed components, it is recommended to set the following policies on Nova to avoid users accessing other users resources:

[root@egi-cloud]# sed -i 's|"admin_or_owner":  "is_admin:True or project_id:%(project_id)s",|"admin_or_owner":  "is_admin:True or project_id:%(project_id)s",\n    "admin_or_user":  "is_admin:True or user_id:%(user_id)s",|g' /etc/nova/policy.json
[root@egi-cloud]# sed -i 's|"default": "rule:admin_or_owner",|"default": "rule:admin_or_user",|g' /etc/nova/policy.json
[root@egi-cloud]# sed -i 's|"compute:get_all": "",|"compute:get": "rule:admin_or_owner",\n    "compute:get_all": "",|g' /etc/nova/policy.json

Registration, validation and certification

As mentioned in the main page, RC services must be registered in the EGI Configuration Management Database (GOCDB). If you are creating a new site for your cloud services, please follow the Resource Centre Registration and Certification with the help of EGI Operations and your reference Resource Infrastructure.

You will need to register the following services (all of them can be provided by the Federated Cloud Appliance):

If offering OCCI interface, the site must register also:

If offering native OpenStack access, you must register:


Site should also declare the following properties using the Site Extension Properties feature:

The installation validation is part of the aforementioned Resource Centre Registration and Certification procedure. After you register the services in GOCDB, EGI Operations will test your services using the site certification manual tests mentioned in the same procedure. It is important to use that guide to test the services published to check that they are behaving properly.

Once the site services are registered in GOCDB (and flagged as "monitored") they will appear in the EGI service monitoring tools. EGI will check the status of the services (see Infrastructure Status for details). Check if your services are present in the EGI service monitoring tools and passing the tests; if you experience any issues (services not shown, services are not OK...) please contact back EGI Operations or your reference Resource Infrastructure.

Integrating OpenNebula

EGI Federated Cloud Site based on OpenNebula is an ordinary OpenNebula installation with some EGI-specific integration components. There are no additional requirements placed on internal site architecture. Follow OpenNebula Documentation if you need advice on how to install and configure OpenNebula itself.

The following OpenNebula versions are supported:

Integration Prerequisites:

Please consider that:

OpenNebulaSite.png

The following components must be installed:

The following ports must be open to allow access to an OpenNebula-based FedCloud sites:

Open Ports for OpenNebula and other components
Port Application Host Note
2633/TCP OpenNebula/XML-RPC OpenNebula Communication between integration components and OpenNebula.
2170/TCP BDII/LDAP cloud-info-provider/BDII EGI Service Discovery/Information System
11443/TCP OCCI/HTTPS rOCCI-server EGI Virtual Machine Management
5000/TCP keystorm/HTTPS keystorm EGI User Management
50505/TCP cloudkeeper/HTTP cloudkeeper EGI Image Management, needs to be accessible from cloudkeeper-one node only
50051/TCP cloudkeeper-one/gRPC cloudkeeper-one EGI Image Management, needs to be accessible from cloudkeeper node only

There are no additional requirements for OpenNebula hosts used to run virtual machines.

This is an overview of service accounts used in an OpenNebula-based site. The names are default and can be changed if required.

Service Accounts
Type Account name Host Use
System accounts rocci rOCCI-server Service account for rOCCI-server. It is only a service account, no access required.
keystorm keystorm Service account for keystorm. It is only a service account, no access required.
apel oneacct-export/APEL Service account for oneacct-export/APEL. Just a service account, no access required.
openldap cloud-info-provider/BDII Service account for cloud-info-provider/BDII. Just a service account, no access required.
cloudkeeper cloudkeeper Service account for cloudkeeper. Just a service account, no access required.
cloudkeeper-one cloudkeeper-one Service account for cloudkeeper-one. Just a service account, no access required.

EGI Virtual Machine Management

Prerequisites

Enable EPEL and install the following packages prior to installation:

 yum install -y epel-release wget

Installation

rOCCI-server is distributed as package for multiple Linux distributions which is available in AppDB. This guide will expect CentOS 7 distribution but installation on any other supported distribution is very similar.

 wget http://repository.egi.eu/community/software/rocci.server/2.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/rocci-server.repo
 yum install -y occi-server

Configuration

mkdir -p /etc/systemd/system/occi-server.socket.d
cat > /etc/systemd/system/occi-server.socket.d/override.conf <<EOS
[Socket]
# lines below are NOT duplicated by mistake
ListenStream=
ListenStream=0.0.0.0:11443
EOS
sed -i 's/HOST=127.0.0.1/HOST=0.0.0.0/g' /etc/occi-server/variables
export HOST_CERT=/path/to/cert                                     # host certificate readable by the rocci user
export HOST_KEY=/path/to/key                                       # host key readable by the rocci user
export ROCCI_SERVER_KEYSTONE_URI=https://localhost:5000/           # URL pointing to keystorm installation
export ROCCI_SERVER_OPENNEBULA_ENDPOINT=http://localhost:2633/RPC2 # URL pointing to OpenNebula installation
export ROCCI_SERVER_ENCRYPTION_TOKEN_CIPHER=                       # crypto options MUST MATCH keystorm's crypto options, see /etc/keystorm/variables
export ROCCI_SERVER_ENCRYPTION_TOKEN_KEY=                          # crypto options MUST MATCH keystorm's crypto options, see /etc/keystorm/variables
export ROCCI_SERVER_ENCRYPTION_TOKEN_IV=                           # crypto options MUST MATCH keystorm's crypto options, see /etc/keystorm/variables
systemctl enable occi-server
systemctl start occi-server

Runtime

/opt/occi-server/bin/oneresource create --endpoint http://one.example.org:2633/RPC2 # --username PRIVILEGED_USER --password PASSWD
# re-run with `--resources /opt/occi-server/embedded/app/rOCCI-server/lib/resources/gpu/` to enable GPU resource templates
DEFAULT_CLUSTER_ID="0"              # Default cluster for this group
DEFAULT_CONNECTIVITY="public"       # Default connectivity for this group: public|nat|private
NETWORK_TYPE="public"               # Supported types: public|nat|private
FLOATING_IP_POOL="yes"
NETWORK_ADDRESS=""                  # e.g., "172.16.100.0"
NETWORK_MASK=""                     # e.g., "255.255.255.0"
GATEWAY=""                          # e.g., "172.16.100.1"

Migration from v1 to v2

In order to migrate from rOCCI-server v1 with Perun-managed user accounts, perform the following steps.

Preparation
Migration
Single-group accounts owned by the same person can be identified as having:

* `NAME` following the naming convention $VONAME_$ID where the same user always has the same $ID
* `TEMPLATE/X509_DN` where the same user always has the same DN
Name of the merged user MUST be a SHA256 digest of the `TEMPLATE/X509_DN` attribute value.

In ruby, SHA256 digest can be generated as:

require 'digest'
Digest::SHA256.hexdigest 'DN_STRING_HERE'
For each user, add the following attributes:

* TEMPLATE/ID
* TEMPLATE/NAME
* TEMPLATE/IDENTITY
* TEMPLATE/AUTHENTICATION

Where

* `TEMPLATE/ID` is a SHA256 digest of the `TEMPLATE/X509_DN` attribute value
* `TEMPLATE/IDENTITY` and `TEMPLATE/NAME` contain the old `TEMPLATE/X509_DN` value
* `TEMPLATE/AUTHENTICATION` is a static value 'voms'

EGI User Management

Prerequisites

Enable EPEL and install the following packages prior to installation:

 yum install -y epel-release wget

Installation

keystorm is distributed as package for multiple Linux distributions which is available in AppDB. This guide will expect CentOS 7 distribution but installation on any other supported distribution is very similar.

 wget http://repository.egi.eu/community/software/keystorm/1.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/keystorm.repo
 yum install -y keystorm

Configuration

export KEYSTORM_OPENNEBULA_ENDPOINT=http://localhost:2633/RPC2     # URL pointing to OpenNebula installation
export KEYSTORM_OPENNEBULA_SECRET=oneadmin:opennebula              # Privileged OpenNebula credentials (with user and group management permissions)
systemctl enable keystorm
systemctl start keystorm
# on Ubuntu/Debian only
a2enmod ssl && \
  a2enmod headers && \
  a2enmod proxy && \
  a2enmod proxy_http && \
  a2enmod remoteip && \
  a2enmod auth_openidc && \
  a2enmod zgridsite
# make sure the following files exist
SSLCertificateFile /etc/grid-security/hostcert.pem
SSLCertificateKeyFile /etc/grid-security/hostkey.pem

# make sure the following directory exists
SSLCACertificatePath /etc/grid-security/certificates
# on Ubuntu/Debian only
systemctl enable apache2
systemctl restart apache2
# on CentOS/SL only
systemctl enable httpd
systemctl start httpd

Runtime

KEYSTORM="YES"                  # Allow keystorm to manage membership for this group

EGI Accounting

Prerequisites

oneacct-export uses Secure Stomp Messenger to send accounting records to the central repository. Please, refer to ssm documentation for installation instructions. By default, accounting records are placed in /var/spool/apel/outgoing/00000000. You have to configure and run ssmsend periodically, this is not handled by oneacct-export.

Enable EPEL and install the following packages prior to oneacct-export installation:

 yum install -y epel-release wget

Installation

oneacct-export is distributed as package for multiple Linux distributions which is available in AppDB. This guide will expect CentOS 7 distribution but installation on any other supported distribution is very similar.

 wget http://repository.egi.eu/community/software/oneacct.export/0.4.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/oneacct-export.repo
 yum install -y oneacct-export

Configuration

 apel:
   site_name: Undefined                     # Usually a short provider name, e.g. CESNET
   cloud_type: OpenNebula                   # CMF type, only OpenNebula is supported
   endpoint: https://localhost.edu:11443/ # Public URL of your OCCI endpoint
 xml_rpc:
   secret: oneadmin:opennebula            # OpenNebula credentials, privileged
   endpoint: http://localhost:2633/RPC2 # OpenNebula XML RPC endpoint
 INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_AD_MPURI"
 INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_DC_IDENTIFIER"
 INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_IL_DC_IDENTIFIER"
 INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_SL_CHECKSUM_SHA512"
 INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_HV_VERSION"
 BENCHMARK_TYPE  = "HEP-SPEC06" # benchmark type
 BENCHMARK_VALUE = "84.46"      # represents a per-core measured value of said benchmark

Usage

 service redis start
 chkconfig redis on
 service oneacct-export-sidekiq start
 chkconfig oneacct-export-sidekiq on
 # This process may take a long time, consider using tmux or screen
 sudo -u apel /usr/bin/oneacct-export-cron --all
 service oneacct-export-cron start
 chkconfig oneacct-export-cron on

This service registers a cron job which will run oneacct-export every 2 hours.

EGI Information System

Sites must publish information to EGI information system which is based on BDII. There is a common bdii provider for all cloud management frameworks. Information on installation and configuration is available in the cloud-bdii-provider README.md and in the Fedclouds BDII instructions, there is a specific section with OpenNebula details.

EGI Image Management

cloudkeeper and cloudkeeper-one are tools used to ensure synchronization of virtual appliances with an OpenNebula-based cloud.

Cloudkeeper-setup.png

Prerequisites

cloudkeeper uses VO-wide image lists provided by AppDB to synchronize virtual appliances to clouds. In order to use VO-wide image lists you need to have a valid access token to AppDB. Check how to access to VO-wide image lists and how to subscribe to a private image list documentation for more information.

 yum install -y centos-release-qemu-ev wget sudo

Installation

Both cloudkeeper and cloudkeeper-one are distributed as packages for multiple Linux distributions which are available in AppDB. This guide will expect CentOS 7 distribution but installation on any other supported distribution is very similar.

wget http://repository.egi.eu/community/software/cloudkeeper/1.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/cloudkeeper.repo
wget http://repository.egi.eu/community/software/cloudkeeper.one/1.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/cloudkeeper-one.repo
yum install -y cloudkeeper cloudkeeper-one

cloudkeeper configuration

cloudkeeper configuration file can be found in /etc/cloudkeeper/cloudkeeper.yml.

image-lists
URLs of image lists containing appliances which you want to synchronize to your cloud. Must contain authentication token.
  image-lists: # List of image lists to sync against
    - https://APPDB_TOKEN:x-oauth-basic@vmcaster.appdb.egi.eu/store/vo/somevo/image.list
    - https://APPDB_TOKEN:x-oauth-basic@vmcaster.appdb.egi.eu/store/vo/othervo/image.list
authentication
Says whether cloudkeeper and cloudkeeper-one will communicate securely via TLS. This requires options certificate, key and backend->certificate to be properly set.
image-dir
Directory where images will be downloaded and converted before uploading to OpenNebula. Directory is cleaned after each appliance registration/update nonetheless, it should provide sufficient free space (some runs may require up to 200GB of free space).
remote-mode
Says whether to serve downloaded images via web server or to copy them locally. Should be true especially if OpenNebula is running on different machine than cloudkeeper and cloudkeeper-one.
nginx->ip-address
IP address on which NGINX will serve images in remote mode. This address MUST be accessible from the machine hosting cloudkeeper-one and your OpenNebula installation.
formats
List of image formats images can be converted to and are supported by the cloud.

cloudkeeper-one configuration

cloudkeeper-one configuration file can be found in /etc/cloudkeeper-one/cloudkeeper-one.yml.

authentication
Says whether cloudkeeper and cloudkeeper-one will communicate securely via TLS. This requires options certificate, key and core->certificate to be properly set.
appliances->tmp-dir
Directory images will be copied to before registration in OpenNebula when in non-remote mode.
appliances->template-dir
Directory for ERB-enabled templates of OpenNebula images and templates used for registration. More information in the next section.
opennebula->datastores
List of OpenNebula datastores images are uploaded to.
opennebula->allow-remote-source
Allows OpenNebula to directly download images in remote mode.

Templates configuration

The directory specified by option appliances->template-dir contains templates for OpenNebula images and templates in files image.erb and template.erb. These files can be customized to register images and templates according to your needs. Files are using standard ERB templating mechanism. By default, these files can be found in /etc/cloudkeeper-one/templates/.

name
Name, under which will the image be registered
appliance
Appliance object. Contains following attributes: identifier, title, description, mpuri, group, ram, core, version, architecture, operating_system, vo, expiration_date, image_list_identifier, attributes.
image
Image object. Contains following attributes: format, uri, checksum, size
name
Name, under which will the template be registered
image_id
ID of the previously registered image (same appliance)
appliance
Appliance object. Same as for image.erb
image
Image object. Same as for image.erb


For compatibility with other integration components, add the following lines to image.rb:

VMCATCHER_EVENT_AD_MPURI="<%= appliance.mpuri %>"
VMCATCHER_EVENT_HV_VERSION="<%= appliance.version %>"
VMCATCHER_EVENT_DC_DESCRIPTION="<%= appliance.description %>"
VMCATCHER_EVENT_DC_TITLE="<%= appliance.title %>"

Usage

systemctl enable cloudkeeper-one
systemctl start cloudkeeper-one

cloudkeeper-one will be now listening for communication from cloudkeeper.

# This MAY take a long time, keep checking for successful exit with `systemctl status cloudkeeper`
systemctl start cloudkeeper
systemctl enable cloudkeeper.timer
systemctl start cloudkeeper.timer

This service registers a systemd timer which will run cloudkeeper approx. every 2 hours.

Registration of services in GOCDB

Site cloud services must be registered in EGI Configuration Management Database (GOCDB). If you are creating a new site for your cloud services, check the PROC09 Resource Centre Registration and Certification procedure. Services can also coexist within an existing (grid) site.

If offering OCCI interface, sites should register the following services:

Installation Validation

You can check your installation following these steps:

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Print/export