Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "MAN10"

From EGIWiki
Jump to navigation Jump to search
Line 571: Line 571:
systemctl enable occi-server
systemctl enable occi-server
systemctl start occi-server
systemctl start occi-server
</pre>
==== Runtime ====
* Set network type on networks used via OCCI:
<pre>
NETWORK_TYPE="public"              # Supported types: public|nat|private
</pre>
* Optionally, set flag for networks that should be treated as public IP pools (for IP reservations):
<pre>
FLOATING_IP_POOL="yes"
</pre>
* Optionally, set additional network attributes:
<pre>
NETWORK_ADDRESS=""                  # e.g., "172.16.100.0"
NETWORK_MASK=""                    # e.g., "255.255.255.0"
GATEWAY=""                          # e.g., "172.16.100.1"
</pre>
</pre>



Revision as of 16:20, 11 October 2017

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home Manuals Procedures Training Other Contact For: VO managers Administrators



Title Cloud Resource Centre Installation Manual
Document link https://wiki.egi.eu/wiki/MAN10
Last modified 19 May 2017
Policy Group Acronym OMB
Policy Group Name Operations Management Board
Contact Group operations-support@mailman.egi.eu
Document Status DRAFT
Approved Date
Procedure Statement This manual provides information on how to set up a Resource Centre providing cloud resources in the EGI infrastructure.
Owner Owner of procedure



Common prerequirements and documentation

General minimal requirements are:

  • Very minimal hardware is required to join. Hardware requirements depend on:
    • the cloud stack you use
    • the amount of resources you want to make available
    • the number of users/use cases you want to support
  • Servers need to authenticate each other in the EGI Federated Cloud context; this is fulfilled using X.509 certificates, so a Resource Centre should be able to obtain server certificates for some services.
  • User and research communities are called Virtual Organisations (VO). At least support for 3 VOs is needed to join as a Resource Centre:
    • ops and dteam, used for operational purposes as per RC OLA
    • fedcloud.egi.eu: this VO provides resources for application prototyping and validation
  • The operating systems supported by the EGI Federated Cloud Management Framework are:
    • Scientific Linux 6, CentOS7 (and in general RHEL-compatible)
    • Ubuntu (and in general Debian-based)

In order to configure Virtual Organisations and private image lists, please consider the following guides to:

Integrating OpenStack

Integration with FedCloud requires a working OpenStack installation as a prerequirement (see http://docs.openstack.org/ for details). There are packages ready to use for most distributions (check for example RDO for RedHat based distributions).

OpenStack integration with FedCloud is known to work with the following versions of OpenStack:

  • Havana (EOL by OpenStack)
  • Icehouse (EOL by OpenStack)
  • Juno (EOL by OpenStack)
  • Kilo (EOL by OpenStack)
  • Liberty (Security supported, EOL 2016-11-17)
  • Mitaka (Current stable release)
  • Newton

See http://releases.openstack.org/ for more details on the OpenStack releases.

Integration components

Which components must be installed and configured depends on the services the RC wants to provide.

  • Keystone must be always available
  • If providing VM Management features (OCCI access or OpenStack access), then Nova, Cinder and Glance must be available; also Neutron is needed, but nova-network can also be used for legacy installations (see here how to configure per-tenant routers with private networks).

Openstack-fedcloud.png

As you can see from the schema above, the integration is performed installing some EGI extensions on top of the OpenStack components.

  • Keystone-VOMS Authorization plugin allow users with a valid VOMS proxy to access the OpenStack deployment
  • OpenStack OCCI Interface (ooi) translates between OpenStack API and OCCI
  • cASO collects accounting data from OpenStack
  • SSM sends the records extracted by cASO to the central accounting database on the EGI Accounting service (APEL)
  • BDII cloud provider registers the RC configuration and description through the EGI Information System to facilitate service discovery
  • vmcatcher checks the EGI App DB for new or updated images that can be provided by the RC to the user communities (VO) supported
  • The vmcatcher hooks (glancepush and OpenStack handler for vmcatcher) push updated subscribed images from vmcatcher to Glance, using Openstack Python API

EGI User Management/AAI (Keystone-VOMS)

Every FedCloud site must support authentication of users with X.509 certificates with VOMS extensions. The Keystone-VOMS extension enables this kind of authentication on Keystone.

Documentation on the installation is available on https://keystone-voms.readthedocs.org/

Notes:

  • You need a host certificate from a recognised CA for your keystone server.
  • Take into account that using keystone-voms plugin will enforce the use of https for your Keystone service, you will need to update your URLs at the Keystone catalog and in the configuration of your services:
    • You will probably need to include your CA to your system's CA bundle to avoid certificate validation issues: /etc/ssl/certs/ca-certificates.crt from the ca-certificates package on Debian/Ubuntu systems or /etc/pki/tls/certs/ca-bundle.crt from the ca-certificates on RH and derived systems. The Federated Cloud OpenStack Client guide includes information on how to do it.
    • replace http with https in auth_[protocol|uri|url] and auth_[host|uri|url] in the nova, cinder, glance and neutron config files (/etc/nova/nova.conf, /etc/nova/api-paste.ini, /etc/neutron/neutron.conf, /etc/neutron/api-paste.ini, /etc/neutron/metadata_agent.ini, /etc/cinder/cinder.conf, /etc/cinder/api-paste.ini, /etc/glance/glance-api.conf, /etc/glance/glance-registry.conf, /etc/glance/glance-cache.conf) and any other service that needs to check keystone tokens.
    • You can update the URLs of the services directly in the database:
mysql> use keystone;
mysql> update endpoint set url="https://<keystone-host>:5000/v2.0" where url="http://<keystone-host>:5000/v2.0";
mysql> update endpoint set url="https://<keystone-host>:35357/v2.0" where url="http://<keystone-host>:35357/v2.0";
  • Support for EGI VOs: VOMS configuration, you should configure fedcloud.egi.eu, dteam and ops VOs.
  • VOMS-Keystone configuration: most sites should enable the autocreate_users option in the [voms] section of Keystone-VOMS configuration. This will enable that new users are automatically created in your local keystone the first time they login into your site.

EGI Virtual Machine Management Interface

EGI currently operates two realms: the Open Standards Realm and the OpenStack Realm. Both are completely integrated with the EGI federation services but they expose different interfaces to offer IaaS capabilities to the users. The Open Standards Realm uses OCCI standard (supported by providers with OpenNebula, OpenStack and Synnefo cloud management frameworks), while the OpenStack Realm uses the OpenStack native Nova API (support limited to OpenStack providers).

You can provide your service in one or both of the realms. For the OpenStack Realm, you just need to declare your endpoint in GOCDB as described below. For the Open Standards realm you will need to deploy an additional service for providing OCCI access.

ooi is the recommended software to provide OCCI for OpenStack (from Juno onwards). Installation and configuration of ooi is available at ooi documentation. Packages for several distributions can be found at ooi entry at EGI's AppDB (recommended version is 0.2.0)

For older OpenStack releases OCCI-OS can be used. Follow the README.md file in the github repo for instructions on installation and configuration. Be sure to select the branch (e.g. stable/icehouse) corresponding to your OpenStack deployment.

Once the OCCI interface is installed, you should register it on your installation (adapt the region and URL to your deployment):

$ openstack service create --name occi --description "OCCI Interface" occi
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | OCCI Interface                   |
| enabled     | True                             |
| id          | 6dfd6a56c9a6456b84e8c86038e58f56 |
| name        | occi                             |
| type        | occi                             |
+-------------+----------------------------------+

$ openstack endpoint create --region RegionOne occi --publicurl http://172.16.4.70:8787/occi1.1

+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |           OCCI service           |
|      id     | 8e6de5d0d7624584bed6bec9bef7c9e0 |
|     name    |             occi_api             |
|     type    |               occi               |
+-------------+----------------------------------+

Integration with EGI FedCloud Appliance

The EGI FedCloud Appliance packages a set of docker containers to federate a OpenStack deployment with some EGI services:

  • Information System (BDII)
  • Accounting (cASO, SSM)
  • Image management (atrope)

You can get the current version of the appliance at AppDB entry. It is available as an OVA file. You can easily extract the VMDK disk of the OVA by untaring the file.

Pre-requisites

The appliance works by querying the public APIs of an existing OpenStack installation. It assumes Keystone-VOMS is installed at that OpenStack and the voms.json file is properly configured.

The appliance uses the following OpenStack APIs:

  • nova, for getting images and flavors available and to get usage information
  • keystone, for authentication and for getting the available tenants
  • glance, for querying, uploading and removing VM images.

Not all services need to be accessed with the same credentials. Each component is individually configured.

A host certificate is to send the accounting information before sending it to the accounting repository. DN of the host certificate must be registered in GOCDB service type eu.egi.cloud.accounting (see the registration section below for more information).

Note:

  • VM Image replication requires large disk space for storing the downloaded images. By default these are stored at /image_data. You can mount a volume at that location.
  • The appliance should be accessible by the EGI Information System. EGI information system will check GOCDB for the exact location of your appliance (see the registration section below for more information).

EGI Accounting (cASO/SSM)

There are two different processes handling the accounting integration:

  • cASO, which connects to the OpenStack deployment to get the usage information, and,
  • ssmsend, which sends that usage information to the central EGI accounting repository.

They are run by cron every hour (cASO) and every six hours (ssmsend).

cASO configuration is stored at /etc/caso/caso.conf. Most default values are ok, but you must set:

  • site_name (line 100)
  • tenants (line 104)
  • credentials to access the accounting data (lines 122-128). Check the cASO documentation for the expected permissions of the user configured here.

The cron job will use the voms mapping file at /etc/voms.json.

cASO will write records to /var/spool/apel where ssmsend will take them.

SSM configuration is available at /etc/apel. Defaults should be ok for most cases. The cron file uses /etc/grid-security for the CAs and the host certificate and private keys (in /etc/grid-security/hostcert.pem and /etc/grid-security/hostkey.pem).

Running the services

Both caso and ssmsend are run via cron scripts. They are located at /etc/cron.d/caso and /etc/crond.d/ssmsend respectively. For convenience there are also two scripts /usr/loca/bin/caso-extract.sh and /usr/local/bin/ssm-send.sh that run the docker container with the proper volumes.

EGI Information System (BDII)

Information discovery provides a real-time view about the actual images and flavors available at the OpenStack for the federation users. It has two components:

  • Resource-Level BDII: which queries the OpenStack deployment to get the information to publish
  • Site-Level BDII: gathers information from several resource-level BDIIs (in this case only 1) and makes it publicly available for the EGI information system.

Resource-level BDII

This is provided by container egifedcloud/cloudbdii. You need to configure:

  • /etc/cloud-info-provider/openstack.rc, with the credentials to query your OpenStack. The user configured just needs to be able to access the lists of images and flavors.
  • /etc/cloud-info-provider/openstack.yaml, this file includes the static information of your deployment. Make sure to set the SITE-NAME as defined in GOCDB.

Site-level BDII

The egifedcloud/sitebdii container runs this process. Configuration files:

  • /etc/sitebdii/glite-info-site-defaults.conf. Set here the name of your site (as defined in GOCDB) and the public hostname where the appliance will be available.
  • /etc/sitebdii/site.cfg. Include here basic information on your site.

Running the services

In order to run the information discovery containers, there is a docker-compose file at /etc/sitebdii/docker-compose.yml. Run it with:

docker-compose -f /etc/sitebdii/docker-compose.yml up -d

Check the status with:

docker-compose -f /etc/sitebdii/docker-compose.yml ps

You should be able to get the BDII information with an LDAP client, e.g.:

ldapsearch -x -p 2170 -h <yourVM.hostname.domain.com> -b o=glue

EGI Image Management (atrope)

The appliance provide VMI replication with atrope, an alternative implementation to vmcatcher. Every 12 hours, the appliance will perform the following actions:

  • download the configured lists in /etc/atrope/hepix.yaml and verify its signature
  • check any changes in the lists and download new images
  • synchronise this information to the configured glance endpoint

Configure the glance credentials in the /etc/atrope/atrope.conf file and add the lists you want to download at the /etc/atrope/hepix.yaml. See the following example for fedcloud.egi.vo list:

# This must match the VO name configured at the voms.json file
fedcloud.egi.eu:
    url: https://vmcaster.appdb.egi.eu/store/vo/fedcloud.egi.eu/image.list
    enabled: true
    # All image lists from AppDB will have this endorser
    endorser:
        dn: '/DC=EU/DC=EGI/C=NL/O=Hosts/O=EGI.eu/CN=appdb.egi.eu'
        ca: "/DC=ORG/DC=SEE-GRID/CN=SEE-GRID CA 2013"
    # You must get this from AppDB
    token: 17580f07-1e33-4a38-94e3-3386daced5be
    # if you want to restrict the images downloaded from the AppDB, you can add here a list of the identifiers
    # check the "dc:identifier" field in the image list file.
    images: []
    # images names will prefixed with this string for easy identification
    prefix: "FEDCLOUD "

Check How to subscribe to a private image list for instructions to get the URL and token. The prefix if specified will be added in the image title in glance. You can define a subset of images to download with the images field.

Running the service

atrope is run via a cron scripts: /etc/cron.d/atrope. For convenience the /usr/loca/bin/atrope-dispatch.sh script runs the docker container with the proper volumes.

Integration with individual components

EGI Accounting (cASO/SSM)

Every cloud RC should publish utilization data to the EGI accounting database. You will need to install cASO, a pluggable extractor of Cloud Accounting Usage Records from OpenStack.

Documentation on how to install and configure cASO is available at https://caso.readthedocs.org/en/latest/

In order to send the records to the accounting database, you will also need to configure SSM, whose documentation can be found at https://github.com/apel/ssm

EGI Information System (BDII)

Sites must publish information to EGI information system which is based on BDII. The BDII can be installed easily directly from the distribution repository, the package is usually named "bdii".

There is a common cloud information provider for all cloud management frameworks that collects the information from the used CMF and send them to the aforementioned BDII. It can be installed on the same machine as the BDII or on another machine. The installation and configuration guide for the cloud information provider can be found in the following Fedclouds BDII instructions; more detailed installation and configuration instructions are available at https://github.com/EGI-FCTF/cloud-bdii-provider

EGI Image Management (vmcatcher, glancepush)

Sites in FedCloud offering VM management capability must give access to VO-endorsed VM images. This functionality is provided with vmcatcher (that is able to subscribe to the image lists available in AppDB) and a set of tools that are able to push the subscribed images into the glance catalog. In order to subscribe to VO-wide image lists, you need to have a valid access token to the AppDB. Check how to access to VO-wide image lists and how to subscribe to a private image list documentation for more information.

Please refer to vmcatcher documentation for installation.

Vmcatcher may be branched to Openstack Glance catalog using python-glancepush tool and Openstack Handler for Vmcatcher event handler. To install and configure glancepush and the handler, you can refer to the following instructions:

[stack@ubuntu]$ wget http://repository.egi.eu/community/software/python.glancepush/0.0.X/releases/generic/0.0.6/python-glancepush-0.0.6.tar.gz
[stack@ubuntu]$ tar -zxvf python-glancepush-0.0.6.tar.gz
[stack@ubuntu]$ python setup.py install
    • for RHEL6 you can run:
[stack@rhel]$ yum localinstall http://repository.egi.eu/community/software/python.glancepush/0.0.X/releases/sl/6/x86_64/RPMS/python-glancepush-0.0.6-1.noarch.rpm
  • Then, configure glancepush directories
[stack@ubuntu]$ sudo mkdir -p /var/spool/glancepush /etc/glancepush/log /etc/glancepush/transform/ /etc/glancepush/clouds /var/log/glancepush
[stack@ubuntu]$ sudo chown stack:stack -R /var/spool/glancepush /etc/glancepush /var/log/glancepush/
  • Copy the file /etc/keystone/voms.json to /etc/glancepush/voms.json. Then create a file in clouds file for every VO to which you are subscribed. For example, if you're subscribed to fedcloud, atlas and lhcb, you'll need 3 files in the /etc/glancepush/clouds directory with the credentials for this VO/tenants, for example:
[general]
# Tenant for this VO. Must match the tenant defined in voms.json file
testing_tenant=egi
# Identity service endpoint (Keystone)
endpoint_url=https://server4-eupt.unizar.es:5000/v2.0
# User Password
password=123456
# User
username=John
# Set this to true if you're NOT using self-signed certificates
is_secure=True
# SSH private key that will be used to perform policy checks (to be done)
ssh_key=Carlos_lxbifi81
# WARNING: Only define the next variable if you're going to need it. Otherwise you may encounter problems
cacert=path_to_your_cert
[stack@ubuntu]$ wget http://repository.egi.eu/community/software/openstack.handler.for.vmcatcher/0.0.X/releases/generic/0.0.7/gpvcmupdate-0.0.7.tar.gz
[stack@ubuntu]$ tar -zxvf gpvcmupdate-0.0.7.tar.gz
[stack@ubuntu]$ python setup.py install

while for RHEL6 you can run:

[stack@rhel]$ yum localinstall http://repository.egi.eu/community/software/openstack.handler.for.vmcatcher/0.0.X/releases/sl/6/x86_64/RPMS/gpvcmupdate-0.0.7-1.noarch.rpm
  • Create the vmcatcher folders for OpenStack
[stack@ubuntu]$ mkdir -p /opt/stack/vmcatcher/cache /opt/stack/vmcatcher/cache/partial /opt/stack/vmcatcher/cache/expired
  • Check that vmcatcher is running properly by listing and subscribing to an image list
[stack@ubuntu]$ export VMCATCHER_RDBMS="sqlite:////opt/stack/vmcatcher/vmcatcher.db"
[stack@ubuntu]$ vmcatcher_subscribe -l
[stack@ubuntu]$ vmcatcher_subscribe -e -s https://vmcaster.appdb.egi.eu/store/vappliance/tinycorelinux/image.list
[stack@ubuntu]$ vmcatcher_subscribe -l
8ddbd4f6-fb95-4917-b105-c89b5df99dda    True    None    https://vmcaster.appdb.egi.eu/store/vappliance/tinycorelinux/image.list
  • Create a CRON wrapper for vmcatcher, named $HOME/gpvcmupdate/vmcatcher_eventHndl_OS_cron.sh, using the following code
#!/bin/bash
#Cron handler for VMCatcher image syncronization script for OpenStack

#Vmcatcher configuration variables
export VMCATCHER_RDBMS="sqlite:////opt/stack/vmcatcher/vmcatcher.db"
export VMCATCHER_CACHE_DIR_CACHE="/opt/stack/vmcatcher/cache"
export VMCATCHER_CACHE_DIR_DOWNLOAD="/opt/stack/vmcatcher/cache/partial"
export VMCATCHER_CACHE_DIR_EXPIRE="/opt/stack/vmcatcher/cache/expired"
export VMCATCHER_CACHE_EVENT="python $HOME/gpvcmupdate/gpvcmupdate.py -D"

#Update vmcatcher image lists
vmcatcher_subscribe -U

#Add all the new images to the cache
for a in `vmcatcher_image -l | awk '{if ($2==2) print $1}'`; do
 vmcatcher_image -a -u $a
done 

#Update the cache
vmcatcher_cache -v -v

#Run glancepush
/usr/bin/glancepush.py
  • Set the newly created file as executable
[stack@ubuntu]$ chmod +x $HOME/gpvcmupdate/vmcatcher_eventHndl_OS_cron.sh
  • Test that the vmcatcher handler is working correctly by running
[stack@ubuntu]$ $HOME/gpvcmupdate/vmcatcher_eventHndl_OS_cron.sh
INFO:main:Defaulting actions as 'expire', and 'download'.
DEBUG:Events:event 'ProcessPrefix' executed 'python /opt/stack/gpvcmupdate/gpvcmupdate.py'
DEBUG:Events:stdout=
DEBUG:Events:stderr=Ignoring ProcessPrefix event.
INFO:DownloadDir:Downloading '541b01a8-94bd-4545-83a8-6ea07209b440'.
DEBUG:Events:event 'AvailablePrefix' executed 'python /opt/stack/gpvcmupdate/gpvcmupdate.py'
DEBUG:Events:stdout=AvailablePrefix
DEBUG:Events:stderr=
INFO:CacheMan:moved file 541b01a8-94bd-4545-83a8-6ea07209b440
DEBUG:Events:event 'AvailablePostfix' executed 'python /opt/stack/gpvcmupdate/gpvcmupdate.py'
DEBUG:Events:stdout=AvailablePostfixCreating Metadata Files
DEBUG:Events:stderr=
DEBUG:Events:event 'ProcessPostfix' executed 'python /opt/stack/gpvcmupdate/gpvcmupdate.py'
DEBUG:Events:stdout=
DEBUG:Events:stderr=Ignoring ProcessPostfix event.


  • Add the following line to the stack user crontab:
50 */6 * * * $HOME/gpvcmupdate/vmcatcher_eventHndl_OS_cron.sh >> /var/log/glancepush/vmcatcher.log 2>&1

NOTES:

  • It is recommended to execute glancepush and vmcatcher_cache as stack or other non-root user.
  • VMcatcher expired images are removed from OS.

Post-installation

After the installation of all the needed components, it is recommended to set the following policies on Nova to avoid users accessing other users resources:

[root@egi-cloud]# sed -i 's|"admin_or_owner":  "is_admin:True or project_id:%(project_id)s",|"admin_or_owner":  "is_admin:True or project_id:%(project_id)s",\n    "admin_or_user":  "is_admin:True or user_id:%(user_id)s",|g' /etc/nova/policy.json
[root@egi-cloud]# sed -i 's|"default": "rule:admin_or_owner",|"default": "rule:admin_or_user",|g' /etc/nova/policy.json
[root@egi-cloud]# sed -i 's|"compute:get_all": "",|"compute:get": "rule:admin_or_owner",\n    "compute:get_all": "",|g' /etc/nova/policy.json

Registration, validation and certification

As mentioned in the main page, RC services must be registered in the EGI Configuration Management Database (GOCDB). If you are creating a new site for your cloud services, please follow the Resource Centre Registration and Certification with the help of EGI Operations and your reference Resource Infrastructure.

You will need to register the following services (all of them can be provided by the Federated Cloud Appliance):

  • Site-BDII. This service collects and publishes site's data for the Information System. Existing sites should already have this registered.
  • eu.egi.cloud.accounting. Register here the host sending the records to the accounting repository (executing SSM send).
  • eu.egi.cloud.vm-metadata.vmcatcher for the VMI replication mechanism. Register here the host providing the replication.

If offering OCCI interface, the site must register also:

  • eu.egi.cloud.vm-management.occi for the OCCI endpoint offered by the site. Please note the special endpoint URL syntax described at GOCDB usage in FedCloud

If offering native OpenStack access, you must register:

  • org.openstack.nova for the Nova endpoint of the site. Please note the special endpoint URL syntax described at GOCDB usage in FedCloud


Site should also declare the following properties using the Site Extension Properties feature:

    1. Max number of virtual cores for VM with parameter name: cloud_max_cores4VM
    2. Max amount of RAM for VM with parameter name: cloud_max_RAM4VM using the format: value+unit, e.g. "16GB".
    3. Max amount of storage that could be mounted in a VM with parameter name: cloud_max_storage4VM using the format: value+unit, e.g. "16GB".

The installation validation is part of the aforementioned Resource Centre Registration and Certification procedure. After you register the services in GOCDB, EGI Operations will test your services using the site certification manual tests mentioned in the same procedure. It is important to use that guide to test the services published to check that they are behaving properly.

Once the site services are registered in GOCDB (and flagged as "monitored") they will appear in the EGI service monitoring tools. EGI will check the status of the services (see Infrastructure Status for details). Check if your services are present in the EGI service monitoring tools and passing the tests; if you experience any issues (services not shown, services are not OK...) please contact back EGI Operations or your reference Resource Infrastructure.

Integrating OpenNebula

EGI Federated Cloud Site based on OpenNebula is an ordinary OpenNebula installation with some EGI-specific integration components. There are no additional requirements placed on internal site architecture. Follow OpenNebula Documentation if you need advice on how to install and configure OpenNebula itself.

The following OpenNebula versions are supported:

  • OpenNebula v5.2.x
  • OpenNebula v5.4.x

Integration Prerequisites:

  • Working OpenNebula installation.
  • Valid IGTF-trusted host certificates for selected hosts.

Please consider that:

  • CDMI storage endpoints are currently not supported for OpenNebula-based sites.
  • GUI integration is not supported.

OpenNebulaSite.png

The following components must be installed:

  • rOCCI-server -- provides a standard virtual machine management interface. Configuration is explained bellow.
  • keystorm -- serves federated authentication and authorization. Configuration is explained bellow.
  • cloudkeeper and cloudkeeper-one, synchronize site with appliances from EGI AppDB Configuration is explained bellow.
  • oneacct-export and apel-ssm -- collect accounting and publishe it into EGI's accounting database. Configuration is explained bellow.
  • cloud-info-provider and BDII, register site in the EGI Information System. Configuration is explained bellow.

The following ports must be open to allow access to an OpenNebula-based FedCloud sites:

Open Ports for OpenNebula and other components
Port Application Host Note
2633/TCP OpenNebula/XML-RPC OpenNebula Communication between integration components and OpenNebula.
2170/TCP BDII/LDAP cloud-info-provider/BDII EGI Service Discovery/Information System
11443/TCP OCCI/HTTPS rOCCI-server EGI Virtual Machine Management
5000/TCP keystorm/HTTPS keystorm EGI User Management
50505/TCP cloudkeeper/HTTP cloudkeeper EGI Image Management, needs to be accessible from cloudkeeper-one node only
50051/TCP cloudkeeper-one/gRPC cloudkeeper-one EGI Image Management, needs to be accessible from cloudkeeper node only

There are no additional requirements for OpenNebula hosts used to run virtual machines.

This is an overview of service accounts used in an OpenNebula-based site. The names are default and can be changed if required.

Service Accounts
Type Account name Host Use
System accounts rocci rOCCI-server Service account for rOCCI-server. It is only a service account, no access required.
keystorm keystorm Service account for keystorm. It is only a service account, no access required.
apel oneacct-export/APEL Service account for oneacct-export/APEL. Just a service account, no access required.
openldap cloud-info-provider/BDII Service account for cloud-info-provider/BDII. Just a service account, no access required.
cloudkeeper cloudkeeper Service account for cloudkeeper. Just a service account, no access required.
cloudkeeper-one cloudkeeper-one Service account for cloudkeeper-one. Just a service account, no access required.

EGI Virtual Machine Management

Prerequisites

Enable EPEL and install the following packages prior to installation:

 yum install -y epel-release wget

Installation

rOCCI-server is distributed as package for multiple Linux distributions which is available in AppDB. This guide will expect CentOS 7 distribution but installation on any other supported distribution is very similar.

  • Register rOCCI-server repositories
 wget http://repository.egi.eu/community/software/rocci.server/2.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/rocci-server.repo
  • Install package
 yum install -y occi-server

Configuration

  • Make rOCCI-server listen on a public interface
mkdir -p /etc/systemd/system/occi-server.socket.d
cat > /etc/systemd/system/occi-server.socket.d/override.conf <<EOS
[Socket]
ListenStream=
ListenStream=0.0.0.0:11443
EOS
sed -i 's/HOST=127.0.0.1/HOST=0.0.0.0/g' /etc/occi-server/variables
  • Uncomment and configure optional parameters in /etc/occi-server/variables
export HOST_CERT=/path/to/cert                                     # host certificate readable by the rocci user
export HOST_KEY=/path/to/key                                       # host key readable by the rocci user
export ROCCI_SERVER_KEYSTONE_URI=https://localhost:5000/           # URL pointing to keystorm installation
export ROCCI_SERVER_OPENNEBULA_ENDPOINT=http://localhost:2633/RPC2 # URL pointing to OpenNebula installation
export ROCCI_SERVER_ENCRYPTION_TOKEN_CIPHER=AES-128-CBC            # crypto options MUST MATCH keystorm's crypto options
export ROCCI_SERVER_ENCRYPTION_TOKEN_KEY=
export ROCCI_SERVER_ENCRYPTION_TOKEN_IV=
  • Enable and start the service
systemctl enable occi-server
systemctl start occi-server

Runtime

  • Set network type on networks used via OCCI:
NETWORK_TYPE="public"               # Supported types: public|nat|private
  • Optionally, set flag for networks that should be treated as public IP pools (for IP reservations):
FLOATING_IP_POOL="yes"
  • Optionally, set additional network attributes:
NETWORK_ADDRESS=""                  # e.g., "172.16.100.0"
NETWORK_MASK=""                     # e.g., "255.255.255.0"
GATEWAY=""                          # e.g., "172.16.100.1"

EGI User Management

Prerequisites

Enable EPEL and install the following packages prior to installation:

 yum install -y epel-release wget

Installation

keystorm is distributed as package for multiple Linux distributions which is available in AppDB. This guide will expect CentOS 7 distribution but installation on any other supported distribution is very similar.

  • Register keystorm repositories
 wget http://repository.egi.eu/community/software/keystorm/1.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/keystorm.repo
  • Install package
 yum install -y keystorm

Configuration

  • Uncomment and configure optional parameters in /etc/keystorm/variables
export HOST_CERT=/path/to/cert                                     # host certificate readable by the rocci user
export HOST_KEY=/path/to/key                                       # host key readable by the rocci user
export KEYSTORM_OPENNEBULA_ENDPOINT=http://localhost:2633/RPC2     # URL pointing to OpenNebula installation
export KEYSTORM_OPENNEBULA_SECRET=oneadmin:opennebula              # Privileged OpenNebula credentials (with user and group management permissions)
export KEYSTORM_CATALOG_ENDPOINTS_URL=https://localhost:11443/     # URL pointing to rOCCI-server installation
  • Enable and start the service
systemctl enable keystorm
systemctl start keystorm
  • Enable support for EGI VOs via VOMS: VOMS configuration
  • Enable support for EGI VOs via OIDC: OIDC configuration
  • In OpenNebula, create empty groups for fedcloud.egi.eu, ops, and dteam with group attributes:
KEYSTORM="YES"                  # Allow keystorm to manage membership on this group
DEFAULT_CLUSTER_ID="0"          # Optionally, set default cluster for this group
DEFAULT_CONNECTIVITY="public"   # Optionally, set default connectivity for this group: public|nat|private

EGI Accounting

Prerequisites

oneacct-export uses Secure Stomp Messenger to send accounting records to the central repository. Please, refer to ssm documentation for installation instructions. By default, accounting records are placed in /var/spool/apel/outgoing/00000000. You have to configure and run ssmsend periodically, this is not handled by oneacct-export.

Enable EPEL and install the following packages prior to oneacct-export installation:

 yum install -y epel-release wget

Installation

oneacct-export is distributed as package for multiple Linux distributions which is available in AppDB. This guide will expect CentOS 7 distribution but installation on any other supported distribution is very similar.

  • Register oneacct-export repositories
 wget http://repository.egi.eu/community/software/oneacct.export/0.4.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/oneacct-export.repo
  • Install package
 yum install -y oneacct-export

Configuration

  • Edit /etc/oneacct-export/conf.yml
 apel:
   site_name: Undefined                     # Usually a short provider name, e.g. CESNET
   cloud_type: OpenNebula                   # CMF type, only OpenNebula is supported
   endpoint: https://localhost.edu:11443/ # Public URL of your OCCI endpoint
 xml_rpc:
   secret: oneadmin:opennebula            # OpenNebula credentials, privileged
   endpoint: http://localhost:2633/RPC2 # OpenNebula XML RPC endpoint
  • Add the following lines to /etc/one/oned.conf and restart OpenNebula
 INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_AD_MPURI"
 INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_DC_IDENTIFIER"
 INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_IL_DC_IDENTIFIER"
 INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_SL_CHECKSUM_SHA512"
 INHERIT_IMAGE_ATTR = "VMCATCHER_EVENT_HV_VERSION"
  • Set benchmark values on CLUSTERs (applies to all hosts in the cluster) or HOSTs (only for that host) in OpenNebula
 BENCHMARK_TYPE  = "HEP-SPEC06" # benchmark type
 BENCHMARK_VALUE = "84.46"      # represents a per-core measured value of said benchmark
  • Use /etc/oneacct-export/groups.include or /etc/oneacct-export/groups.exclude to control which information gets exported. Specify one group name per line.

Usage

  • Enable and register service 'redis'
 service redis start
 chkconfig redis on
  • Enable and register service 'oneacct-export-sidekiq'
 service oneacct-export-sidekiq start
 chkconfig oneacct-export-sidekiq on
  • Perform the first export manually
 # This process may take a long time, consider using tmux or screen
 sudo -u apel /usr/bin/oneacct-export-cron --all
  • Enable and register service 'oneacct-export-cron'
 service oneacct-export-cron start
 chkconfig oneacct-export-cron on

This service registers a cron job which will run oneacct-export every 2 hours.

EGI Information System

Sites must publish information to EGI information system which is based on BDII. There is a common bdii provider for all cloud management frameworks. Information on installation and configuration is available in the cloud-bdii-provider README.md and in the Fedclouds BDII instructions, there is a specific section with OpenNebula details.

EGI Image Management

cloudkeeper and cloudkeeper-one are tools used to ensure synchronization of virtual appliances with an OpenNebula-based cloud.

Cloudkeeper-setup.png

Prerequisites

cloudkeeper uses VO-wide image lists provided by AppDB to synchronize virtual appliances to clouds. In order to use VO-wide image lists you need to have a valid access token to AppDB. Check how to access to VO-wide image lists and how to subscribe to a private image list documentation for more information.

  • Install recent qemu-img and wget
 yum install -y centos-release-qemu-ev wget sudo

Installation

Both cloudkeeper and cloudkeeper-one are distributed as packages for multiple Linux distributions which are available in AppDB. This guide will expect CentOS 7 distribution but installation on any other supported distribution is very similar.

  • Register cloudkeeper and cloudkeeper-one repositories
wget http://repository.egi.eu/community/software/cloudkeeper/1.5.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/cloudkeeper.repo
wget http://repository.egi.eu/community/software/cloudkeeper.one/1.2.x/releases/repofiles/sl-7-x86_64.repo -O /etc/yum.repos.d/cloudkeeper-one.repo
  • Install cloudkeeper and cloudkeeper-one
yum install -y cloudkeeper cloudkeeper-one

cloudkeeper configuration

cloudkeeper configuration file can be found in /etc/cloudkeeper/cloudkeeper.yml.

image-lists
URLs of image lists containing appliances which you want to synchronize to your cloud. Must contain authentication token.
  image-lists: # List of image lists to sync against
    - https://APPDB_TOKEN:x-oauth-basic@vmcaster.appdb.egi.eu/store/vo/somevo/image.list
    - https://APPDB_TOKEN:x-oauth-basic@vmcaster.appdb.egi.eu/store/vo/othervo/image.list
authentication
Says whether cloudkeeper and cloudkeeper-one will communicate securely via TLS. This requires options certificate, key and backend->certificate to be properly set.
image-dir
Directory where images will be downloaded and converted before uploading to OpenNebula. Directory is cleaned after each appliance registration/update nonetheless, it should provide sufficient free space (some runs may require up to 200GB of free space).
remote-mode
Says whether to serve downloaded images via web server or to copy them locally. Should be true especially if OpenNebula is running on different machine than cloudkeeper and cloudkeeper-one.
nginx->ip-address
IP address on which NGINX will serve images in remote mode. This address MUST be accessible from the machine hosting cloudkeeper-one and your OpenNebula installation.
formats
List of image formats images can be converted to and are supported by the cloud.

cloudkeeper-one configuration

cloudkeeper-one configuration file can be found in /etc/cloudkeeper-one/cloudkeeper-one.yml.

authentication
Says whether cloudkeeper and cloudkeeper-one will communicate securely via TLS. This requires options certificate, key and core->certificate to be properly set.
appliances->tmp-dir
Directory images will be copied to before registration in OpenNebula when in non-remote mode.
appliances->template-dir
Directory for ERB-enabled templates of OpenNebula images and templates used for registration. More information in the next section.
opennebula->datastores
List of OpenNebula datastores images are uploaded to.
opennebula->allow-remote-source
Allows OpenNebula to directly download images in remote mode.

Templates configuration

The directory specified by option appliances->template-dir contains templates for OpenNebula images and templates in files image.erb and template.erb. These files can be customized to register images and templates according to your needs. Files are using standard ERB templating mechanism. By default, these files can be found in /etc/cloudkeeper-one/templates/.

  • image.erb available variables:
name
Name, under which will the image be registered
appliance
Appliance object. Contains following attributes: identifier, title, description, mpuri, group, ram, core, version, architecture, operating_system, vo, expiration_date, image_list_identifier, attributes.
image
Image object. Contains following attributes: format, uri, checksum, size
  • template.erb available variables:
name
Name, under which will the template be registered
image_id
ID of the previously registered image (same appliance)
appliance
Appliance object. Same as for image.erb
image
Image object. Same as for image.erb


For compatibility with other integration components, add the following lines to image.rb:

VMCATCHER_EVENT_AD_MPURI="<%= appliance.mpuri %>"
VMCATCHER_EVENT_HV_VERSION="<%= appliance.version %>"
VMCATCHER_EVENT_DC_DESCRIPTION="<%= appliance.description %>"
VMCATCHER_EVENT_DC_TITLE="<%= appliance.title %>"

Usage

  • Start and enable cloudkeeper-one service
service cloudkeeper-one start
chkconfig cloudkeeper-one on

cloudkeeper-one will be now listening for communication from cloudkeeper.

  • Perform the first synchronization manually
# This process may take a long time, consider using tmux or screen
sudo -u cloudkeeper /opt/cloudkeeper/bin/cloudkeeper
  • Start and enable cloudkeeper-cron service
service cloudkeeper-cron start
chkconfig cloudkeeper-cron on

This service registers a cron job which will run cloudkeeper every 2 hours.

Registration of services in GOCDB

Site cloud services must be registered in EGI Configuration Management Database (GOCDB). If you are creating a new site for your cloud services, check the PROC09 Resource Centre Registration and Certification procedure. Services can also coexist within an existing (grid) site.

If offering OCCI interface, sites should register the following services:

  • eu.egi.cloud.vm-management.occi for the OCCI endpoint offered by the site. Please note the special endpoint URL syntax described at GOCDB usage in FedCloud
  • eu.egi.cloud.accounting (host should be your OCCI machine)
  • eu.egi.cloud.vm-metadata.vmcatcher (also host is your OCCI machine)
  • Site should also declare the following properties using the Site Extension Properties feature:
    1. Max number of virtual cores for VM with parameter name: cloud_max_cores4VM
    2. Max amount of RAM for VM with parameter name: cloud_max_RAM4VM using the format: value+unit, e.g. "16GB".
    3. Max amount of storage that could be mounted in a VM with parameter name: cloud_max_storage4VM using the format: value+unit, e.g. "16GB".

Installation Validation

You can check your installation following these steps:

  • Check in ARGO-Mon2 that your services are listed and are passing the tests. If all the tests are OK, your installation is already in good shape.
  • Check that you are publishing cloud information in your site BDII:
    ldapsearch -x -h <site bdii host> -p 2170 -b Glue2GroupID=cloud,Glue2DomainID=<your site name>,o=glue
  • Check that all the images listed in the AppDB page for fedlcoud.egi.eu VO  are listed in your BDII. This sample query will return all the template IDs registered in your BDII:
    ldapsearch -x -h <site bdii host> -p 2170 -b Glue2GroupID=cloud,Glue2DomainID=<your site name>,o=glue objectClass=GLUE2ApplicationEnvironment GLUE2ApplicationEnvironmentRepository
  • Try to start one of those images in your cloud. You can do it with `onetemplate instantiate` or OCCI commands, the result should be the same.
  • Execute the site certification manual tests against your endpoints.
  • Check in the accounting portal that your site is listed and the values reported look consistent with the usage of your site.