Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "HOWTO15 How to configure the Federated Cloud BDII"

From EGIWiki
Jump to navigation Jump to search
(47 intermediate revisions by 7 users not shown)
Line 1: Line 1:
[[Category: Technology ]]
{{Template:Op menubar}} {{Template:Doc_menubar}} {{TOC_right}}
[[Category: Fedcloud-tf]]
File attached to this wiki page:
# script file for the LDIF (python). [https://docs.google.com/file/d/0B7LpvREXG9c-RDdEZUZidVJva0k/edit?usp=sharing (single-info-system-maker-extended-v2.py) Download here].
# Configuration file for the LDAP server. [https://docs.google.com/a/egi.eu/file/d/0B7LpvREXG9c-d2g4MmhGUDRDdXM/edit?usp=sharing (slapd.conf) Download here].
# GLUE20.schema file, containing the GLUE20 LDAP definitions. [https://dc2.safesync.com/FZMqMvz/Peter%20Solagna/cache/GLUE20.schema?a=KhPgEaqz32g download here].


'''Note:'''The configuration file, slapd.conf has been tested with these openldap package:
= Purpose  =
*openldap-2.4.23-20.el6.x86_64
*openldap-servers-2.4.23-20.el6.x86_64
*openldap-clients-2.4.23-20.el6.x86_64
For other openldap releases it may need to be tuned.


This page provides instructions on how to configure the Federated Cloud Production resource-BDII.


Once you have installed the ldap server you should be able to load the
= Installation  =
ldap data with:


$ slapadd  -f /etc/openldap/slapd.conf -l cloud-ldif.ldi
Documentation and general installation guidelines for the cloud-info-provider are available at https://github.com/EGI-FCTF/cloud-bdii-provider


and to start the LDAP server with:
If you don't have already a BDII-site service in production, you will need to install also the <code>bdii</code> package (available in EPEL repo).


  $ slapd -f /etc/openldap/slapd.conf -h ldap://full.hostname:2170  -d 0  > file-name 2>&1 &
If you are using OpenStack as provider, you also need to install the <code>python-novaclient</code> package.


=== Refresh the data ===
If you want to install from packages, here follow some guidelines.
To refresh the data contained in the LDAP server:
# Stop slapd process
# Delete in the database directory all the files (but '''not''' the DB_CONFIG file)
# Run again the slapadd command
# Start again slapd process


== Data published with the current script ==
== RHEL/CentOS/ScientificLinux  ==
The following table is describing the data that must be feeded in the script.


(Please, note that the example are randomly chosen :) )
*Add EPEL repository according to the instructions at https://fedoraproject.org/wiki/EPEL
*Add the cloud-info-provider repository to yum:


{| border="1" cellspacing="5" cellpadding="5" class="wikitable" style="border-collapse: collapse; border:1px solid black;"
wget http://repository.egi.eu/community/software/cloud.info.provider/0.x/releases/repofiles/sl-6-x86_64.repo \
!Variable name
    -O /etc/yum.repos.d/cloud-info-provider.repo
!Description
!Examples
|-
|provider['site_name']
|This is the name of the cloud site, if it was a Grid site it would have been the GOCDB name.
|Self explanatory
|-
|provider['www']
|Address of the information web site of the resource provider, or the cloud service.
|
|-
|provider['Country']
|Country where the cloud infrastructure is located
|"Italy"
|-
|provider['product_name']
|The name of' the cloud middleware used to provide
|'OpenNebula
|-
|provider['type']
|The type of cloud service provided. The more common is Infrastructure As A Service
|'IaaS'
|-
|provider['product_vers']
|The version of the cloud middleware deployed
|'3.0'
|-
|provider['production_l']
|This specify if the service is a production one. Could be production or test.
|'production'
|-
|provider['capabilities']
|This variable contains a list of strings "('string1','string2','string3,..)", those strings describe the capabilities provided by the cloud service. Please note that the strings are not formalized anywhere, new labels should be agreed within the Task Force, for the moment.
|('cloud.managementSystem','cloud.vm.uploadImage', 'cloud.data.cdmi')
|-
|provider['total_cpus']
|The total number of cores available in the cloud service
|''Only numeric values''
|-
|provider['vm_name']
|The name of the virtual machine manager, it is the VM hypervisor that actually instantiate the virtual machines, underneath the cloud layer.
|XEN
|-
|provider['vm_version']
|The version of the virtual machine manager
|1.1
|-
|provider['endpoints']
|List of the endpoints to reach the different cloud services. This is a list of python dictionaries. Basically the format is "({endpoint1},{endpoint2},{endpoint3}...). the structure of the single ''{endpoint}'' bits will be described in an additional table.
|
|-
|provider['execution_env']
|List of the execution environments available in the cloud service, that is the different virtual machines images available to be instantiated by the user. It is a list of python dictionaries. Basically the format is "({execution environment1},{execution environment2},{execution environment3}...). the structure of the single ''{execution environment}'' bits will be described in an additional table.
|
|-
|}


*Install the package


===Endpoint instance===
yum install cloud-info-provider-service
It is a python dictionary, the format is: {'label1':'value1' , 'label2':'value2'....}.


If a cloud service provide different interfaces, such as OCCI and EC2, it should publish different endpoints with different implementation.
== For Debian/Ubuntu 6/7/8  ==


{| border="1" cellspacing="5" cellpadding="5" class="wikitable" style="border-collapse: collapse; border:1px solid black;"
*Add AppDB's repository:
!Label
!Description
!Examples
|-
|endpoint_url
|The URL to reach the service endpoint, it should contain the protocol (e.g. ''https://'') and the port, if it is not the standard port for the protocol.
|https://one.cloud.gwdg.de:8443
|-
|endpoint_interface
|The interface implemented by the endpoint.
|OCCI
|-
|service_type_name/version
|The scripts fills this with the 'product_name' variable described above
|
|-
|interface_version
|The version of the specification of the interface implemented (for example OCCI has a 1.0 or 1.2 specification)
|1.2
|-
|endpoint_technology
|This is the architecture implemented by the web service. Examples are SOAP and REST.
|REST
|}


sudo wget http://repository.egi.eu/community/software/cloud.info.provider/0.x/releases/repofiles/ubuntu-precise-amd64.list \
        -O /etc/apt/sources.list.d/cloud-info-provider.list


===Execution environment instance===
*Add AppDB key:
If a cloud service provide a list of default virtual images that can be instantiate by the user, they should be advertised with a list of execution environment.
It is a python dictionary, the format is: {'label1':'value1' , 'label2':'value2'....}.


{| border="1" cellspacing="5" cellpadding="5" class="wikitable" style="border-collapse: collapse; border:1px solid black;"
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E2E992EB352D3E14
!Label
 
!Description
*Install the package
!Examples
 
|-
sudo apt-get update
|memory
sudo apt-get install python-cloud-info-provider
|The maximum amount of memory that can be configured in a single VM instance (MB)
 
|1600
= Configuration  =
|-
 
|os_family
Use one of the template files in <code>/etc/cloud-info-provider</code> as basis for creating your own YAML file with the static information of your resources. E.g:  
|This is the operating system type provided in the virtual image, the currently formalized families are: linux, windows, macosx, solaris
 
|linux
cp /etc/cloud-info-provider/sample.openstack.yaml /etc/cloud-info-provider/bdii.yaml
|-
 
|os_name
Each middleware has its own options to fetch the dynamic information, check the <code>--help</code> option for more information. Some additional notes are given in the following sections.
|The name of the operating system.
 
|suse
== General  ==
|-
 
|os_version
*Site name will be fetched from <code>site</code> -&gt; <code>name</code> in the template file. Set it to the name defined in GOCDB. Alternatively, the site name can be fetched from /etc/glite-info-static/site/site.cfg (or by the file set with the --glite-site-info-static option)
|The version of the operating system
 
|6.0.4
== OpenNebula ==
|-
 
|platform
*Use the <code>--middleware opennebularocci</code> option to activate this provider
|The processor architecture (i386,x86-64, sparc, powerpc)
 
|i386
*Use the <code>sample.opennebularocci.yaml</code> configuration sample. Change the following lines:
|-
<pre>
|max_cores
site:
|the maximum number of cores that can be configured in a single VM instnace (if there are no policies in places, it should be the number of cores in the hypervisor host). In case of heterogeneous machines in the cluster it should be equal to the higher number of cores available in a single machine.
    # Your site name, as in GODCB (if omitted or set to None, this value is
|24
    # retreived from /etc/glite-info-static/site/site.cfg )
|-
    #name: SITE_NAME
|max_speed
</pre>
|The maximum clock speed of the virtual CPU configurable (MHz)
<pre>
|3000
compute:
|-
    # Total number of cores available
|physical_cpu
    total_cores: 0
|The number of real CPUs of the virtual machine, currently I am not sure about how much this information makes sense for a virtual machine. Can be left '999'.
    # Total RAM available (GB)
|999
    total_ram: 0
|}
    # Hypervisor name (e.g. KVM, Xen, etc.)
    hypervisor: Foo Hypervisor
    # Hypervisor version
    hypervisor_version: 0.0.0
    # ...
    # Middleware version
    middleware_version: 0.0
</pre>
<pre>
    endpoints:
        # ...
 
        # Host serving your Virtual Machine Management interface (rOCCI-server)
        https://cloud-service01.example.org:11443:
            endpoint_url: https://cloud-service01.example.org:11443
</pre>
<pre>
    # Images are retreived automatically by the endpoint
    images:
        defaults:
            #...
            # os_tpl schema prefix must match ROCCI_SERVER_OPENNEBULA_SCHEMA_NAMESPACE in rOCCI-server, only the host part
            schema: http://schemas.cloud-service01.example.org/occi/infrastructure/os_tpl
</pre>
 
*You need to specify connection parameters to the OpenNebula XML-RPC interface:
**<code>--on-auth</code> parameter should contain the authorization parameters for an existing user with full read permissions on the image disks. If the user has been created with the <code>core</code> driver, this parameter shall be set to <code>&lt;username&gt;:&lt;password&gt;</code>.
**<code>--on-rpcxml-endpoint</code> shall contain the address of the RPCv2 endpoint. Usually it is <code><nowiki>http://<hostname>:2633/RPC2</nowiki></code>. If not on a secure network, it is suggested to provide this interface via https, since the <code>on-auth</code> parameter will be sent in clear text to the server.
 
*Compute templates can be gathered in two ways. One option does not preclude the other and the resulting templates will be the merge of the two.
**directly from remote OpenNebula (with rOCCI-server installed and configured), by setting the <code>--rocci-remote-templates</code> flag
**manually by placing them in the configuration file
 
*With the <code>--cloudkeeper-images</code> flag, OS templates can be filtered so only the cloudkeeper ones are published.
 
== OpenStack  ==
 
*Use the <code>--middleware openstack</code> option to activate this provider
 
*The OpenStack provider uses python-novaclient (needs to be installed separately)
**<code>--os-username</code>, <code>--os-password</code>, <code>--auth-tenant-name</code>, <code>--os-auth-url</code>, <code>--os-cacert</code>, <code>--insecure</code> options to the cloud-provider allow to set the connection parameters. Alternatively you can use environment variables (e.g. <code>OS_USERNAME</code>) as with other OpenStack clients
**<code>--insecure</code> should not be used in production!
 
*Be sure that keystone contains the OCCI endpoint, otherwise it will not be published by the BDII. You can check this via the command <code>keystone service-list</code>. To create a new service and endpoint, you can run
<pre>keystone service-create --name occi_api --type occi --description 'Nova OCCI Service'
 
keystone endpoint-create --service_id &lt;service-id&gt; --region RegionOne --publicurl http://$HOSTNAME:8787/ \
                                        --internalurl http://$HOSTNAME:8787/ --adminurl http://$HOSTNAME:8787/
</pre>
where the ''service-id'' is the one obtained from <code>keystone service-list</code>.  
 
*By default, the provider script will filter images without marketplace uri defined into the marketplace or vmcatcher_event_ad_mpuri property. If you want to list all the images templates (included local snapshots), set the variable 'require_marketplace_id: false' under 'compute' -&gt; 'images' -&gt; 'defaults' in the YAML configuration file.
 
== Create the provider  ==
 
*Create the file <code>/var/lib/bdii/gip/provider/cloud-info-provider</code> that calls the provider with the correct options for your site, for example:
<pre>#!/bin/sh
 
cloud-info-provider-service --yaml /etc/cloud-info-provider/openstack.yaml \
                            --middleware openstack \
                            --os-username &lt;username&gt; --os-password &lt;passwd&gt; \
                            --os-tenant-name &lt;tenant&gt; --os-auth-url &lt;url&gt;
</pre>
*Give execution permission:
 
chmod +x /var/lib/bdii/gip/provider/cloud-info-provider
 
*and test it:
 
/var/lib/bdii/gip/provider/cloud-info-provider
 
*That should return the complete LDIF describing your site. Now you can start the bdii service
 
service bdii start
 
*And check that the information is being published
 
ldapsearch -x -h localhost -p 2170 -b o=glue
 
== Add your resource BDII to the site-BDII  ==
 
Add your cloud-info-provider to your site-BDII by adding a new URL like this:
 
ldap://&lt;cloud-info-provier-hostname&gt;:2170/GLUE2GroupID=cloud,o=glue
 
Check how to set up your Site-BDII at [[MAN01 How to publish Site Information]] for information on how to add the URL.
 
[[Category:Operations_Manuals]]

Revision as of 13:36, 11 October 2017

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home • Manuals • Procedures • Training • Other • Contact ► For: VO managers • Administrators



Purpose

This page provides instructions on how to configure the Federated Cloud Production resource-BDII.

Installation

Documentation and general installation guidelines for the cloud-info-provider are available at https://github.com/EGI-FCTF/cloud-bdii-provider

If you don't have already a BDII-site service in production, you will need to install also the bdii package (available in EPEL repo).

If you are using OpenStack as provider, you also need to install the python-novaclient package.

If you want to install from packages, here follow some guidelines.

RHEL/CentOS/ScientificLinux

wget http://repository.egi.eu/community/software/cloud.info.provider/0.x/releases/repofiles/sl-6-x86_64.repo \
   -O /etc/yum.repos.d/cloud-info-provider.repo
  • Install the package
yum install cloud-info-provider-service

For Debian/Ubuntu 6/7/8

  • Add AppDB's repository:
sudo wget http://repository.egi.eu/community/software/cloud.info.provider/0.x/releases/repofiles/ubuntu-precise-amd64.list \
        -O /etc/apt/sources.list.d/cloud-info-provider.list
  • Add AppDB key:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E2E992EB352D3E14
  • Install the package
sudo apt-get update
sudo apt-get install python-cloud-info-provider

Configuration

Use one of the template files in /etc/cloud-info-provider as basis for creating your own YAML file with the static information of your resources. E.g: 

cp /etc/cloud-info-provider/sample.openstack.yaml /etc/cloud-info-provider/bdii.yaml

Each middleware has its own options to fetch the dynamic information, check the --help option for more information. Some additional notes are given in the following sections.

General

  • Site name will be fetched from site -> name in the template file. Set it to the name defined in GOCDB. Alternatively, the site name can be fetched from /etc/glite-info-static/site/site.cfg (or by the file set with the --glite-site-info-static option)

OpenNebula

  • Use the --middleware opennebularocci option to activate this provider
  • Use the sample.opennebularocci.yaml configuration sample. Change the following lines:
site:
    # Your site name, as in GODCB (if omitted or set to None, this value is
    # retreived from /etc/glite-info-static/site/site.cfg )
    #name: SITE_NAME

compute:
    # Total number of cores available
    total_cores: 0
    # Total RAM available (GB)
    total_ram: 0
    # Hypervisor name (e.g. KVM, Xen, etc.)
    hypervisor: Foo Hypervisor
    # Hypervisor version
    hypervisor_version: 0.0.0
    # ...
    # Middleware version
    middleware_version: 0.0

    endpoints:
        # ...

        # Host serving your Virtual Machine Management interface (rOCCI-server)
        https://cloud-service01.example.org:11443:
            endpoint_url: https://cloud-service01.example.org:11443

    # Images are retreived automatically by the endpoint
    images:
        defaults:
            #...
            # os_tpl schema prefix must match ROCCI_SERVER_OPENNEBULA_SCHEMA_NAMESPACE in rOCCI-server, only the host part
            schema: http://schemas.cloud-service01.example.org/occi/infrastructure/os_tpl
  • You need to specify connection parameters to the OpenNebula XML-RPC interface:
    • --on-auth parameter should contain the authorization parameters for an existing user with full read permissions on the image disks. If the user has been created with the core driver, this parameter shall be set to <username>:<password>.
    • --on-rpcxml-endpoint shall contain the address of the RPCv2 endpoint. Usually it is http://<hostname>:2633/RPC2. If not on a secure network, it is suggested to provide this interface via https, since the on-auth parameter will be sent in clear text to the server.
  • Compute templates can be gathered in two ways. One option does not preclude the other and the resulting templates will be the merge of the two.
    • directly from remote OpenNebula (with rOCCI-server installed and configured), by setting the --rocci-remote-templates flag
    • manually by placing them in the configuration file
  • With the --cloudkeeper-images flag, OS templates can be filtered so only the cloudkeeper ones are published.

OpenStack

  • Use the --middleware openstack option to activate this provider
  • The OpenStack provider uses python-novaclient (needs to be installed separately)
    • --os-username, --os-password, --auth-tenant-name, --os-auth-url, --os-cacert, --insecure options to the cloud-provider allow to set the connection parameters. Alternatively you can use environment variables (e.g. OS_USERNAME) as with other OpenStack clients
    • --insecure should not be used in production!
  • Be sure that keystone contains the OCCI endpoint, otherwise it will not be published by the BDII. You can check this via the command keystone service-list. To create a new service and endpoint, you can run
keystone service-create --name occi_api --type occi --description 'Nova OCCI Service'

keystone endpoint-create --service_id <service-id> --region RegionOne --publicurl http://$HOSTNAME:8787/ \
                                         --internalurl http://$HOSTNAME:8787/ --adminurl http://$HOSTNAME:8787/

where the service-id is the one obtained from keystone service-list.

  • By default, the provider script will filter images without marketplace uri defined into the marketplace or vmcatcher_event_ad_mpuri property. If you want to list all the images templates (included local snapshots), set the variable 'require_marketplace_id: false' under 'compute' -> 'images' -> 'defaults' in the YAML configuration file.

Create the provider

  • Create the file /var/lib/bdii/gip/provider/cloud-info-provider that calls the provider with the correct options for your site, for example:
#!/bin/sh

cloud-info-provider-service --yaml /etc/cloud-info-provider/openstack.yaml \
                            --middleware openstack \
                            --os-username <username> --os-password <passwd> \
                            --os-tenant-name <tenant> --os-auth-url <url>
  • Give execution permission:
chmod +x /var/lib/bdii/gip/provider/cloud-info-provider
  • and test it:
/var/lib/bdii/gip/provider/cloud-info-provider
  • That should return the complete LDIF describing your site. Now you can start the bdii service
service bdii start
  • And check that the information is being published
ldapsearch -x -h localhost -p 2170 -b o=glue

Add your resource BDII to the site-BDII

Add your cloud-info-provider to your site-BDII by adding a new URL like this: 

ldap://<cloud-info-provier-hostname>:2170/GLUE2GroupID=cloud,o=glue

Check how to set up your Site-BDII at MAN01 How to publish Site Information for information on how to add the URL.

Retrieved from "https://wiki.egi.eu/w/index.php?title=HOWTO15_How_to_configure_the_Federated_Cloud_BDII&oldid=97293"