Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "HOWTO15 How to configure the Federated Cloud BDII"

From EGIWiki
Jump to navigation Jump to search
Line 1: Line 1:
[[Category: Technology ]]
[[Category: Technology ]]
[[Category: Fedcloud-tf]]
[[Category: Fedcloud-tf]]
= Purpose =
= Purpose =
This page provides instructions on how to configure the Federated Cloud Production BDII.
= Installation guide =
== Pre-requisites ==
This guide has the following pre-requisites:
* Site-BDII, with support to GLUE2 schema specifications. If you do not have already a site BDII installed from EMI or UMD you can follow the installation guide below
* Python 2.2.x
== Install Site-BDII ==
If you have already a production BDII (eg. for existing Grid or storage resources), you can skip this step. Otherwise, here is a quick guide on how to install and configure a site BDII:
=== For RHEL/CentOS/ScientificLinux 5/6 ===
# Install UMD repository according to the instructions here: http://repository.egi.eu/category/umd_releases/distribution/umd-3/
# Install Site-BDII packages: <code>yum install bdii bdii-config-site</code>
# Edit the file <code>/etc/glite-info-static/site/site.cfg</code> with your site information
# Start the BDII service: <code>service bdii restart; chkconfig bdii on</code>
# Configure your GOCDB site information the 'GIIS URL' with the address of your site BDII and the base schema, (eg: ldap://prisma-cloud.ba.infn.it:2170/GLUE2DomainID=PRISMA-INFN-BARI,o=glue )
=== For Debian/Ubuntu 6/7/8 ===
# Install BDII package (<code>apt-get install -y bdii</code>) ''NOTE:'' If BDII gives you a 'could not open config file' error, try to install the latest version from Debian Sid (https://packages.debian.org/unstable/main/bdii) or disable Apparmor (<code>service apparmor teardown</code>), then reinstall the bdii package
# Install BDII site configuration files (from AppDB repository):
#:<code>root@ubuntu:~# echo 'deb http://repository.egi.eu/community/software/cloud.info.provider/0.x/releases/debian wheezy main' > /etc/apt/sources.list.d/cloud-info-provider-appdb-repo.list</code>
#:<code>root@ubuntu:~# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E2E992EB352D3E14</code>
#:<code>root@ubuntu:~# apt-get update</code>
#:<code>root@ubuntu:~# apt-get install bdii-config-site glite-info-site glite-info-static glite-info-provider-service glite-info-provider-ldap</code>
# Edit the file <code>/etc/glite-info-static/site/site.cfg</code> with your site information
# Restart the BDII service: <code>service bdii restart</code>
# Configure your GOCDB site information the 'GIIS URL' with the address of your site BDII and the base schema, (eg: ldap://prisma-cloud.ba.infn.it:2170/GLUE2DomainID=PRISMA-INFN-BARI,o=glue )


== Install the cloud resource provider script ==
This page provides instructions on how to configure the Federated Cloud Production resource-BDII.
For filling the BDII with the cloud resource information, you need to install the cloud resource provider script.


=== For RHEL/CentOS/ScientifcLinux 6.x ===
= Installation =
#Install EPEL (follow instructions [https://fedoraproject.org/wiki/EPEL here])
#Install cloud provider script via RPM
#:<code>yum localinstall http://github.com/EGI-FCTF/BDIIscripts/raw/master/rpm/cloud-info-provider-service-0.3-1.el6.noarch.rpm</code>


=== For Debian/Ubuntu 6/7/8 ===
Packages for the cloud-info-provider are available at [https://appdb.egi.eu/store/software/cloud.info.provider EGI's AppDB]. They will install bdii as a dependency.


# Install AppDB repository:
== RHEL/CentOS/ScientificLinux ==
#:<code>root@ubuntu:~# echo 'deb http://repository.egi.eu/community/software/cloud.info.provider/0.x/releases/debian wheezy main' > /etc/apt/sources.list.d/cloud-info-provider-appdb-repo.list</code>
#:<code>root@ubuntu:~# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E2E992EB352D3E14</code>
#:<code>root@ubuntu:~# apt-get update</code>
# Install dependencies:
#:<code>root@ubuntu:~# apt-get install -y python-yaml</code>
# Install cloud provider script
#:<code>root@ubuntu:~# apt-get install -y cloud-info-provider-service</code>


=== For other OSes (Install from sources) ===
* Add EPEL repository according to the instructions at https://fedoraproject.org/wiki/EPEL
* Add the cloud-info-provider repository to yum:
wget http://repository.egi.eu/community/software/cloud.info.provider/0.x/candidates/0.4/repofiles/sl-6-x86_64-candidate.repo \
      -O /etc/yum.repos.d/cloud-info-provider.repo
* Install the package
yum install cloud-info-provider-service


git clone http://github.com/EGI-FCTF/BDIIscripts/
== For Debian/Ubuntu 6/7/8 ==
cd BDIIscripts
pip install -e .


= Configuration guide =
* Add AppDB's repository:
sudo wget http://repository.egi.eu/community/software/cloud.info.provider/0.x/candidates/0.4/repofiles/ubuntu-precise-amd64-candidate.list \
          -O /etc/apt/sources.list.d/cloud-info-provider.list
* Add AppDB key:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E2E992EB352D3E14
* Install the package
sudo apt-get update
sudo apt-get install python-cloud-info-provider


== Configure middleware backend ==
= Configuration =
The cloud provider script information is retrieved partially from a static configuration file and partially from the cloud middleware directly. Thus, the configuration depends on which middleware you have installed.


=== OpenNebula ===
Use one of the template files in <code>/etc/cloud-info-provider</code> as basis for creating your own YAML file with the static information of your resources. E.g:
''NOTE:'' This is a pure OpenNebula installation. If you have installed OpenNebula via rOCCI, refer to the [[#OpenNebula via rOCCI|OpenNebula via rOCCI]] guide.
cp /etc/cloud-info-provider/sample.openstack.yaml /etc/cloud-info-provider/bdii.yaml


* Copy the sample provider configuration file to the default software configuration file
Each middleware has its own options to fetch the dynamic information, check the <code>--help</code> option for more information. Some additional notes are given in the following sections.
cp /opt/cloud-info-provider/etc/sample.opennebula.yaml /opt/cloud-info-provider/etc/bdii.yaml
* Edit the <code>/opt/cloud-info-provider/etc/bdii.yaml</code> configuration, setting up the site permanent information and the OpenNebula connection information. Most of the information to be provider is self explanatory or specified in the comments. Below there is a set of notes who can be relevant during the configuration.


''Configuration notes:''
== General ==
* Keep always full_bdii_ldif set to False
* Site name will be fetched from <code>site</code> -> <code>name</code> in the template file. Set it to the name defined in GOCDB. Alternatively, the site name can be fetched from /etc/glite-info-static/site/site.cfg (or by the file set with the --glite-site-info-static option)
* You need to specify connection parameters to the OpenNebula XML-RPC interface. ''on_auth'' should contain the authorization parameters for an existing user with full read permissions on the image disks. If the user has been created with the ''core'' driver, this parameter shall be set to ''<username>:<password>''. ''on_rpcxml_endpoint'' shall contain the address of the RPCv2 endpoint. Usually it is ''http://myipaddress:2633/RPC2'' . If not on a secure network, it is suggested to provide this interface via https, since the on_auth parameter will be sent in clear text to the server.
* ''site'' parameters can be left commented, since they will be automatically retreived from the ''/etc/glite-info-static/site/site.cfg'' configuration file
* Compute templates can be ignored, since OpenNebula has no concept of resource flavours
* Object storage services (STorage-as-a-service) can be set statically. As they are not provided by OpenNebula, they can be ignored or set to the ones provided by other middleware.


=== OpenNebula via rOCCI ===
== OpenNebula + rOCCI ==


* Copy the sample provider configuration file to the default software configuration file
* You need to specify connection parameters to the OpenNebula XML-RPC interface:
cp /opt/cloud-info-provider/etc/sample.opennebularocci.yaml /opt/cloud-info-provider/etc/bdii.yaml
** <code>--on_auth</code> parameter should contain the authorization parameters for an existing user with full read permissions on the image disks. If the user has been created with the <code>core</code> driver, this parameter shall be set to <code><username>:<password></code>.
* Edit the ''/opt/cloud-info-provider/etc/bdii.yaml'' configuration, setting up the site permanent information and the OpenStack connection information. Most of the information to be provider is self explanatory or specified in the comments. Below there is a set of notes who can be relevant during the configuration.
** <code>--on_rpcxml_endpoint</code> shall contain the address of the RPCv2 endpoint. Usually it is <code><nowiki>http://<hostname>:2633/RPC2</nowiki></code>. If not on a secure network, it is suggested to provide this interface via https, since the <code>on_auth</code> parameter will be sent in clear text to the server.


''Configuration notes:''
* Compute templates can be gathered in two ways: directly from rOCCI configuration, by setting up the <code>--template_dir</code> option pointing to the rOCCI configuration folder or manually by placing them in the configuration file. One option does not preclude the other and the resulting templates will be the merge of the two.
* Keep always full_bdii_ldif set to False
* You need to specify connection parameters to the OpenStack interface. ''on_auth'' should contain the authorization parameters for an existing user with full read permissions on the image disks. If the user has been created with the ''core'' driver, this parameter shall be set to ''<username>:<password>''. ''on_rpcxml_endpoint'' shall contain the address of the RPCv2 endpoint. Usually it is ''http://myipaddress:2633/RPC2'' . If not on a secure network, it is suggested to provide this interface via https, since the on_auth parameter will be sent in clear text to the server.
* ''site'' parameters can be left commented, since they will be automatically retreived from the ''/etc/glite-info-static/site/site.cfg'' configuration file
* Compute templates can be gathered in two ways: directly from rOCCI configuration, by setting up the ''template_dir'' parameter to the rOCCI configuration folder or manually by placing them in the configuration file. One option does not preclude the other and the resulting templates will be the merge of the two.
* Images are retrieved from the OpenNebula templates, to mimic the behavior of rOCCI.
* Object storage services (STorage-as-a-service) can be set statically. As they are not provided by OpenNebula, they can be ignored or set to the ones provided by other middleware.


=== OpenStack ===


* Install the Nova Python SDK (needed by the OpenStack driver). You should have packages for that. In RHEL, they are provided by EPEL and you can install them via
== OpenStack ==
yum install -y python-novaclient


* Copy the sample provider configuration file to the default software configuration file
* The OpenStack provider uses python-novaclient (already included as a dependency in deb packages, rpm based systems will need to install it separately):
cp /opt/cloud-info-provider/etc/sample.openstack.yaml /opt/cloud-info-provider/etc/bdii.yaml
** <code>--os-username</code>, <code>--os-password</code>, <code>--auth-tenant-name</code>, <code>--os-auth-url</code>, <code>--os-cacert</code>, <code>--insecure</code> options to the cloud-provider allow to set the connection parameters. Alternatively you can use environment variables (e.g. <code>OS_USERNAME</code>) as with other OpenStack clients
* Edit the <code>/opt/cloud-info-provider/etc/bdii.yaml</code> configuration, setting up the site permanent information and the OpenStack connection information. Most of the information to be provider is self explanatory or specified in the file comments. Below there is a set of notes who can be relevant during the configuration.
** <code>--insecure</code> should not be used in production!


''Configuration notes:''
* Be sure that keystone contains the OCCI endpoint, otherwise it will not be published by the BDII. You can check this via the command <code>keystone service-list</code>. To create a new service and endpoint, you can run
* Keep always full_bdii_ldif set to False
<pre>
* You need to specify connection parameters to the OpenStack Auth service (Keystone). OpenStack will then get the API endpoints from Keystone.
keystone service-create --name nova --type occi --description 'Nova OCCI Service'
* Be sure that keystone contains the OCCI endpoint, otherwise it will not be published by the BDII. You can check this via the command <code>keystone service-list</code>. To create a new service and endpoint, you can run <code>keystone service-create --name nova --type occi --description 'Nova OCCI Service'</code> and then <code>keystone endpoint-create --service_id 8e6de5d0d7624584bed6bec9bef7c9e0 --region RegionOne --publicurl http://$HOSTNAME:8787/ --internalurl http://$HOSTNAME:8787/ --adminurl http://$HOSTNAME:8787/</code> where the ''service_id'' is the one obtained from <code>keystone service-list</code>
keystone endpoint-create --service_id <service-id> --region RegionOne --publicurl http://$HOSTNAME:8787/ --internalurl http://$HOSTNAME:8787/ --adminurl http://$HOSTNAME:8787/
* By default, the provider script will filter images who have no marketplace uri defined into the marketplace or vmcatcher_event_ad_mpuri property. If you want to list all the images templates (included local snapshots), set the variable 'require_marketplace_id: false' under 'compute' -> 'images' -> 'defaults' in the YAML configuration file
</pre>  
* In production environments, it is recommended to set the ''insecure'' parameter in the options to ''False'' and uncomment the ''os_cacert''
where the ''service-id'' is the one obtained from <code>keystone service-list</code>.
* ''site'' parameters can be left commented, since they will be automatically retreived from the ''/etc/glite-info-static/site/site.cfg'' configuration file


=== Other ===
* By default, the provider script will filter images without marketplace uri defined into the marketplace or vmcatcher_event_ad_mpuri property. If you want to list all the images templates (included local snapshots), set the variable 'require_marketplace_id: false' under 'compute' -> 'images' -> 'defaults' in the YAML configuration file.
For all the other middleware, you can setup all the middleware information statically. To do so:


* Copy the sample provider configuration file to the default software configuration file
== Create the provider ==
cp /opt/cloud-info-provider/etc/sample.static.yaml /opt/cloud-info-provider/etc/bdii.yaml
* Edit the <code>/opt/cloud-info-provider/etc/bdii.yaml</code> configuration, setting up all the site compute and storage resource information. Most of the information to be provider is self explanatory or specified in the comments. Below there is a set of notes who can be relevant during the configuration.


''Configuration notes:''
* Create the file <code>/var/lib/bdii/gip/provider/cloud-info-provider</code> that calls the provider with the correct options for your site, for example:
* Keep always full_bdii_ldif set to False
<pre>
* ''site'' parameters can be left commented, since they will be automatically retreived from the ''/etc/glite-info-static/site/site.cfg'' configuration file
#!/bin/sh


== Test configuration ==
cloud-info-provider-service --yaml /etc/cloud-info-provider/openstack.yaml \
Run manually the cloud-provider script and check that the output is correctly imported into the BDII. To do so, execute
                            --middleware openstack \
                            --os-username <username> --os-password <passwd> \
                            --os-tenant-name <tenant> --os-auth-url <url>
</pre>
* Give execution permission:
chmod +x /var/lib/bdii/gip/provider/cloud-info-provider
* and test it:
/var/lib/bdii/gip/provider/cloud-info-provider
* That should return the complete LDIF describing your site. Now you can start the bdii service
service bdii start
* And check that the information is being published
ldapsearch -x -h localhost -p 2170 -b o=glue


/usr/bin/cloud-info-provider-service > cloud-ldif.ldif
== Add your resource BDII to the site-BDII ==


Then open the cloud-ldif.ldif fiel and check that no error is present. Then, import the data into the LDAP via
Information on how to set up your Site-BDII is available at [[MAN01 How to publish Site Information]]


ldapdelete -H ldap://full.hostname:2170 'GLUE2GroupID=cloud,GLUE2DomainID=<your site>,o=glue' -D 'your LDAP admin DN' -W
Add your cloud-info-provider to your site-BDII by adding a new URL like this:
ldapadd -f cloud-ldif.ldif -H ldap://full.hostname:2170 -D 'your LDAP admin DN' -W


and check that the cloud data has been successfully added via
ldap://<cloud-info-provier-hostname>:2170/GLUE2GroupID=cloud,o=glue


ldapsearch -x -H ldap://full.hostname:2170 -b 'GLUE2GroupID=cloud,GLUE2DomainID=<your site>,o=glue'
== Alternative: use your cloud-provider as site-BDII ==


== Enable provider ==
{{Template:Block-comment
To enable the provider, just link the executable to the BDII provider directory (as default: <code>/var/lib/bdii/gip/provider/</code> or the BDII_PROVIDER_DIR path set into <code>/etc/bdii/bdii.conf</code>)
| name=Warning
| text=The recommended deployment is having an independent site-BDII
  }}


ln -fs /usr/bin/cloud-info-provider-service /var/lib/bdii/gip/provider/
If you don't have an existing site-BDII and you want to generate both the resource information and the site information with the cloud-bdii-provider you can add in the /var/lib/bdii/gip/provider/cloud-info-provider the <code>--full-bdii-info</code> option to the cloud-info-provider-service. The YAML file must contain all your site information as described in the templates.

Revision as of 17:42, 6 October 2014


Purpose

This page provides instructions on how to configure the Federated Cloud Production resource-BDII.

Installation

Packages for the cloud-info-provider are available at EGI's AppDB. They will install bdii as a dependency.

RHEL/CentOS/ScientificLinux

wget http://repository.egi.eu/community/software/cloud.info.provider/0.x/candidates/0.4/repofiles/sl-6-x86_64-candidate.repo \
     -O /etc/yum.repos.d/cloud-info-provider.repo
  • Install the package
yum install cloud-info-provider-service

For Debian/Ubuntu 6/7/8

  • Add AppDB's repository:
sudo wget http://repository.egi.eu/community/software/cloud.info.provider/0.x/candidates/0.4/repofiles/ubuntu-precise-amd64-candidate.list \
          -O /etc/apt/sources.list.d/cloud-info-provider.list
  • Add AppDB key:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E2E992EB352D3E14
  • Install the package
sudo apt-get update
sudo apt-get install python-cloud-info-provider

Configuration

Use one of the template files in /etc/cloud-info-provider as basis for creating your own YAML file with the static information of your resources. E.g:

cp /etc/cloud-info-provider/sample.openstack.yaml /etc/cloud-info-provider/bdii.yaml

Each middleware has its own options to fetch the dynamic information, check the --help option for more information. Some additional notes are given in the following sections.

General

  • Site name will be fetched from site -> name in the template file. Set it to the name defined in GOCDB. Alternatively, the site name can be fetched from /etc/glite-info-static/site/site.cfg (or by the file set with the --glite-site-info-static option)

OpenNebula + rOCCI

  • You need to specify connection parameters to the OpenNebula XML-RPC interface:
    • --on_auth parameter should contain the authorization parameters for an existing user with full read permissions on the image disks. If the user has been created with the core driver, this parameter shall be set to <username>:<password>.
    • --on_rpcxml_endpoint shall contain the address of the RPCv2 endpoint. Usually it is http://<hostname>:2633/RPC2. If not on a secure network, it is suggested to provide this interface via https, since the on_auth parameter will be sent in clear text to the server.
  • Compute templates can be gathered in two ways: directly from rOCCI configuration, by setting up the --template_dir option pointing to the rOCCI configuration folder or manually by placing them in the configuration file. One option does not preclude the other and the resulting templates will be the merge of the two.


OpenStack

  • The OpenStack provider uses python-novaclient (already included as a dependency in deb packages, rpm based systems will need to install it separately):
    • --os-username, --os-password, --auth-tenant-name, --os-auth-url, --os-cacert, --insecure options to the cloud-provider allow to set the connection parameters. Alternatively you can use environment variables (e.g. OS_USERNAME) as with other OpenStack clients
    • --insecure should not be used in production!
  • Be sure that keystone contains the OCCI endpoint, otherwise it will not be published by the BDII. You can check this via the command keystone service-list. To create a new service and endpoint, you can run
keystone service-create --name nova --type occi --description 'Nova OCCI Service'
keystone endpoint-create --service_id <service-id> --region RegionOne --publicurl http://$HOSTNAME:8787/ --internalurl http://$HOSTNAME:8787/ --adminurl http://$HOSTNAME:8787/

where the service-id is the one obtained from keystone service-list.

  • By default, the provider script will filter images without marketplace uri defined into the marketplace or vmcatcher_event_ad_mpuri property. If you want to list all the images templates (included local snapshots), set the variable 'require_marketplace_id: false' under 'compute' -> 'images' -> 'defaults' in the YAML configuration file.

Create the provider

  • Create the file /var/lib/bdii/gip/provider/cloud-info-provider that calls the provider with the correct options for your site, for example:
#!/bin/sh

cloud-info-provider-service --yaml /etc/cloud-info-provider/openstack.yaml \
                            --middleware openstack \
                            --os-username <username> --os-password <passwd> \
                            --os-tenant-name <tenant> --os-auth-url <url>
  • Give execution permission:
chmod +x /var/lib/bdii/gip/provider/cloud-info-provider
  • and test it:
/var/lib/bdii/gip/provider/cloud-info-provider
  • That should return the complete LDIF describing your site. Now you can start the bdii service
service bdii start
  • And check that the information is being published
ldapsearch -x -h localhost -p 2170 -b o=glue

Add your resource BDII to the site-BDII

Information on how to set up your Site-BDII is available at MAN01 How to publish Site Information

Add your cloud-info-provider to your site-BDII by adding a new URL like this:

ldap://<cloud-info-provier-hostname>:2170/GLUE2GroupID=cloud,o=glue

Alternative: use your cloud-provider as site-BDII

Warning:
The recommended deployment is having an independent site-BDII


If you don't have an existing site-BDII and you want to generate both the resource information and the site information with the cloud-bdii-provider you can add in the /var/lib/bdii/gip/provider/cloud-info-provider the --full-bdii-info option to the cloud-info-provider-service. The YAML file must contain all your site information as described in the templates.