Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "MAN10"

From EGIWiki
Jump to navigation Jump to search
 
(336 intermediate revisions by 10 users not shown)
Line 2: Line 2:


{{Ops_procedures
{{Ops_procedures
|Doc_title = Setting up Cloud Resource Centre
|Doc_title = Cloud Resource Centre Installation Manual
|Doc_link = [[MAN09|https://wiki.egi.eu/wiki/MAN10]]
|Doc_link = [[MAN09|https://wiki.egi.eu/wiki/MAN10]]
|Version =  19 August 2014
|Version =  19 May 2017
|Policy_acronym = OMB
|Policy_acronym = OMB
|Policy_name = Operations Management Board
|Policy_name = Operations Management Board
Line 10: Line 10:
|Doc_status = DRAFT
|Doc_status = DRAFT
|Approval_date =  
|Approval_date =  
|Procedure_statement = This manual provides information on how to set up Cloud Resource Centre.
|Procedure_statement = This manual provides information on how to set up a Resource Centre providing cloud resources in the EGI infrastructure.
}}  
}}  


{{Template:Block-comment
| name=Warning
| text=The installation manual is now available at https://docs.egi.eu/. Information below just points to the relevant sections of that manual
}}


= Introduction  =
EGI cloud supports 3 middlewares. It means you can base your cloud site installation on one of the following cloud software:
*OpenNebula
*OpenStack
*Synnefo
If you want to install an EGI Cloud Site please have a look at our EGI Cloud Site Installation Manuals below.
''<span style="color: rgb(51,102,255);">Note: EGI Cloud Site Installation Manual is a step-by-step instruction for Cloud Site Admin. The manual is not meant to be a comprehensive on topics related to the installation, it is a collection of steps taken by someone to install an EGI cloud site starting from a scratch. Commands executed should be made available for someone to copy&amp;paste and easily follow up. At some initial stage the manual may not cover all cases, but it is meant to be extended by other site admins while following up the manual. It is a living document.
</span>''
<br>
= The manuals  =
<span style="color: rgb(51,102,255);">'''''Current issues:'''''</span>
*<span style="color: rgb(51,102,255);">''Documentation for cloud components is written with assumption that the admin knows where (machine, neighbour components) this components should be installed. It is missing the general cloud site deployment context''</span>
*<span style="color: rgb(51,102,255);">''Documentation should address the prerequsities part.''</span><span style="color: rgb(51,102,255);">&nbsp;</span>
*<span style="color: rgb(51,102,255);">''Documentation should address the constraints and limitations part i.e. supported operating systems, software versions.''</span>
*<span style="color: rgb(51,102,255);">''Documentation should provide a contact person (per component) which can be contacted in case of questions/problems.&nbsp;''</span><span style="color: rgb(51,102,255);">&nbsp;</span>
*<span style="color: rgb(51,102,255);">''Documentation should provide commands for checking validity of installation.''</span><span style="color: rgb(51,102,255);">
</span>
== Prerequisities &amp; Limitations  ==
Whatever cloud stack you choose you need to prepare some things at the begining:
#Hardware (minimal hw requirements for small cloud site e.g up to 100 VMs): <br>
##number of physical machines, performance/capacity requirements: RAM size
##disk space - how big, where must be connected, performance of network links (images are heavy!)
#DNS names, X.509 certificates
#Register in fedcloud VO
#Registration in AppDB to have access to private EGI VM image repository
#What operating systems are supported
=Cloud management frameworks=
== OpenStack  ==
{| style="border:1px solid black; background-color:yellow; color: black; padding:5px; font-size:140%; width: 90%; margin: auto;"
| style="padding-right: 15px; padding-left: 15px;" |
|[[File:Baustelle.png]] This part is '''under construction'''.
|}
EGI Cloud site can be based on OpenStack software with some EGI extensions. See deployment schema (''Note: <span style="color: rgb(51,102,255);">high level description on what modules are to be put on which machines.</span>'')
=== OpenStack installation ===
Integration with FedCloud requires a working OpenStack installation. Follow the general documentation at http://docs.openstack.org/, there are packages ready to use for most distributions (check for example [https://openstack.redhat.com/Main_Page RDO] for RedHat based distributions).
==== Requirements and Limitations ====
OpenStack integration with FedCloud is known to work with the following versions of OpenStack:
* ''Havana'' (EOL by OpenStack, should not be used in production)
* '''Icehouse'''
* '''Juno'''
Suggested list of services to provide FedCloud integration:
* Keystone service must be available in any case.
* If providing OCCI access (VM management):
** Nova
** Cinder
** Glance
** Neutron (nova-network can also be used for legacy installations), [http://docs.openstack.org/havana/install-guide/install/yum/content/section_networking-routers-with-private-networks.html Per-tenant routers with private networks] configuration is known to work.
* If providing CDMI access (Object storage):
** Swift
==== AAI integration in OpenStack ====
Every FedCloud site must support authentication of users with X.509 certificates with VOMS extensions. The [https://ifca.github.io/keystone-voms Keystone-VOMS] extension enables this kind of authentication on Keystone.
* Pre-requisites: you will need a valid host certificate from a EUGridPMA CA.
* Installation: documentation on the installation is available at [https://keystone-voms.readthedocs.org/ Keystone-voms documentation]. Make sure to use the correct documentation for your OpenStack version.
* Take into account that using keystone-voms plugin will enforce the use of https for your Keystone service, you will need to update your URLs at the Keystone catalog and in the configuration of your services (check <code>[keystone_authtoken]</code> in nova, cinder, glance config files and any other service that needs to check keystone tokens)
<span style="color: rgb(51,102,255);">''change database? other services''</span>
* VOs: Every FedCloud site is expected to support [http://operations-portal.egi.eu/vo/view/voname/fedcloud.egi.eu fedcloud.egi.eu], [http://operations-portal.egi.eu/vo/view/voname/dteam dteam] and [http://operations-portal.egi.eu/vo/view/voname/ops ops] VOs. You should configure this VO in your <code>/etc/keystone/voms.json</code> file. Make sure that the tenant you are mapping the VO to exists. Below there is a sample <code>voms.json</code> file, adapt it with the appropriate names of your tenants:
{
    "fedcloud.egi.eu": {
        "tenant": "VO:fedcloud.egi.eu"
    },
    "dteam": {
        "tenant": "VO:dteam"
    },
    "ops": {
        "tenant": "VO:ops"
    }
}
You also need to include the appropriate <code>.lsc</code> files for each VO at <code>/etc/grid-security/vomsdir/</code>:
<pre>
mkdir -p /etc/grid-security/vomsdir/fedcloud.egi.eu
cat > /etc/grid-security/vomsdir/fedcloud.egi.eu/voms1.egee.cesnet.cz << EOF
/DC=org/DC=terena/DC=tcs/OU=Domain Control Validated/CN=voms1.egee.cesnet.cz
/C=NL/O=TERENA/CN=TERENA eScience SSL CA
EOF
cat > /etc/grid-security/vomsdir/fedcloud.egi.eu/voms2.grid.cesnet.cz << EOF
/DC=org/DC=terena/DC=tcs/OU=Domain Control Validated/CN=voms2.grid.cesnet.cz
/C=NL/O=TERENA/CN=TERENA eScience SSL CA
EOF
mkdir -p /etc/grid-security/vomsdir/dteam
cat > /etc/grid-security/vomsdir/dteam/voms.hellasgrid.gr << EOF
/C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms.hellasgrid.gr
/C=GR/O=HellasGrid/OU=Certification Authorities/CN=HellasGrid CA 2006
EOF
cat > /etc/grid-security/vomsdir/dteam/voms2.hellasgrid.gr << EOF
/C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms2.hellasgrid.gr
/C=GR/O=HellasGrid/OU=Certification Authorities/CN=HellasGrid CA 2006
EOF
mkdir -p /etc/grid-security/vomsdir/ops
cat > /etc/grid-security/vomsdir/dteam/lcg-voms2.cern.ch << EOF
/DC=ch/DC=cern/OU=computers/CN=lcg-voms2.cern.ch
/DC=ch/DC=cern/CN=CERN Grid Certification Authority
EOF
cat > /etc/grid-security/vomsdir/dteam/voms2.cern.ch << EOF
/DC=ch/DC=cern/OU=computers/CN=voms2.cern.ch
/DC=ch/DC=cern/CN=CERN Grid Certification Authority
EOF
</pre>
* VOMS-Keystone configuration: most sites should enable the <code>autocreate_users</code> option in the <code>[voms]</code> section of [https://keystone-voms.readthedocs.org/en/latest/configuration.html Keystone-VOMS configuration]. This will enable that new users are automatically created in your local keystone the first time they login into your site.
==== OCCI Support ====
OCCI is the EGI-approved access method for computing resources that VM management cloud services must expose. [https://github.com/EGI-FCTF/occi-os OCCI-OS] is the recommended software to provide this capability.
OCCI-OS can be installed from the github repo (recommended) or by using pip (packages may not be up-to-date!). The module must be installed on the machines hosting your nova-api. Installation instructions are available in the <code>README.md</code> file of the repo. Before installing OCCI-OS, you should manually install pyssf (<code>pip install pyssf</code>). If installing from the github repo, '''be sure to select the appropriate branch for your OpenStack installation''', e.g. for an OpenStack Icehouse installation:
<pre>
$ pip install pyssf
$ git clone https://github.com/EGI-FCTF/occi-os.git -b stable/icehouse
Cloning into 'occi-os'...
remote: Counting objects: 1312, done.
remote: Total 1312 (delta 0), reused 0 (delta 0), pack-reused 1312
Receiving objects: 100% (1312/1312), 357.53 KiB | 0 bytes/s, done.
Resolving deltas: 100% (752/752), done.
Checking connectivity... done.
$ cd occi-os
$ python setup.py install
running install
running bdist_egg
running egg_info
creating openstackocci_icehouse.egg-info
...
Finished processing dependencies for openstackocci-icehouse==1.0
Configuration is also detailed in the [https://github.com/EGI-FCTF/occi-os/#configuration OCCI-OS readme file].
</pre>
==== EGI Accounting ====
Every cloud site must publish utilization data to the EGI accounting database. You will need to install [https://github.com/IFCA/caso cASO], a pluggable extractor of Cloud Accounting Usage Records from OpenStack.
* Latest version is available at PyPi: (https://pypi.python.org/pypi/caso/), you can install it with <code>pip install caso</code>.
* Check the [http://caso.readthedocs.org/en/latest/ cASO documentation] includes how to install and configure OpenStack for generating the accounting records.
* Source code available at [https://github.com/IFCA/caso cASO github repo]
* Packages for Ubuntu distributions are build at [https://build.opensuse.org/project/show/home:aloga:cloud:integration OpenSUSE build service home:aloga:cloud:integration project]
In order to send the records to the accounting database, you will also need to configure SSM. Follow the [https://wiki.egi.eu/wiki/Fedcloud-tf:WorkGroups:Scenario4#Publishing_Records publishing records documentation at the accounting scenario]
==== EGI Information System ====
Sites must publish information to EGI information system which is based on BDII. There is a common [https://github.com/EGI-FCTF/cloud-bdii-provider bdii provider] for all cloud management frameworks. Information on installation and configuration is available in [https://github.com/EGI-FCTF/cloud-bdii-provider/blob/master/README.md the cloud-bdii-provider README.md] and in the [[Fedclouds BDII instructions]], there is a [[Fedclouds_BDII_instructions#OpenStack|specific section with OpenStack details]].
==== EGI Image Management ====
#EGI Image Management<br>Each cloud site must give access to EGI-approved VM images. An image clarifying functions and relations between vmcaster, vmcatcher, glance, glancepush and openstack handler for vmcatcher would be well welcome.''<br>''
##Registration in AppDB to have access to private EGI VM image repository - ''<span style="color: rgb(51,102,255);">missin</span><span style="color: rgb(51,102,255);">g, please describe steps to be done by Site Admin, should be as prerequisite steps because it requires manual step by someone</span>''
##<s>''VMCatcher - allows users to subscribe to VMs (unclear). [https://github.com/hepix-virtualisation/vmcatcher https://github.com/hepix-virtualisation/vmcatcher]''</s>
##Install EGI-customized version of glancepush: instruction: [https://wiki.egi.eu/wiki/Fedcloud-tf:WorkGroups:Scenario8:Configuration#OpenStack https://wiki.egi.eu/wiki/Fedcloud-tf:WorkGroups:Scenario8:Configuration#OpenStack] software repo: [https://appdb.egi.eu/store/software/python.glancepush/releases/0.0.x https://appdb.egi.eu/store/software/python.glancepush/releases/0.0.x]
##Install <span class="external text">Openstack handler for vmcatcher</span>: instruction:&nbsp;: [https://wiki.egi.eu/wiki/Fedcloud-tf:WorkGroups:Scenario8:Configuration#OpenStack https://wiki.egi.eu/wiki/Fedcloud-tf:WorkGroups:Scenario8:Configuration#OpenStack] software repo: [https://appdb.egi.eu/store/software/openstack.handler.for.vmcatcher https://appdb.egi.eu/store/software/openstack.handler.for.vmcatcher]
==== Registration of services in GOCDB ====
#EGI Configuration Management Database (GOCDB)<br>Each cloud site must register services in EGI configuration management database which is GOCDB<br>''<span style="color: rgb(51,102,255);">Need information if cloud site must be separate from grid site or can be share.</span>''<span style="color: rgb(51,102,255);">'' Shouldn't GOCDB step be earlier to allow nagios monitoring?''</span><br>
##''[https://wiki.egi.eu/wiki/Fedcloud-tf:WorkGroups:Scenario5#GOCDB Registering endpoints https://wiki.egi.eu/wiki/Fedcloud-tf:WorkGroups:Scenario5#GOCDB] - <span style="color: rgb(51,102,255);">ok, but we need info on what exactly has been registered in gocDB like "I have regiesterd os.acme.org as</span>''<span style="color: rgb(51,102,255);"> with type eu.egi.cloud.accounting.</span>
##Registering SiteExtension Properties -''<span style="color: rgb(51,102,255);"> is this still valid requirement? missing instruction</span>''
=== Installation Validation ===
#<span style="color: rgb(51,102,255);">Installation validaton - this is a new step: ''<span style="color: rgb(51,102,255);"><span style="color: rgb(51,51,51);"><span style="color: rgb(51,102,255);">describe steps performed by site admin that confirms the site installation is working well according to EGI requirements</span></span></span>''<span style="color: rgb(51,102,255);">.&nbsp;''It is better to have it as a separate, final step for all checks''</span></span>
##<span style="color: rgb(51,51,51);">Nagios step - ''<span style="color: rgb(51,102,255);">missing</span>''</span>
##<span style="color: rgb(51,102,255);"><span style="color: rgb(51,51,51);">check accounting <span style="color: rgb(51,102,255);"><span style="color: rgb(51,51,51);">''<span style="color: rgb(51,102,255);">- missing</span>''</span></span><br></span></span>
##<span style="color: rgb(51,102,255);"><span style="color: rgb(51,51,51);">check vmcatcher subscription <span style="color: rgb(51,102,255);"><span style="color: rgb(51,51,51);">''<span style="color: rgb(51,102,255);">- missing</span>''</span></span><br></span></span>
##<span style="color: rgb(51,102,255);"><span style="color: rgb(51,51,51);">check BDII publishing ''<span style="color: rgb(51,102,255);">- missing</span>''<br></span></span>
##<span style="color: rgb(51,102,255);"><span style="color: rgb(51,51,51);">check OCCI ''[https://wiki.egi.eu/wiki/HOWTO04_Site_Certification_Manual_tests#Cloud_Compute_.28OCCI.29_checks It is possible to reuse https://wiki.egi.eu/wiki/HOWTO04_Site_Certification_Manual_tests#Cloud_Compute_.28OCCI.29_checks]''</span></span>
##<span style="color: rgb(51,102,255);"><span style="color: rgb(51,51,51);">check CDMI <span style="color: rgb(51,102,255);">''It is possible to reuse [https://wiki.egi.eu/wiki/HOWTO04_Site_Certification_Manual_tests#Cloud_Storage_.28CDMI.29_checks <span style="color: rgb(51,102,255);">https://wiki.egi.eu/wiki/HOWTO04_Site_Certification_Manual_tests#Cloud_Storage_.28CDMI.29_checks</span>]''</span></span></span>
<span style="color: rgb(51,102,255);"><span style="color: rgb(51,51,51);"><span style="color: rgb(51,102,255);"><span style="color: rgb(51,102,255);">Support for CDMI
</span></span></span></span>
<span style="color: rgb(51,102,255);"><span style="color: rgb(51,51,51);"><span style="color: rgb(51,102,255);"><span style="color: rgb(51,102,255);">''Maybe it is better to have separated part on CDMI and not mix it with OCCI.''
</span></span></span></span>
== OpenNebula  ==
EGI Cloud site is based on OpenNebula software with some EGI extensions.See Deployment Schema (''Note: here we need high level explanation on what modules are to be put on which machines.'')
Stages of installation ''(similar for every middleware):''
#OpenNebula installation with X.509 support
##''Be consistent on requirements WHICH OpenNebula version is supported.''
##Unfortunately this manual does not cover OpenNebula installation. You need to do this by yourself but this is well described here: [http://docs.opennebula.org/4.4/ http://docs.opennebula.org/4.4/]
##Configure X.509 support according to [http://docs.opennebula.org/4.4/administration/authentication/x509_auth.html http://docs.opennebula.org/4.4/administration/authentication/x509_auth.html]
#Support for OCCI - EGI-approved access method
##''Described here: [https://wiki.egi.eu/wiki/Fedcloud-tf:WorkGroups:_Federated_AAI:OpenNebula https://wiki.egi.eu/wiki/Fedcloud-tf:WorkGroups:_Federated_AAI:OpenNebula] but missing the context of EGI Cloud site installation i.e. what modules should be put on which machines, and which commands executed. <br>''
#EGI User Authentication/Authorization
##''You need to integrate with Perun. Described here [https://github.com/EGI-FCTF/fctf-perun https://github.com/EGI-FCTF/fctf-perun] but missing context of EGI Cloud site installation and missing commands to be executed.''
#EGI Image Management
##''<span style="color: rgb(255,0,0);">Missing in the instructions for OpenNebula.</span> [https://wiki.egi.eu/wiki/Fedcloud-tf:WorkGroups:Scenario8:Configuration#VMcatcher Seems we have in https://wiki.egi.eu/wiki/Fedcloud-tf:WorkGroups:Scenario8:Configuration#VMcatcher] section for Opennebula. Still no common context.''
#EGI Accounting
##''[https://github.com/EGI-FCTF/opennebula-cloudacc Described here: https://github.com/EGI-FCTF/opennebula-cloudacc] but missing context of EGI Cloud site installation - complete with info on which host the commands should be executed.''
#EGI Information System
##''[https://wiki.egi.eu/wiki/Fedclouds_BDII_instructions Described here https://wiki.egi.eu/wiki/Fedclouds_BDII_instructions] but again missing context of where these commands should be executed.''
#EGI Configuration Management Database (GOCDB)
##''Manual not available. We need information on OpenNebula-specific service types to be registered in GOCDB.''
<span style="color: rgb(255,0,0);">What with support for CDMI in Opennebula?</span>
== Synnefo  ==
<span style="color: rgb(51,102,255);">''There are installation guides e.g. [https://www.synnefo.org/docs/synnefo/latest/install-guide-debian.html#install-guide-debian <span style="color: rgb(51,102,255);">https://www.synnefo.org/docs/synnefo/latest/install-guide-debian.html#install-guide-debian</span>] but it seems there is no EGI-specific installation guides.''</span>
<br>
= Revision History  =
{| border="3"
|-
! Version
! Authors
! Date
! Comments
|-
|
|
|
|
|}


[[Category:Operations_Manuals]]
[[Category:Operations_Manuals]]

Latest revision as of 16:15, 12 April 2021

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home • Manuals • Procedures • Training • Other • Contact ► For: VO managers • Administrators



Title Cloud Resource Centre Installation Manual
Document link https://wiki.egi.eu/wiki/MAN10
Last modified 19 May 2017
Policy Group Acronym OMB
Policy Group Name Operations Management Board
Contact Group operations-support@mailman.egi.eu
Document Status DRAFT
Approved Date
Procedure Statement This manual provides information on how to set up a Resource Centre providing cloud resources in the EGI infrastructure.
Owner Owner of procedure


Warning:
The installation manual is now available at https://docs.egi.eu/. Information below just points to the relevant sections of that manual

Retrieved from "https://wiki.egi.eu/w/index.php?title=MAN10&oldid=112877"