Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Fedcloud-tf:WorkGroups:Contextualisation"

From EGIWiki
Jump to navigation Jump to search
 
(15 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Fedcloud-tf:Menu}} {{Fedcloud-tf:WorkGroups:Menu}} {{TOC_right}}  
{{Fedcloud-tf:Menu}} {{Fedcloud-tf:WorkGroups:Menu}} {{TOC_right}}  


<font color="red">Leader: Enol Fernandez </font>
<font color="red"></font>
 
= [Closed] =
 
<font color="red">Leader: Enol Fernandez </font>  


== Scope  ==
== Scope  ==


This workgroup deals with the VM contextualization open issues, namely being able to pass user-defined data to the VM.
This workgroup deals with the VM contextualization open issues, namely being able to pass user-defined data to the VM.  


== Roadmap  ==
== Roadmap  ==


* Evaluate mechanisms available in the cloud middleware to provide user-defined data
*Evaluate mechanisms available in the cloud middleware to provide user-defined data  
* Evaluate possible OCCI extensions to pass user-defined data
*Evaluate possible OCCI extensions to pass user-defined data  
* Evaluate CIMI as alternative API (already includes user-data features)
*Evaluate CIMI as alternative API (already includes user-data features)  
* Track implementation of OCCI extensions/CIMI on the cloud middleware
*Track implementation of OCCI extensions/CIMI on the cloud middleware
 
== Contextualization  ==
 
Contextualization is the process of installing, configuring and preparing software upon boot time on a pre-defined virtual machine image (e.g. setting the hostname, IP addresses, SSH aithorized keys, starting services, installing applications, etc.). We have identified as a requirement to contextualize images the possibility of passing user provided data to the VM when they are instantiated. Hence there are two things to be defined:


== Contextualization ==
*how to pass data upon VM creation (the exact type and format of the data should not be relevant, it should be up to the user)
*how to retrieve those data from the running VM


Contextualization is the process of installing, configuring and preparing software upon boot time on a pre-defined virtual machine image (e.g. setting the hostname, IP addresses, SSH aithorized keys, starting services, installing applications, etc.).  We have identified as a requirement to contextualize images the possibility of passing user provided data to the VM when they are instantiated. Hence there are two things to be defined:
For passing the data we have proposed the use of a new OCCI mixin that has an attribute to hold the data to pass to the image. The second part, related to retrieving the data, is more dependent of the back-end implementation. There are different methods in the systems in place in FedCloud, but tools like cloud-init handle those possible differences in a transparent way for the users.
* how to pass data upon VM creation (the exact type and format of the data should not be relevant, it should be up to the user)
* how to retrieve those data from the running VM


For passing the data we have proposed the use of a new OCCI mixin that has an attribute to hold the data to pass to the image. The second part, related to retrieving the data, is more dependent of the back-end implementation. There are different methods in the systems in place in FedCloud, but tools like cloud-init handle those possible differences in a transparent way for the users.
== OCCI support  ==


== OCCI support ==
FedCloud has agreed on two mixins for pushing user defined data into the VM instances:


FedCloud has agreed on two mixins for pushing user defined data into the VM instances:
*<code><nowiki>Category: user_data; scheme="http://schemas.openstack.org/compute/instance#"; class="mixin"</nowiki></code>, and
*<code><nowiki>Category: public_key; scheme="http://schemas.openstack.org/instance/credentials#"; class="mixin"</nowiki></code>


* <code><nowiki>Category: user_data; scheme="http://schemas.openstack.org/compute/instance#"; class="mixin"</nowiki></code>, and
The <code>user_data</code> mixin has an attribute <code>org.openstack.compute.user_data</code> where the use can specify the base64 encoded data that wants to be available at the VM upon instantiation.
* <code><nowiki>Category: public_key; scheme="http://schemas.openstack.org/instance/credentials#"; class="mixin"</nowiki></code>


The <code>user_data</code> mixin has an attribute <code>org.openstack.compute.user_data</code> where the use can specify the base64 encoded data that wants to be available at the VM upon instantiation.
In the case of the <code>public_key</code> mixin, there are two attributes to be specified:


In the case of the <code>public_key</code> mixin, there are two attributes to be specified:
*<code>org.openstack.credentials.publickey.name</code> with a name for the public key, and  
* <code>org.openstack.credentials.publickey.name</code> with a name for the public key, and
*<code>org.openstack.credentials.publickey.data</code> with the public key itself.
* <code>org.openstack.credentials.publickey.data</code> with the public key itself.


These are specified as headers in the HTTP request for creating a VM. A sample curl request could look like this (credential managing is missing):
These are specified as headers in the HTTP request for creating a VM. A sample curl request could look like this (credential managing is missing):  
<pre>
<pre>curl -H 'Category: compute;scheme="http://schemas.ogf.org/occi/infrastructure#";class="kind"' \
curl -H 'Category: compute;scheme="http://schemas.ogf.org/occi/infrastructure#";class="kind"' \
     -H 'Category: myVM;scheme="http://my.occi.service/occi/infrastructure/os_tpl#";class="mixin"' \
     -H 'Category: myVM;scheme="http://my.occi.service/occi/infrastructure/os_tpl#";class="mixin"' \
     -H 'Category: public_key;scheme="http://schemas.openstack.org/instance/credentials#";class="mixin"' \
     -H 'Category: public_key;scheme="http://schemas.openstack.org/instance/credentials#";class="mixin"' \
Line 45: Line 50:
     -H 'X-OCCI-Attribute: org.openstack.compute.user_data="VGhpcyBpcyBhIHNpbXBsZSB0ZXN0IQ=="' \
     -H 'X-OCCI-Attribute: org.openstack.compute.user_data="VGhpcyBpcyBhIHNpbXBsZSB0ZXN0IQ=="' \
     https://your.endpoint.com:1234/
     https://your.endpoint.com:1234/
</pre>
</pre>  
=== rOCCI client  ===


=== rOCCI client ===
[https://github.com/gwdg/rOCCI-cli rOCCI client] includes support for both extensions via the <code>-T</code>/<code>--context</code> command line argument. The client does not need to previously b64encode the data. The curl command above could be translated into the following
<pre>occi -e https://your.endpoint.com:1234 --action create --resource compute \
    --attribute occi.core.title='ContextVM' --mixin os_tpl#myVM
    --context user_data='This is just a test!' --context public_key='ssh-rsa BAA...zxe ==user@host'
</pre>
the user data and public keys can be specified as files:
<pre>--context user_data="file:///path/to/file" --context public_key="file:///path/to/publickey"
</pre>
=== OCCI-OS (OpenStack)  ===


[https://github.com/gwdg/rOCCI-cli rOCCI client] includes support for both extensions via the <code>-T</code>/<code>--context</code> command line argument. The client does not need to previously b64encode the data. The curl command above could be translated into the following
[https://github.com/EGI-FCTF/occi-os OCCI-OS] supports the extensions for pushing user data and keys for the <code>stable/folsom</code> and <code>stable/havana</code> versions.  
<pre>
occi -e https://your.endpoint.com:1234 --action create --resource compute \
    --attributes title='ContextVM' --mixin os_tpl#myVM
    --context user_data='This is just a test!',public_key='ssh-rsa BAA...zxe ==user@host'
</pre>


the user data and public keys can be specified as files:
=== rOCCI-server (OpenNebula and others)  ===
<pre>
--context user_data="file:///path/to/file",public_key="file:///path/to/publickey"
</pre>


=== OCCI-OS (OpenStack) ===
[http://gwdg.github.io/rOCCI-server/ rOCCI-server] also supports the extensions since version <code>4.0.0</code> of rOCCI framework.


[https://github.com/EGI-FCTF/occi-os OCCI-OS] supports the extensions for pushing user data and keys for the <code>stable/folsom</code> and <code>stable/havana</code> versions.
=== Synnefo  ===


=== rOCCI-server (OpenNebula and others) ===
[https://www.synnefo.org/ Synnefo] also supports the extension in the latests versions.


[http://gwdg.github.io/rOCCI-server/ rOCCI-server] also supports the extensions since version <code>4.0.0</code> of rOCCI framework.
== Using the data at the instances ==
 
=== Synnefo ===
[https://www.synnefo.org/] also supports the extension in the latests versions.
 
== Using the data at the instances ==


The proposed mixin does not impose any format on the data, thus the VM image creators are free to use this blob as it suites their needs. For most use cases, [https://help.ubuntu.com/community/CloudInit cloud-init] is the best way to avoid dealing with the details that each middleware has for providing the data in the VM instance.  
The proposed mixin does not impose any format on the data, thus the VM image creators are free to use this blob as it suites their needs. For most use cases, [https://help.ubuntu.com/community/CloudInit cloud-init] is the best way to avoid dealing with the details that each middleware has for providing the data in the VM instance.  


=== cloud-init ===
=== cloud-init ===


This task force recommends the use of cloud-init for the contextualization of VMs. Cloud-init frees the user from managing the specific ways for handling the contextualization information and it's widely available in most OS versions and IaaS cloud platforms. The latest versions support OpenNebula contextualization mechanisms (EC2/OpenStack was supported since long ago)
This task force recommends the use of cloud-init for the contextualization of VMs. Cloud-init frees the user from managing the specific ways for handling the contextualization information and it's widely available in most OS versions and IaaS cloud platforms. The latest versions support OpenNebula contextualization mechanisms. OpenStack and Synnefo contextualization are supported in most cloud-init versions (datasources are EC2 and NoCloud).


By default cloud-init will:
When creating a VM that is meant to run in different cloud middlewares leave the &lt;cloud&gt;datasource_list&lt;/cloud&gt; undefined (so it will try all the available ones), or set at least the following:
<pre>[ NoCloud, OpenNebula, Ec2, None ]
</pre>
<br> By default cloud-init will:  


- put the ssh-key into the <code>~/.ssh/authorized_keys</code> of root user (or equivalent)
- put the ssh-key into the <code>~/.ssh/authorized_keys</code> of root user (or equivalent)  


- if the user provided data is a script, it will be executed upon instantiation. More complex use-cases are supported, check [http://cloudinit.readthedocs.org/ cloud-init documentation] for examples.
- if the user provided data is a script, it will be executed upon instantiation. More complex use-cases are supported, check [http://cloudinit.readthedocs.org/ cloud-init documentation] for examples.  


<br>


==== cloud-init and OpenNebula ====
==== cloud-init and OpenNebula ====


The rOCCI server pushes into the OpenNebula context the base64 encoded user data provided by the client, however the default cloud-init is expecting clear text. We are working on the solution to this problem, in the meantime you can use the fedcloud packages (see below):
The rOCCI server pushes into the OpenNebula context the base64 encoded user data provided by the client, however the default cloud-init is expecting clear text. This is fixed for the upcoming 0.7.5 version of cloud-init, in the meantime while it's released you can use the fedcloud packages (see below):  
==== Packages ====


Custom FedCloud packages for some distributions are available at [https://appdb.egi.eu/store/software/fedcloud.cloud.init EGI's AppDB FedCloud cloud-init]. These packages already manage the user data correctly for OpenNebula.
==== Packages  ====


=== Contextualizing without cloudinit ===
Custom FedCloud packages for some distributions are available at [https://appdb.egi.eu/store/software/fedcloud.cloud.init EGI's AppDB FedCloud cloud-init]. These packages already manage the user data correctly for OpenNebula.


==== OpenStack ====
=== Contextualizing without cloudinit  ===


OpenStack provides the user provided data at the meta-data server available at <code><nowiki>http://169.254.169.254</nowiki></code> for every instance.
==== OpenStack ====
With the EC2 API, user data is available at <code><nowiki>http://169.254.169.254/latest/user-data</nowiki></code>, it can be fetched with curl:
<pre>
curl http://169.254.169.254/latest/user-data
</pre>


OpenStack API provides the same data at <code><nowiki>http://169.254.169.254/openstack/latest/user_data</nowiki></code>.
OpenStack provides the user provided data at the meta-data server available at <code><nowiki>http://169.254.169.254</nowiki></code> for every instance. With the EC2 API, user data is available at <code><nowiki>http://169.254.169.254/latest/user-data</nowiki></code>, it can be fetched with curl:
<pre>curl http://169.254.169.254/latest/user-data
</pre>
OpenStack API provides the same data at <code><nowiki>http://169.254.169.254/openstack/latest/user_data</nowiki></code>.  


Public keys are available at <code><nowiki>http://169.254.169.254/latest/meta-data/public-keys</nowiki></code> for EC2 API or <code><nowiki>http://169.254.169.254/openstack/latest/meta_data.json</nowiki></code>.
Public keys are available at <code><nowiki>http://169.254.169.254/latest/meta-data/public-keys</nowiki></code> for EC2 API or <code><nowiki>http://169.254.169.254/openstack/latest/meta_data.json</nowiki></code>.  


For getting OpenSSH key with EC2:
For getting OpenSSH key with EC2:  
<pre>
<pre>curl http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key
curl http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key
ssh-rsa  [...]==  xxx@host
ssh-rsa  [...]==  xxx@cloud
&lt;pre&gt;
<pre>


OpenStack API returns a json dictionary with a "public_keys" dictionary inside, where the available keys are stored:
OpenStack API returns a json dictionary with a "public_keys" dictionary inside, where the available keys are stored:
<pre>
&lt;pre&gt;
curl http://169.254.169.254/openstack/latest/meta_data.json
curl http://169.254.169.254/openstack/latest/meta_data.json
{... "public_keys": {"enolkey": "ssh-rsa [...]== xxx@cloud\n"}...}
{... "public_keys": {"enolkey": "ssh-rsa [...]== xxx@host\n"}...}
</pre>
</pre>  
See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html and http://docs.openstack.org/user-guide/content/user-data.html for more info.


See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html and http://docs.openstack.org/user-guide/content/user-data.html for more info.
==== OpenNebula  ====


==== OpenNebula ====
OpenNebula provides contextualization information through an ISO image that is available in the VM instance. The contents of the ISO image include a context.sh file where a set of variables are defined according to the VM template.


OpenNebula provides contextualization information through an ISO image that is available in the VM instance. The contents of the ISO image include a context.sh file where a set of variables are defined according to the VM template.
The iso filesystem is normally available at <code>/dev/sr0</code> where a <code>context.sh</code> file contains all the context variables declared in the OpenNebula template.  


The iso filesystem is normally available at <code>/dev/sr0</code> where a <code>context.sh</code> file contains all the context variables declared in the OpenNebula template.
The <code>USER_DATA</code> variable contains a base64 enconded blob (can be decoded with <code>base64 -d</code>) with the data provided by the user with the OCCI client. The <code>SSH_PUBLIC_KEYS</code> variable contain the user public keys.  


The <code>USER_DATA</code> variable contains a base64 enconded blob (can be decoded with <code>base64 -d</code>) with the data provided by the user with the OCCI client. The <code>SSH_PUBLIC_KEYS</code> variable contain the user public keys.
For using this context just mount the filesystem and source the context file:  
 
<pre>$ mount -o ro /dev/sr0 /mnt/
For using this context just mount the filesystem and source the context file:
<pre>
$ mount -o ro /dev/sr0 /mnt/
$ cat /mnt/context.sh   
$ cat /mnt/context.sh   
# Context variables generated by OpenNebula
# Context variables generated by OpenNebula
Line 149: Line 148:
$ . /mnt/context.sh
$ . /mnt/context.sh
$ echo $SSH_PUBLIC_KEY
$ echo $SSH_PUBLIC_KEY
ssh-rsa [...]== xxx@cloud
ssh-rsa [...]== xxx@host
$ echo $USER_DATA | base64 -d
$ echo $USER_DATA | base64 -d
This is my user data
This is my user data
</pre>
</pre>  
See http://opennebula.org/documentation:archives:rel3.8:cong for more information.
 
==== Synnefo  ====
 
Synnefo provides context data in a [http://cloudinit.readthedocs.org/en/latest/topics/datasources.html#no-cloud cloud-init NoCloud] compatible format.
 
Two files are injected into the VMs: <code>/var/lib/cloud/seed/nocloud-net/meta-data</code>, which is a <code>yaml</code> file that contains the user provided public keys:
<pre>public-keys: ssh-rsa [...]== xxx@host
</pre>
And <code>/var/lib/cloud/seed/nocloud-net/user-data</code>, which contains a blob with the data provided by the user with the OCCI client.
 
== Setting up the middleware for contextualization  ==
 
=== OpenStack  ===
 
Default OpenStack installations do not require any additional steps for providing contextualization.
 
=== OpenNebula  ===
 
Cloud-init documentation includes [http://cloudinit.readthedocs.org/en/latest/topics/datasources.html#opennebula information on the OpenNebula configuration]. Check also OpenNebula [http://opennebula.org/documentation:documentation:context_overview contextualization overview] for more information.
 
== Windows Contextualization  ==
 
Windows VM also support contextualization of the machine on machine boot using [https://github.com/stackforge/cloudbase-init cloudbase-init] software. This software similarly to the cloud-init available for linux machines uses the available meta-data and user-data and executes a set of plugins to configure the virtual machine.
 
Cloudbase-init supports OpenStack meta-data and there is work in progress to support other cloud management frameworks, including OpenNebula.
 
=== Installation  ===
 
Cloudbase-init is distributed with a MSI installer build after every commit in the repo:
 
*[https://www.cloudbase.it/downloads/CloudbaseInitSetup_Beta_x64.msi 64-bit version]
*[https://www.cloudbase.it/downloads/CloudbaseInitSetup_Beta_x86.msi 32-bit version]
 
During the installation, you can configure:
 
*Name of a user that will be created and upon boot up (if it is an already existing user, cloudbase-init will only change its password)
*Network adapter / serial ports to use (defaults should be safe)
 
Normally is not needed to execute sysprep on the image after the installation of cloudbase-init
 
Once installed the machine can be uploaded/snapshoted to be started by the users.
 
=== Contextualization  ===
 
The features of cloudbase-init are not as extensive as the Linux cloud-init, but it can still perform the actions described below. For an up-to-date list and documentation, check the [https://github.com/stackforge/cloudbase-init#plugins Plugin documentation of cloudbase-init], there is also documentation available at [http://wiki.cloudbase.it/cloudbaseinit-doc cloudbaseinit wiki]
 
==== User configuration  ====
 
cloudbase-init by default will configure the user account configured during contextualization. Take into account that the configured group '''must''' exist before rebooting the machine.
 
===== Specifying the user password via OCCI  =====
 
Cloudbase-init can set the user password if specified in the context of the VM. OpenStack does automatically generate a random password on machine creation but this is not available to the OCCI interface. We have created a new mixin that allows to specify the password you want the user to have as follows:
 
<code><nowiki>Category: admin_pwd; scheme="http://schemas.openstack.org/compute/instance#"; class="mixin"</nowiki></code>
 
And the password is specified with the attribute:
 
<code><nowiki>org.openstack.credentials.admin_pwd</nowiki></code>
 
For using this mixin on VM creation, you can use a command line similar to this:
<pre>occi -e $OCCI_ENDPOINT -x $X509_USER_PROXY -X -n x509 \
    -a  create -r compute \
    -M $OS_TPL -M $RES_TPL \
    -t occi.core.title="$VM_TITLE" \
    -T user_data="$USER_DATA" \
    -M http://schemas.openstack.org/compute/instance#admin_pwd \
    -t org.openstack.credentials.admin_pwd="&lt;secret&gt;"
</pre>
==== User-data  ====
 
cloudbase-init will try to execute as a script whatever is passed in the user-data. It uses the first line of the file to determine which kind of script is passed for execution:
 
*For Windows batch, the first line of the file must be
 
rem cmd
 
*For Powershell, the first line must be:
 
#ps1_sysnative


See http://opennebula.org/documentation:archives:rel3.8:cong for more information.
or for a x86 PowerShell


== Setting up the middleware for contextualization ==
#ps1_x86


=== OpenStack ===
*For Bash (it must be preinstalled on the VM!), the first line must ''begin'' with


Default OpenStack installations do not require any additional steps for providing contextualization.
#!


=== OpenNebula ===
for example, it can be


Cloud-init documentation includes [http://cloudinit.readthedocs.org/en/latest/topics/datasources.html#opennebula information on the OpenNebula configuration]. Check also OpenNebula [http://opennebula.org/documentation:documentation:context_overview contextualization overview] for more information.
#!/bin/bash


== Other info ==
The script will be executed with admin privileges. Take into account that the file must have DOS format, so if you are editing them on a UNIX-based machine you should make sure that the endline terminators are in the correct format.


=== CIMI ===
==== Forcing re-execution of contextualization  ====


CIMI standard does include contextualization features. We have deployed an instance of OpenStack for testing the CIMI implementation from this [https://blueprints.launchpad.net/nova/+spec/cimi blueprint]. A simple [https://github.com/enolfc/cimi/commit/17635bf4fac24dd86af0fe86dd7300a5c588fded modification] adds the support for the `userData` parameter in machine creation.
cloudbase-init will execute the plugins only on the first time the image is booted. If you need to re-execute them you should remove in the Windows registry the following key:


Sample code for the request (python code with requests package):
HKEY_LOCAL_MACHINE\Software\Cloudbase Solutions\Cloudbase-Init


<nowiki>
on 64 bit versions of Windows, the key is:  
headers = {'content-type': 'application/json',
                  'accept': 'application/json',
                  'X-Auth-Token': keystone_token}


HKEY_LOCAL_MACHINE\Software\Wow6432Node\Cloudbase Solutions\Cloudbase-Init


img_body = {
The next time cloudbase-init is executed (due to a reboot or by restarting the service), it will execute the plugins again.
      'name': 'cimi-test-server',
      'description': 'no desc',
      'machineTemplate': {
          'machineImage': {'href': image_id},
          'machineConfig': {'href': conf_id},
          'userData': base64.b64encode('this is just a test'),
    }
}
r = requests.post(cimi_url + '/machineCollection',
                            headers=headers,
                            data=json.dumps(img_body))
</nowiki>

Latest revision as of 09:27, 20 April 2015

Main Roadmap and Innovation Technology For Users For Resource Providers Media


Workbenches: Open issues
Scenario 1
VM Management
Scenario 2
Data Management
Scenario 3
Information Systems
Scenario 4
Accounting
Scenario 5
Monitoring
Scenario 6
Notification
Scenario 7
Federated AAI
Scenario 8
VM Image Management
Scenario 9
Brokering
Scenario 10
Contextualisation
Scenario 11
Security



[Closed]

Leader: Enol Fernandez

Scope

This workgroup deals with the VM contextualization open issues, namely being able to pass user-defined data to the VM.

Roadmap

  • Evaluate mechanisms available in the cloud middleware to provide user-defined data
  • Evaluate possible OCCI extensions to pass user-defined data
  • Evaluate CIMI as alternative API (already includes user-data features)
  • Track implementation of OCCI extensions/CIMI on the cloud middleware

Contextualization

Contextualization is the process of installing, configuring and preparing software upon boot time on a pre-defined virtual machine image (e.g. setting the hostname, IP addresses, SSH aithorized keys, starting services, installing applications, etc.). We have identified as a requirement to contextualize images the possibility of passing user provided data to the VM when they are instantiated. Hence there are two things to be defined:

  • how to pass data upon VM creation (the exact type and format of the data should not be relevant, it should be up to the user)
  • how to retrieve those data from the running VM

For passing the data we have proposed the use of a new OCCI mixin that has an attribute to hold the data to pass to the image. The second part, related to retrieving the data, is more dependent of the back-end implementation. There are different methods in the systems in place in FedCloud, but tools like cloud-init handle those possible differences in a transparent way for the users.

OCCI support

FedCloud has agreed on two mixins for pushing user defined data into the VM instances:

  • Category: user_data; scheme="http://schemas.openstack.org/compute/instance#"; class="mixin", and
  • Category: public_key; scheme="http://schemas.openstack.org/instance/credentials#"; class="mixin"

The user_data mixin has an attribute org.openstack.compute.user_data where the use can specify the base64 encoded data that wants to be available at the VM upon instantiation.

In the case of the public_key mixin, there are two attributes to be specified:

  • org.openstack.credentials.publickey.name with a name for the public key, and
  • org.openstack.credentials.publickey.data with the public key itself.

These are specified as headers in the HTTP request for creating a VM. A sample curl request could look like this (credential managing is missing):

curl -H 'Category: compute;scheme="http://schemas.ogf.org/occi/infrastructure#";class="kind"' \
     -H 'Category: myVM;scheme="http://my.occi.service/occi/infrastructure/os_tpl#";class="mixin"' \
     -H 'Category: public_key;scheme="http://schemas.openstack.org/instance/credentials#";class="mixin"' \
     -H 'Category: user_data;scheme="http://schemas.openstack.org/compute/instance#";class="mixin"' \
     -H 'X-OCCI-Attribute: org.openstack.credentials.publickey.name="my_key"' \ 
     -H 'X-OCCI-Attribute: org.openstack.credentials.publickey.data="ssh-rsa BAA...zxe ==user@host" \
     -H 'X-OCCI-Attribute: org.openstack.compute.user_data="VGhpcyBpcyBhIHNpbXBsZSB0ZXN0IQ=="' \
     https://your.endpoint.com:1234/

rOCCI client

rOCCI client includes support for both extensions via the -T/--context command line argument. The client does not need to previously b64encode the data. The curl command above could be translated into the following

occi -e https://your.endpoint.com:1234 --action create --resource compute \
     --attribute occi.core.title='ContextVM' --mixin os_tpl#myVM 
     --context user_data='This is just a test!' --context public_key='ssh-rsa BAA...zxe ==user@host'

the user data and public keys can be specified as files:

--context user_data="file:///path/to/file" --context public_key="file:///path/to/publickey"

OCCI-OS (OpenStack)

OCCI-OS supports the extensions for pushing user data and keys for the stable/folsom and stable/havana versions.

rOCCI-server (OpenNebula and others)

rOCCI-server also supports the extensions since version 4.0.0 of rOCCI framework.

Synnefo

Synnefo also supports the extension in the latests versions.

Using the data at the instances

The proposed mixin does not impose any format on the data, thus the VM image creators are free to use this blob as it suites their needs. For most use cases, cloud-init is the best way to avoid dealing with the details that each middleware has for providing the data in the VM instance.

cloud-init

This task force recommends the use of cloud-init for the contextualization of VMs. Cloud-init frees the user from managing the specific ways for handling the contextualization information and it's widely available in most OS versions and IaaS cloud platforms. The latest versions support OpenNebula contextualization mechanisms. OpenStack and Synnefo contextualization are supported in most cloud-init versions (datasources are EC2 and NoCloud).

When creating a VM that is meant to run in different cloud middlewares leave the <cloud>datasource_list</cloud> undefined (so it will try all the available ones), or set at least the following:

[ NoCloud, OpenNebula, Ec2, None ]


By default cloud-init will:

- put the ssh-key into the ~/.ssh/authorized_keys of root user (or equivalent)

- if the user provided data is a script, it will be executed upon instantiation. More complex use-cases are supported, check cloud-init documentation for examples.


cloud-init and OpenNebula

The rOCCI server pushes into the OpenNebula context the base64 encoded user data provided by the client, however the default cloud-init is expecting clear text. This is fixed for the upcoming 0.7.5 version of cloud-init, in the meantime while it's released you can use the fedcloud packages (see below):

Packages

Custom FedCloud packages for some distributions are available at EGI's AppDB FedCloud cloud-init. These packages already manage the user data correctly for OpenNebula.

Contextualizing without cloudinit

OpenStack

OpenStack provides the user provided data at the meta-data server available at http://169.254.169.254 for every instance. With the EC2 API, user data is available at http://169.254.169.254/latest/user-data, it can be fetched with curl:

curl http://169.254.169.254/latest/user-data

OpenStack API provides the same data at http://169.254.169.254/openstack/latest/user_data.

Public keys are available at http://169.254.169.254/latest/meta-data/public-keys for EC2 API or http://169.254.169.254/openstack/latest/meta_data.json.

For getting OpenSSH key with EC2:

curl http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key
ssh-rsa  [...]==  xxx@host
<pre>

OpenStack API returns a json dictionary with a "public_keys" dictionary inside, where the available keys are stored:
<pre>
curl http://169.254.169.254/openstack/latest/meta_data.json
{... "public_keys": {"enolkey": "ssh-rsa [...]== xxx@host\n"}...}

See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html and http://docs.openstack.org/user-guide/content/user-data.html for more info.

OpenNebula

OpenNebula provides contextualization information through an ISO image that is available in the VM instance. The contents of the ISO image include a context.sh file where a set of variables are defined according to the VM template.

The iso filesystem is normally available at /dev/sr0 where a context.sh file contains all the context variables declared in the OpenNebula template.

The USER_DATA variable contains a base64 enconded blob (can be decoded with base64 -d) with the data provided by the user with the OCCI client. The SSH_PUBLIC_KEYS variable contain the user public keys.

For using this context just mount the filesystem and source the context file:

$ mount -o ro /dev/sr0 /mnt/
$ cat /mnt/context.sh  
# Context variables generated by OpenNebula
DISK_ID='1'
ETH0_BROADCAST='193.144.35.127'
ETH0_DNS='193.144.34.209'
ETH0_GATEWAY='193.144.35.1'
ETH0_IP='193.144.35.111'
ETH0_MASK='255.255.255.128'
ETH0_NETWORK='193.144.35.128'
SSH_PUBLIC_KEY='ssh-rsa [...]== xxx@cloud'
TARGET='hdd'
TYPE='CDROM'
USER_DATA='VGhpcyBpcyBteSB1c2VyIGRhdGEK'
$ . /mnt/context.sh
$ echo $SSH_PUBLIC_KEY
ssh-rsa [...]== xxx@host
$ echo $USER_DATA | base64 -d
This is my user data

See http://opennebula.org/documentation:archives:rel3.8:cong for more information.

Synnefo

Synnefo provides context data in a cloud-init NoCloud compatible format.

Two files are injected into the VMs: /var/lib/cloud/seed/nocloud-net/meta-data, which is a yaml file that contains the user provided public keys:

public-keys: ssh-rsa [...]== xxx@host

And /var/lib/cloud/seed/nocloud-net/user-data, which contains a blob with the data provided by the user with the OCCI client.

Setting up the middleware for contextualization

OpenStack

Default OpenStack installations do not require any additional steps for providing contextualization.

OpenNebula

Cloud-init documentation includes information on the OpenNebula configuration. Check also OpenNebula contextualization overview for more information.

Windows Contextualization

Windows VM also support contextualization of the machine on machine boot using cloudbase-init software. This software similarly to the cloud-init available for linux machines uses the available meta-data and user-data and executes a set of plugins to configure the virtual machine.

Cloudbase-init supports OpenStack meta-data and there is work in progress to support other cloud management frameworks, including OpenNebula.

Installation

Cloudbase-init is distributed with a MSI installer build after every commit in the repo:

During the installation, you can configure:

  • Name of a user that will be created and upon boot up (if it is an already existing user, cloudbase-init will only change its password)
  • Network adapter / serial ports to use (defaults should be safe)

Normally is not needed to execute sysprep on the image after the installation of cloudbase-init

Once installed the machine can be uploaded/snapshoted to be started by the users.

Contextualization

The features of cloudbase-init are not as extensive as the Linux cloud-init, but it can still perform the actions described below. For an up-to-date list and documentation, check the Plugin documentation of cloudbase-init, there is also documentation available at cloudbaseinit wiki

User configuration

cloudbase-init by default will configure the user account configured during contextualization. Take into account that the configured group must exist before rebooting the machine.

Specifying the user password via OCCI

Cloudbase-init can set the user password if specified in the context of the VM. OpenStack does automatically generate a random password on machine creation but this is not available to the OCCI interface. We have created a new mixin that allows to specify the password you want the user to have as follows:

Category: admin_pwd; scheme="http://schemas.openstack.org/compute/instance#"; class="mixin"

And the password is specified with the attribute:

org.openstack.credentials.admin_pwd

For using this mixin on VM creation, you can use a command line similar to this:

occi -e $OCCI_ENDPOINT -x $X509_USER_PROXY -X -n x509 \
     -a  create -r compute \
     -M $OS_TPL -M $RES_TPL \
     -t occi.core.title="$VM_TITLE" \
     -T user_data="$USER_DATA" \
     -M http://schemas.openstack.org/compute/instance#admin_pwd \
     -t org.openstack.credentials.admin_pwd="<secret>"

User-data

cloudbase-init will try to execute as a script whatever is passed in the user-data. It uses the first line of the file to determine which kind of script is passed for execution:

  • For Windows batch, the first line of the file must be
rem cmd
  • For Powershell, the first line must be:
#ps1_sysnative

or for a x86 PowerShell

#ps1_x86
  • For Bash (it must be preinstalled on the VM!), the first line must begin with
#!

for example, it can be

#!/bin/bash

The script will be executed with admin privileges. Take into account that the file must have DOS format, so if you are editing them on a UNIX-based machine you should make sure that the endline terminators are in the correct format.

Forcing re-execution of contextualization

cloudbase-init will execute the plugins only on the first time the image is booted. If you need to re-execute them you should remove in the Windows registry the following key:

HKEY_LOCAL_MACHINE\Software\Cloudbase Solutions\Cloudbase-Init

on 64 bit versions of Windows, the key is:

HKEY_LOCAL_MACHINE\Software\Wow6432Node\Cloudbase Solutions\Cloudbase-Init

The next time cloudbase-init is executed (due to a reboot or by restarting the service), it will execute the plugins again.