Fedcloud-tf:WorkGroups:Contextualisation

Main

Roadmap and Innovation

Technology

For Users

For Resource Providers

Media

Workbenches:

Open issues

Scenario 1 VM Management

Scenario 2 Data Management

Scenario 3 Information Systems

Scenario 4 Accounting

Scenario 5 Monitoring

Scenario 6 Notification

Scenario 7 Federated AAI

Scenario 8 VM Image Management

Scenario 9 Brokering

Scenario 10 Contextualisation

Scenario 11 Security

Leader: Enol Fernandez

Scope

This workgroup deals with the VM contextualization open issues, namely being able to pass user-defined data to the VM.

Roadmap

• Evaluate mechanisms available in the cloud middleware to provide user-defined data
• Evaluate possible OCCI extensions to pass user-defined data
• Evaluate CIMI as alternative API (already includes user-data features)
• Track implementation of OCCI extensions/CIMI on the cloud middleware

Contextualization

Contextualization is the process of installing, configuring and preparing software upon boot time on a pre-defined virtual machine image (e.g. setting the hostname, IP addresses, SSH aithorized keys, starting services, installing applications, etc.). We have identified as a requirement to contextualize images the possibility of passing user provided data to the VM when they are instantiated. Hence there are two things to be defined:

• how to pass data upon VM creation (the exact type and format of the data should not be relevant, it should be up to the user)
• how to retrieve those data from the running VM

For passing the data we have proposed the use of a new OCCI mixin that has an attribute to hold the data to pass to the image. The second part, related to retrieving the data, is more dependent of the back-end implementation. There are different methods in the systems in place in FedCloud, but tools like cloud-init handle those possible differences in a transparent way for the users.

OCCI support

FedCloud has agreed on two mixins for pushing user defined data into the VM instances:

• Category: user_data; scheme="http://schemas.openstack.org/compute/instance#"; class="mixin", and
• Category: public_key; scheme="http://schemas.openstack.org/instance/credentials#"; class="mixin"

The user_data mixin has an attribute org.openstack.compute.user_data where the use can specify the base64 encoded data that wants to be available at the VM upon instantiation.

In the case of the public_key mixin, there are two attributes to be specified:

• org.openstack.credentials.publickey.name with a name for the public key, and
• org.openstack.credentials.publickey.data with the public key itself.

These are specified as headers in the HTTP request for creating a VM. A sample curl request could look like this (credential managing is missing):

curl -H 'Category: compute;scheme="http://schemas.ogf.org/occi/infrastructure#";class="kind"' \
     -H 'Category: myVM;scheme="http://my.occi.service/occi/infrastructure/os_tpl#";class="mixin"' \
     -H 'Category: public_key;scheme="http://schemas.openstack.org/instance/credentials#";class="mixin"' \
     -H 'Category: user_data;scheme="http://schemas.openstack.org/compute/instance#";class="mixin"' \
     -H 'X-OCCI-Attribute: org.openstack.credentials.publickey.name="my_key"' \ 
     -H 'X-OCCI-Attribute: org.openstack.credentials.publickey.data="ssh-rsa BAA...zxe ==user@host" \
     -H 'X-OCCI-Attribute: org.openstack.compute.user_data="VGhpcyBpcyBhIHNpbXBsZSB0ZXN0IQ=="' \
     https://your.endpoint.com:1234/

rOCCI client

rOCCI client includes support for both extensions via the -T/--context command line argument. The client does not need to previously b64encode the data. The curl command above could be translated into the following

occi -e https://your.endpoint.com:1234 --action create --resource compute \
     --attributes title='ContextVM' --mixin os_tpl#myVM 
     --context user_data='This is just a test!',public_key='ssh-rsa BAA...zxe ==user@host'

the user data and public keys can be specified as files:

--context user_data="file:///path/to/file",public_key="file:///path/to/publickey"

OCCI-OS (OpenStack)

OCCI-OS supports the extensions for pushing user data and keys for the stable/folsom and stable/havana versions.

rOCCI-server (OpenNebula and others)

rOCCI-server also supports the extensions since version 4.0.0 of rOCCI framework.

Using the data at the instances

The proposed mixin does not impose any format on the data, thus the VM image creators are free to use this blob as it suites their needs. For most use cases, cloud-init is the best way to avoid dealing with the details that each middleware has for providing the data in the VM instance.

cloud-init

This task force recommends the use of cloud-init for the contextualization of VMs. Cloud-init frees the user from managing the specific ways for handling the contextualization information and it's widely available in most OS versions and IaaS cloud platforms. The latest versions support OpenNebula contextualization mechanisms (EC2/OpenStack was supported since long ago)

By default cloud-init will: - put the ssh-key into the ~/.ssh/authorized_keys of root user (or equivalent) - if the user provided data is a script, it will be executed upon instantiation. More complex use-cases are supported, check cloud-init documentation for examples.

cloud-init and OpenNebula

The rOCCI server pushes into the OpenNebula context the base64 encoded user data provided by the client, however the default cloud-init is expecting clear text. We are working on the solution to this problem, in the meantime you can use the fedcloud packages (see below):

Packages

Custom FedCloud packages for some distributions are available at EGI's AppDB FedCloud cloud-init. These packages already manage the user data correctly for OpenNebula.

Contextualizing without cloudinit

OpenStack

OpenStack provides the user provided data at the meta-data server available at http://169.254.169.254 for every instance. With the EC2 API, user data is available at http://169.254.169.254/latest/user-data, it can be fetched with curl:

curl http://169.254.169.254/latest/user-data

OpenStack API provides the same data at http://169.254.169.254/openstack/latest/user_data.

Public keys are available at http://169.254.169.254/latest/meta-data/public-keys for EC2 API or http://169.254.169.254/openstack/latest/meta_data.json.

For getting OpenSSH key with EC2:

curl http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key
ssh-rsa  [...]==  xxx@cloud
<pre>

OpenStack API returns a json dictionary with a "public_keys" dictionary inside, where the available keys are stored:
<pre>
curl http://169.254.169.254/openstack/latest/meta_data.json
{... "public_keys": {"enolkey": "ssh-rsa [...]== xxx@cloud\n"}...}

See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html and http://docs.openstack.org/user-guide/content/user-data.html for more info.

OpenNebula

OpenNebula provides contextualization information through an ISO image that is available in the VM instance. The contents of the ISO image include a context.sh file where a set of variables are defined according to the VM template.

The iso filesystem is normally available at /dev/sr0 where a context.sh file contains all the context variables declared in the OpenNebula template.

The USER_DATA variable contains a base64 enconded blob (can be decoded with base64 -d) with the data provided by the user with the OCCI client. The SSH_PUBLIC_KEYS variable contain the user public keys.

For using this context just mount the filesystem and source the context file:

$ mount -o ro /dev/sr0 /mnt/
$ cat /mnt/context.sh  
# Context variables generated by OpenNebula
DISK_ID='1'
ETH0_BROADCAST='193.144.35.127'
ETH0_DNS='193.144.34.209'
ETH0_GATEWAY='193.144.35.1'
ETH0_IP='193.144.35.111'
ETH0_MASK='255.255.255.128'
ETH0_NETWORK='193.144.35.128'
SSH_PUBLIC_KEY='ssh-rsa [...]== xxx@cloud'
TARGET='hdd'
TYPE='CDROM'
USER_DATA='VGhpcyBpcyBteSB1c2VyIGRhdGEK'
$ . /mnt/context.sh
$ echo $SSH_PUBLIC_KEY
ssh-rsa [...]== xxx@cloud
$ echo $USER_DATA | base64 -d
This is my user data

See http://opennebula.org/documentation:archives:rel3.8:cong for more information.

Setting up the middleware for contextualization

OpenStack

Default OpenStack installations do not require any additional steps for providing contextualization.

OpenNebula

Cloud-init documentation includes information on the OpenNebula configuration. Check also OpenNebula contextualization overview for more information.

Other info

CIMI

CIMI standard does include contextualization features. We have deployed an instance of OpenStack for testing the CIMI implementation from this blueprint. A simple modification adds the support for the `userData` parameter in machine creation.

Sample code for the request (python code with requests package):

headers = {'content-type': 'application/json',
                  'accept': 'application/json',
                  'X-Auth-Token': keystone_token}


 img_body = {
      'name': 'cimi-test-server',
      'description': 'no desc',
      'machineTemplate': {
          'machineImage': {'href': image_id},
          'machineConfig': {'href': conf_id},
          'userData': base64.b64encode('this is just a test'),
     }
 }
 
r = requests.post(cimi_url + '/machineCollection',
                            headers=headers,
                            data=json.dumps(img_body))