Difference between revisions of "Fedcloud-tf:WorkGroups:Scenario8"
m (→OpenStack) |
|||
Line 282: | Line 282: | ||
==== OpenStack ==== | ==== OpenStack ==== | ||
vmcatcher may be branched to Openstack Glance catalog using [https://github.com/EGI-FCTF/glancepush glancepush] tool. Basically: | vmcatcher may be branched to Openstack Glance catalog using [https://github.com/EGI-FCTF/glancepush glancepush] tool. Basically: | ||
# glancepush-vmcatcher uses vmcatcher's event handler to signal glancepush that a new image was updated in vmcatcher's cache | |||
# glancepush uploads the new image to glance in a quarantined zone (only the configured tenant will have access to the image) | |||
# glancepush instanciates a VM with the new image | |||
# glancepush pass policy compliance tests inside the VM (no hardcoded secrets, packages up to date...) | |||
# if the tests pass, the image will be released publicly in the catalog or stalled otherwise. | |||
Please | Please read the full documentation [https://github.com/EGI-FCTF/glancepush/wiki here]. | ||
The binaries for glancepush and glancepush-vmcatcher are located [ftp://ftp.in2p3.fr/ccin2p3/egi-acct-osdriver/ IN2P3-CC ftp repository] | The binaries for glancepush and glancepush-vmcatcher are located [ftp://ftp.in2p3.fr/ccin2p3/egi-acct-osdriver/ IN2P3-CC ftp repository] | ||
Revision as of 11:11, 5 April 2013
Main | Roadmap and Innovation | Technology | For Users | For Resource Providers | Media |
Leader: Kostas Koumantaros, EGI-InSPIRE SA2
Collaborators
Role | Institution | Name |
---|---|---|
Scenario leader | EGI-InSPIRE SA2 | Kostas Koumantaros |
Collaborator | GRIF | Michel Jouvin |
Collaborator | TCD | Stuart Kenny |
Roadmap
- Investigate how to do double endorsement
- Investigate x509 + VOMS authentication
Scope
This workbench deals with the issues around setting up a VM Marketplace to:
- Provide a publicly searchable place for VMs that may provide the application that is needed
- Provide a common place to add a token of endorsement to a pertinent VM
Marketplace Howto
Register an image with the EGI.eu Marketplace
(Modified version of instructions compiled by Boris Parak. The original version can be found here)
Install and configure stratuslab-cli-tools
This part is very straight-forward, we need stratuslab-cli-tools. So
cd ~ mkdir stratuslab cd stratuslab wget http://repo.stratuslab.eu:8081/content/repositories/centos-6.2-releases/eu/stratuslab/pkgs/stratuslab-cli-user-pkg/2.2/stratuslab-cli-user-pkg-2.2.tar.gz tar xvf stratuslab-cli-user-pkg-2.2.tar.gz
and then conclude the installation process by appending the following to ~/.bashrc
# STRATUSLAB-CLI-TOOLS export PATH=$PATH:~/stratuslab/bin export PYTHONPATH=$PYTHONPATH:~/stratuslab/lib/stratuslab/python
RPMs for the client are also available from the StratusLab yum repositories, see http://yum.stratuslab.eu/. Packages are provided for CentOS 6.2, OpenSuse 12.1 and Fedora 16.
Get demo images
There are two images required for the demo. Each resource provider should upload a metadata entry for each. The first is the BNCweb image, which is available from https://appliance-repo.egi.eu/images/base/egi-bncweb/1.0/egi-bncweb.img. The second is a plain Debian 6 image (https://appliance-repo.egi.eu/images/base/Debian-6.0.5-x86_64-base/1.0/debian-6.0.5-x86_64-base.img).
Upload the image into your cloud
appliance Repo
Here are the steps for uploading an image to the appliance repo, which you can register to the EGI Marketplace as described below (ref?): The server uses the fedloud.egi.eu voms for authentication. You can register here (https://perun.metacentrum.cz/perun-registrar-cert/?vo=fedcloud.egi.eu). You will also need the hellasgrid-ca-chain.pem file so that curl can verify the server's certificate.
1. Create the directory where you want to place your image:
curl --cacert ~/path/to/hellasgrid-ca-chain.pem --cert client.pem -X MKCOL https://appliance-repo.egi.eu/images/base/SL-5.7-x86_64-base curl --cacert ~/path/to/hellasgrid-ca-chain.pem --cert client.pem -X MKCOL https://appliance-repo.egi.eu/images/base/SL-5.7-x86_64-base/1.0
2. upload the image:
curl --cacert /path/to/hellasgrid-ca-chain.pem -T /path/to/image --cert client.pem https://appliance-repo.egi.eu/images/base/SL-5.5-x86_64-base/1.0/
NOTES:
Curl assumes that your cert.pem file contains your private key and certificate concatenated, if that not the case you will get a ""curl: (58) unable to set private key file: /file" error. A workaround is to create separate files for the private key and certificate. For example you can create the files using your pkcs12 certificate using openssl:
openssl pkcs12 -in MULTICERT.p12 -out client.pem -clcerts -nokeys openssl pkcs12 -in MULTICERT.p12 -out key.pem -nocerts
and issue the curl commands by:
curl --cacert ~/path/to/hellasgrid-ca-chain.pem --key key.pem --cert client.pem
e.g.
curl --cacert ~/path/to/hellasgrid-ca-chain.pem --key key.pem --cert client.pem -X MKCOL https://appliance-repo.egi.eu/images/base/SL-5.7-x86_64-base
You can generate the hellasgrid-ca-chain.pem file by:
- wget http://repository.egi.eu/sw/production/cas/1/current/repo-files/EGI-trustanchors.repo | mv EGI-trustanchors.repo /etc/yum.repos/
- yum install yum install ca_HellasGrid-CA-2006 ca_HellasGrid-Root
- cat /etc/grid-security/certificates/HellasGrid-Root.pem /etc/grid-security/certificates/HellasGrid-CA-2006.pem > /path/to/new/hellasgrid-ca-chain.pem
Other
This step is different for every cloud platform. For instance, in OpenNebula v3.4+ you can use Sunstone GUI to upload images directly, in previous versions you have to upload the image to the frontend and then register it.
Sice FedCloud-TF will be using OCCI to access the cloud, you must provide a location of the image that is OCCI-compatible. To find the right link you can browse through all the storage elements registered in your OCCI server
https://occi.host:port/storage/
checking the occi.core.title attribute for the right name. You should end up with something like
https://occi.host:port/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511
Build the metadata
The EGI.eu Marketplace stores only metadata which points to the image, provide basic information and integrity verification. Since RDF is not the most user-friendly format, we can use stratus-build-metadata to generate a template
stratus-build-metadata --author='##YOUR_NAME##' --type=base --os=Ubuntu --os-version=11.04 --os-arch=x86_64 \ --image-version=1.0 --hypervisor=xen --format=raw --comment='BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##' \ --compression=none --location='https://occi.host:port/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511' egi-bncweb.img
Note: stratus-build-metadata needs the image to compute checksums, you can download it here egi-bncweb.img
Modify the metadata
Now we can check/modify the metadata, the most important elements are dcterms:valid and dcterms:title.
The correct format for dcterms:title is EGI-##IMAGE_NAME##-##SITE_NAME##. This field will need to be manually added to the metadata file. You can also modify the validity date as required.
Metadata from the EGI.eu Marketplace cannot be removed, it can only expire. It is also possible to deprecate an entry. This might be necessary, if for example, a security issue is detected with the image, or if you simply wish to no longer endorse the image. Instructions for the stratus-deprecate-image command can be found here.
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:dcterms="http://purl.org/dc/terms/" xmlns:slterms="http://mp.stratuslab.eu/slterms#" xmlns:slreq="http://mp.stratuslab.eu/slreq#" xml:base="http://mp.stratuslab.eu/"> <rdf:Description rdf:about="#DtRwHZzoo1xFKtk-iL51t6RNQ9Q"> <dcterms:identifier>DtRwHZzoo1xFKtk-iL51t6RNQ9Q</dcterms:identifier> <slreq:bytes>14680064000</slreq:bytes> <slreq:checksum rdf:parseType="Resource"> <slreq:algorithm>MD5</slreq:algorithm> <slreq:value>144fff2477673aa1d883f0a3ba89f273</slreq:value> </slreq:checksum> <slreq:checksum rdf:parseType="Resource"> <slreq:algorithm>SHA-1</slreq:algorithm> <slreq:value>3b51c07673a28d7114ab64fa22f9d6de91350f50</slreq:value> </slreq:checksum> <slreq:checksum rdf:parseType="Resource"> <slreq:algorithm>SHA-256</slreq:algorithm> <slreq:value>8bde348c81e5a2aa5aa51b8d39a30ad137d0482decd5960cd95594d224a45bdd</slreq:value> </slreq:checksum> <slreq:checksum rdf:parseType="Resource"> <slreq:algorithm>SHA-512</slreq:algorithm> <slreq:value>e780f2aa6922bc7cfdaae4a5e410f6b499bef5c83314bcd760b082b625860834c4942de9d096c7aa83cdad0411c47686f2e7d0fcc65f816475f6525db28b236d</slreq:value> </slreq:checksum> <slreq:endorsement rdf:parseType="Resource"/> <dcterms:title>EGI-BNCweb-##YOUR_SITE##</dcterms:title> <dcterms:type>base</dcterms:type> <slterms:kind>machine</slterms:kind> <slterms:os>Ubuntu</slterms:os> <slterms:os-version>11.04</slterms:os-version> <slterms:os-arch>x86_64</slterms:os-arch> <slterms:version>1.0</slterms:version> <dcterms:compression>none</dcterms:compression> <slterms:location>https://occi.host:port/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511</slterms:location> <dcterms:format>raw</dcterms:format> <dcterms:creator>##YOUR_NAME##</dcterms:creator> <dcterms:created>2012-06-12T12:36:25Z</dcterms:created> <dcterms:valid>2012-06-14T12:36:25Z</dcterms:valid> <dcterms:description>BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##</dcterms:description> <slterms:hypervisor>xen</slterms:hypervisor> <dcterms:publisher>##YOUR_SITE##</dcterms:publisher> </rdf:Description> </rdf:RDF>
Notice:
These fields should be checked: <dcterms:title>EGI-BNCweb-##YOUR_SITE##</dcterms:title> <dcterms:creator>##YOUR_NAME##</dcterms:creator> <dcterms:description>BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##</dcterms:description> and <dcterms:publisher>##YOUR_SITE##</dcterms:publisher>
Modify Metadata (OCCI 1.1 servers)
Warning:
These changes are required for TF2012 demo.
<slterms:location>https://occi.host:port/storage/##STORAGE ID##</slterms:location> <dcterms:requires>https://occi.host:port/network/##NETWORK ID##</dcterms:requires>
- Optional:
Set <dcterms:valid> field to be used until TF demo:
<dcterms:valid>2012-10-02T09:55:00Z</dcterms:valid>
Modify Metadata (rOCCI or OCCI OpenStack servers)
Warning:
These changes are required for TF2012 demo
<dcterms:requires>https://rocci.host:port</dcterms:requires>
- Optional:
Set <dcterms:valid> field to be used until TF demo:
<dcterms:valid>2012-10-02T09:55:00Z</dcterms:valid>
Sign the metadata
To establish the origin of the image, we have to sign the metadata with a personal certificate (ideally the one registered with EGI.eu). Before doing this you should familiarise yourself with the EGI Security Policy for the Endorsement and Operation of Virtual Machine Images.
stratus-sign-metadata --p12-cert=##FULL_PATH_TO_usercred.p12## egi-bncweb.xml
Register the metadata with the EGI.eu Marketplace
And to complete the process, we have to upload the metadata to the EGI.eu Marketplace with stratus-upload-metadata
stratus-upload-metadata --marketplace-endpoint=marketplace.egi.eu egi-bncweb.xml
or manually at
http://marketplace.egi.eu/upload
Howto update and change old metadata
To update uploaded metadata just modify the metadata file, sign it again, and then upload. It is basically the same procedure as uploading new metadata. Only the most recent entry for a particular image identifier/email address is displayed.
VMcatcher
VMcatcher allows users to subscribe to virtual machine Virtual Machine image lists, cache the images referenced to in the Virtual Machine Image List, validate the images list with x509 based public key cryptography, and validate the images against sha512 hashes in the images lists and provide events for further applications to process updates or expiries of virtual machine images without having to further validate the images.
Installation
Usage
Event Handlers
OpenNebula
vmcatcher_eventHndlExpl_ON is a VMcatcher event handler for OpenNebula to store or disable images based on VMcatcher response. The easy way to setup vmcatcher_eventHndlExpl_ON is creating a cron (use /etc/cron.d/vmcatcher as example):
export VMCATCHER_RDBMS="sqlite:////var/lib/one/vmcatcher.db" export VMCATCHER_CACHE_DIR_CACHE="/var/lib/one/cache" export VMCATCHER_CACHE_DIR_DOWNLOAD="/var/lib/one/cache/partial" export VMCATCHER_CACHE_DIR_EXPIRE="/var/lib/one/cache/expired" export VMCATCHER_CACHE_EVENT="python <vmcatcher_eventHndlExpl_ON path>/vmcatcher_eventHndlExpl_ON" 50 */6 * * * oneadmin (/usr/bin/vmcatcher_subscribe -U; /usr/bin/vmcatcher_cache ) >> /var/log/vmcatcher.log 2>&1
NOTES:
- vmcatcher_cache must be executed as oneadmin user.
- Environment variables can be used to set default values but the command line options will override any set environment options. Set these env variables for oneadmin user: VMCATCHER_RDBMS, VMCATCHER_CACHE_DIR_CACHE, VMCATCHER_CACHE_DIR_DOWNLOAD, VMCATCHER_CACHE_DIR_EXPIRE and VMCATCHER_CACHE_EVENT.
- vmcatcher_eventHndlExpl_ON generates ON image templates. These templates are available from $VMCATCHER_CACHE_DIR_CACHE/templates (templates nomenclature $VMCATCHER_EVENT_DC_IDENTIFIER.one)
- The new ON images include VMCATCHER_EVENT_DC_IDENTIFIER = <VMCATCHER_UUID> tag. This tag is used to identify Fedcloud VM images.
- VMcatcher expired images are set as disabled by ON. It is up to the RP to remove disabled images or assign the new ones to a specific ON group or user.
OpenStack
vmcatcher may be branched to Openstack Glance catalog using glancepush tool. Basically:
- glancepush-vmcatcher uses vmcatcher's event handler to signal glancepush that a new image was updated in vmcatcher's cache
- glancepush uploads the new image to glance in a quarantined zone (only the configured tenant will have access to the image)
- glancepush instanciates a VM with the new image
- glancepush pass policy compliance tests inside the VM (no hardcoded secrets, packages up to date...)
- if the tests pass, the image will be released publicly in the catalog or stalled otherwise.
Please read the full documentation here. The binaries for glancepush and glancepush-vmcatcher are located IN2P3-CC ftp repository
Links
HEPIX Virtualisation Working Group