Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Fedcloud-tf:WorkGroups:Scenario8:Configuration"

From EGIWiki
Jump to navigation Jump to search
Line 315: Line 315:
==== OpenStack  ====
==== OpenStack  ====


vmcatcher may be branched to Openstack Glance catalog using [https://github.com/EGI-FCTF/glancepush glancepush] tool and [https://github.com/EGI-FCTF/glancepush-vmcatcher2 glancepush-vmcatcher] event handler. To install and configure glancepush and the handler, you can refer to the following instructions:
Vmcatcher may be branched to Openstack Glance catalog using [https://appdb.egi.eu/store/software/python.glancepush python-glancepush] tool and [https://appdb.egi.eu/store/software/openstack.handler.for.vmcatcher Openstack Handler for Vmcatcher] event handler. To install and configure glancepush and the handler, you can refer to the following instructions:


* Install the latest release of glancepush via the [https://appdb.egi.eu/store/software/glancepush/releases/ AppDB repository]. As example, for Debian, you can run
* Install the latest release of glancepush via the [https://appdb.egi.eu/store/software/python.glancepush/releases/0.0.x AppDB repository]. For debian based systems, just download the tarball, extract it, and execute python setup.py install
  [stack@ubuntu]$ wget http://repository.egi.eu/community/software/glancepush/0.x/releases/debian/dists/wheezy/main/binary-amd64/glancepush_0.3_all.deb
  [stack@ubuntu]$ wget http://repository.egi.eu/community/software/python.glancepush/0.0.X/releases/generic/0.0.6/python-glancepush-0.0.6.tar.gz
  [stack@ubuntu]$ sudo dpkg -i glancepush_0.3_all.deb
  [stack@ubuntu]$ tar -zxvf python-glancepush-0.0.6.tar.gz
[stack@ubuntu]$ python setup.py install
while for RHEL6 you can run:
while for RHEL6 you can run:
  [stack@rhel]$ yum localinstall http://repository.egi.eu/community/software/glancepush/0.x/releases/sl/6/x86_64/RPMS/glancepush-0.3-1.noarch.rpm
  [stack@rhel]$ yum localinstall http://repository.egi.eu/community/software/python.glancepush/0.0.X/releases/sl/6/x86_64/RPMS/python-glancepush-0.0.6-1.noarch.rpm


* Then, configure glancepush directories
* Then, configure glancepush directories
  [stack@ubuntu]$ sudo mkdir -p /var/spool/glancepush /var/log/glancepush/ /etc/glancepush/transform/ /etc/glancepush/clouds
  [stack@ubuntu]$ sudo mkdir -p /var/spool/glancepush /etc/glancepush/log /etc/glancepush/transform/ /etc/glancepush/clouds
  [stack@ubuntu]$ sudo chown stack:stack -R /var/spool/glancepush /etc/glancepush /var/log/glancepush/
  [stack@ubuntu]$ sudo chown stack:stack -R /var/spool/glancepush /etc/glancepush /var/log/glancepush/


* Configure openstack cloud credentials
* Copy the file /etc/keystone/voms.json to /etc/glancepush/voms.json. Then create a file in clouds file for every VO to which you are subscribed. For example, if you're subscribed to fedcloud, atlas and lhcb, you'll need 3 files in the /etc/glancepush/clouds directory with the credentials for this VO/tenants, for example:
[stack@ubuntu]$ cat << EOF > /etc/glancepush/clouds/openstack
testing_tenant='admin'
[general]
test_policy=false
  # Tenant for this VO. Must match the tenant defined in voms.json file
EOF
  testing_tenant=egi
[stack@ubuntu]$ cat << EOF > /etc/glancepush/clouds/openstack.creds
  # Identity service endpoint (Keystone)
  export OS_USERNAME='admin'
  endpoint_url=https://server4-eupt.unizar.es:5000/v2.0
  export OS_PASSWORD='devstack'
# User Password
  export OS_TENANT_NAME='admin'
password=123456
  export OS_AUTH_URL='http://192.168.56.103:5000/v2.0'
# User
  EOF
username=John
  [stack@ubuntu]$ ssh-keygen -b 2048 -t rsa -f /etc/glancepush/clouds/openstack.key #Do not specify any passphrase
# Set this to true if you're NOT using self-signed certificates
  is_secure=True
  # SSH private key that will be used to perform policy checks (to be done)
ssh_key=Carlos_lxbifi81
# WARNING: Only define the next variable if you're going to need it. Otherwise you may encounter problems
cacert=path_to_your_cert


* Edit the /etc/glancepush/glancepushrc and add <code>target_clouds=openstack</code>:
* Install [https://appdb.egi.eu/store/software/openstack.handler.for.vmcatcher Openstack handler for vmcatcher]. For debian based systems, just download the tarball, extract it and execute python setup.py install
  [stack@ubuntu]$ echo 'target_clouds=openstack' >> /etc/glancepush/glancepushrc
  [stack@ubuntu]$ wget http://repository.egi.eu/community/software/openstack.handler.for.vmcatcher/0.0.X/releases/generic/0.0.7/gpvcmupdate-0.0.7.tar.gz
 
  [stack@ubuntu]$ tar -zxvf gpvcmupdate-0.0.7.tar.gz
* Install glancepush handler for vmcatcher
  [stack@ubuntu]$ python setup.py install
  [stack@ubuntu]$ mkdir gpvcmupdate
while for RHEL6 you can run:
  [stack@ubuntu]$ cd gpvcmupdate/
  [stack@rhel]$ yum localinstall http://repository.egi.eu/community/software/openstack.handler.for.vmcatcher/0.0.X/releases/generic/0.0.7/gpvcmupdate-0.0.7.tar.gz
  [stack@ubuntu]$ wget https://github.com/cgimeno/Openstack-handler-for-vmcatcher/raw/master/gpvcmupdate.py -O gpvcmupdate.py


* Create the vmcatcher folders for OpenStack
* Create the vmcatcher folders for OpenStack
Line 382: Line 387:
   
   
  #Run glancepush
  #Run glancepush
  /usr/bin/gpupdate
  /usr/bin/glancepush.py


* Set teh newly created fiel as executable
* Set the newly created file as executable
  [stack@ubuntu]$ chmod +x $HOME/gpvcmupdate/vmcatcher_eventHndl_OS_cron.sh
  [stack@ubuntu]$ chmod +x $HOME/gpvcmupdate/vmcatcher_eventHndl_OS_cron.sh


Line 405: Line 410:
  DEBUG:Events:stdout=
  DEBUG:Events:stdout=
  DEBUG:Events:stderr=Ignoring ProcessPostfix event.
  DEBUG:Events:stderr=Ignoring ProcessPostfix event.
+-----------------------------------------------+----------------------------------------------------------------------------------+
 
| Property                                      | Value                                                                            |
+-----------------------------------------------+----------------------------------------------------------------------------------+
| Property 'version'                            | 1.0                                                                              |
| Property 'vmcatcher_event_ad_mpuri'          | https://appdb.egi.eu/store/vm/image/541b01a8-94bd-4545-83a8-6ea07209b440:1278/  |
| Property 'vmcatcher_event_dc_description'    | None                                                                            |
| Property 'vmcatcher_event_dc_identifier'      | 541b01a8-94bd-4545-83a8-6ea07209b440                                            |
| Property 'vmcatcher_event_dc_title'          | Image for TinyCoreLinux [Other/TinyCoreLinux/QEMU-KVM]                          |
| Property 'vmcatcher_event_filename'          | 541b01a8-94bd-4545-83a8-6ea07209b440                                            |
| Property 'vmcatcher_event_hv_format'          | OVA                                                                              |
| Property 'vmcatcher_event_hv_hypervisor'      | QEMU-KVM                                                                        |
| Property 'vmcatcher_event_hv_size'            | 14641152                                                                        |
| Property 'vmcatcher_event_hv_uri'            | http://appliance-repo.egi.eu/images/base/tinylinux-core/0.1/CoreLinux.ova        |
| Property 'vmcatcher_event_hv_version'        | 0.1                                                                              |
| Property 'vmcatcher_event_il_dc_identifier'  | 8ddbd4f6-fb95-4917-b105-c89b5df99dda                                            |
| Property 'vmcatcher_event_sl_arch'            | x86_64                                                                          |
| Property 'vmcatcher_event_sl_checksum_sha512' | 5d7ea14a4a57c5fb945c21e44d3067678a6b899f69df4691cde5bef8ab57e2bcb6358f610683c309 |
|                                              | 675efe5e431285aa3b71b2eea9ccb117bcc0eeba71965cbc                                |
| Property 'vmcatcher_event_sl_comments'        | None                                                                            |
| Property 'vmcatcher_event_sl_os'              | Linux                                                                            |
| Property 'vmcatcher_event_sl_osversion'      | TinyCoreLinux                                                                    |
| Property 'vmcatcher_event_type'              | AvailablePostfix                                                                |
| checksum                                      | ad510abc09c2e1528a28f095d4c64aca                                                |
| container_format                              | bare                                                                            |
| created_at                                    | 2014-07-21T11:40:11                                                              |
| deleted                                      | False                                                                            |
| deleted_at                                    | None                                                                            |
| disk_format                                  | qcow2                                                                            |
| id                                            | 79abc374-6374-4874-bd67-065c06f6de48                                            |
| is_public                                    | True                                                                            |
| min_disk                                      | 10                                                                              |
| min_ram                                      | 0                                                                                |
| name                                          | Image_for_TinyCoreLinux_[Other_TinyCoreLinux_QEMU-KVM]                          |
| owner                                        | feb390ff104247bdb42a41827a807874                                                |
| protected                                    | False                                                                            |
| size                                          | 15007744                                                                        |
| status                                        | active                                                                          |
| updated_at                                    | 2014-07-21T11:40:15                                                              |
+-----------------------------------------------+----------------------------------------------------------------------------------+


* Add the following line to the stack user crontab:
* Add the following line to the stack user crontab:
Line 455: Line 422:
=== Links  ===
=== Links  ===


[https://github.com/hepix-virtualisation/vmcatcher VMcatcher]  
[https://github.com/hepix-virtualisation/vmcatcher VMcatcher]
 
[https://github.com/cgimeno/Openstack-handler-for-vmcatcher Openstach Handler for vmcatcher repository]
 
[https://github.com/cgimeno/python-glancepush Python-glancepush repository]


[http://grid.desy.de/vm/hepix/vwg/doc/html/index.shtml HEPIX Virtualisation Working Group]  
[http://grid.desy.de/vm/hepix/vwg/doc/html/index.shtml HEPIX Virtualisation Working Group]  

Revision as of 10:54, 28 November 2014

Register an image with the EGI.eu Marketplace

(Modified version of instructions compiled by Boris Parak. The original version can be found here)

Install and configure stratuslab-cli-tools

This part is very straight-forward, we need stratuslab-cli-tools. So

cd ~
mkdir stratuslab
cd stratuslab
wget http://repo.stratuslab.eu:8081/content/repositories/centos-6.2-releases/eu/stratuslab/pkgs/stratuslab-cli-user-pkg/2.2/stratuslab-cli-user-pkg-2.2.tar.gz
tar xvf stratuslab-cli-user-pkg-2.2.tar.gz

and then conclude the installation process by appending the following to ~/.bashrc

# STRATUSLAB-CLI-TOOLS
export PATH=$PATH:~/stratuslab/bin
export PYTHONPATH=$PYTHONPATH:~/stratuslab/lib/stratuslab/python

RPMs for the client are also available from the StratusLab yum repositories, see http://yum.stratuslab.eu/. Packages are provided for CentOS 6.2, OpenSuse 12.1 and Fedora 16.

Get demo images

There are two images required for the demo. Each resource provider should upload a metadata entry for each. The first is the BNCweb image, which is available from https://appliance-repo.egi.eu/images/base/egi-bncweb/1.0/egi-bncweb.img. The second is a plain Debian 6 image (https://appliance-repo.egi.eu/images/base/Debian-6.0.5-x86_64-base/1.0/debian-6.0.5-x86_64-base.img).

Upload the image into your cloud

appliance Repo

Here are the steps for uploading an image to the appliance repo, which you can register to the EGI Marketplace as described below (ref?): The server uses the fedloud.egi.eu voms for authentication. You can register here (https://perun.metacentrum.cz/perun-registrar-cert/?vo=fedcloud.egi.eu). You will also need the hellasgrid-ca-chain.pem file so that curl can verify the server's certificate.

1. Create the directory where you want to place your image:

 curl --cacert ~/path/to/hellasgrid-ca-chain.pem --cert client.pem -X MKCOL https://appliance-repo.egi.eu/images/base/SL-5.7-x86_64-base
curl --cacert ~/path/to/hellasgrid-ca-chain.pem --cert client.pem -X MKCOL https://appliance-repo.egi.eu/images/base/SL-5.7-x86_64-base/1.0

2. upload the image:

 curl --cacert /path/to/hellasgrid-ca-chain.pem -T /path/to/image --cert client.pem https://appliance-repo.egi.eu/images/base/SL-5.5-x86_64-base/1.0/

NOTES:

Curl assumes that your cert.pem file contains your private key and certificate concatenated, if that not the case you will get a ""curl: (58) unable to set private key file: /file" error. A workaround is to create separate files for the private key and certificate. For example you can create the files using your pkcs12 certificate using openssl:

 openssl pkcs12 -in MULTICERT.p12 -out client.pem -clcerts -nokeys
openssl pkcs12 -in MULTICERT.p12 -out key.pem -nocerts

and issue the curl commands by:

 curl --cacert ~/path/to/hellasgrid-ca-chain.pem --key key.pem --cert client.pem

e.g.

 curl --cacert ~/path/to/hellasgrid-ca-chain.pem --key key.pem --cert client.pem -X MKCOL https://appliance-repo.egi.eu/images/base/SL-5.7-x86_64-base

You can generate the hellasgrid-ca-chain.pem file by:

  1. wget http://repository.egi.eu/sw/production/cas/1/current/repo-files/EGI-trustanchors.repo | mv EGI-trustanchors.repo /etc/yum.repos/
  2. yum install  yum install ca_HellasGrid-CA-2006 ca_HellasGrid-Root
  3. cat /etc/grid-security/certificates/HellasGrid-Root.pem /etc/grid-security/certificates/HellasGrid-CA-2006.pem > /path/to/new/hellasgrid-ca-chain.pem

Other

This step is different for every cloud platform. For instance, in OpenNebula v3.4+ you can use Sunstone GUI to upload images directly, in previous versions you have to upload the image to the frontend and then register it.

Sice FedCloud-TF will be using OCCI to access the cloud, you must provide a location of the image that is OCCI-compatible. To find the right link you can browse through all the storage elements registered in your OCCI server

https://occi.host:port/storage/

checking the occi.core.title attribute for the right name. You should end up with something like

https://occi.host:port/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511

Build the metadata

The EGI.eu Marketplace stores only metadata which points to the image, provide basic information and integrity verification. Since RDF is not the most user-friendly format, we can use stratus-build-metadata to generate a template

stratus-build-metadata --author='##YOUR_NAME##' --type=base --os=Ubuntu --os-version=11.04 --os-arch=x86_64 \
--image-version=1.0 --hypervisor=xen --format=raw --comment='BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##' \ 
--compression=none --location='https://occi.host:port/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511' egi-bncweb.img

Note: stratus-build-metadata needs the image to compute checksums, you can download it here egi-bncweb.img

Modify the metadata

Now we can check/modify the metadata, the most important elements are dcterms:valid and dcterms:title.

The correct format for dcterms:title is EGI-##IMAGE_NAME##-##SITE_NAME##. This field will need to be manually added to the metadata file. You can also modify the validity date as required.

Metadata from the EGI.eu Marketplace cannot be removed, it can only expire. It is also possible to deprecate an entry. This might be necessary, if for example, a security issue is detected with the image, or if you simply wish to no longer endorse the image. Instructions for the stratus-deprecate-image command can be found here.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:slterms="http://mp.stratuslab.eu/slterms#"
    xmlns:slreq="http://mp.stratuslab.eu/slreq#"
    xml:base="http://mp.stratuslab.eu/">

    <rdf:Description rdf:about="#DtRwHZzoo1xFKtk-iL51t6RNQ9Q">

        <dcterms:identifier>DtRwHZzoo1xFKtk-iL51t6RNQ9Q</dcterms:identifier>

        <slreq:bytes>14680064000</slreq:bytes>

        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>MD5</slreq:algorithm>
            <slreq:value>144fff2477673aa1d883f0a3ba89f273</slreq:value>
        </slreq:checksum>
        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>SHA-1</slreq:algorithm>
            <slreq:value>3b51c07673a28d7114ab64fa22f9d6de91350f50</slreq:value>
        </slreq:checksum>
        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>SHA-256</slreq:algorithm>
            <slreq:value>8bde348c81e5a2aa5aa51b8d39a30ad137d0482decd5960cd95594d224a45bdd</slreq:value>
        </slreq:checksum>
        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>SHA-512</slreq:algorithm>
            <slreq:value>e780f2aa6922bc7cfdaae4a5e410f6b499bef5c83314bcd760b082b625860834c4942de9d096c7aa83cdad0411c47686f2e7d0fcc65f816475f6525db28b236d</slreq:value>
        </slreq:checksum>

        <slreq:endorsement rdf:parseType="Resource"/>

        <dcterms:title>EGI-BNCweb-##YOUR_SITE##</dcterms:title>
        <dcterms:type>base</dcterms:type>
        <slterms:kind>machine</slterms:kind>

        <slterms:os>Ubuntu</slterms:os>
        <slterms:os-version>11.04</slterms:os-version>
        <slterms:os-arch>x86_64</slterms:os-arch>
        <slterms:version>1.0</slterms:version>
        <dcterms:compression>none</dcterms:compression>
        <slterms:location>https://occi.host:port/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511</slterms:location>

        <dcterms:format>raw</dcterms:format>

        <dcterms:creator>##YOUR_NAME##</dcterms:creator>

        <dcterms:created>2012-06-12T12:36:25Z</dcterms:created>
        <dcterms:valid>2012-06-14T12:36:25Z</dcterms:valid>

        <dcterms:description>BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##</dcterms:description>

        <slterms:hypervisor>xen</slterms:hypervisor>

        <dcterms:publisher>##YOUR_SITE##</dcterms:publisher>
        
    </rdf:Description>
</rdf:RDF>

Notice:
These fields should be checked: <dcterms:title>EGI-BNCweb-##YOUR_SITE##</dcterms:title> <dcterms:creator>##YOUR_NAME##</dcterms:creator> <dcterms:description>BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##</dcterms:description> and <dcterms:publisher>##YOUR_SITE##</dcterms:publisher>



Modify Metadata (OCCI 1.1 servers)

Warning:
These changes are required for TF2012 demo.


<slterms:location>https://occi.host:port/storage/##STORAGE ID##</slterms:location>
<dcterms:requires>https://occi.host:port/network/##NETWORK ID##</dcterms:requires>
  • Optional:

Set <dcterms:valid> field to be used until TF demo:

<dcterms:valid>2012-10-02T09:55:00Z</dcterms:valid>

Modify Metadata (rOCCI or OCCI OpenStack servers)

Warning:
These changes are required for TF2012 demo



<dcterms:requires>https://rocci.host:port</dcterms:requires>
  • Optional:

Set <dcterms:valid> field to be used until TF demo:

<dcterms:valid>2012-10-02T09:55:00Z</dcterms:valid>


Sign the metadata

To establish the origin of the image, we have to sign the metadata with a personal certificate (ideally the one registered with EGI.eu). Before doing this you should familiarise yourself with the EGI Security Policy for the Endorsement and Operation of Virtual Machine Images.

stratus-sign-metadata --p12-cert=##FULL_PATH_TO_usercred.p12## egi-bncweb.xml

Register the metadata with the EGI.eu Marketplace

And to complete the process, we have to upload the metadata to the EGI.eu Marketplace with stratus-upload-metadata

stratus-upload-metadata --marketplace-endpoint=marketplace.egi.eu egi-bncweb.xml

or manually at

http://marketplace.egi.eu/upload

Howto update and change old metadata

To update uploaded metadata just modify the metadata file, sign it again, and then upload. It is basically the same procedure as uploading new metadata. Only the most recent entry for a particular image identifier/email address is displayed.

VMcatcher

VMcatcher allows users to subscribe to virtual machine Virtual Machine image lists, cache the images referenced to in the Virtual Machine Image List, validate the images list with x509 based public key cryptography, and validate the images against sha512 hashes in the images lists and provide events for further applications to process updates or expiries of virtual machine images without having to further validate the images.

Installation & Usage

Please refer to vmcatcher documentation here

Event Handlers

OpenNebula

vmcatcher_eventHndlExpl_ON is a VMcatcher event handler for OpenNebula to store or disable images based on VMcatcher response. The followign guide will show how to install and configure vmCatcher handler as oneadmin user, directly from github. The configuration will automatically syncronize OpenNebula Image datastore with the registered vmcatcher images.

  • Install pre-requisites for VMCatcher handler
[oneadmin@one-sandbox] sudo yum install -y qemu-img
  • Install VMcatcher handler from github
[oneadmin@one-sandbox]$ mkdir $HOME/vmcatcher_eventHndlExpl_ON
[oneadmin@one-sandbox]$ cd $HOME/vmcatcher_eventHndlExpl_ON
[oneadmin@one-sandbox]$ wget http://github.com/grid-admin/vmcatcher_eventHndlExpl_ON/archive/v0.0.8.zip -O vmcatcher_eventHndlExpl_ON.zip
[oneadmin@one-sandbox]$ unzip vmcatcher_eventHndlExpl_ON.zip
[oneadmin@one-sandbox]$ mv vmcatcher_eventHndlExpl_ON*/* ./
[oneadmin@one-sandbox]$ rmdir vmcatcher_eventHndlExpl_ON-*
  • Create the vmcatcher folders for ON (do not use /var/lib/one/ or other OpenNebula default directories for the vmcatcher cache, since you cannot import images into OpenNebula from these directories. Also, since this directory will host a copy of all the images downloaded via vmcatcher, it is suggested to place the directory into a separate disk)
[oneadmin@one-sandbox]$ sudo mkdir -p /opt/vmcatcher-ON/cache /opt/vmcatcher-ON/cache/partial /opt/vmcatcher-ON/cache/expired /opt/vmcatcher-ON/cache/templates
[oneadmin@one-sandbox]$ sudo chown oneadmin:oneadmin -R /opt/vmcatcher-ON
  • Check that vmcatcher is running properly by listing and subscribing to an image list
[oneadmin@one-sandbox]$ export VMCATCHER_RDBMS="sqlite:////opt/vmcatcher-ON/vmcatcher.db"
[oneadmin@one-sandbox]$ vmcatcher_subscribe -l
[oneadmin@one-sandbox]$ vmcatcher_subscribe -e -s https://vmcaster.appdb.egi.eu/store/vappliance/tinycorelinux/image.list
[oneadmin@one-sandbox]$ vmcatcher_subscribe -l
8ddbd4f6-fb95-4917-b105-c89b5df99dda    True    None    https://vmcaster.appdb.egi.eu/store/vappliance/tinycorelinux/image.list
  • Create a CRON wrapper for vmcatcher, named /var/lib/one/vmcatcher_eventHndlExpl_ON/vmcatcher_eventHndl_ON_cron.sh, using the following code
#!/bin/bash
#Cron handler for VMCatcher image syncronization script for OpenNebula

#Vmcatcher configuration variables
export VMCATCHER_RDBMS="sqlite:////opt/vmcatcher-ON/vmcatcher.db"
export VMCATCHER_CACHE_DIR_CACHE="/opt/vmcatcher-ON/cache"
export VMCATCHER_CACHE_DIR_DOWNLOAD="/opt/vmcatcher-ON/cache/partial"
export VMCATCHER_CACHE_DIR_EXPIRE="/opt/vmcatcher-ON/cache/expired"
export VMCATCHER_CACHE_EVENT="python $HOME/vmcatcher_eventHndlExpl_ON/vmcatcher_eventHndl_ON"

#Update vmcatcher image lists
vmcatcher_subscribe -U

#Add all the new images to the cache
for a in `vmcatcher_image -l | awk '{if ($2==2) print $1}'`; do
  vmcatcher_image -a -u $a
done

#Update the cache
vmcatcher_cache -v -v
  • Test that the vmcatcher handler is working correctly by running
[oneadmin@one-sandbox]$ chmod +x $HOME/vmcatcher_eventHndlExpl_ON/vmcatcher_eventHndl_ON_cron.sh
[oneadmin@one-sandbox]$ $HOME/vmcatcher_eventHndlExpl_ON/vmcatcher_eventHndl_ON_cron.sh
INFO:main:Defaulting actions as 'expire', and 'download'.
DEBUG:Events:event 'ProcessPrefix' executed 'python /var/lib/one/vmcatcher_eventHndlExpl_ON/vmcatcher_eventHndl_ON'
DEBUG:Events:stdout=
DEBUG:Events:stderr=2014-07-16 12:25:49,586;   DEBUG; vmcatcher_eventHndl_ON; main -- Processing event 'ProcessPrefix'
2014-07-16 12:25:49,586; WARNING; vmcatcher_eventHndl_ON; main -- Ignoring event 'ProcessPrefix'

INFO:DownloadDir:Downloading '541b01a8-94bd-4545-83a8-6ea07209b440'.
DEBUG:Events:event 'AvailablePrefix' executed 'python /var/lib/one/vmcatcher_eventHndlExpl_ON/vmcatcher_eventHndl_ON'
DEBUG:Events:stdout=
DEBUG:Events:stderr=2014-07-16 12:26:00,522;   DEBUG; vmcatcher_eventHndl_ON; main -- Processing event 'AvailablePrefix'
2014-07-16 12:26:00,522; WARNING; vmcatcher_eventHndl_ON; main -- Ignoring event 'AvailablePrefix'

INFO:CacheMan:moved file 541b01a8-94bd-4545-83a8-6ea07209b440
DEBUG:Events:event 'AvailablePostfix' executed 'python /var/lib/one/vmcatcher_eventHndlExpl_ON/vmcatcher_eventHndl_ON'
DEBUG:Events:stdout=
DEBUG:Events:stderr=2014-07-16 12:26:00,567;   DEBUG; vmcatcher_eventHndl_ON; main -- Processing event 'AvailablePostfix'
2014-07-16 12:26:00,567;   DEBUG; vmcatcher_eventHndl_ON; HandleAvailablePostfix -- Starting HandleAvailablePostfix for '541b01a8-94bd-4545-83a8-6ea07209b440'
2014-07-16 12:26:00,571;    INFO; vmcatcher_eventHndl_ON; UntarFile -- /opt/vmcatcher-ON/cache/541b01a8-94bd-4545-83a8-6ea07209b440 is an OVA file. Extracting files...
2014-07-16 12:26:00,599;    INFO; vmcatcher_eventHndl_ON; UntarFile -- Converting /opt/vmcatcher-ON/cache/templates/541b01a8-94bd-4545-83a8-6ea07209b440/CoreLinux-disk1.vmdk to raw format.
2014-07-16 12:26:00,641;    INFO; vmcatcher_eventHndl_ON; UntarFile -- New RAW image created: /opt/vmcatcher-ON/cache/templates/541b01a8-94bd-4545-83a8-6ea07209b440/CoreLinux-disk1.vmdk.raw
2014-07-16 12:26:00,642;    INFO; vmcatcher_eventHndl_ON; HandleAvailablePostfix -- Creating template file /opt/vmcatcher-ON/cache/templates/541b01a8-94bd-4545-83a8-6ea07209b440.one
2014-07-16 12:26:00,780;    INFO; vmcatcher_eventHndl_ON; getImageListXML -- Getting image list: oneimage list --xml
2014-07-16 12:26:00,784;    INFO; vmcatcher_eventHndl_ON; HandleAvailablePostfix -- There is not a previous image with the same UUID in the OpenNebula infrastructure
2014-07-16 12:26:00,785;    INFO; vmcatcher_eventHndl_ON; HandleAvailablePostfix -- Instantiating template: oneimage create -d default /opt/vmcatcher-ON/cache/templates/541b01a8-94bd-4545-83a8-6ea07209b440.one | cut -d ':' -f 2

DEBUG:Events:event 'ProcessPostfix' executed 'python /var/lib/one/vmcatcher_eventHndlExpl_ON/vmcatcher_eventHndl_ON'
DEBUG:Events:stdout=
DEBUG:Events:stderr=2014-07-16 12:26:01,077;   DEBUG; vmcatcher_eventHndl_ON; main -- Processing event 'ProcessPostfix'
2014-07-16 12:26:01,077; WARNING; vmcatcher_eventHndl_ON; main -- Ignoring event 'ProcessPostfix'
  • Add the following line to the oneadmin user crontab:
50 */6 * * * $HOME/vmcatcher_eventHndlExpl_ON/vmcatcher_eventHndl_ON_cron.sh >> /var/log/vmcatcher.log 2>&1

NOTES:

  • vmcatcher_cache must be executed as oneadmin user.
  • Environment variables can be used to set default values but the command line options will override any set environment options. Set these env variables for oneadmin user: VMCATCHER_RDBMS, VMCATCHER_CACHE_DIR_CACHE, VMCATCHER_CACHE_DIR_DOWNLOAD, VMCATCHER_CACHE_DIR_EXPIRE and VMCATCHER_CACHE_EVENT.
  • vmcatcher_eventHndlExpl_ON generates ON image templates. These templates are available from $VMCATCHER_CACHE_DIR_CACHE/templates (templates nomenclature $VMCATCHER_EVENT_DC_IDENTIFIER.one)
  • The new ON images include VMCATCHER_EVENT_DC_IDENTIFIER = <VMCATCHER_UUID> tag. This tag is used to identify Fedcloud VM images.
  • VMcatcher expired images are set as disabled by ON. It is up to the RP to remove disabled images or assign the new ones to a specific ON group or user.

OpenStack

Vmcatcher may be branched to Openstack Glance catalog using python-glancepush tool and Openstack Handler for Vmcatcher event handler. To install and configure glancepush and the handler, you can refer to the following instructions:

  • Install the latest release of glancepush via the AppDB repository. For debian based systems, just download the tarball, extract it, and execute python setup.py install
[stack@ubuntu]$ wget http://repository.egi.eu/community/software/python.glancepush/0.0.X/releases/generic/0.0.6/python-glancepush-0.0.6.tar.gz
[stack@ubuntu]$ tar -zxvf python-glancepush-0.0.6.tar.gz
[stack@ubuntu]$ python setup.py install

while for RHEL6 you can run:

[stack@rhel]$ yum localinstall http://repository.egi.eu/community/software/python.glancepush/0.0.X/releases/sl/6/x86_64/RPMS/python-glancepush-0.0.6-1.noarch.rpm
  • Then, configure glancepush directories
[stack@ubuntu]$ sudo mkdir -p /var/spool/glancepush /etc/glancepush/log /etc/glancepush/transform/ /etc/glancepush/clouds
[stack@ubuntu]$ sudo chown stack:stack -R /var/spool/glancepush /etc/glancepush /var/log/glancepush/
  • Copy the file /etc/keystone/voms.json to /etc/glancepush/voms.json. Then create a file in clouds file for every VO to which you are subscribed. For example, if you're subscribed to fedcloud, atlas and lhcb, you'll need 3 files in the /etc/glancepush/clouds directory with the credentials for this VO/tenants, for example:
[general]
# Tenant for this VO. Must match the tenant defined in voms.json file
testing_tenant=egi
# Identity service endpoint (Keystone)
endpoint_url=https://server4-eupt.unizar.es:5000/v2.0
# User Password
password=123456
# User
username=John
# Set this to true if you're NOT using self-signed certificates
is_secure=True
# SSH private key that will be used to perform policy checks (to be done)
ssh_key=Carlos_lxbifi81
# WARNING: Only define the next variable if you're going to need it. Otherwise you may encounter problems
cacert=path_to_your_cert
[stack@ubuntu]$ wget http://repository.egi.eu/community/software/openstack.handler.for.vmcatcher/0.0.X/releases/generic/0.0.7/gpvcmupdate-0.0.7.tar.gz
[stack@ubuntu]$ tar -zxvf gpvcmupdate-0.0.7.tar.gz
[stack@ubuntu]$ python setup.py install

while for RHEL6 you can run:

[stack@rhel]$ yum localinstall http://repository.egi.eu/community/software/openstack.handler.for.vmcatcher/0.0.X/releases/generic/0.0.7/gpvcmupdate-0.0.7.tar.gz
  • Create the vmcatcher folders for OpenStack
[stack@ubuntu]$ mkdir -p /opt/stack/vmcatcher/cache /opt/stack/vmcatcher/cache/partial /opt/stack/vmcatcher/cache/expired
  • Check that vmcatcher is running properly by listing and subscribing to an image list
[stack@ubuntu]$ export VMCATCHER_RDBMS="sqlite:////opt/stack/vmcatcher/vmcatcher.db"
[stack@ubuntu]$ vmcatcher_subscribe -l
[stack@ubuntu]$ vmcatcher_subscribe -e -s https://vmcaster.appdb.egi.eu/store/vappliance/tinycorelinux/image.list
[stack@ubuntu]$ vmcatcher_subscribe -l
8ddbd4f6-fb95-4917-b105-c89b5df99dda    True    None    https://vmcaster.appdb.egi.eu/store/vappliance/tinycorelinux/image.list
  • Create a CRON wrapper for vmcatcher, named $HOME/gpvcmupdate/vmcatcher_eventHndl_OS_cron.sh, using the following code
#!/bin/bash
#Cron handler for VMCatcher image syncronization script for OpenStack

#Vmcatcher configuration variables
export VMCATCHER_RDBMS="sqlite:////opt/stack/vmcatcher/vmcatcher.db"
export VMCATCHER_CACHE_DIR_CACHE="/opt/stack/vmcatcher/cache"
export VMCATCHER_CACHE_DIR_DOWNLOAD="/opt/stack/vmcatcher/cache/partial"
export VMCATCHER_CACHE_DIR_EXPIRE="/opt/stack/vmcatcher/cache/expired"
export VMCATCHER_CACHE_EVENT="python $HOME/gpvcmupdate/gpvcmupdate.py -D"

#Update vmcatcher image lists
vmcatcher_subscribe -U

#Add all the new images to the cache
for a in `vmcatcher_image -l | awk '{if ($2==2) print $1}'`; do
  vmcatcher_image -a -u $a
done 

#Update the cache
vmcatcher_cache -v -v

#Run glancepush
/usr/bin/glancepush.py
  • Set the newly created file as executable
[stack@ubuntu]$ chmod +x $HOME/gpvcmupdate/vmcatcher_eventHndl_OS_cron.sh
  • Test that the vmcatcher handler is working correctly by running
[stack@ubuntu]$ $HOME/gpvcmupdate/vmcatcher_eventHndl_OS_cron.sh
INFO:main:Defaulting actions as 'expire', and 'download'.
DEBUG:Events:event 'ProcessPrefix' executed 'python /opt/stack/gpvcmupdate/gpvcmupdate.py'
DEBUG:Events:stdout=
DEBUG:Events:stderr=Ignoring ProcessPrefix event.
INFO:DownloadDir:Downloading '541b01a8-94bd-4545-83a8-6ea07209b440'.
DEBUG:Events:event 'AvailablePrefix' executed 'python /opt/stack/gpvcmupdate/gpvcmupdate.py'
DEBUG:Events:stdout=AvailablePrefix
DEBUG:Events:stderr=
INFO:CacheMan:moved file 541b01a8-94bd-4545-83a8-6ea07209b440
DEBUG:Events:event 'AvailablePostfix' executed 'python /opt/stack/gpvcmupdate/gpvcmupdate.py'
DEBUG:Events:stdout=AvailablePostfixCreating Metadata Files
DEBUG:Events:stderr=
DEBUG:Events:event 'ProcessPostfix' executed 'python /opt/stack/gpvcmupdate/gpvcmupdate.py'
DEBUG:Events:stdout=
DEBUG:Events:stderr=Ignoring ProcessPostfix event.


  • Add the following line to the stack user crontab:
50 */6 * * * $HOME/gpvcmupdate/vmcatcher_eventHndl_OS_cron.sh >> /var/log/glancepush/vmcatcher.log 2>&1

NOTES:

  • It is recommended to execute glancepush and vmcatcher_cache as stack or other non-root user.
  • VMcatcher expired images are removed from OS.

Links

VMcatcher

Openstach Handler for vmcatcher repository

Python-glancepush repository

HEPIX Virtualisation Working Group

Part IV. Virtual Machine Image Transfer

github OpenNebula VMcatcher event handler