Difference between revisions of "Fedcloud-tf:WorkGroups:VM Marketplace"

From EGIWiki
Jump to: navigation, search
(Register an image with the EGI.eu Marketplace)
m
 
(27 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
{{Fedcloud-tf:Menu}} {{Fedcloud-tf:WorkGroups:Menu}} {{TOC_right}}  
 
{{Fedcloud-tf:Menu}} {{Fedcloud-tf:WorkGroups:Menu}} {{TOC_right}}  
  
<font color="red">Leader: Kostas Koumantaros, EGI-InSPIRE SA2 </font>
+
<font color="red">Leader: Kostas Koumantaros, EGI-InSPIRE SA2 </font>  
  
 
== Collaborators  ==
 
== Collaborators  ==
Line 12: Line 12:
 
|-
 
|-
 
| Scenario leader  
 
| Scenario leader  
| EGI-InSPIRE SA2
+
| EGI-InSPIRE SA2  
 
| Kostas Koumantaros
 
| Kostas Koumantaros
 
|-
 
|-
Line 19: Line 19:
 
| Michel Jouvin
 
| Michel Jouvin
 
|-
 
|-
| Collaborator
+
| Collaborator  
| TCD
+
| TCD  
 
| Stuart Kenny
 
| Stuart Kenny
 
|}
 
|}
Line 26: Line 26:
 
== Roadmap  ==
 
== Roadmap  ==
  
* Investigate how to do double endorsement
+
*Investigate how to do double endorsement
  
* Investigate x509 + VOMS authentication
+
*Investigate x509 + VOMS authentication
  
== Scope ==
+
== Scope ==
  
This workbench deals with the issues around setting up a VM Marketplace to:
+
This workbench deals with the issues around setting up a VM Marketplace to:  
* Provide a publicly searchable place for VMs that may provide the application that is needed
 
* Provide a common place to add a token of endorsement to a pertinent VM
 
  
== Marketplace Howto ==
+
*Provide a publicly searchable place for VMs that may provide the application that is needed
 +
*Provide a common place to add a token of endorsement to a pertinent VM
 +
 
 +
== Marketplace Howto ==
 +
 
 +
== Register an image with the EGI.eu Marketplace  ==
 +
 
 +
(''Modified version of instructions compiled by Boris Parak. The original version can be found [http://meta.cesnet.cz/wiki/FedCloudDocumentation:How_to_upload_images_to_the_EGI.eu_Marketplace here]'')
 +
 
 +
=== Install and configure stratuslab-cli-tools  ===
 +
 
 +
This part is very straight-forward, we need ''stratuslab-cli-tools''. So
  
== Register an image with the EGI.eu Marketplace ==
 
=== Install and configure stratuslab-cli-tools ===
 
This part is very straight-forward, we need ''stratuslab-cli-tools''. So
 
 
  cd ~
 
  cd ~
 
  mkdir stratuslab
 
  mkdir stratuslab
 
  cd stratuslab
 
  cd stratuslab
  wget http://repo.stratuslab.eu:8081/content/repositories/fedora-14-releases/eu/stratuslab/pkgs/stratuslab-cli-user-pkg/1.27/stratuslab-cli-user-pkg-1.27.tar.gz
+
  wget http://repo.stratuslab.eu:8081/content/repositories/centos-6.2-releases/eu/stratuslab/pkgs/stratuslab-cli-user-pkg/2.2/stratuslab-cli-user-pkg-2.2.tar.gz
  tar xvf stratuslab-cli-user-pkg-1.27.tar.gz
+
  tar xvf stratuslab-cli-user-pkg-2.2.tar.gz
and then conclude the installation process by appending the following to ''~/.bashrc''
+
 
 +
and then conclude the installation process by appending the following to ''~/.bashrc''  
 +
 
 
  # STRATUSLAB-CLI-TOOLS
 
  # STRATUSLAB-CLI-TOOLS
 
  export PATH=$PATH:~/stratuslab/bin
 
  export PATH=$PATH:~/stratuslab/bin
 
  export PYTHONPATH=$PYTHONPATH:~/stratuslab/lib/stratuslab/python
 
  export PYTHONPATH=$PYTHONPATH:~/stratuslab/lib/stratuslab/python
  
=== Upload the image into your cloud ===
+
RPMs for the client are also available from the StratusLab yum repositories, see http://yum.stratuslab.eu/. Packages are provided for CentOS 6.2, OpenSuse 12.1 and Fedora 16.
This step is different for every cloud platform. For instance, in OpenNebula v3.4+ you can use Sunstone GUI to upload images directly, in previous versions you have to upload the image to the frontend and then register it.
+
 
 +
=== Get demo images  ===
 +
 
 +
There are two images required for the demo. Each resource provider should upload a metadata entry for each. The first is the BNCweb image, which is available from https://appliance-repo.egi.eu/images/base/egi-bncweb/1.0/egi-bncweb.img. The second is a plain Debian 6 image (https://appliance-repo.egi.eu/images/base/Debian-6.0.5-x86_64-base/1.0/debian-6.0.5-x86_64-base.img).  
  
Sice FedCloud-TF will be using OCCI to access the cloud, you must provide a location of the image that is OCCI-compatible. You can use [https://oerc.basecamphq.com/projects/7732005/file/122665694/get-occi-link.pl get-occi-link.pl]. This has only been tested with OpenNebula and the rOCCI server.
+
=== Upload the image into your cloud  ===
perl get-occi-link.pl -host=http://<occi_host> -port=<occi_port> -image="<image_name>"
+
 
or find the right link manually by going through all the storage elements registered in your OCCI server
+
==== appliance Repo  ====
  https://occi.host:10443/storage/-/
+
 
checking the ''occi.core.title'' attribute for the right name. You shloud end up with something like
+
Here are the steps for uploading an image to the appliance repo, which you can register to the EGI Marketplace as described below (ref?): The server uses the fedloud.egi.eu voms for authentication. You can register here (https://perun.metacentrum.cz/perun-registrar-cert/?vo=fedcloud.egi.eu). You will also need the hellasgrid-ca-chain.pem file so that curl can verify the server's certificate.
  https://occi.host:10443/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511
+
 
 +
1. Create the directory where you want to place your image:
 +
 
 +
  curl --cacert ~/path/to/hellasgrid-ca-chain.pem --cert client.pem -X MKCOL https://appliance-repo.egi.eu/images/base/SL-5.7-x86_64-base
 +
curl --cacert ~/path/to/hellasgrid-ca-chain.pem --cert client.pem -X MKCOL https://appliance-repo.egi.eu/images/base/SL-5.7-x86_64-base/1.0
 +
 
 +
2. upload the image:
 +
 
 +
  curl --cacert /path/to/hellasgrid-ca-chain.pem -T /path/to/image --cert client.pem https://appliance-repo.egi.eu/images/base/SL-5.5-x86_64-base/1.0/
 +
 
 +
NOTES:
 +
 
 +
Curl assumes that your cert.pem file contains your private key and certificate concatenated, if that not the case you will get a ""curl: (58) unable to set private key file: /file" error. A workaround is to create separate files for the private key and certificate. For example you can create the files using your pkcs12 certificate using openssl:
 +
 
 +
  openssl pkcs12 -in MULTICERT.p12 -out client.pem -clcerts -nokeys
 +
openssl pkcs12 -in MULTICERT.p12 -out key.pem -nocerts
 +
 
 +
and issue the curl commands by:
 +
 
 +
  curl --cacert ~/path/to/hellasgrid-ca-chain.pem --key key.pem --cert client.pem
 +
 
 +
e.g.  
 +
 
 +
  curl --cacert ~/path/to/hellasgrid-ca-chain.pem --key key.pem --cert client.pem -X MKCOL https://appliance-repo.egi.eu/images/base/SL-5.7-x86_64-base
 +
 
 +
You can generate the hellasgrid-ca-chain.pem file by:
 +
 
 +
#wget http://repository.egi.eu/sw/production/cas/1/current/repo-files/EGI-trustanchors.repo | mv EGI-trustanchors.repo /etc/yum.repos/
 +
#yum install &nbsp;yum install ca_HellasGrid-CA-2006 ca_HellasGrid-Root
 +
#cat /etc/grid-security/certificates/HellasGrid-Root.pem /etc/grid-security/certificates/HellasGrid-CA-2006.pem &gt; /path/to/new/hellasgrid-ca-chain.pem
 +
 
 +
==== Other  ====
 +
 
 +
This step is different for every cloud platform. For instance, in OpenNebula v3.4+ you can use Sunstone GUI to upload images directly, in previous versions you have to upload the image to the frontend and then register it.
 +
 
 +
Sice FedCloud-TF will be using OCCI to access the cloud, you must provide a location of the image that is OCCI-compatible. To find the right link you can browse through all the storage elements registered in your OCCI server  
 +
 
 +
  https://occi.host:port/storage/
 +
 
 +
checking the ''occi.core.title'' attribute for the right name. You should end up with something like  
 +
 
 +
  https://occi.host:port/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511
 +
 
 +
=== Build the metadata  ===
 +
 
 +
The EGI.eu Marketplace stores only metadata which points to the image, provide basic information and integrity verification. Since RDF is not the most user-friendly format, we can use ''stratus-build-metadata'' to generate a template
  
=== Build the metadata ===
 
The EGI.eu Marketplace stores only metadata which points to the image, provide basic information and integrity verification. Since RDF is not the most user-friendly format, we can use ''stratus-build-metadata'' to generate a template
 
 
  stratus-build-metadata --author='##YOUR_NAME##' --type=base --os=Ubuntu --os-version=11.04 --os-arch=x86_64 \
 
  stratus-build-metadata --author='##YOUR_NAME##' --type=base --os=Ubuntu --os-version=11.04 --os-arch=x86_64 \
 
  --image-version=1.0 --hypervisor=xen --format=raw --comment='BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##' \  
 
  --image-version=1.0 --hypervisor=xen --format=raw --comment='BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##' \  
  --compression=none --location='https://occi.host:10443/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511' egi-bncweb.img
+
  --compression=none --location='https://occi.host:port/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511' egi-bncweb.img
'''Note:''' stratus-build-metadata needs the image to compute checksums, you can download it here [https://appliance-repo.egi.eu/images/base/egi-bncweb/1.0/egi-bncweb.img egi-bncweb.img]
+
 
 +
'''Note:''' stratus-build-metadata needs the image to compute checksums, you can download it here [https://appliance-repo.egi.eu/images/base/egi-bncweb/1.0/egi-bncweb.img egi-bncweb.img]  
 +
 
 +
=== Modify the metadata  ===
  
=== Modify the metadata ===
 
 
Now we can check/modify the metadata, the most important elements are ''dcterms:valid'' and ''dcterms:title''.  
 
Now we can check/modify the metadata, the most important elements are ''dcterms:valid'' and ''dcterms:title''.  
  
The correct format for ''dcterms:title'' is ''EGI-##IMAGE_NAME##-##SITE_NAME##''. This field will need to be manually added to the metadata file. You can also modify the validity date as required.
+
The correct format for ''dcterms:title'' is ''EGI-##IMAGE_NAME##-##SITE_NAME##''. This field will need to be manually added to the metadata file. You can also modify the validity date as required.  
  
'''Metadata from the EGI.eu Marketplace cannot be removed, it can only expire.''' It is also possible to ''deprecate'' an entry. This might be necessary, if for example, a security issue is detected with the image, or if you simply wish to no longer endorse the image. Instructions for the stratus-deprecate-image command can be found [http://stratuslab.eu/doku.php/ref-doc:user-cli#stratus-deprecate-metadata here].
+
'''Metadata from the EGI.eu Marketplace cannot be removed, it can only expire.''' It is also possible to ''deprecate'' an entry. This might be necessary, if for example, a security issue is detected with the image, or if you simply wish to no longer endorse the image. Instructions for the stratus-deprecate-image command can be found [http://stratuslab.eu/doku.php/ref-doc:user-cli#stratus-deprecate-metadata here].  
 
+
<pre>&lt;?xml version="1.0" encoding="UTF-8" standalone="no"?&gt;
<pre>
+
&lt;rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
 
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
 
 
     xmlns:dcterms="http://purl.org/dc/terms/"
 
     xmlns:dcterms="http://purl.org/dc/terms/"
 
     xmlns:slterms="http://mp.stratuslab.eu/slterms#"
 
     xmlns:slterms="http://mp.stratuslab.eu/slterms#"
 
     xmlns:slreq="http://mp.stratuslab.eu/slreq#"
 
     xmlns:slreq="http://mp.stratuslab.eu/slreq#"
     xml:base="http://mp.stratuslab.eu/">
+
     xml:base="http://mp.stratuslab.eu/"&gt;
  
     <rdf:Description rdf:about="#DtRwHZzoo1xFKtk-iL51t6RNQ9Q">
+
     &lt;rdf:Description rdf:about="#DtRwHZzoo1xFKtk-iL51t6RNQ9Q"&gt;
  
         <dcterms:identifier>DtRwHZzoo1xFKtk-iL51t6RNQ9Q</dcterms:identifier>
+
         &lt;dcterms:identifier&gt;DtRwHZzoo1xFKtk-iL51t6RNQ9Q&lt;/dcterms:identifier&gt;
  
         <slreq:bytes>14680064000</slreq:bytes>
+
         &lt;slreq:bytes&gt;14680064000&lt;/slreq:bytes&gt;
  
         <slreq:checksum rdf:parseType="Resource">
+
         &lt;slreq:checksum rdf:parseType="Resource"&gt;
             <slreq:algorithm>MD5</slreq:algorithm>
+
             &lt;slreq:algorithm&gt;MD5&lt;/slreq:algorithm&gt;
             <slreq:value>144fff2477673aa1d883f0a3ba89f273</slreq:value>
+
             &lt;slreq:value&gt;144fff2477673aa1d883f0a3ba89f273&lt;/slreq:value&gt;
         </slreq:checksum>
+
         &lt;/slreq:checksum&gt;
         <slreq:checksum rdf:parseType="Resource">
+
         &lt;slreq:checksum rdf:parseType="Resource"&gt;
             <slreq:algorithm>SHA-1</slreq:algorithm>
+
             &lt;slreq:algorithm&gt;SHA-1&lt;/slreq:algorithm&gt;
             <slreq:value>3b51c07673a28d7114ab64fa22f9d6de91350f50</slreq:value>
+
             &lt;slreq:value&gt;3b51c07673a28d7114ab64fa22f9d6de91350f50&lt;/slreq:value&gt;
         </slreq:checksum>
+
         &lt;/slreq:checksum&gt;
         <slreq:checksum rdf:parseType="Resource">
+
         &lt;slreq:checksum rdf:parseType="Resource"&gt;
             <slreq:algorithm>SHA-256</slreq:algorithm>
+
             &lt;slreq:algorithm&gt;SHA-256&lt;/slreq:algorithm&gt;
             <slreq:value>8bde348c81e5a2aa5aa51b8d39a30ad137d0482decd5960cd95594d224a45bdd</slreq:value>
+
             &lt;slreq:value&gt;8bde348c81e5a2aa5aa51b8d39a30ad137d0482decd5960cd95594d224a45bdd&lt;/slreq:value&gt;
         </slreq:checksum>
+
         &lt;/slreq:checksum&gt;
         <slreq:checksum rdf:parseType="Resource">
+
         &lt;slreq:checksum rdf:parseType="Resource"&gt;
             <slreq:algorithm>SHA-512</slreq:algorithm>
+
             &lt;slreq:algorithm&gt;SHA-512&lt;/slreq:algorithm&gt;
             <slreq:value>e780f2aa6922bc7cfdaae4a5e410f6b499bef5c83314bcd760b082b625860834c4942de9d096c7aa83cdad0411c47686f2e7d0fcc65f816475f6525db28b236d</slreq:value>
+
             &lt;slreq:value&gt;e780f2aa6922bc7cfdaae4a5e410f6b499bef5c83314bcd760b082b625860834c4942de9d096c7aa83cdad0411c47686f2e7d0fcc65f816475f6525db28b236d&lt;/slreq:value&gt;
         </slreq:checksum>
+
         &lt;/slreq:checksum&gt;
  
         <slreq:endorsement rdf:parseType="Resource"/>
+
         &lt;slreq:endorsement rdf:parseType="Resource"/&gt;
  
         <dcterms:title>EGI-BNCweb-##YOUR_SITE##</dcterms:title>
+
         &lt;dcterms:title&gt;EGI-BNCweb-##YOUR_SITE##&lt;/dcterms:title&gt;
         <dcterms:type>base</dcterms:type>
+
         &lt;dcterms:type&gt;base&lt;/dcterms:type&gt;
         <slterms:kind>machine</slterms:kind>
+
         &lt;slterms:kind&gt;machine&lt;/slterms:kind&gt;
  
         <slterms:os>Ubuntu</slterms:os>
+
         &lt;slterms:os&gt;Ubuntu&lt;/slterms:os&gt;
         <slterms:os-version>11.04</slterms:os-version>
+
         &lt;slterms:os-version&gt;11.04&lt;/slterms:os-version&gt;
         <slterms:os-arch>x86_64</slterms:os-arch>
+
         &lt;slterms:os-arch&gt;x86_64&lt;/slterms:os-arch&gt;
         <slterms:version>1.0</slterms:version>
+
         &lt;slterms:version&gt;1.0&lt;/slterms:version&gt;
         <dcterms:compression>none</dcterms:compression>
+
         &lt;dcterms:compression&gt;none&lt;/dcterms:compression&gt;
         <slterms:location>https://occi.host:10443/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511</slterms:location>
+
         &lt;slterms:location&gt;https://occi.host:port/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511&lt;/slterms:location&gt;
  
         <dcterms:format>raw</dcterms:format>
+
         &lt;dcterms:format&gt;raw&lt;/dcterms:format&gt;
  
         <dcterms:creator>##YOUR_NAME##</dcterms:creator>
+
         &lt;dcterms:creator&gt;##YOUR_NAME##&lt;/dcterms:creator&gt;
  
         <dcterms:created>2012-06-12T12:36:25Z</dcterms:created>
+
         &lt;dcterms:created&gt;2012-06-12T12:36:25Z&lt;/dcterms:created&gt;
         <dcterms:valid>2012-06-14T12:36:25Z</dcterms:valid>
+
         &lt;dcterms:valid&gt;2012-06-14T12:36:25Z&lt;/dcterms:valid&gt;
  
         <dcterms:description>BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##</dcterms:description>
+
         &lt;dcterms:description&gt;BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##&lt;/dcterms:description&gt;
  
         <slterms:hypervisor>xen</slterms:hypervisor>
+
         &lt;slterms:hypervisor&gt;xen&lt;/slterms:hypervisor&gt;
  
         <dcterms:publisher>StratusLab</dcterms:publisher>
+
         &lt;dcterms:publisher&gt;##YOUR_SITE##&lt;/dcterms:publisher&gt;
 
          
 
          
     </rdf:Description>
+
     &lt;/rdf:Description&gt;
</rdf:RDF>
+
&lt;/rdf:RDF&gt;
</pre>
+
</pre>  
 +
{{Template:Block-comment
 +
| name=Notice
 +
| text=These fields should be checked: ''&lt;dcterms:title&gt;EGI-BNCweb-##YOUR_SITE##&lt;/dcterms:title&gt;'' ''&lt;dcterms:creator&gt;##YOUR_NAME##&lt;/dcterms:creator&gt;'' ''&lt;dcterms:description&gt;BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##&lt;/dcterms:description&gt;'' and ''&lt;dcterms:publisher&gt;##YOUR_SITE##&lt;/dcterms:publisher&gt;''
 +
}}
 +
 
 +
<br>
 +
 
 +
==== Modify Metadata (OCCI 1.1 servers)  ====
 +
 
 +
{{Template:Block-comment
 +
| name=Warning
 +
| text=These changes are required for TF2012 demo.
 +
}}
 +
 
 +
&lt;slterms:location&gt;https://occi.host:port/storage/##STORAGE ID##&lt;/slterms:location&gt;
 +
&lt;dcterms:requires&gt;https://occi.host:port/network/##NETWORK ID##&lt;/dcterms:requires&gt;
 +
 
 +
*'''Optional:'''
 +
 
 +
Set &lt;dcterms:valid&gt; field to be used until TF demo:
 +
 
 +
&lt;dcterms:valid&gt;2012-10-02T09:55:00Z&lt;/dcterms:valid&gt;
 +
 
 +
==== Modify Metadata (rOCCI or OCCI OpenStack servers)  ====
 +
 
 +
{{Template:Block-comment
 +
| name=Warning
 +
| text=These changes are required for TF2012 demo
 +
}}
 +
 
 +
<br>
 +
 
 +
&lt;dcterms:requires&gt;https://rocci.host:port&lt;/dcterms:requires&gt;
 +
 
 +
*'''Optional:'''
 +
 
 +
Set &lt;dcterms:valid&gt; field to be used until TF demo:
 +
 
 +
&lt;dcterms:valid&gt;2012-10-02T09:55:00Z&lt;/dcterms:valid&gt;
 +
 
 +
<br>  
 +
 
 +
=== Sign the metadata  ===
  
=== Sign the metadata ===
+
To establish the origin of the image, we have to sign the metadata with a personal certificate (ideally the one registered with EGI.eu). '''Before''' doing this you should familiarise yourself with the [https://documents.egi.eu/public/ShowDocument?docid=771 EGI Security Policy for the Endorsement and Operation of Virtual Machine Images].  
To establish the origin of the image, we have to sign the metadata with a personal certificate (ideally the one registered with EGI.eu). '''Before''' doing this you should familiarise yourself with the [https://documents.egi.eu/public/ShowDocument?docid=771 EGI Security Policy for the Endorsement and Operation of Virtual Machine Images].
 
  
 
  stratus-sign-metadata --p12-cert=##FULL_PATH_TO_usercred.p12## egi-bncweb.xml
 
  stratus-sign-metadata --p12-cert=##FULL_PATH_TO_usercred.p12## egi-bncweb.xml
  
=== Register the metadata with the EGI.eu Marketplace ===
+
=== Register the metadata with the EGI.eu Marketplace ===
And to complete the process, we have to upload the metadata to the EGI.eu Marketplace with ''stratus-upload-metadata''
+
 
 +
And to complete the process, we have to upload the metadata to the EGI.eu Marketplace with ''stratus-upload-metadata''  
 +
 
 
  stratus-upload-metadata --marketplace-endpoint=marketplace.egi.eu egi-bncweb.xml
 
  stratus-upload-metadata --marketplace-endpoint=marketplace.egi.eu egi-bncweb.xml
or manually at
+
 
 +
or manually at  
 +
 
 
  http://marketplace.egi.eu/upload
 
  http://marketplace.egi.eu/upload
 +
 +
=== Howto update and change old metadata  ===
 +
 +
To update uploaded metadata just modify the metadata file, sign it again, and then upload. It is basically the same procedure as uploading new metadata. Only the most recent entry for a particular image identifier/email address is displayed.
 +
 +
== [https://github.com/hepix-virtualisation/vmcatcher VMcatcher]  ==
 +
 +
VMcatcher allows users to subscribe to virtual machine Virtual Machine image lists, cache the images referenced to in the Virtual Machine Image List, validate the images list with x509 based public key cryptography, and validate the images against sha512 hashes in the images lists and provide events for further applications to process updates or expiries of virtual machine images without having to further validate the images.
 +
 +
=== Installation  ===
 +
 +
=== Usage  ===
 +
 +
=== Event Handlers  ===
 +
 +
==== OpenNebula  ====
 +
 +
==== OpenStack  ====
 +
 +
=== Links  ===
 +
 +
[http://grid.desy.de/vm/hepix/vwg/doc/html/index.shtml HEPIX Virtualisation Working Group]
 +
 +
[http://grid.desy.de/vm/hepix/vwg/doc/html/imagetransfer/imagetransfer.shtml Part&nbsp;IV.&nbsp;Virtual Machine Image Transfer]
 +
 +
[https://github.com/robertord/Fedcloud/blob/master/vmcatcher_eventHndlExpl_ON github OpenNebula VMcatcher event handler]

Latest revision as of 17:28, 12 March 2013

Main Roadmap and Innovation Technology For Users For Resource Providers Media


Workbenches: Open issues
Scenario 1
VM Management
Scenario 2
Data Management
Scenario 3
Information Systems
Scenario 4
Accounting
Scenario 5
Monitoring
Scenario 6
Notification
Scenario 7
Federated AAI
Scenario 8
VM Image Management
Scenario 9
Brokering
Scenario 10
Contextualisation
Scenario 11
Security



Leader: Kostas Koumantaros, EGI-InSPIRE SA2

Collaborators

Role Institution Name
Scenario leader EGI-InSPIRE SA2 Kostas Koumantaros
Collaborator GRIF Michel Jouvin
Collaborator TCD Stuart Kenny

Roadmap

  • Investigate how to do double endorsement
  • Investigate x509 + VOMS authentication

Scope

This workbench deals with the issues around setting up a VM Marketplace to:

  • Provide a publicly searchable place for VMs that may provide the application that is needed
  • Provide a common place to add a token of endorsement to a pertinent VM

Marketplace Howto

Register an image with the EGI.eu Marketplace

(Modified version of instructions compiled by Boris Parak. The original version can be found here)

Install and configure stratuslab-cli-tools

This part is very straight-forward, we need stratuslab-cli-tools. So

cd ~
mkdir stratuslab
cd stratuslab
wget http://repo.stratuslab.eu:8081/content/repositories/centos-6.2-releases/eu/stratuslab/pkgs/stratuslab-cli-user-pkg/2.2/stratuslab-cli-user-pkg-2.2.tar.gz
tar xvf stratuslab-cli-user-pkg-2.2.tar.gz

and then conclude the installation process by appending the following to ~/.bashrc

# STRATUSLAB-CLI-TOOLS
export PATH=$PATH:~/stratuslab/bin
export PYTHONPATH=$PYTHONPATH:~/stratuslab/lib/stratuslab/python

RPMs for the client are also available from the StratusLab yum repositories, see http://yum.stratuslab.eu/. Packages are provided for CentOS 6.2, OpenSuse 12.1 and Fedora 16.

Get demo images

There are two images required for the demo. Each resource provider should upload a metadata entry for each. The first is the BNCweb image, which is available from https://appliance-repo.egi.eu/images/base/egi-bncweb/1.0/egi-bncweb.img. The second is a plain Debian 6 image (https://appliance-repo.egi.eu/images/base/Debian-6.0.5-x86_64-base/1.0/debian-6.0.5-x86_64-base.img).

Upload the image into your cloud

appliance Repo

Here are the steps for uploading an image to the appliance repo, which you can register to the EGI Marketplace as described below (ref?): The server uses the fedloud.egi.eu voms for authentication. You can register here (https://perun.metacentrum.cz/perun-registrar-cert/?vo=fedcloud.egi.eu). You will also need the hellasgrid-ca-chain.pem file so that curl can verify the server's certificate.

1. Create the directory where you want to place your image:

 curl --cacert ~/path/to/hellasgrid-ca-chain.pem --cert client.pem -X MKCOL https://appliance-repo.egi.eu/images/base/SL-5.7-x86_64-base
curl --cacert ~/path/to/hellasgrid-ca-chain.pem --cert client.pem -X MKCOL https://appliance-repo.egi.eu/images/base/SL-5.7-x86_64-base/1.0

2. upload the image:

 curl --cacert /path/to/hellasgrid-ca-chain.pem -T /path/to/image --cert client.pem https://appliance-repo.egi.eu/images/base/SL-5.5-x86_64-base/1.0/

NOTES:

Curl assumes that your cert.pem file contains your private key and certificate concatenated, if that not the case you will get a ""curl: (58) unable to set private key file: /file" error. A workaround is to create separate files for the private key and certificate. For example you can create the files using your pkcs12 certificate using openssl:

 openssl pkcs12 -in MULTICERT.p12 -out client.pem -clcerts -nokeys
openssl pkcs12 -in MULTICERT.p12 -out key.pem -nocerts

and issue the curl commands by:

 curl --cacert ~/path/to/hellasgrid-ca-chain.pem --key key.pem --cert client.pem

e.g.

 curl --cacert ~/path/to/hellasgrid-ca-chain.pem --key key.pem --cert client.pem -X MKCOL https://appliance-repo.egi.eu/images/base/SL-5.7-x86_64-base

You can generate the hellasgrid-ca-chain.pem file by:

  1. wget http://repository.egi.eu/sw/production/cas/1/current/repo-files/EGI-trustanchors.repo | mv EGI-trustanchors.repo /etc/yum.repos/
  2. yum install  yum install ca_HellasGrid-CA-2006 ca_HellasGrid-Root
  3. cat /etc/grid-security/certificates/HellasGrid-Root.pem /etc/grid-security/certificates/HellasGrid-CA-2006.pem > /path/to/new/hellasgrid-ca-chain.pem

Other

This step is different for every cloud platform. For instance, in OpenNebula v3.4+ you can use Sunstone GUI to upload images directly, in previous versions you have to upload the image to the frontend and then register it.

Sice FedCloud-TF will be using OCCI to access the cloud, you must provide a location of the image that is OCCI-compatible. To find the right link you can browse through all the storage elements registered in your OCCI server

https://occi.host:port/storage/

checking the occi.core.title attribute for the right name. You should end up with something like

https://occi.host:port/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511

Build the metadata

The EGI.eu Marketplace stores only metadata which points to the image, provide basic information and integrity verification. Since RDF is not the most user-friendly format, we can use stratus-build-metadata to generate a template

stratus-build-metadata --author='##YOUR_NAME##' --type=base --os=Ubuntu --os-version=11.04 --os-arch=x86_64 \
--image-version=1.0 --hypervisor=xen --format=raw --comment='BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##' \ 
--compression=none --location='https://occi.host:port/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511' egi-bncweb.img

Note: stratus-build-metadata needs the image to compute checksums, you can download it here egi-bncweb.img

Modify the metadata

Now we can check/modify the metadata, the most important elements are dcterms:valid and dcterms:title.

The correct format for dcterms:title is EGI-##IMAGE_NAME##-##SITE_NAME##. This field will need to be manually added to the metadata file. You can also modify the validity date as required.

Metadata from the EGI.eu Marketplace cannot be removed, it can only expire. It is also possible to deprecate an entry. This might be necessary, if for example, a security issue is detected with the image, or if you simply wish to no longer endorse the image. Instructions for the stratus-deprecate-image command can be found here.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:slterms="http://mp.stratuslab.eu/slterms#"
    xmlns:slreq="http://mp.stratuslab.eu/slreq#"
    xml:base="http://mp.stratuslab.eu/">

    <rdf:Description rdf:about="#DtRwHZzoo1xFKtk-iL51t6RNQ9Q">

        <dcterms:identifier>DtRwHZzoo1xFKtk-iL51t6RNQ9Q</dcterms:identifier>

        <slreq:bytes>14680064000</slreq:bytes>

        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>MD5</slreq:algorithm>
            <slreq:value>144fff2477673aa1d883f0a3ba89f273</slreq:value>
        </slreq:checksum>
        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>SHA-1</slreq:algorithm>
            <slreq:value>3b51c07673a28d7114ab64fa22f9d6de91350f50</slreq:value>
        </slreq:checksum>
        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>SHA-256</slreq:algorithm>
            <slreq:value>8bde348c81e5a2aa5aa51b8d39a30ad137d0482decd5960cd95594d224a45bdd</slreq:value>
        </slreq:checksum>
        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>SHA-512</slreq:algorithm>
            <slreq:value>e780f2aa6922bc7cfdaae4a5e410f6b499bef5c83314bcd760b082b625860834c4942de9d096c7aa83cdad0411c47686f2e7d0fcc65f816475f6525db28b236d</slreq:value>
        </slreq:checksum>

        <slreq:endorsement rdf:parseType="Resource"/>

        <dcterms:title>EGI-BNCweb-##YOUR_SITE##</dcterms:title>
        <dcterms:type>base</dcterms:type>
        <slterms:kind>machine</slterms:kind>

        <slterms:os>Ubuntu</slterms:os>
        <slterms:os-version>11.04</slterms:os-version>
        <slterms:os-arch>x86_64</slterms:os-arch>
        <slterms:version>1.0</slterms:version>
        <dcterms:compression>none</dcterms:compression>
        <slterms:location>https://occi.host:port/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511</slterms:location>

        <dcterms:format>raw</dcterms:format>

        <dcterms:creator>##YOUR_NAME##</dcterms:creator>

        <dcterms:created>2012-06-12T12:36:25Z</dcterms:created>
        <dcterms:valid>2012-06-14T12:36:25Z</dcterms:valid>

        <dcterms:description>BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##</dcterms:description>

        <slterms:hypervisor>xen</slterms:hypervisor>

        <dcterms:publisher>##YOUR_SITE##</dcterms:publisher>
        
    </rdf:Description>
</rdf:RDF>

Notice:
These fields should be checked: <dcterms:title>EGI-BNCweb-##YOUR_SITE##</dcterms:title> <dcterms:creator>##YOUR_NAME##</dcterms:creator> <dcterms:description>BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##</dcterms:description> and <dcterms:publisher>##YOUR_SITE##</dcterms:publisher>



Modify Metadata (OCCI 1.1 servers)

Warning:
These changes are required for TF2012 demo.


<slterms:location>https://occi.host:port/storage/##STORAGE ID##</slterms:location>
<dcterms:requires>https://occi.host:port/network/##NETWORK ID##</dcterms:requires>
  • Optional:

Set <dcterms:valid> field to be used until TF demo:

<dcterms:valid>2012-10-02T09:55:00Z</dcterms:valid>

Modify Metadata (rOCCI or OCCI OpenStack servers)

Warning:
These changes are required for TF2012 demo



<dcterms:requires>https://rocci.host:port</dcterms:requires>
  • Optional:

Set <dcterms:valid> field to be used until TF demo:

<dcterms:valid>2012-10-02T09:55:00Z</dcterms:valid>


Sign the metadata

To establish the origin of the image, we have to sign the metadata with a personal certificate (ideally the one registered with EGI.eu). Before doing this you should familiarise yourself with the EGI Security Policy for the Endorsement and Operation of Virtual Machine Images.

stratus-sign-metadata --p12-cert=##FULL_PATH_TO_usercred.p12## egi-bncweb.xml

Register the metadata with the EGI.eu Marketplace

And to complete the process, we have to upload the metadata to the EGI.eu Marketplace with stratus-upload-metadata

stratus-upload-metadata --marketplace-endpoint=marketplace.egi.eu egi-bncweb.xml

or manually at

http://marketplace.egi.eu/upload

Howto update and change old metadata

To update uploaded metadata just modify the metadata file, sign it again, and then upload. It is basically the same procedure as uploading new metadata. Only the most recent entry for a particular image identifier/email address is displayed.

VMcatcher

VMcatcher allows users to subscribe to virtual machine Virtual Machine image lists, cache the images referenced to in the Virtual Machine Image List, validate the images list with x509 based public key cryptography, and validate the images against sha512 hashes in the images lists and provide events for further applications to process updates or expiries of virtual machine images without having to further validate the images.

Installation

Usage

Event Handlers

OpenNebula

OpenStack

Links

HEPIX Virtualisation Working Group

Part IV. Virtual Machine Image Transfer

github OpenNebula VMcatcher event handler