Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Fedcloud-tf:WorkGroups:VM Marketplace"

From EGIWiki
Jump to navigation Jump to search
Line 38: Line 38:
== Marketplace Howto ==
== Marketplace Howto ==


=== Publishing metadata for images stored in RP local image store ===
== Register an image with the EGI.eu Marketplace ==
=== Install and configure stratuslab-cli-tools ===
This part is very straight-forward, we need ''stratuslab-cli-tools''. So
cd ~
mkdir stratuslab
cd stratuslab
wget http://repo.stratuslab.eu:8081/content/repositories/fedora-14-releases/eu/stratuslab/pkgs/stratuslab-cli-user-pkg/1.27/stratuslab-cli-user-pkg-1.27.tar.gz
tar xvf stratuslab-cli-user-pkg-1.27.tar.gz
and then conclude the installation process by appending the following to ''~/.bashrc''
# STRATUSLAB-CLI-TOOLS
export PATH=$PATH:~/stratuslab/bin
export PYTHONPATH=$PYTHONPATH:~/stratuslab/lib/stratuslab/python


In this case we are publishing metadata describing an image that is stored in a Resource Providers local image store. The metadata will contain an OCCI storage link that can be used by a user when creating a compute resource at that provider.
=== Upload the image into your cloud ===
This step is different for every cloud platform. For instance, in OpenNebula v3.4+ you can use Sunstone GUI to upload images directly, in previous versions you have to upload the image to the frontend and then register it.


# Install the StratusLab command line client.
Sice FedCloud-TF will be using OCCI to access the cloud, we have to provide them with a location of the image that is OCCI-compatible. You can use [https://oerc.basecamphq.com/projects/7732005/file/122665694/get-occi-link.pl get-occi-link.pl]
#: See http://www.stratuslab.org/doku.php/tutorial:installation
perl get-occi-link.pl -host=http://<occi_host> -port=<occi_port> -image="<image_name>"
#: This provides the commands needed to create metadata and to upload to the Marketplace
or find the right link manually by going through all the storage elements registered in your OCCI server
# Obtain the OCCI URL for the image. The script [https://oerc.basecamphq.com/projects/7732005/file/122665694/get-occi-link.pl get-occi-link.pl] will do this:
https://carach5.ics.muni.cz:10443/storage/-/
#: perl get-occi-link.pl -host=<nowiki>http://<occi_host></nowiki> -port=<occi_port> -image="<image_name>"
checking the ''occi.core.title'' attribute for the right name. You shloud end up with something like
# To create the metadata you must have access to the image file. The file passed to stratus-build-metadata should be uncompressed. The location should be set as the OCCI URL retrieved in the previous step.
https://carach5.ics.muni.cz:10443/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511
#: stratus-build-metadata
 
#:    --author=<author>
=== Build the metadata ===
#:    --os=<os>
The EGI.eu Marketplace stores only metadata which points to the image, provide basic information and integrity verification. Since RDF is not the most user-friendly format, we can use ''stratus-build-metadata'' to generate a template
#:    --os-version=<os version>
stratus-build-metadata --author='##YOUR_NAME##' --type=base --os=Ubuntu --os-version=11.04 --os-arch=x86_64 \
#:     --os-arch=<arch>
--image-version=1.0 --hypervisor=xen --format=raw --comment='BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##' \
#:     --image-version=<version>
--compression=none --location='https://carach5.ics.muni.cz:10443/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511' egi-bncweb.img
#:    --comment="A description of the image"
'''Note:''' stratus-build-metadata needs the image to compute checksums, you can download it here [https://appliance-repo.egi.eu/images/base/egi-bncweb/1.0/egi-bncweb.img egi-bncweb.img]
#:    --compression=<compression>
 
#:     '''--location=<occi url>'''
=== Modify the metadata ===
#:     </path/to/image>
Now we can check/modify the metadata, the most important elements are ''dcterms:valid'' and ''dcterms:title''.
# The metadata must now be signed.
 
#: stratus-sign-metadata
The correct format for ''dcterms:title'' is ''EGI-##IMAGE_NAME##-##SITE_NAME##''. This field will need to be manually added to the metadata file. You can also modify the validity date as required.
#:     --p12-cert=</path/to/cert>  
 
#:     --p12-password=<password>  
'''Metadata from the EGI.eu Marketplace cannot be removed, it can only expire.''' It is also possible to ''deprecate'' an entry. This might be necessary, if for example, a security issue is detected with the image, or if you simply wish to no longer endorse the image. Instructions for the stratus-deprecate-image command can be found [http://stratuslab.eu/doku.php/ref-doc:user-cli#stratus-deprecate-metadata here].
#:     --email=<endorser email>  
 
#:    <metadata file>
<pre>
# And finally uploaded
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
#: stratus-upload-metadata
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
#:    --marketplace-endpoint http://marketplace.egi.eu
    xmlns:dcterms="http://purl.org/dc/terms/"
#:    <metadata file>
     xmlns:slterms="http://mp.stratuslab.eu/slterms#"
    xmlns:slreq="http://mp.stratuslab.eu/slreq#"
    xml:base="http://mp.stratuslab.eu/">
 
     <rdf:Description rdf:about="#DtRwHZzoo1xFKtk-iL51t6RNQ9Q">
 
        <dcterms:identifier>DtRwHZzoo1xFKtk-iL51t6RNQ9Q</dcterms:identifier>
 
        <slreq:bytes>14680064000</slreq:bytes>
 
        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>MD5</slreq:algorithm>
            <slreq:value>144fff2477673aa1d883f0a3ba89f273</slreq:value>
        </slreq:checksum>
        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>SHA-1</slreq:algorithm>
            <slreq:value>3b51c07673a28d7114ab64fa22f9d6de91350f50</slreq:value>
        </slreq:checksum>
        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>SHA-256</slreq:algorithm>
            <slreq:value>8bde348c81e5a2aa5aa51b8d39a30ad137d0482decd5960cd95594d224a45bdd</slreq:value>
        </slreq:checksum>
        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>SHA-512</slreq:algorithm>
            <slreq:value>e780f2aa6922bc7cfdaae4a5e410f6b499bef5c83314bcd760b082b625860834c4942de9d096c7aa83cdad0411c47686f2e7d0fcc65f816475f6525db28b236d</slreq:value>
        </slreq:checksum>
 
        <slreq:endorsement rdf:parseType="Resource"/>
 
        <dcterms:title>EGI-BNCweb-##YOUR_SITE##</dcterms:title>
        <dcterms:type>base</dcterms:type>
        <slterms:kind>machine</slterms:kind>
 
        <slterms:os>Ubuntu</slterms:os>
        <slterms:os-version>11.04</slterms:os-version>
        <slterms:os-arch>x86_64</slterms:os-arch>
        <slterms:version>1.0</slterms:version>
        <dcterms:compression>none</dcterms:compression>
        <slterms:location>https://carach5.ics.muni.cz:10443/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511</slterms:location>
 
        <dcterms:format>raw</dcterms:format>
 
        <dcterms:creator>##YOUR_NAME##</dcterms:creator>
 
        <dcterms:created>2012-06-12T12:36:25Z</dcterms:created>
        <dcterms:valid>2012-06-14T12:36:25Z</dcterms:valid>
 
        <dcterms:description>BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##</dcterms:description>
 
        <slterms:hypervisor>xen</slterms:hypervisor>
 
        <dcterms:publisher>StratusLab</dcterms:publisher>
       
     </rdf:Description>
</rdf:RDF>
</pre>
 
=== Sign the metadata ===
To establish the origin of the image, we have to sign the metadata with a personal certificate (ideally the one registered with EGI.eu). '''Before''' doing this you should familiarise yourself with the [https://documents.egi.eu/public/ShowDocument?docid=771 EGI Security Policy for the Endorsement and Operation of Virtual Machine Images].
 
stratus-sign-metadata --p12-cert=##FULL_PATH_TO_usercred.p12## egi-bncweb.xml
 
=== Register the metadata with the EGI.eu Marketplace ===
And to complete the process, we have to upload the metadata to the EGI.eu Marketplace with ''stratus-upload-metadata''
stratus-upload-metadata --marketplace-endpoint=marketplace.egi.eu egi-bncweb.xml
or manually at
http://marketplace.egi.eu/upload

Revision as of 13:27, 17 July 2012

Main Roadmap and Innovation Technology For Users For Resource Providers Media


Workbenches: Open issues
 		Scenario 1
VM Management 		Scenario 2
Data Management 		Scenario 3
Information Systems 		Scenario 4
Accounting 		Scenario 5
Monitoring 		Scenario 6
Notification 		Scenario 7
Federated AAI 		Scenario 8
VM Image Management 		Scenario 9
Brokering 		Scenario 10
Contextualisation 		Scenario 11
Security



Leader: Kostas Koumantaros, EGI-InSPIRE SA2

Collaborators

Role Institution Name
Scenario leader EGI-InSPIRE SA2 Kostas Koumantaros
Collaborator GRIF Michel Jouvin
Collaborator TCD Stuart Kenny

Roadmap

  • Investigate how to do double endorsement
  • Investigate x509 + VOMS authentication

Scope

This workbench deals with the issues around setting up a VM Marketplace to:

  • Provide a publicly searchable place for VMs that may provide the application that is needed
  • Provide a common place to add a token of endorsement to a pertinent VM

Marketplace Howto

Register an image with the EGI.eu Marketplace

Install and configure stratuslab-cli-tools

This part is very straight-forward, we need stratuslab-cli-tools. So 

cd ~
mkdir stratuslab
cd stratuslab
wget http://repo.stratuslab.eu:8081/content/repositories/fedora-14-releases/eu/stratuslab/pkgs/stratuslab-cli-user-pkg/1.27/stratuslab-cli-user-pkg-1.27.tar.gz
tar xvf stratuslab-cli-user-pkg-1.27.tar.gz

and then conclude the installation process by appending the following to ~/.bashrc 

# STRATUSLAB-CLI-TOOLS
export PATH=$PATH:~/stratuslab/bin
export PYTHONPATH=$PYTHONPATH:~/stratuslab/lib/stratuslab/python

Upload the image into your cloud

This step is different for every cloud platform. For instance, in OpenNebula v3.4+ you can use Sunstone GUI to upload images directly, in previous versions you have to upload the image to the frontend and then register it.

Sice FedCloud-TF will be using OCCI to access the cloud, we have to provide them with a location of the image that is OCCI-compatible. You can use get-occi-link.pl 

perl get-occi-link.pl -host=http://<occi_host> -port=<occi_port> -image="<image_name>"

or find the right link manually by going through all the storage elements registered in your OCCI server 

https://carach5.ics.muni.cz:10443/storage/-/

checking the occi.core.title attribute for the right name. You shloud end up with something like 

https://carach5.ics.muni.cz:10443/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511

Build the metadata

The EGI.eu Marketplace stores only metadata which points to the image, provide basic information and integrity verification. Since RDF is not the most user-friendly format, we can use stratus-build-metadata to generate a template 

stratus-build-metadata --author='##YOUR_NAME##' --type=base --os=Ubuntu --os-version=11.04 --os-arch=x86_64 \
--image-version=1.0 --hypervisor=xen --format=raw --comment='BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##' \ 
--compression=none --location='https://carach5.ics.muni.cz:10443/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511' egi-bncweb.img

Note: stratus-build-metadata needs the image to compute checksums, you can download it here egi-bncweb.img

Modify the metadata

Now we can check/modify the metadata, the most important elements are dcterms:valid and dcterms:title.

The correct format for dcterms:title is EGI-##IMAGE_NAME##-##SITE_NAME##. This field will need to be manually added to the metadata file. You can also modify the validity date as required.

Metadata from the EGI.eu Marketplace cannot be removed, it can only expire. It is also possible to deprecate an entry. This might be necessary, if for example, a security issue is detected with the image, or if you simply wish to no longer endorse the image. Instructions for the stratus-deprecate-image command can be found here. 

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:slterms="http://mp.stratuslab.eu/slterms#"
    xmlns:slreq="http://mp.stratuslab.eu/slreq#"
    xml:base="http://mp.stratuslab.eu/">

    <rdf:Description rdf:about="#DtRwHZzoo1xFKtk-iL51t6RNQ9Q">

        <dcterms:identifier>DtRwHZzoo1xFKtk-iL51t6RNQ9Q</dcterms:identifier>

        <slreq:bytes>14680064000</slreq:bytes>

        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>MD5</slreq:algorithm>
            <slreq:value>144fff2477673aa1d883f0a3ba89f273</slreq:value>
        </slreq:checksum>
        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>SHA-1</slreq:algorithm>
            <slreq:value>3b51c07673a28d7114ab64fa22f9d6de91350f50</slreq:value>
        </slreq:checksum>
        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>SHA-256</slreq:algorithm>
            <slreq:value>8bde348c81e5a2aa5aa51b8d39a30ad137d0482decd5960cd95594d224a45bdd</slreq:value>
        </slreq:checksum>
        <slreq:checksum rdf:parseType="Resource">
            <slreq:algorithm>SHA-512</slreq:algorithm>
            <slreq:value>e780f2aa6922bc7cfdaae4a5e410f6b499bef5c83314bcd760b082b625860834c4942de9d096c7aa83cdad0411c47686f2e7d0fcc65f816475f6525db28b236d</slreq:value>
        </slreq:checksum>

        <slreq:endorsement rdf:parseType="Resource"/>

        <dcterms:title>EGI-BNCweb-##YOUR_SITE##</dcterms:title>
        <dcterms:type>base</dcterms:type>
        <slterms:kind>machine</slterms:kind>

        <slterms:os>Ubuntu</slterms:os>
        <slterms:os-version>11.04</slterms:os-version>
        <slterms:os-arch>x86_64</slterms:os-arch>
        <slterms:version>1.0</slterms:version>
        <dcterms:compression>none</dcterms:compression>
        <slterms:location>https://carach5.ics.muni.cz:10443/storage/a39a1d08-bff8-5a62-ba68-a1cd76bb4511</slterms:location>

        <dcterms:format>raw</dcterms:format>

        <dcterms:creator>##YOUR_NAME##</dcterms:creator>

        <dcterms:created>2012-06-12T12:36:25Z</dcterms:created>
        <dcterms:valid>2012-06-14T12:36:25Z</dcterms:valid>

        <dcterms:description>BNCWeb appliance for the OGF35 demo available at ##YOUR_SITE##</dcterms:description>

        <slterms:hypervisor>xen</slterms:hypervisor>

        <dcterms:publisher>StratusLab</dcterms:publisher>
        
    </rdf:Description>
</rdf:RDF>

Sign the metadata

To establish the origin of the image, we have to sign the metadata with a personal certificate (ideally the one registered with EGI.eu). Before doing this you should familiarise yourself with the EGI Security Policy for the Endorsement and Operation of Virtual Machine Images. 

stratus-sign-metadata --p12-cert=##FULL_PATH_TO_usercred.p12## egi-bncweb.xml

Register the metadata with the EGI.eu Marketplace

And to complete the process, we have to upload the metadata to the EGI.eu Marketplace with stratus-upload-metadata 

stratus-upload-metadata --marketplace-endpoint=marketplace.egi.eu egi-bncweb.xml

or manually at 

http://marketplace.egi.eu/upload
Retrieved from "https://wiki.egi.eu/w/index.php?title=Fedcloud-tf:WorkGroups:VM_Marketplace&oldid=38443"