From EGIWiki
Revision as of 13:54, 25 October 2013 by Mhaggel (talk | contribs)
Jump to: navigation, search

The mission

Since the 4th quarter of 2013, the EGI Applications Database (hereforth EGI AppDB) has been extended to include a community-driven service namely the Vistual Appliance marketplace, which:

  • stores and provides to the public, Virtual Appliances related metadata
  • acts as distribution mechanism between the submitters, the community and the Resource Providers within the EGI FedCloud infrastructure


The EGI AppDB service has been extended to support Virtual Appliances in the form of the Virtual Appliance Marketplace. The Virtual Appliance Marketplace brings about a new category of software entries, called virtual appliances, which are, in all practical manners, clean-and mean virtual machine images designed to run on a virtualization platform, that provide a software solution out-of-the-box, ready to be used with minimal or no set-up needed weithing the EGI Federated Cloud infrastruture. AppDB's Virtual Appliance Marketplace will provide the ground for managing and publishing versioned repositories of virtual appliances, in a way that integrates with the existing HEPiX VMCaster / VMCatcher framework, currently in use by the EGI. Besides basic features embedded in the AppDB portal itself, such as creating, publishing, enabling/disabling, and archiving or deleting VA versions, it will also use the HEPiX VMCaster command line tool for uploading full VA versions, as well as a web-based dashboard for monitoring the individual uploads injected from the command line.

Context diagram

AppDB Portal: Using the AppDB portal:
  • a typical visitor is able to search, find and download, ready to be used virtual machine images that constitute the latest Visrtual Appliance version
  • a VMI submitter/user is able to submit/upload VA version metadata (in the form of signed image lists)
vmcaster@appdb module: This module satisfies three functions:
  1. acts as the AppDB interface for accepting individual uploads injected by a VMI submitter/user using the HEPiX VMCaster command line tool.
  2. offers a web-based dashboard for monitoring the individual uploads injected via the aforementioned tool.
  3. compiles the signed image lists per each published VA version and make them available for subscription by the Resource Providers (using the HEPiX VMCatchertool)
AppDB API: By using the AppDB API, external services and/or applications are able to interact with our system.


Useful terms

  • The Virtual Appliances (VAs) are ready-to-run virtual machines packaged with an operating system and software application(s)
  • A Virtual Appliance version (VA version) provides a list of the versioned Virtual Machines Images (VMIs) that constitutes the said Virtual Appliance. Each VA version is further distributed and therefore mapped, to a single -- and unique within the EGI AppDB -- Virtual Machine Image list.
  • A Virtual Machine Image List is the SMIME X509 signed, JSON representation of a Virtual Appliance Version.
  • A Virtual Machine Image (VMI) is the software implementation of a machine (i.e. a computer) that executes software application(s) just like a physical machine

The Virtual Appliance Version workflow

Below you may find a graphical representation of the workflow defining the lifecycle of a Virtual Appliance Version.



In order to start using the EGI AppDB as a Virtual Appliance Marketplace you will need:

  1. Obtain a  valid EGI SSO account
  2. Login to the EGI AppDB portal and claim your profile
  3. Register your Virtual Appliance. The Virtual Appliance instance within the AppDB, provides the high level details of your work and will act as the placeholder for the released Virtual Appliance versions.
  4. In case you would like to acquire write access to an already registered Virtual Appliance, then please make submit a request to 'Join as a contact' to the said VA; upon acceptance by the owner, you will have gained write permissions and therefore be able to modify VA metadata and manage its versions.

Quick start guides & HowTos

The Quick start guides and HowTos listed in this section, cover the usage of the EGI AppDB service as Virtual Appliance Marketplace, either through the AppDB portal (the graphical way) or by using the HEPiX VMCaster command line tool.

Through the AppDB portal

portal: Register a Virtual Appliance (VA)

1. Login to the EGI AppDB .
2. On the top left corner there is the option to Register a New Virtual Appliance
Register new va.png

3. Fill in the mandatory Virtual Appliance high level metadata.
- The VA name - should be unique in the EGI AppDB, so that a unique canonical URL to be assigned
- A short description (one sentence)
- One or more discipline's' that best characterize/classify the newly registered VA. This will help us further to promote your work within the specific scientific communities.
- Full description and/or full abstract.
- Optionally, we advise that you, to set the logo that will be displayed under the details dialog of the VA.
Register new va dialog.png

4. Click on the save button and thats it!! Additional information, such as associated VOs, countries, middleware(s), tags, people etc, will be available for edit after saving.

portal: Create a new VA version

1. Click on the Manage VA versions option or alternatively, go directly to the Virtual Appliances Versions tab; both are available at the top of the VA dialog.
Manage va version.png

2. From the VA Versions tab you should be able to either inspect your latest published version, optionally going through previously released versions or to create a new one.
Manage va version options.png

3. Upon navigating to the Unpublished/New Version nested tab, you will be prompted to select either one of the: create a new version or create an update of the published one.
Manage va version create options.png

4. Click on the Create New button (Creating an update will be described below)
Manage va version create new.png

5. Add one -or more- new groups and add one or more new images per group.
Manage va version create new group image.png

6. Fill in the metadata.
At the time of this writing, the needed metadata are:
  • Version level
    • Version number
    • Expiration date. (When the version will expire)
    • Description
  • Group level
    • A Group title [example: My Scientific Linux images]
    • Group notes (optional) [example: For Root password please contact me.]
  • Image level
    • Image version
    • Location, to a public URI (only http/https protocols are supoorted)
    • Automatic integrity check (ON / OFF[default])
      • if unchecked (OFF), then you will have to provide the SHA512 checksum and the filesize manually
      • if checked (ON), then the system will make an attempt to download the image and calculate the SHA512 checksum and filesize automatically.
For more details see tht FAQ entry What is the Autocheck image file metadata integrity.
    • Title for the Image
    • Description (optional)
    • Comments (optional)
    • OS [example: Linux]
    • OS Version [example: Scientific Linux 6]
    • Architecture [example: x86_64]
    • Format [example: raw]
    • Minimum cores (optional)
    • Minimum ram (optional)
    • Hypervisor

7. Click Save.
Manage va version create new save.png

portal: Create an update of a VA version

1. Repeat steps 1-3 as described in Create new VA version
2. Click on the Create update button
Manage va version create update.png

3. Edit the metadata at any level (Version, Group or Image) and when done, click on save.
  • You are highly advised to increase at least the revision number by one eg. if your latest version, it is tagged as 1.0.0 you should change it to something like 1.0.1 or 1.0.0-1.
  • In case of an update, you will also be prompted to once more set the expiration date of the new version.
Manage va version edit an update.png

portal: Verify a VA version

This option is available to the user if and only if at least one of the participating images has the Autocheck image file metadata integrity checked.
1. Click on the Verify button or on Verify & Publish to perform verification and publishin in one step.
Manage va version verify.png

2a. If all the images having the Autocheck image file metadata integrity checked, have been successfully verified, then either the VA version is ready to be published or its being directly published in case the Verify & Publish button was initially clicked.
Manage va version verify status-success.png

2a. If at least one image having the Autocheck image file metadata integrity checked, could not be verified, then the associated VA version can not be published. For that image, the sha512 checksum and the filesize should be manually filled.
Manage va version verify status failed.png

3. Upon successful verification the Publish option/button should be available.
Manage va version verify status-publish.png

portal: Publish a VA version

  • Upon clicking on the Publish button:
    • The given VA version becomes the Latest one
    • An image list file is generated and becomes available for the Resource Providers
    • The previously published VA Version gets to the Previous/Archived versions tab and remains there for historical and auditing purposes.

Manage va version publish.png

portal: Delete a VA version

  • As long as a VA version has not been published yet, it may can be deleted.(Please note that this action can not be undone.)
Manage va version delete.png

portal: Show/enable or Hide/disable a VA version

  • This operation is available only for published VA versions.
  • The result of using the Show/enable or Hide/disable functions is to:
    • Show/Hide the VA version metadata to/from the public
    • Enable/Disable the VA version image list for the Resource Providers (and from the public in general)
Manage va version show-hide.png

Using the VMCaster command line tool (CLI)

The reader of this section should:
  • Be familiar with the HEPiX vmcaster and vmcatcher toolset.
  • Have access to a machine with the aforementioned tools, correctly installed.

vmcaster CLI: Configure the egiappdb protocol

  • Edit or create the .vmcaster.cfg file in your home directory and append the following content.
[user@quido-x:~]$ cat ~/.vmcaster.cfg 

server = "vmcaster.appdb.egi.eu" 
protocol = "egiappdb" 
uriMatch = ".*" 
uriReplace = "egiappdb://_YOUR_EGI_SSO_USERNAME_@vmcaster.appdb.egi.eu" 

vmcaster CLI: Create a new VA version

1. Create a new image list as described in the VMCaster documentation.
Lets assume that the identifier of the new image list is: 6b470d15-835a-42ec-b568-002393f4dc58,
and that identifier of one of the included images is: 05535860-71b1-48cf-94e3-af05f169be7b
2. Get the corresponding ID assigned to your Virtual Appliance, using the EGI AppDB portal.
Cli va id.png

3. Associate the produced image list with the fetched VA ID, by issuing the following command:
vmcaster  --select-imagelist _IMAGE_LIST_IDENTIFIER_ --key-set-imagelist  "ad:appid" --key-value-imagelist "_VA_ID_"

full example:

vmcaster  --select-imagelist 6b470d15-835a-42ec-b568-002393f4dc58 --key-set-imagelist "ad:appid" --key-value-imagelist "771"
4. Optionally, perform some sort of grouping with the included images.
vmcaster  --select-image _IMAGE_IDENTIFIER_ --key-set-image  "ad:group" --key-value-image "_IMAGE_GROUP_"

full example:

vmcaster  --select-image 05535860-71b1-48cf-94e3-af05f169be7b --key-set-image "ad:group" --key-value-image "Scientific Linux Images"
5. Visually inspect the image list, as a cross-check.
vmcaster  --select-imagelist _IMAGE_LIST_IDENTIFIER_ --show-imagelist

full example:

vmcaster  --select-imagelist 6b470d15-835a-42ec-b568-002393f4dc58 --show-imagelist
6. Upload/Submit the image list to the vmcaster@appdb end point.
NOTE: vmcaster will prompt you for a passphrase and an 'Appdb password'. The first is for signing your submitted image list and the second is your personal EGI SSO account password.
vmcaster  --select-imagelist _IMAGE_LIST_IDENTIFIER_ --upload-imagelist
Enter passphrase:[_YOUR_x509_CERT_PASSPHRASE_]
Appdb password for 'user':[_YOUR_EGI_SSO_PASSWORD_]

full example:

vmcaster  --select-imagelist 6b470d15-835a-42ec-b568-002393f4dc58 ---upload-imagelist
7. Upon completion, the vmcaster tool will prompt with an error message & code in case of a failure or no message in case of success.
In any case, a URL will be provided, directing the user to the vmcaster@appdb Dashboard, where he/she can see more details about the status of the submission.
vmcaster  --select-imagelist 6b470d15-835a-42ec-b568-002393f4dc58 --upload-imagelist
Enter passphrase:[_YOUR_x509_CERT_PASSPHRASE_]
Appdb password for 'user':[_YOUR_EGI_SSO_PASSWORD_]
8. If all's gone well, using your favorite web browser, navigate to the EGI AppDB portal and Publish the submitted VA version

vmcaster CLI: Create an update of a VA version

  • Starting with an image list that has been successfully submitted & published to the EGI AppDB,
1. add a new or remove/modify an existing image from the given image list, by using the commands as they describe in the HEPiX vmcaster documentation.
2. follow steps 4-8 as described in vmcaster cli: Create new VA version

vmcaster CLI: Publish a VA version

  • Currently this action is not supported through the vmcaster tool but it will in the near future.
In the meantime, you may use the EGI AppDB portal in order to Publish the submitted VA version.

Information for the Resource Providers

To be defined.

Frequently Asked Questions (FAQ)

What is the Autocheck image file metadata integrity

Autocheck image file metadata integrity option is available on per image basis and if checked (optional) it will command our system (upon verification - not upon saving) to temporarily download the image from the Location given, to our/AppDB servers, and calculate the sha512 checksum and the filesize. By following this option the user does not have to fill in the respective metadata values manually.
There are quite a few cases that this process could fail. For example:
  • The image is not accessible neither over http nor https protocol (at the time being only http(s) protocol is supported).
  • The http server requires authentication
  • The Location given was not found.
  • etc..
One more thing that the user should be aware of, is the time needed for this process to be finalized. Downloading a couple of hundreds of data over http and calculate the sha512 checksum of the downloaded image file, could be considered as a time consuming process. Depending on the bandwidth of the network it could take from a few seconds up to a few tenths of minutes.
Manage va version autocheck metadata integrity.png

VMCaster CLI Error Codes

This section lists the resulting errors that might occurred when using the VMCaster tool in order to upload VA versions in the form of signed image lists, to the EGI AppDB service.

  • In any case, please do not hesitate to contact us.
Error Code Description Corrective actions
1001 You are not authorized to perform this action.
Please follow steps 1 & 2 as described in the Prerequisites section.
No image list file submitted.
Please read the section Using the VMCaster command line tool and check out:
  • how the VMCaster should be configured,
  • how it can be used to upload either new VA Versions or updates.
The image file you have submitted, does not NOT follow a valid JSON format and can not being further processed.
Check the format of the produced image list.
Internal error.
It seams that the submitted image list is not valid. Please consider contacting us.

Also please consider sending us either the vmcaster@appdb dashboard reference URL or the submission ID.

The application id (ad:appid) is missing.
You have not provide an appid. Please read section Using the VMCaster command line tool.
The image list identifier (dc:identifier) is not valid.
You have provide a invalid image list identifier (dc:identifier). Please read  section Using the VMCaster command line tool.
The application id (ad:appid) you have provided it is not associated with any Virtual Appliance.
You have not provide a valid appid. The application id (ad:appid) you have provided it is not associated with any Virtual Appliance. Please read section Using the VMCaster command line tool.
You DO NOT have access to submit versions for the given Virtual Appliance.
You are not authorized to perform this action on the given Virtual Appliance.

Please follow the steps 1, 2 and 4 as described at the Prerequisites section.

Internal error. Post data are not correct.
It seams that the submitted image list is not valid. Please consider contacting us.

Also please consider to send us either the vmcaster@appdb dashboard reference URL or the submission ID.

Internal error.
Internal error. Uknown action submitted.