Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Fedcloud-tf:WorkGroups:Scenario8:AppDB-VA-Marketplace"

From EGIWiki
Jump to navigation Jump to search
Line 260: Line 260:


<br>  
<br>  
::*Set the VMCASTER_EGIAPPDB_AUTHN approprietly (the '''default''' value is '''x509''')
<pre>
  $ export VMCASTER_EGIAPPDB_AUTHN=x509 ## for using x509 as AuthN mechanism ('''default''')
  $ export VMCASTER_EGIAPPDB_AUTHN=sso ## for using EGI SSO as AuthN mechanism
</pre>


::*Edit or create the <code>.vmcaster.cfg</code> file in your home directory and append the following content.
::*Edit or create the <code>.vmcaster.cfg</code> file in your home directory and append the following content.

Revision as of 14:06, 23 December 2013


The mission

Since the 4th quarter of 2013, the EGI Applications Database (hereforth EGI AppDB) has been extended to include a community-driven service namely the Vistual Appliance marketplace, which:

  • stores and provides to the public, Virtual Appliances related metadata
  • acts as distribution mechanism between the submitters, the community and the Resource Providers within the EGI FedCloud infrastructure


Prologue

The EGI AppDB service has been extended to support Virtual Appliances in the form of the Virtual Appliance Marketplace. The Virtual Appliance Marketplace brings about a new category of software entries, called virtual appliances, which are, in all practical manners, clean-and mean virtual machine images designed to run on a virtualization platform, that provide a software solution out-of-the-box, ready to be used with minimal or no set-up needed within the EGI Federated Cloud infrastruture. AppDB's Virtual Appliance Marketplace will provide the ground for managing and publishing versioned repositories of virtual appliances, in a way that integrates with the existing HEPiX VMCaster / VMCatcher framework, currently in use by the EGI. Besides basic features embedded in the AppDB portal itself, such as creating, publishing, enabling/disabling, and archiving or deleting VA versions, it will also use the HEPiX VMCaster command line tool for uploading full VA versions, as well as a web-based dashboard for monitoring the individual uploads injected from the command line.

Context diagram


AppDB Portal: Using the AppDB portal:
  • a typical visitor is able to search, find and download, ready to be used virtual machine images that constitute the latest Virtual Appliance version
  • a VMI submitter/user is able to submit/upload VA version metadata (in the form of signed image lists)
vmcaster@appdb module: This module satisfies three functions:
  1. acts as the AppDB interface for accepting individual uploads injected by a VMI submitter/user using the HEPiX VMCaster command line tool.
  2. offers a web-based dashboard for monitoring the individual uploads injected via the aforementioned tool.
  3. compiles the signed image lists per each published VA version and make them available for subscription by the Resource Providers (using the HEPiX VMCatchertool)
AppDB API: By using the AppDB API, external services and/or applications are able to interact with our system.
AppDB-VM-big-picture.png


Workflow

Useful terms

  • Virtual Appliances (VAs) are ready-to-run virtual machines packaged with an operating system and software application(s)
  • A Virtual Appliance version (VA version) provides a list of the versioned Virtual Machines Images (VMIs) that constitutes the said Virtual Appliance. Each VA version is further distributed and therefore mapped, to a single -- and unique within the EGI AppDB -- Virtual Machine Image list.
  • A Virtual Machine Image List is the SMIME X509 signed, JSON representation of a Virtual Appliance Version.
  • A Virtual Machine Image (VMI) is the software implementation of a machine (i.e. a computer) that executes software application(s) just like a physical machine

The Virtual Appliance Version workflow

Below you may find a graphical representation of the workflow defining the lifecycle of a Virtual Appliance Version.

Appdb-vm-workflow.png


Prerequisites

In order to start using the EGI AppDB as a Virtual Appliance Marketplace you will need to:

  1. Obtain a  valid EGI SSO account AND/OR be owener of a valid X509 certificate signed by an IGTF Certificate Authority
  2. Login to the EGI AppDB portal and claim your profile
  3. Register your Virtual Appliance. The Virtual Appliance instance within the AppDB, provides the high level details of your work and will act as the placeholder for the released Virtual Appliance versions.
  4. In case you would like to acquire write access to an already registered Virtual Appliance, then please make submit a request to 'Join as a contact' to the said VA; upon acceptance by the owner, you will have gained write permissions and therefore be able to modify VA metadata and manage its versions.


Quick start guides & HowTos

The Quick start guides and HowTos listed in this section, cover the usage of the EGI AppDB service as Virtual Appliance Marketplace, either through the AppDB portal (the graphical way) or by using the HEPiX VMCaster command line tool.

Through the AppDB portal

portal: Register a Virtual Appliance (VA)


1. Login to the EGI AppDB .
2. On the top left corner there is the option to Register a New Virtual Appliance
Register new va.png


3. Fill in the mandatory Virtual Appliance high level metadata.
- The VA name - should be unique in the EGI AppDB, so that a unique canonical URL may be assigned
- A short description (one sentence)
- One or more discipline's' that best characterize/classify the newly registered VA. This will help us further promote your work within the specific scientific communities.
- Full description and/or full abstract.
- Optionally, we advise that you, to set the logo that will be displayed under the details dialog of the VA.
Register new va dialog.png


4. Click on the save button and thats it!! Additional information, such as associated VOs, countries, middleware(s), tags, people etc, will be available for edit after saving.


portal: Create a new VA version


1. Click on the Manage VA versions option or alternatively, go directly to the Virtual Appliances Versions tab; both are available at the top of the VA dialog.
Manage va version.png


2. From the VA Versions tab you should be able to either inspect your latest published version, optionally going through previously released versions or to create a new one.
Manage va version options.png


3. Upon navigating to the Working/New Version nested tab and by clicking on the Create New Version button, you will be able to start a new VA version, a clone of the latest published one (if exists), and therefore:


  • add/remove virtual machine images
  • modify the metadata of the existed ones
  • reuse images from previous versions, either 'as is' or for updating them (see more details at the FAQ item When should i reuse an image)
  • modify version wide metadata like version number, expiration date etc.
Manage va version create new.png


4. Add one -or more- new groups and add one or more new images per group.
Manage va version create new group image.png


5. Fill in and/or Update the metadata.
At the time of this writing, the needed metadata are:
  • Version level
    • Version number
    • Expiration date. (When the version will expire)
    • Description
  • Group level
    • A Group title [example: My Scientific Linux images]
    • Group notes (optional) [example: For Root password please contact me.]
  • Image level
    • Image version
    • Location, to a public URI (only http/https protocols are supoorted)
    • Automatic integrity check (ON / OFF[default])
      • if unchecked (OFF), then you will have to provide the SHA512 checksum and the filesize manually
      • if checked (ON), then the system will make an attempt to download the image and calculate the SHA512 checksum and filesize automatically.
For more details see tht FAQ entry What is the Autocheck image file metadata integrity.
    • Title for the Image
    • Description (optional)
    • Comments (optional)
    • OS [example: Linux]
    • OS Version [example: Scientific Linux 6]
    • Architecture [example: x86_64]
    • Format [example: raw]
    • Minimum cores (optional)
    • Recommended cores (optional)
    • Minimum ram (optional)
    • Recommended ram (optional)
    • Hypervisor


7. Click Save.
Manage va version create new save.png


portal: Verify a VA version


This option is available to the user if and only if at least one of the participating images has the Autocheck image file metadata integrity checked.
1. Click on the Verify button or on Verify & Publish to perform verification and publishing in one step.
Manage va version verify.png


2a. If all the images having the Autocheck image file metadata integrity checked, have been successfully verified, then either the VA version is ready to be published or its being directly published in case the Verify & Publish button was initially clicked.
Manage va version verify status-success.png


2a. If at least one image having the Autocheck image file metadata integrity checked, could not be verified, then the associated VA version can not be published. For that image, the sha512 checksum and the filesize should be manually filled.
Manage va version verify status failed.png


3. Upon successful verification the Publish option/button should be available.
Manage va version verify status-publish.png





portal: Publish a VA version


  • Upon clicking on the Publish button:
    • The given VA version becomes the Latest one
    • An image list file is generated and becomes available for the Resource Providers
    • The previously published VA Version gets to the Previous/Archived versions tab and remains there for historical and auditing purposes.


Manage va version publish.png


portal: Delete a VA version


  • As long as a VA version has not been published yet, it may be deleted.(Please note that this action can not be undone.)
Manage va version delete.png


portal: Show/enable or Hide/disable a VA version


  • This operation is available only for published VA versions.
  • The result of using the Show/enable or Hide/disable functions is to:
    • Show/Hide the VA version metadata to/from the public
    • Enable/Disable the VA version image list for the Resource Providers (and from the public in general)
Manage va version show-hide.png



Using the VMCaster command line tool (CLI)


The reader of this section should:


vmcaster CLI: Configure the egiappdb protocol


  • Set the VMCASTER_EGIAPPDB_AUTHN approprietly (the default value is x509)
   $ export VMCASTER_EGIAPPDB_AUTHN=x509 ## for using x509 as AuthN mechanism ('''default''')

   $ export VMCASTER_EGIAPPDB_AUTHN=sso ## for using EGI SSO as AuthN mechanism
  • Edit or create the .vmcaster.cfg file in your home directory and append the following content.
[user@quido-x:~]$ cat ~/.vmcaster.cfg 

[appdb] 
server = "vmcaster.appdb.egi.eu" 
protocol = "egiappdb" 
uriMatch = ".*" 
uriReplace = "egiappdb://_YOUR_EGI_SSO_USERNAME_@vmcaster.appdb.egi.eu/vmlist/submit/" 

## _YOUR_EGI_SSO_USERNAME_ is needed only if you are planning to use the EGI SSO as authentication mechanism. In case that you will use a valid X509 certificate, there is no need for _YOUR_EGI_SSO_USERNAME_ 



vmcaster CLI: Create a new VA version


1. Use the dedicated VMCaster image list identifier produced automatically by the AppDB service (see How to get the VMCaster image list identifier using the AppDB portal).

Notice:
Lets assume that the image list identifier is: 6b470d15-835a-42ec-b568-002393f4dc58



2. Create a new image list as described in the VMCaster documentation, using the aforementioned image list identifier.


3. Add/remove/modify images as described in the VMCaster documentation.

Notice:
Lets assume that the identifier of one of the included images is: 05535860-71b1-48cf-94e3-af05f169be7b



4. Optionally, perform some sort of grouping with the included images.
vmcaster  --select-image _IMAGE_IDENTIFIER_ --key-set-image  "ad:group" --key-value-image "_IMAGE_GROUP_"

full example:

vmcaster  --select-image 05535860-71b1-48cf-94e3-af05f169be7b --key-set-image "ad:group" --key-value-image "My Scientific Linux Images"


5. Visually inspect the image list, as a cross-check.
vmcaster  --select-imagelist _IMAGE_LIST_IDENTIFIER_ --show-imagelist

full example:

vmcaster  --select-imagelist 6b470d15-835a-42ec-b568-002393f4dc58 --show-imagelist


6. Upload/Submit the image list to the vmcaster@appdb end point.
NOTE: vmcaster will prompt you for a passphrase. The passphrase depends on the AuthN mechanism (EGI SSO OR X509) you have choose.
vmcaster  --select-imagelist _IMAGE_LIST_IDENTIFIER_ --upload-imagelist
Enter passphrase:[_YOUR_x509_CERT_PASSPHRASE_]
Appdb password for 'user':[_YOUR_EGI_SSO_PASSWORD_]

full example:

vmcaster  --select-imagelist 6b470d15-835a-42ec-b568-002393f4dc58 ---upload-imagelist


7. Upon completion, the vmcaster tool will prompt with an error message & code in case of a failure or no message in case of success.
In any case, a URL will be provided, directing the user to the vmcaster@appdb Dashboard, where he/she can see more details about the status of the submission.
vmcaster  --select-imagelist 6b470d15-835a-42ec-b568-002393f4dc58 --upload-imagelist
Enter passphrase:[_YOUR_x509_CERT_PASSPHRASE_]
Appdb password for 'user':[_YOUR_EGI_SSO_PASSWORD_]
details:https://vmcaster.appdb.egi.eu/vmlist/status/item/87
submission_id:87


8. If all's gone well, using your favorite web browser, navigate to the EGI AppDB portal and Publish the submitted VA version



vmcaster CLI: Publish a VA version


  • In case you would like to upload and publish you image list in one move
use the flag --flag "egiappdb:publish=true" while uploading.
vmcaster  --select-imagelist _IMAGE_LIST_IDENTIFIER_ --upload-imagelist --flag "egiappdb:publish=true"

full example:

vmcaster  --select-imagelist 6b470d15-835a-42ec-b568-002393f4dc58 --upload-imagelist --flag "egiappdb:publish=true"



Information for the Resource Providers

To be defined.


Frequently Asked Questions (FAQ)

What is the Autocheck image file metadata integrity


Autocheck image file metadata integrity option is available on per image basis and if checked (optional) it will command our system (upon verification - not upon saving) to temporarily download the image from the Location given, to our/AppDB servers, and calculate the sha512 checksum and the filesize. By following this option the user does not have to fill in the respective metadata values manually.
There are quite a few cases that this process could fail. For example:
  • The image is not accessible neither over http nor https protocol (at the time being only http(s) protocol is supported).
  • The http server requires authentication
  • The Location given was not found.
  • etc..
One more thing that the user should be aware of, is the time needed for this process to be finalized. Downloading a couple of hundreds of data over http and calculate the sha512 checksum of the downloaded image file, could be considered as a time consuming process. Depending on the bandwidth of the network it could take from a few seconds up to a few tenths of minutes.
Manage va version autocheck metadata integrity.png

When should i reuse an image?

You are strongly advised to reuse an image that was published in the past, in the following two cases:
A: reuse it 'as is' in the new Virtual Appliance version (use-case: republishing an image)
B: use it as a starting point/template for describing an image update (use-case: image update)
manage_reuse_image.png

How to get the VMCaster image list identifier using the AppDB portal

1. Using the AppDB portal, navigate to the registered Virtual Appliance
2. On the right side of the title pane click on the identifier control.
Cli va id.png



Who has write access to a Virtual Appliance?

Write access to a Virtual Appliance (VA) and its versions, has:
  • The Owner of the VA
  • Everyone that is listed as a contact point,
  • People of the Managers list
  • NGI representatives full control to every VA associated to their country




VMCaster CLI Error Codes

This section lists the resulting errors that might occurred when using the VMCaster tool in order to upload VA versions in the form of signed image lists, to the EGI AppDB service.

  • In any case, please do not hesitate to contact us.
Error Code Description Corrective actions
ERROR_ADVMC_1001_UNAUTHORIZED
1001 You are not authorized to perform this action.
Please follow steps 1 & 2 as described in the Prerequisites section.
ERROR_ADVMC_1002_NO_FILE_SUBMITTED
1002
No image list file submitted.
Please read the section Using the VMCaster command line tool and check out:
  • how the VMCaster should be configured,
  • how it can be used to upload either new VA Versions or updates.
ERROR_ADVMC_1003_NOT_VALID_JSON
1003
The image file you have submitted, does not NOT follow a valid JSON format and can not being further processed.
Check the format of the produced image list.
ERROR_ADVMC_1004_XML_VALIDATOR
1004
Internal error. Not correct or well formed, image list submitted.
It seams that the submitted image list is not valid. Please consider contacting us.

Also please consider sending us either the vmcaster@appdb dashboard reference URL or the submission ID.

ERROR_ADVMC_1005_MISSING_APPDB_ID
1005
Internal error: The application id (ad:appid) is missing.
It seams that the submitted image list is not valid. Please consider contacting us.

Also please consider to send us either the vmcaster@appdb dashboard reference URL or the submission ID.

ERROR_ADVMC_1006_INVALID_IMAGELIST_IDENTIFIER
1006
The provided image list identifier (dc:identifier) is not valid.
You have provide a invalid image list identifier (dc:identifier). Please read  section Using the VMCaster command line tool and the FAQ item: How to get the VMCaster image list identifier using the AppDB portal.
ERROR_ADVMC_1007_NO_VA_APPDB_ID
1007
Internal error. The application id (ad:appid) related with the provided image list identifier (dc:identifier), it is not correct.

It seams that the submitted image list is not valid. Please consider contacting us.

Also please consider to send us either the vmcaster@appdb dashboard reference URL or the submission ID.

ERROR_ADVMC_1008_NO_PERMISSION_TO_THE_VA
1008
You DO NOT have access to submit versions for the given Virtual Appliance.
You are not authorized to perform this action on the given Virtual Appliance.

Please follow the steps 1, 2 and 4 as described at the Prerequisites section.

ERROR_ADVMC_1009_INVALID_WRAPPER_XML
1009
Internal error. Post data are not correct.
It seams that the submitted image list is not valid. Please consider contacting us.

Also please consider to send us either the vmcaster@appdb dashboard reference URL or the submission ID.

ERROR_ADVMC_1010_INVALID_WRAPPER_SCHEMA
1010
Internal error.
ERROR_ADVMC_1011_UKNOWN_ACTION
1011
Internal error. Uknown action submitted.