Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Federated Cloud APIs and SDKs"

From EGIWiki
Jump to navigation Jump to search
(Replaced content with "{{Fedcloud_Menu}} {{TOC_right}} Category:Federated_Cloud Please check the new [https://egi-federated-cloud.readthedocs.io/ EGI Federated Cloud documentation] for mor...")
Tag: Replaced
 
(33 intermediate revisions by 3 users not shown)
Line 2: Line 2:
[[Category:Federated_Cloud]]
[[Category:Federated_Cloud]]


This page includes pointers for application development on the EGI Federated Cloud
Please check the new [https://egi-federated-cloud.readthedocs.io/ EGI Federated Cloud documentation] for more information.
 
= Discovery of resources =
 
== Service Endpoints ==
 
[[GOCDB]] maintains the available services in the EGI infrastructure. It offers a [https://goc.egi.eu web based view] and [[GOCDB/PI/Technical_Documentation|API access]] for browsing/querying the available endpoints. Relevant service types for EGI Cloud are (links will bring you directly to the GOCDB XML results for querying that specific service type):
 
* '''[https://goc.egi.eu/gocdbpi/public/?method=get_service_endpoint&service_type=eu.egi.cloud.vm-management.occi eu.egi.cloud.vm-management.occi]''' EGI cloud virtual machine management based on OCCI 1.1 specification http://occi-wg.org
* '''[https://goc.egi.eu/gocdbpi/public/?method=get_service_endpoint&service_type=eu.egi.cloud.storage-management.cdmi eu.egi.cloud.storage-management.cdmi]''' EGI cloud data management interface based on CDMI 1.0.2 specification http://www.snia.org/cdmi
* '''[https://goc.egi.eu/gocdbpi/public/?method=get_service_endpoint&service_type=org.openstack.nova org.openstack.nova]''' OpenStack Nova provides VM management services.
 
You may also check higher-level service types from brokers:
* '''[https://goc.egi.eu/gocdbpi/public/?method=get_service_endpoint&service_type=eu.egi.cloud.broker.compss eu.egi.cloud.broker.compss]''' A super-scalar broker service and programming model for Grids and Clouds. From Barcelona Supercomputing Centre (BSC).
* '''[https://goc.egi.eu/gocdbpi/public/?method=get_service_endpoint&service_type=eu.egi.cloud.broker.proprietary.slipstream eu.egi.cloud.broker.proprietary.slipstream]''' A broker supporting autoscaling and elasticity of virtualised applications in federated Clouds. From SixSq AG.
* '''[https://goc.egi.eu/gocdbpi/public/?method=get_service_endpoint&service_type=eu.egi.cloud.broker.vmdirac eu.egi.cloud.broker.vmdirac]''' A IaaS cloud broker extending Dirac to automated VM based application deployment. From Universitat Autònoma de Barcelona (UAB).
 
== BDII ==
 
The common information system deployed at EGI is based on the Berkeley Database Information Index (BDII) with a hierarchical structure distributed over the whole infrastructure. Information on OCCI resources are published through the [https://github.com/EGI-FCTF/cloud-bdii-provider cloud BDII information provider] including the available endpoints and their characteristics. AppDB queries the EGI Information service to display usage details for instantiating the different sites. This information is also used for operational purposes (e.g. the monitoring depends on certain data to be present in order to properly operate).
 
Information can be gathered from any of the available [[Top-BDII_list_for_NGI|Top BDIIs]].
 
You can use a ldap client to query the BDII. Here are some example queries.
 
List of sites providing IaaS:
ldapsearch -x -H ldap://lcg-bdii.cern.ch:2170 \
    -b GLUE2GroupID=grid,o=glue "GLUE2ServiceType=IaaS" \
    GLUE2ServiceAdminDomainForeignKey
 
OCCI endpoints for a given site <code>SITE</code>:
ldapsearch -x -H ldap://lcg-bdii.cern.ch:2170 \
    -b GLUE2DomainID=<site name>,GLUE2GroupID=grid,o=glue \
    GLUE2EndpointInterfaceName=OCCI GLUE2EndpointUR
 
Get all OCCI resource templates for a given site:
ldapsearch -x -H ldap://lcg-bdii.cern.ch:2170 \
    -b GLUE2DomainID=<site-name>,GLUE2GroupID=grid,o=glue \
    objectClass=GLUE2ExecutionEnvironment GLUE2ResourceID
 
Most of this information is also available through AppDB which provides a web-based interface.
 
== AppDB ==
 
<!--
'''TBC'''
[https://appdb.egi.eu/browse/cloud AppDB cloud marketplace] allows you to browse the available Virtual Appliances.
-->
 
The [https://appdb.egi.eu/ EGI Application Database (AppDB)] has recently evolved its functionalities from its currently catalogue of applications and virtual machines (VMs) to include a Graphical User Interface [https://dashboard.appdb.egi/vmops|(GUI)] allowing authorized users to perform basic VM management operations.
 
=== The VMops dashboard ===
 
==== Highlighted features for end-users ====
 
*User identification
**Based on the aforementioned identification, provide a personalized list of VMs available for the user based on his VO membership and his role within the VO (ex. vm_operator@fedcloud.egi.eu)
*Create a new topology with one or more instances of a single VM
*Attach additional storage to the VM instances
*Deploy/Un-deploy a topology
*Start/Stop a topology (start/stop all the VMs instances of a topology)
*Start/Stop a single VM instance
*Provide personalized view of topologies and/or VM instances
 
==== Highlighted technical features ====
*TOSCA compatible
*[https://appdb.egi.eu/store/software/infrastructure.manager.im Infrastructure Manager (IM)] as the OCCI communication layer
*REST APIs for integration with other services
*Single-Sign-On with the portal and the rest of the AppDB sub-services
*Scalable architecture (one front-end, many back-ends for load balancing)
 
= VM Management =
== OCCI ==
 
OCCI is the standard API for accessing the EGI Federated Cloud resources.
 
=== API ===
 
The [http://occi-wg.org/ Open Cloud Computing Interface (OCCI)] is a RESTful Protocol and API designed to facilitate interoperable access to, and query of, cloud-based resources across multiple resource providers and heterogeneous environments. The formal specification is maintained and actively worked on by OGF’s OCCI-WG.
 
EGI Federated Cloud uses v1.1 of OCCI's specification, which is defined in three documents:
* '''[http://ogf.org/documents/GFD.183.pdf OCCI Core]''' that describes the formal definition of the OCCI core model,
* '''[http://ogf.org/documents/GFD.185.pdf OCCI HTTP Rendering]''' defines how to interact with the OCCI Core Model using the RESTful OCCI API
* '''[http://ogf.org/documents/GFD.184.pdf OCCI Infrastructure]''' contains the definition of the OCCI Infrastructure extension for the IaaS domain
 
The OCCI Infrastructure is further extended with two new mixins for contextualization in EGI Federated Cloud:
 
{| class="wikitable"
|-
! term !! scheme !! attributes
|-
| <code>user_data</code>
| <code><nowiki>http://schemas.openstack.org/compute/instance#</nowiki></code>
| <code>org.openstack.compute.user_data</code>: string that holds base64 encoded data to be available at the VM upon instantiation
|-
| <code>public_key</code>
| <code><nowiki>http://schemas.openstack.org/instance/credentials#</nowiki></code>
| <code>org.openstack.credentials.publickey.name</code>: string with the name of the public key (optional)<br/><code>org.openstack.credentials.publickey.data</code>: string with the public key
|}
 
The upcoming v1.2 of OCCI will be also supported on EGI resources.
 
=== SDKs ===
 
SDKs helps you to create applications that use the EGI Federated Cloud in the language of your choice. If your language or use case is not supported by the following SDKs, you can use directly the API. Requests for new SDKs are welcomed.
 
==== Java ====
 
; [https://github.com/EGI-FCTF/jOCCI-api jOCCI]
: jOCCI is an independent OCCI implementation by the authors of rOCCI. It satisfies the demand for a Java library, replacing the previous alternative based on JRuby. Currently in development, it is available as source from Github. Check this [[EGI_Federated_Cloud_jOCCI_APIs|detailed guide on how to use jOCCI with FedCloud]].
 
==== Ruby ====
 
; [https://github.com/EGI-FCTF/rOCCI-api rOCCI]
: rOCCI framework implements the OCCI class structure in Ruby, which allows developers to work with OCCI concepts natively. rOCCI-api takes care of transporting OCCI messages, currently supporting HTTP as transport protocol. rOCCI-api handles EGI Fedederated Cloud authentication on behalf of the developer.
 
=== CLI ===
 
[[rOCCI:ROCCI|rOCCI]] offers a Command-Line Interface to interact with OCCI compliant servers with support to EGI Federated Cloud authentication and extensions to the standard (contextualization). [[HOWTO11|HOWTO 11]] contains detailed instructions on how to install and use it.
 
== OpenStack ==
 
OpenStack sites of the EGI Federated Cloud can provide access through the native OpenStack API.
 
=== API ===
 
OpenStack API documentation is available at [http://developer.openstack.org/ OpenStack developer pages]. EGI Federated Cloud supports the usage of the [http://developer.openstack.org/api-ref-compute-v2.1.html Compute (nova) v2.1 API]. Using the nova API requires a valid token first, that can be obtained as [[#Keystone_VOMS_authentication|described below]].
 
 
=== SDKs ===
 
==== python ====
; OpenStackClient
: [http://docs.openstack.org/developer/python-openstackclient/ OpenStackClient (aka OSC)] can be used as SDK. Check [[#CLI_2|below]] how to install the plugin for VOMS authentication.
 
<!--
; libcloud
: we are evaluating the effort to provide support for libcloud
-->
 
<!--
=== Java ===
; jclouds
: We are working on a jclouds extension that allows to use OpenStack resources with X.509 proxies for authentication
-->
 
=== CLI ===
 
[http://docs.openstack.org/developer/python-openstackclient/ OpenStackClient (aka OSC)] is a command-line client for OpenStack that brings the command set for Compute, Identity, Image, Object Store and Volume APIs together in a single shell with a uniform command structure. A [https://github.com/enolfc/openstack-voms-auth-type VOMS authentication plugin] is available for this CLI, you can install it with pip:
 
pip install openstack-voms-auth-type
 
For using VOMS authentication you need to specify params <code>--os-auth-type</code> with <code>v2voms</code> and <code>--os-x509-user-proxy</code> with the location of your proxy:
 
openstack --os-auth-type v2voms --os-x509-user-proxy /tmp/x509up_u1001
 
A complete command line for getting the list of VMs on a site follows (os_auth_url and os_project_name needs to be adjusted for each endpoint):
<pre>
openstack --os-auth-type v2voms --os-x509-user-proxy  /tmp/x509up_u1001 \
          --os-auth-url https://keystone.ifca.es:5000/v2.0 \
          --os-project-name VO:fedcloud.egi.eu  \
          server list
</pre>
 
In order to get the proper value for the project name,  you can use the following command line:
<pre>
openstack --os-auth-url <keystone_url> \
          --os-auth-type v2voms \
          --os-x509-user-proxy /tmp/x509up_u1001 \
          project list
</pre>
 
Keystone URL depends on the resource provider and is published in GOCDB
 
==== CA Certificates ====
 
If you get SSL errors when using the OpenStack CLI, this is probably due to missing CA certificates (or badly configured). This can be avoided by using the <code>--insecure</code> options to the command but this is not recommended. requests is the python library used by the client to connect to the different endpoints and this library uses a CA certificates bundle. Depending on how you have installed the requests library this bundle will be located in different places, check with the following command:
 
<pre>
python -m requests.certs
</pre>
 
If the output of that command is <code>/etc/ssl/certs/ca-certificates.crt</code> (Debian derivatives) or <code>/etc/pki/tls/certs/ca-bundle.crt</code> (RH derivatives) you are using system requests library which uses the default's bundle. If the output is similar to <code>/usr/local/lib/python2.7/dist-packages/requests/cacert.pem</code>, then you are using internal requests bundle. The procedure for adding the CAs for each of these cases is described below. If you don't have the CA certificates installed in your machine, you can get them from [http://repository.egi.eu/?category_name=cas UMD EGI Core Trust Anchor Distribution].
 
===== System requests library (debian/ubuntu) =====
<pre>
cd /usr/local/share/ca-certificates
for f in /etc/grid-security/certificates/*.pem ; do ln -s $f $(basename $f .pem).crt; done
update-ca-certificates
</pre>
 
===== System requests library (RH derivatives) =====
<pre>
cd /etc/pki/ca-trust/source/anchors
ln -s /etc/grid-security/certificates/*.pem .
update-ca-trust extract
</pre>
 
===== Requests internal bundle =====
<pre>
cat /etc/grid-security/certificates/*.pem >> $(python -m requests.certs)
</pre>
 
== Authentication ==
 
EGI Federated Cloud uses X.509 proxy certificates with VOMS extensions, but each provider in the federation has its own mechanisms to accept those credentials. To avoid dealing with the differences in authentication for each provider we recommend using the SDKs and CLIs described above that automatically handle the differences. The current mechanisms to get access to the services are:
 
; rOCCI server (OpenNebula)
: rOCCI expects that the connection to the endpoint is done with the user proxy as client-side certificate. rOCCI will authenticate/authorize the client with that certificate.
 
; OCCI-OS/ooi/snf-occi (OpenStack/Synnefo)
: These systems expect that a valid token is sent to the endpoint in the <code>X-Auth-Token</code> header, if no valid token is found, the endpoint should return in the <code>WWW-Authenticate</code> header the URI of the Keystone service used to obtain the tokens in the folowing form: <code>WWW-Authenticate: Keystone uri=&lt;KEYSTONE URI&gt;</code> where &lt;KEYSTONE URI&gt; is the location of the server (e.g. <code><nowiki>https://keystone.ifca.es:5000/</nowiki></code>). Keystone supports several authentication methods, see next section on how to get a token with a proxy certificate with Keystone.
 
=== Keystone VOMS authentication ===
 
[http://ifca.github.io/keystone-voms/ Keystone-VOMS] supports getting tokens with X.509 proxy certificates with proxy extensions. In order to get an unscoped token, you must '''POST''' a JSON request to  <code>/v2.0/tokens</code> of your Keystone server with the proxy as client-side certificate and with the following body:
<pre>
{
    "auth": {
        "voms": "true"
    }
}
</pre>
 
Response from server is documented at [http://developer.openstack.org/api-ref-identity-v2.html#authenticate-v2.0 Keystone API v2 Authenticate]. Unscoped tokens allow discovery of the supported tenants for the user (check [http://developer.openstack.org/api-ref-identity-v2.html#listTenants Keystone API v2, list tenants]) but not the usage of other OpenStack services. For that you will need to get a scoped one by submitting a '''POST''' with a JSON document like this one:
<pre>
{
    "auth": {
        "voms": "true",
        "tenantName": "TenantForTheVo",
    }
}
</pre>
 
See an example of the complete authentication process at the [http://keystone-voms.readthedocs.org/en/stable/test.html Keystone-VOMS documentation]

Latest revision as of 14:12, 3 March 2020