Federated Cloud OpenStack Providers
Overview | For users | For resource providers | Infrastructure status | Site-specific configuration | Architecture |
OpenStack is the cloud management framework powering most of the EGI Federated Cloud providers. These providers offer native OpenStack APIs with EGI-specific AAI integration so you can have Single Sign-On across all the federation.
This page describes how to access OpenStack providers of the EGI Cloud Compute service with different tools. For providers joining the federation, check the integration documentation for details
Authentication
EGI Check-in
Providers natively integrated with EGI Check-in support OpenID Connect for authentication on Keystone with the OS-FEDERATION extension of Keystone v3 API. The OS-FEDERATION has a set of API calls that allow getting an unscoped token and Keystone assume the URLs for these calls to be protected by some authentication mechanism (SAML, OpenID Connect, ...) that will make sure the user is valid. Once the user is granted by the underlying authentication, Keystone will perform the authorisation based on a mapping configured by the provider. Mappings restrict the allowed users and maps them to local groups or specific local accounts. If the mapping is successful, an unscoped token will be returned. This is a regular OpenStack unscoped token that can be scoped to any of the allowed projects as with any other authentication mechanism.
Obtaining an access token
Access tokens can obtained via several mechanisms, usually involving the use of a web server and a browser. Command line clients/APIs without access to a browser or interactive prompt for user authentication can use refresh tokens. A refresh token is a special token that is used to generate additional access tokens. This allows you to have short-lived access tokens without having to collect credentials every single time one expires. You can request this token alongside the access and/or ID tokens as part of a user’s initial authentication flow.
In the case of EGI Check-in, we have created a special client meant to obtain your personal refresh token and client credentials that will allow the obtention of access tokens as needed. You can access the client at https://aai.egi.eu/fedcloud/ and click on 'Authorise' to log in with your Check-in credentials to obtain:
- a client id
- a client secret
- a refresh token
All of them can be used to obtain the needed access token:
$ curl -X POST -u '<client id>':'<client secret>' \ -d 'client_id=<client id>&<client secret>&grant_type=refresh_token&refresh_token=<refresh token>&scope=openid%20email%20profile' \ 'https://aai.egi.eu/oidc/token' | python -m json.tool; % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2066 0 1743 100 323 2970 550 --:--:-- --:--:-- --:--:-- 2974 { "access_token": "<your access token>, "expires_in": 3599, "id_token": "<your id token>", "scope": "openid profile email", "token_type": "Bearer" }
Authentication using OpenStack Client
Once you have your access token, you can easily use it for authenticating with the OpenStack client using the v3oidcaccesstoken
auth type as follows:
$ openstack --auth-type v3oidcaccesstoken \ --os-protocol oidc --identity-provider egi.eu \ --os-auth-url <keystone url> \ --os-access-token <your access token> \ --os-project-id <your project id> <command..>
Discovering projects
You may not know which projects are available to your user at a given provider, but you can easily discover them using the Keystone API. The following python code will just do that for you (requires requests library). It expects these variables to be defined in the environment:
The script expects your credentials to be available in the environment:
CHECKIN_CLIENT_ID
: Your Check-in client id (get it from https://aai.egi.eu/fedcloud)CHECKIN_CLIENT_SECRET
: Your Check-in client secret (get it from https://aai.egi.eu/fedcloud)CHECKIN_REFRESH_TOKEN
: Your Check-in refresh token (get it from https://aai.egi.eu/fedcloud)OS_AUTH_URL
: Keystone URL (depends on the provider, you can get it in GOC-DB)
#!/usr/bin/env python # in case someone is trying this with python 2 from __future__ import print_function import base64 import json import os import requests # URL Parse try: # Python 2.x from urlparse import urlparse, urlunparse except ImportError: # Python 3.x from urllib.parse import urlparse, urlunparse def get_access_token(client_id, client_secret, refresh_token): refresh_data = { 'client_id': client_id, 'client_secret': client_secret, 'grant_type': 'refresh_token', 'refresh_token': refresh_token, 'scope': 'openid email profile', } r = requests.post("https://aai.egi.eu/oidc/token", auth=(client_id, client_secret), data=refresh_data) return r.json()['access_token'] def get_keystone_url(os_auth_url, path): url = urlparse(os_auth_url) prefix = url.path.rstrip('/') if prefix.endswith('v2.0') or prefix.endswith('v3'): prefix = os.path.dirname(prefix) path = os.path.join(prefix, path) return urlunparse((url[0], url[1], path, url[3], url[4], url[5])) def get_unscoped_token(os_auth_url, access_token): url = get_keystone_url( os_auth_url, "/v3/OS-FEDERATION/identity_providers/egi.eu/protocols/oidc/auth") r = requests.post(url, headers={'Authorization': 'Bearer %s' % access_token}) return r.headers['X-Subject-Token'] def get_projects(os_auth_url, unscoped_token): url = get_keystone_url(os_auth_url, "/v3/auth/projects") r = requests.get(url, headers={'X-Auth-Token': unscoped_token}) return r.json()['projects'] def main(): # read from environment client_id = os.environ.get('CHECKIN_CLIENT_ID', '') client_secret = os.environ.get('CHECKIN_CLIENT_SECRET', '') refresh_token = os.environ.get('CHECKIN_REFRESH_TOKEN', '') os_auth_url = os.environ.get('OS_AUTH_URL', '') access_token = get_access_token(client_id, client_secret, refresh_token) projects = get_projects(os_auth_url, get_unscoped_token(os_auth_url, access_token)) for p in projects: print('ID: %(id)s - Name: %(name)s - Enabled: %(enabled)s' % p) if __name__ == '__main__': main()
Sample usage:
$ export CHECKIN_CLIENT_ID=xxx $ export CHECKIN_CLIENT_SECRET=secret $ export CHECKIN_REFRESH_TOKEN=othersecret $ export OS_AUTH_URL=OS_AUTH_URL=https://sbgcloud.in2p3.fr:5000 $ python get-projects.py ID: a5eb30bba2c2497b90645fb199e34b39 - Name: EGI_FCTF - Enabled: True
Obtaining OpenStack tokens for other tools
Most OpenStack client allow authentication with tokens, so you can easily use them with EGI Cloud providers just doing a first step for obtaining the token. With the OpenStack client you can use the following command to set the OS_TOKEN variable with the needed token:
$ OS_TOKEN=$(openstack --auth-type v3oidcaccesstoken \ --os-protocol oidc --identity-provider egi.eu \ --os-auth-url <keystone url> \ --os-access-token <your access token> \ --os-project-id <your project id> token issue -c id -f value)
You can refresh the access token and obtain an OpenStack token in a single script similar to this (python requires requests):
#!/usr/bin/env python # in case someone is trying this with python 2 from __future__ import print_function import base64 import json import os import requests # URL Parse try: # Python 2.x from urlparse import urlparse, urlunparse, urljoin except ImportError: # Python 3.x from urllib.parse import urlparse, urlunparse, urljoin def get_access_token(checkin_url, client_id, client_secret, refresh_token): refresh_data = { 'client_id': client_id, 'client_secret': client_secret, 'grant_type': 'refresh_token', 'refresh_token': refresh_token, 'scope': 'openid email profile', } r = requests.post(urljoin(checkin_url, "oidc/token"), auth=(client_id, client_secret), data=refresh_data) return r.json()['access_token'] def get_keystone_url(os_auth_url, path): url = urlparse(os_auth_url) prefix = url.path.rstrip('/') if prefix.endswith('v2.0') or prefix.endswith('v3'): prefix = os.path.dirname(prefix) path = os.path.join(prefix, path) return urlunparse((url[0], url[1], path, url[3], url[4], url[5])) def get_unscoped_token(os_auth_url, access_token): url = get_keystone_url( os_auth_url, "/v3/OS-FEDERATION/identity_providers/egi.eu/protocols/oidc/auth") r = requests.post(url, headers={'Authorization': 'Bearer %s' % access_token}) return r.headers['X-Subject-Token'] def get_scoped_token(os_auth_url, os_project_id, unscoped_token): url = get_keystone_url(os_auth_url, "/v3/auth/tokens") token_body = { "auth": { "identity": { "methods": ["token"], "token": {"id": unscoped_token} }, "scope": {"project": {"id": os_project_id}} } } r = requests.post(url, headers={'content-type': 'application/json'}, data=json.dumps(token_body)) return r.headers['X-Subject-Token'] def main(): # read from environment checkin_url = os.environ.get('CHECKIN_URL', 'https://aai.egi.eu') client_id = os.environ.get('CHECKIN_CLIENT_ID', '') client_secret = os.environ.get('CHECKIN_CLIENT_SECRET', '') refresh_token = os.environ.get('CHECKIN_REFRESH_TOKEN', '') os_auth_url = os.environ.get('OS_AUTH_URL', '') os_project_id = os.environ.get('OS_PROJECT_ID', '') access_token = get_access_token(checkin_url, client_id, client_secret, refresh_token) token = get_scoped_token(os_auth_url, os_project_id, get_unscoped_token(os_auth_url, access_token)) print(token, end='') if __name__ == '__main__': main()
The script expects your credentials to be available in the environment:
CHECKIN_CLIENT_ID
: Your Check-in client id (get it from https://aai.egi.eu/fedcloud)CHECKIN_CLIENT_SECRET
: Your Check-in client secret (get it from https://aai.egi.eu/fedcloud)CHECKIN_REFRESH_TOKEN
: Your Check-in refresh token (get it from https://aai.egi.eu/fedcloud)OS_AUTH_URL
: Keystone URL (depends on the provider, you can get it in GOC-DB)OS_PROJECT_ID
: OpenStack project to use (See script above for obtaining it)
Optionally set the CHECKIN_URL
to the Check-in endpoint (https://aai-dev.eu.eu/
if testing on the devel environment)
Sample usage (script named get-token.py):
export CHECKIN_CLIENT_ID=xxx export CHECKIN_CLIENT_SECRET=secret export CHECKIN_REFRESH_TOKEN=othersecret export OS_AUTH_URL=OS_AUTH_URL=https://sbgcloud.in2p3.fr:5000 export OS_PROJECT_ID=a5eb30bba2c2497b90645fb199e34b39 export OS_TOKEN=$(python get-token.py)
Detailed workflow
You can safely skip this section if not interested in the internals.
Clients willing to authenticate with an OpenStack provider should perform:
- a request to
http://<keystone_url>/v3/OS-FEDERATION/identity_providers/egi.eu/protocols/oidc/auth
with the headerAuthentication: Bearer <access token>
, where<access token>
is a valid OAuth2.0 Access Token obtained from EGI Check-in. This call if successful will return aX-Subject-Token
header with an unscoped token to be used in subsequent calls.
curl -i -H 'Authorization: Bearer <OAuth2.0 acces token>' \ https://<keystone_url>/v3/OS-FEDERATION/identity_providers/egi.eu/protocols/oidc/auth HTTP/1.1 201 Created Date: Wed, 01 Aug 2018 15:23:15 GMT Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 X-Subject-Token: <unscoped token> Vary: X-Auth-Token x-openstack-request-id: req-fae2789d-35ba-4020-9487-378ba1e3fafb Content-Length: 491 Content-Type: application/json {"token": {"issued_at": "2018-08-01T15:23:15.000000Z", "audit_ids": ["j-B2cjhwQ9ePBjJdntWnvw"], "methods": ["oidc"], "expires_at": "2018-08-01T16:23:15.000000Z", "user": {"OS-FEDERATION": {"identity_provider": {"id": "egi.eu"}, "protocol": {"id": "oidc"}, "groups": [{"id": "c526ee5d3d224c0780f7877b3597d014"}]}, "domain": {"id": "Federated", "name": "Federated"}, "id": "84b1c6812ab746c89e282ac854824712", "name": "529a87e5ce04cd5ddd7161734d02df0e2199a11452430803e714cb1309cc3907@egi.eu"}}}
- A request to
https://<keystone url>/v3/auth/projects
get supported projects with the obtained token in aX-Auth-Token
header.
curl -H 'X-Auth-Token: <unscoped token>' https://<keystone-url>/v3/auth/projects | python -mjson.tool % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 384 100 384 0 0 1534 0 --:--:-- --:--:-- --:--:-- 1536 { "links": { "next": null, "previous": null, "self": "https://<keystone url>/v3/auth/projects" }, "projects": [ { "description": null, "domain_id": "default", "enabled": true, "id": "a5eb30bba2c2497b90645fb199e34b39", "is_domain": false, "links": { "self": "https://<keystone url>/v3/projects/a5eb30bba2c2497b90645fb199e34b39" }, "name": "EGI_FCTF", "parent_id": "default" } ] }
- Lastly, the token can be scoped to a project with the information from the previous call by performing a
POST
request tohttps://<keystone url>/v3/auth/tokens
with a json body as shown below and will return the scoped token in aX-Subject-Token
header:
{ "auth": { "identity": { "methods": [ "token" ], "token": { "id": "<unscped token>" } }, "scope": { "project": { "id": "<project id>" } } } }
Sample curl command:
curl -i -H 'content-type: application/json' -X POST \ -d '{"auth": {"identity": {"methods": ["token"], "token":{ "id": "token"}}, "scope": {"project": {"id": "project id"}}}}' \ https://<keystone url>/v3/auth/tokens HTTP/1.1 201 Created Date: Wed, 01 Aug 2018 15:45:34 GMT Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 X-Subject-Token: <your scoped token> Vary: X-Auth-Token x-openstack-request-id: req-e2d54fa1-e3d2-451d-a5a4-e5ee22f7d8c3 Content-Length: 8464 Content-Type: application/json <...extra token information returned by keystone...>
Legacy VOs (VOMS)
Keystone-VOMS supports getting tokens with X.509 proxy certificates with VOMS extensions. You can either use directly the OpenStack CLI tools with a custom plugin or get tokens programmatically via the API.
OpenStack CLI Plugin
OpenStackClient (aka OSC) is a command-line client for OpenStack that brings the command set for Compute, Identity, Image, Object Store and Volume APIs together in a single shell with a uniform command structure. A VOMS authentication plugin is available for this CLI, you can install it with pip:
pip install openstack-voms-auth-type
For using VOMS authentication you need to specify params --os-auth-type
with v2voms
and --os-x509-user-proxy
with the location of your proxy:
openstack --os-auth-type v2voms --os-x509-user-proxy /tmp/x509up_u1001
Get the projects that support your VO with (<keystone_url>
needs to be adjusted for each endpoint):
openstack --os-auth-url <keystone_url> \ --os-auth-type v2voms \ --os-x509-user-proxy /tmp/x509up_u1001 \ project list
And get a valid access token to interact with other services with (in this example https://keystone.ifca.es:5000/v2.0
is used as the Keystone URL and VO:fedcloud.egi.eu
as the project):
openstack --os-auth-type v2voms --os-x509-user-proxy /tmp/x509up_u1001 \ --os-auth-url https://keystone.ifca.es:5000/v2.0 \ --os-project-name VO:fedcloud.egi.eu \ token issue
Other options of the CLI should also work as expected, for example for getting a list of VMs on a site:
openstack --os-auth-type v2voms --os-x509-user-proxy /tmp/x509up_u1001 \ --os-auth-url https://keystone.ifca.es:5000/v2.0 \ --os-project-name VO:fedcloud.egi.eu \ server list
API
In order to get an unscoped token, you must POST a JSON request to /v2.0/tokens
of your Keystone server with the proxy as client-side certificate and with the following body:
{ "auth": { "voms": "true" } }
Response from server is documented at Keystone API v2 Authenticate. Unscoped tokens allow discovery of the supported tenants for the user (check Keystone API v2, list tenants) but not the usage of other OpenStack services. For that you will need to get a scoped one by submitting a POST with a JSON document like this one:
{ "auth": { "voms": "true", "tenantName": "TenantForTheVo", } }
See an example of the complete authentication process at the Keystone-VOMS documentation
Other OpenStack clients
Terraform
Terraform OpenStack provider accepts tokens for authentication and will use the value in the OS_TOKEN
environment variable if defined. Just configure your provider, but do not include user/password informaton:
# Configure the OpenStack Provider provider "openstack" { project_id = "xxxx" auth_url = "http://keystone_url:5000/" } # Create a server resource "openstack_compute_instance_v2" "test-server" { # ... }
when launching Terraform, set the OS_TOKEN environment variable to a valid OpenStack token. You can also set OS_AUTH_URL and OS_PROJECT_ID in the environment and not set it in the configuration file as shown below:
$ cat simple.tf provider "openstack" { } data "openstack_images_image_v2" "ubuntu16" { most_recent = true properties { APPLIANCE_MPURI = "https://appdb.egi.eu/store/vo/image/8df7ba00-8467-57aa-bf1e-05754a2a73bf:6428/" } } data "openstack_compute_flavor_v2" "small" { vcpus = 1 ram = 2048 disk = 20 } resource "openstack_compute_instance_v2" "vm" { name = "testvm" image_id = "${data.openstack_images_image_v2.ubuntu16.id}" flavor_id = "${data.openstack_compute_flavor_v2.small.id}" security_groups = ["default"] } $ export CHECKIN_CLIENT_ID=xxx $ export CHECKIN_CLIENT_SECRET=secret $ export CHECKIN_REFRESH_TOKEN=othersecret $ export OS_AUTH_URL=https://sbgcloud.in2p3.fr:5000 $ export OS_PROJECT_ID=a5eb30bba2c2497b90645fb199e34b39 $ export OS_TOKEN=$(python get-token.py) $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.openstack_compute_flavor_v2.small: Refreshing state... data.openstack_images_image_v2.ubuntu16: Refreshing state... ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: + openstack_compute_instance_v2.vm id: <computed> access_ip_v4: <computed> access_ip_v6: <computed> all_metadata.%: <computed> availability_zone: <computed> flavor_id: "2" flavor_name: <computed> force_delete: "false" image_id: "ceb0434d-37af-4d1f-9efe-13f6f9937df2" image_name: <computed> name: "testvm" network.#: <computed> power_state: "active" region: <computed> security_groups.#: "1" security_groups.3814588639: "default" stop_before_destroy: "false" Plan: 1 to add, 0 to change, 0 to destroy. ------------------------------------------------------------------------ Note: You didn't specify an "-out" parameter to save this plan, so Terraform can't guarantee that exactly these actions will be performed if "terraform apply" is subsequently run.
Note that as in the example above you can get images using information from AppDB if needed.
libcloud
Apache libcloud supports OpenStack and both the EGI Check-in and VOMS mechanisms for authentication against the resources by setting ex_force_auth_version
to 3.x_oidc_access_token
or 2.0_voms
respectively. Check the documentation on connecting to OpenStack for details. See below two code samples for using them
EGI Check-in
import requests from libcloud.compute.types import Provider from libcloud.compute.providers import get_driver refresh_data = { 'client_id': '<your client_id>', 'client_secret': '<your client_secret>', 'grant_type': 'refresh_token', 'refresh_token': '<your refresh_token>', 'scope': 'openid email profile', } r = requests.post("https://aai.egi.eu/oidc/token", auth=(client_id, client_secret), data=refresh_data) access_token = r.json()['access_token'] OpenStack = get_driver(Provider.OPENSTACK) # first parameter is the identity provider: "egi.eu" # Second parameter is the access_token # The protocol 'oidc' is specified in ex_tenant_name, and tenant/project cannot be selected :( driver = OpenStack('egi.eu', access_token, ex_tenant_name='oidc', ex_force_auth_url='https://keystone_url:5000', ex_force_auth_version='3.x_oidc_access_token')
VOMS
from libcloud.compute.types import Provider from libcloud.compute.providers import get_driver OpenStack = get_driver(Provider.OPENSTACK) # assume your proxy is available at /tmp/x509up_u1000 # you can obtain a proxy with the voms-proxy-init command # no need for username driver = OpenStack(None, '/tmp/x509up_u1000', ex_tenant_name='EGI_FCTF', ex_force_auth_url='https://sbgcloud.in2p3.fr:5000', ex_force_auth_version='2.0_voms')