Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Federated Cloud OpenStack Providers

From EGIWiki
Jump to navigation Jump to search
Overview For users For resource providers Infrastructure status Site-specific configuration Architecture




OpenStack is the cloud management framework powering most of the EGI Federated Cloud providers. These providers offer native OpenStack APIs with EGI-specific AAI integration so you can have Single Sign-On across all the federation.

This page describes how to access OpenStack providers of the EGI Cloud Compute service with different tools. For providers joining the federation, check the integration documentation for details

Authentication

EGI Check-in

Providers natively integrated with EGI Check-in support OpenID Connect for authentication on Keystone with the OS-FEDERATION extension of Keystone v3 API. The OS-FEDERATION has a set of API calls that allow getting an unscoped token and Keystone assume the URLs for these calls to be protected by some authentication mechanism (SAML, OpenID Connect, ...) that will make sure the user is valid. Once the user is granted by the underlying authentication, Keystone will perform the authorisation based on a mapping configured by the provider. Mappings restrict the allowed users and maps them to local groups or specific local accounts. If the mapping is successful, an unscoped token will be returned. This is a regular OpenStack unscoped token that can be scoped to any of the allowed projects as with any other authentication mechanism.

Obtaining an access token

Access tokens can obtained via several mechanisms, usually involving the use of a web server and a browser. Command line clients/APIs without access to a browser or interactive prompt for user authentication can use refresh tokens. A refresh token is a special token that is used to generate additional access tokens. This allows you to have short-lived access tokens without having to collect credentials every single time one expires. You can request this token alongside the access and/or ID tokens as part of a user’s initial authentication flow.

In the case of EGI Check-in, we have created a special client meant to obtain your personal refresh token and client credentials that will allow the obtention of access tokens as needed. You can access the client at https://aai.egi.eu/fedcloud/ and click on 'Authorise' to log in with your Check-in credentials to obtain:

  • a client id
  • a client secret
  • a refresh token

All of them can be used to obtain the needed access token:

$ curl -X POST -u '<client id>':'<client secret>'  \
       -d 'client_id=<client id>&<client secret>&grant_type=refresh_token&refresh_token=<refresh token>&scope=openid%20email%20profile' \
       'https://aai.egi.eu/oidc/token' | python -m json.tool;
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2066    0  1743  100   323   2970    550 --:--:-- --:--:-- --:--:--  2974
{
    "access_token": "<your access token>,
    "expires_in": 3599,
    "id_token": "<your id token>",
    "scope": "openid profile email",
    "token_type": "Bearer"
}

Authentication using OpenStack Client

Once you have your access token, you can easily use it for authenticating with the OpenStack client using the v3oidcaccesstoken auth type as follows:

$ openstack --auth-type v3oidcaccesstoken \
            --os-protocol oidc --identity-provider egi.eu \
            --os-auth-url <keystone  url> \
            --os-access-token <your access token> \
            --os-project-id <your project id> <command..>

Discovering projects

You may not know which projects are available to your user at a given provider, but you can easily discover them using the Keystone API. The following python code will just do that for you (requires requests library). It expects these variables to be defined in the environment:

The script expects your credentials to be available in the environment:

#!/usr/bin/env python

# in case someone is trying this with python 2
from __future__ import print_function

import base64
import json
import os
import requests

# URL Parse
try:
    # Python 2.x
    from urlparse import urlparse, urlunparse
except ImportError:
    # Python 3.x
    from urllib.parse import urlparse, urlunparse



def get_access_token(client_id, client_secret, refresh_token):
    refresh_data = {
        'client_id': client_id, 
        'client_secret': client_secret,
        'grant_type': 'refresh_token',
        'refresh_token': refresh_token,
        'scope': 'openid email profile',
    }
    r = requests.post("https://aai.egi.eu/oidc/token",
                      auth=(client_id, client_secret), data=refresh_data)
    return r.json()['access_token']

def get_keystone_url(os_auth_url, path):
    url = urlparse(os_auth_url)
    prefix = url.path.rstrip('/')
    if prefix.endswith('v2.0') or prefix.endswith('v3'):
        prefix = os.path.dirname(prefix)
    path = os.path.join(prefix, path)
    return urlunparse((url[0], url[1], path, url[3], url[4], url[5]))


def get_unscoped_token(os_auth_url, access_token):
    url = get_keystone_url(
        os_auth_url,
        "/v3/OS-FEDERATION/identity_providers/egi.eu/protocols/oidc/auth")
    r = requests.post(url,
                      headers={'Authorization': 'Bearer %s' % access_token})
    return r.headers['X-Subject-Token']

def get_projects(os_auth_url, unscoped_token):
    url = get_keystone_url(os_auth_url, "/v3/auth/projects")
    r = requests.get(url, headers={'X-Auth-Token': unscoped_token})
    return r.json()['projects']


def main():
    # read from environment
    client_id = os.environ.get('CHECKIN_CLIENT_ID', '')
    client_secret = os.environ.get('CHECKIN_CLIENT_SECRET', '')
    refresh_token = os.environ.get('CHECKIN_REFRESH_TOKEN', '')

    os_auth_url = os.environ.get('OS_AUTH_URL', '')

    access_token = get_access_token(client_id, client_secret, refresh_token)
    projects = get_projects(os_auth_url,
                            get_unscoped_token(os_auth_url, access_token))
    for p in projects:
        print('ID: %(id)s - Name: %(name)s - Enabled: %(enabled)s' % p)

if __name__ == '__main__':
    main()

Obtaining OpenStack tokens for other tools

Most OpenStack client allow authentication with tokens, so you can easily use them with EGI Cloud providers just doing a first step for obtaining the token. With the OpenStack client you can use the following command to set the OS_TOKEN variable with the needed token:

$ OS_TOKEN=$(openstack --auth-type v3oidcaccesstoken \
            --os-protocol oidc --identity-provider egi.eu \
            --os-auth-url <keystone  url> \
            --os-access-token <your access token> \
            --os-project-id <your project id> token issue -c id -f value)

You can refresh the access token and obtain an OpenStack token in a single script similar to this (python requires requests):

#!/usr/bin/env python

# in case someone is trying this with python 2
from __future__ import print_function

import base64
import json
import os
import requests

# URL Parse
try:
    # Python 2.x
    from urlparse import urlparse, urlunparse
except ImportError:
    # Python 3.x
    from urllib.parse import urlparse, urlunparse



def get_access_token(client_id, client_secret, refresh_token):
    refresh_data = {
        'client_id': client_id, 
        'client_secret': client_secret,
        'grant_type': 'refresh_token',
        'refresh_token': refresh_token,
        'scope': 'openid email profile',
    }
    r = requests.post("https://aai.egi.eu/oidc/token",
                      auth=(client_id, client_secret), data=refresh_data)
    return r.json()['access_token']

def get_keystone_url(os_auth_url, path):
    url = urlparse(os_auth_url)
    prefix = url.path.rstrip('/')
    if prefix.endswith('v2.0') or prefix.endswith('v3'):
        prefix = os.path.dirname(prefix)
    path = os.path.join(prefix, path)
    return urlunparse((url[0], url[1], path, url[3], url[4], url[5]))


def get_unscoped_token(os_auth_url, access_token):
    url = get_keystone_url(
        os_auth_url,
        "/v3/OS-FEDERATION/identity_providers/egi.eu/protocols/oidc/auth")
    r = requests.post(url,
                      headers={'Authorization': 'Bearer %s' % access_token})
    return r.headers['X-Subject-Token']

def get_scoped_token(os_auth_url, os_project_id, unscoped_token):
    url = get_keystone_url(os_auth_url, "/v3/auth/tokens")
    token_body = {
        "auth": {
            "identity": {
                "methods": ["token"],
                "token": {"id": unscoped_token}
            },
            "scope": {"project": {"id": os_project_id}}
        }
    }
    r = requests.post(url, headers={'content-type': 'application/json'},
                      data=json.dumps(token_body))
    return r.headers['X-Subject-Token']


def main():
    # read from environment
    client_id = os.environ.get('CHECKIN_CLIENT_ID', '')
    client_secret = os.environ.get('CHECKIN_CLIENT_SECRET', '')
    refresh_token = os.environ.get('CHECKIN_REFRESH_TOKEN', '')

    os_auth_url = os.environ.get('OS_AUTH_URL', '')
    os_project_id = os.environ.get('OS_PROJECT_ID', '')

    access_token = get_access_token(client_id, client_secret, refresh_token)
    token = get_scoped_token(os_auth_url, os_project_id,
                             get_unscoped_token(os_auth_url, access_token))
    print(token, end='')

if __name__ == '__main__':
    main()


The script expects your credentials to be available in the environment:

Sample usage (script named get-token.py):

export CHECKIN_CLIENT_ID=xxx
export CHECKIN_CLIENT_SECRET=secret
export CHECKIN_REFRESH_TOKEN=othersecret
export OS_AUTH_URL=OS_AUTH_URL=https://sbgcloud.in2p3.fr:5000
export OS_PROJECT_ID=a5eb30bba2c2497b90645fb199e34b39
export OS_TOKEN=$(python get-token.py)

Detailed workflow

You can safely skip this section if not interested in the internals.

Clients willing to authenticate with an OpenStack provider should perform:

  • a request to http://<keystone_url>/v3/OS-FEDERATION/identity_providers/egi.eu/protocols/oidc/auth with the header Authentication: Bearer <access token>, where <access token> is a valid OAuth2.0 Access Token obtained from EGI Check-in. This call if successful will return a X-Subject-Token header with an unscoped token to be used in subsequent calls.
curl -i -H 'Authorization: Bearer <OAuth2.0 acces token>' \
    https://<keystone_url>/v3/OS-FEDERATION/identity_providers/egi.eu/protocols/oidc/auth
HTTP/1.1 201 Created
Date: Wed, 01 Aug 2018 15:23:15 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
X-Subject-Token: <unscoped token>
Vary: X-Auth-Token
x-openstack-request-id: req-fae2789d-35ba-4020-9487-378ba1e3fafb
Content-Length: 491
Content-Type: application/json

{"token": {"issued_at": "2018-08-01T15:23:15.000000Z", "audit_ids": ["j-B2cjhwQ9ePBjJdntWnvw"], "methods": ["oidc"], 
 "expires_at": "2018-08-01T16:23:15.000000Z", "user": {"OS-FEDERATION": {"identity_provider": {"id": "egi.eu"},
 "protocol": {"id": "oidc"}, "groups": [{"id": "c526ee5d3d224c0780f7877b3597d014"}]}, 
 "domain": {"id": "Federated", "name": "Federated"}, "id": "84b1c6812ab746c89e282ac854824712", "name": 
 "529a87e5ce04cd5ddd7161734d02df0e2199a11452430803e714cb1309cc3907@egi.eu"}}}
  • A request to https://<keystone url>/v3/auth/projects get supported projects with the obtained token in a X-Auth-Token header.
curl -H 'X-Auth-Token: <unscoped token>' https://<keystone-url>/v3/auth/projects | python -mjson.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   384  100   384    0     0   1534      0 --:--:-- --:--:-- --:--:--  1536
{
    "links": {
        "next": null,
        "previous": null,
        "self": "https://<keystone url>/v3/auth/projects"
    },
    "projects": [
        {
            "description": null,
            "domain_id": "default",
            "enabled": true,
            "id": "a5eb30bba2c2497b90645fb199e34b39",
            "is_domain": false,
            "links": {
                "self": "https://<keystone url>/v3/projects/a5eb30bba2c2497b90645fb199e34b39"
            },
            "name": "EGI_FCTF",
            "parent_id": "default"
        }
    ]
}
  • Lastly, the token can be scoped to a project with the information from the previous call by performing a POST request to https://<keystone url>/v3/auth/tokens with a json body as shown below and will return the scoped token in a X-Subject-Token header:
{
    "auth": {
        "identity": {
            "methods": [
                "token"
            ],
            "token": {
                "id": "<unscped token>"
            }
        },
        "scope": {
            "project": {
                "id": "<project id>"
            }
        }
    }
}

Sample curl command:

curl -i -H 'content-type: application/json'   -X POST \
     -d '{"auth": {"identity": {"methods": ["token"], "token":{ "id": "token"}}, "scope": {"project": {"id": "project id"}}}}'  \
     https://<keystone url>/v3/auth/tokens
HTTP/1.1 201 Created
Date: Wed, 01 Aug 2018 15:45:34 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
X-Subject-Token: <your scoped token>
Vary: X-Auth-Token
x-openstack-request-id: req-e2d54fa1-e3d2-451d-a5a4-e5ee22f7d8c3
Content-Length: 8464
Content-Type: application/json

<...extra token information returned by keystone...>

Legacy VOs (VOMS)

Other OpenStack clients

Terraform

Terraform OpenStack provider accepts tokens for authentication and will use the value in the OS_TOKEN environment variable if defined. Just configure your provider as usual:

provider "openstack" {
}