Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Applications on Demand Service - information for providers

From EGIWiki
Jump to navigation Jump to search
Alert.png This article is Deprecated and has been moved to https://docs.egi.eu/users/applications-on-demand/.



Applications on Demand Service menu: Home Documentation for providers Documentation for developers Architecture




Applications on Demand (AoD) Service Information pages

The EGI Applications on Demand service (AoD) is the EGI’s response to the requirements of researchers who are interested in using applications in a on-demand fashion together with the compute and storage environment needed to compute and store data.

The Service can be accessed through the EGI Marketplace


Who can contribute to the service ?

The EGI Applications on Demand service is open for any resource providers (e.g. HTC, cloud) and technological providers (e.g. Science Gateways/portal). Interested providers have to follows the technical steps reported in the below sections.

Overview

To join the EGI Applications on Demand service as resource or technological providers the following steps must be verified:

  • Science Gateways/Portals and resource providers must accept and follow the platform security policy: https://documents.egi.eu/public/ShowDocument?docid=2734
  • Science Gateways/Portals and resource providers must sign OLA with EGI.eu.
  • Science Gateway/Portals providers must integrate with the User Registration Portal to enable the single sign-on capability for users.
  • Science Gateway/Portals providers must integrate with the per-user subproxy solution to offer traceable user authentication towards the e-infrastructure VO.
  • Science Gateways/Portals must implement user resource usage quota (to prohibit a user consuming all the resources from the platform).
  • Resource providers must support the per-user subproxy solution and join the e-infrastructure VO.

The below subsections provide guidance to complete these steps.

How to join the EGI Applications on Demand service

In the following sub-sections are reported the technical steps to join the EGI Applications on Demand service as providers (e.g.: resource and technological providers).

How to join as a technological provider

The integration of new Science Gateways/Applications hosting frameworks into the EGI Applications on Demand (AoD) service is a six-step process:

  • Step 1. Register your Science Gateways/Applications hosting frameworks with the development instance of EGI AAI Check-In service or Unity.
  • Step 2. Get a clientID and secretKey. These credentials will be used to identify the new framework and implement a secure connection with it.
  • Step 3. Configure your Science Gateways/Applications hosting frameworks to use the UserID provided by the authentication framework to generate Per-User Sub-Proxies (PUSP) to secure user interaction with the EGI resources (see next section).

Lastly, the technical integration is complete when the service support team together with the technology provider:

  • Step 4. Register the Science Gateways/Applications hosting frameworks in the EGI Service Registry (GOCDB) to activate the availability and reliability monitoring for the framework with the EGI ARGO service.
  • Step 5. Setup a specific support unit in the EGI Helpdesk.
  • Step 6. Sign an Operational Level Agreement (OLA) defining for example the availability and reliability targets, helpdesk ticket response times.


In the next two sections, are introduced the steps to register the new Science Gateways/Applications hosting frameworks with the AAI framework.

Connecting Science Gateway/Applications hosting framework with the URP DEPRECATED

Connecting the Science Gateway/Applications hosting framework with the EGI AAI Check-In service

  • Register the new Science Gateway/Applications hosting framework as Service Provider (SP) in the EGI AAI Check-In service following the integration flow described in this wiki.
  • Two different protocols are supported: SAML and OpenID Connect

In the next section, it is shown an example how to register a Liferay-based Science Gateway in the EGI AAI Check-In (DEV) service using the OpenID Connect protocol.

OpenID Connect Service Provider

From https://wiki.egi.eu/wiki/AAI_guide_for_SPs#OpenID_Connect_Service_Provider:

  • Get the new ClientID and secretKey
  • Change the OID settings and claims:
[..]
authC = new ClientSecretBasic(new ClientID(<YOUR_CLIENT_ID>), new Secret(<YOUR_SECRET_KEY>));
this.state = state;
try {
    callback = new URI("https://csgf.egi.eu/c/portal/login"); # <= Add here the URL of the Science Gateway
    oauthS = new URI("https://aai-dev.egi.eu/oidc/authorize");
    tokenS = new URI("https://aai-dev.egi.eu/oidc/token");
    userS = new URI("https://aai-dev.egi.eu/oidc/userinfo");
    tokenCertSign = new URI("https://aai-dev.egi.eu/oidc/jwk");
    issuer = "https://aai-dev.egi.eu/oidc/";
    aud = "csgf"; # <= Add here the Client ID of the Science Gateway
} ...
  • Create a new user group in Liferay and change the setting:
[..]
UserGroup uGroup = UserGroupLocalServiceUtil.getUserGroup(companyId, "EGIUsers");

For further information, the module to enable Liferay to support OIDC protocol is available on GitHub.

REST APIs for managing VO membership information in Check-In

The EGI AAI Check-In service provides a REST API that allows clients to manage membership information only for the VOs they are authoritative for. This API are used by the service providers of the EGI Applications on Demand (AoD) service to manage the members of the vo.access.egi.eu VO.

Requirements:

  • Members of the VO should be identified via their EGI ePUID
  • Membership should be limited to the specified period
  • Different membership status values should be supported, namely: 'Active', 'Expired', 'Deleted'
  • CheckIn should automatically change the membership status from 'Active' to 'Expired' beyond the validity period


Authentication: The REST client is authenticated via username/password credentials transmitted over HTTPS using the Basic Authentication scheme. More sophisticated authentication mechanisms, such as OpenID Connect/OAuth 2.0 access tokens, may be supported in the future.

Methods:

1. Adding a user to a VO requires specifying the user’s EGI ID, the name of the VO ("vo.access.egi.eu" in the case of LToS), the status ("Active") and the valid from/through dates. All these parameters are mandatory. Here is an example using curl (see attached ltos-add.json file):

]$ curl -vX POST https://aai.egi.eu.eu/api/v1/VoMembers \
  -user "example-client":"veryverysecret" \
  --data @ltos-add.json \
  --header "Content-Type: application/json"

2. Retrieving the VO membership information for a given EGI ID:

]$ curl -vX GET https://aai.egi.eu.eu/api/v1/VoMembers/01234567890123456789@egi.eu \
  —user "example-client":"veryverysecret"

{"id":85,"epuid":"01234567890123456789@egi.eu","vo_id":"vo.access.egi.eu","valid_from":"2017-05-20T22:00:00.000Z","valid_through":"2017-06-21T22

Beyond the valid_through date, the status will be automatically changed to "Expired". So, when querying for VO membership information, it’s important to check that the status is actually set to "Active", and of course that the vo_id equals "vo.access.egi.eu".

3. Updating existing VO membership record:

]$ curl -vX PUT https://aai.egi.eu.eu/api/v1/VoMembers \
  --user "example-client":"veryverysecret"  \
  --data @ltos-add.json \
  --header "Content-Type: application/json"

The request body is the same as the one used for adding new members but update requires using PUT instead of POST.

4. Removing VO member: Same as the update but requires setting the membership status to "Deleted".

An example of ltos-add.json file is the following:

{
  "RequestType": "VoMembers",
  "Version": "1.0",
  "VoMembers": [
    {
      "Version": "1.0",
      "VoId": "vo.access.egi.eu",
      "Person": {
        "Type": "CO",
        "Id": "025166931789a0f57793a6092726c2ad89387a4cc167e7c63c5d85fc91021d18@egi.eu"
      },
      "Status": "Active",
      "ValidFrom": "2017-05-20",
      "ValidThrough": "2017-12-31"
    }
  ]
}

Connecting the Science Gateways/Portals with Per-User Sub-Proxies

The Applications on Demand Infrastructure uses per-user sub-proxies (PSUPs) for user's authentication. Any connected Science Gateway must generate per-user sub proxies for their users and must use these for any interaction with VO resources on behalf of the users.

A Science Gateway can generate PSUPs in two ways:

  1. From a robot certificate that is physically hosted on the gateway server itself. OR
  2. From a remote robot certificate that is hosted in the e-Token Server by INFN Catania.

We recommend you to choose the first option. If there is an IGTF CA in your country which issues robot certificates then obtain a robot from this CA. If such robots are not available in your country or region, then EGI can issue a robot for you from the SEEGRID catch-all CA. The next subsections provide details information to complete these steps.

Generic requirements

The Per-User Sub-Proxy (PUSP) and End-Entity Certificate (EEC) must satisfy the following requirements:

  1. The EEC is a valid robot certificate:
  2. The PUSP is RFC 3820 compliant, i.e. no legacy GT2 or GT3 proxies
  3. The PUSP is the first proxy delegation
  4. If the same user enters via the same portal, he must get the same PUSP DN
  5. No two distinct identified users will have the same PUSP DN.

A robot EEC that generates PUSP credentials SHOULD NOT be used for any other purpose; for example, it should not be used to generate non-PUSP proxy credentials and should not be use for direct authenticating.

The machine/service that will take care of PUSP generation and management should respect the following rules:

  1. Documented response procedures in case of incidents (that are periodically tested).
  2. A listed/accredited CSIRT team.
  3. Internal risk assessment and an actuarial team to calculate the effective risk

Using a robot certificate from your national IGTF CA

  1. Obtain a robot certificate from your national IGTF Certification Authority following the instructions here.
  2. Register the robot in the vo.access.egi.eu VO: https://perun.metacentrum.cz/cert/registrar/?vo=vo.access.egi.eu
  3. Generate proxies from the robot using this script: https://ndpfsvn.nikhef.nl/viewvc/mwsec/trunk/lcmaps-plugins-robot/tools/

Obtaining a robot certificate from EGI catch-all CA

  1. Contact applications-platform-support@mailman.egi.eu and send a short description of your gateway service and the way it would be integrated with platform resources. The team will arrange a robot certificate for your gateway from the SEEGRID CA (which operates as a 'catch-all' CA in EGI) and will register this in the VO and in the e-Token Server in Italy.
  2. Register the robot in the vo.access.egi.eu VO: https://perun.metacentrum.cz/cert/registrar/?vo=vo.access.egi.eu
  3. Generate proxies from the robot using this script: https://ndpfsvn.nikhef.nl/viewvc/mwsec/trunk/lcmaps-plugins-robot/tools/

Instructions to use the e-Token Server

  1. Contact applications-platform-support@mailman.egi.eu and send a short justification why you would like to use the eToken server (instead of hosting the robot certificate locally). Describe your gateway service and the way it would be integrated with platform resources. The team will arrange a robot certificate for your gateway from the SEEGRID CA and will register it in vo.access.egi.eu.
  2. Provide applications-platform-support@mailman.egi.eu with the static IP address of your gateway server, so proxy requests can be authorized from this address on the e-Token Server.
  3. Generate proxies from the e-Token server following this guideline:

There are two available e-Token Server instances for availability and reliability reasons:

  • etokenserver.ct.infn.it
  • etokenserver2.ct.infn.it

The following rest API is available to get a PUSP given a unique identifier:

https://[eToken Server instance]:8443/eTokenServer/eToken/[Robot Certificate ID]?voms=[VO]:/[VO]&proxy-renewal=[true|false]&disable-voms-proxy=[true|false]&rfc-proxy=[true|false]&cn-label=user:[user unique identifier]
  • Robot cetificate ID: it is the ID of your robot certificate in the e-Token server. It will be generated after the setup of your robot into the e-Token Server.
  • VO: the VO you want to use to perform any action on the EGI infrastructure. The robot certificate must be a member of this VO.
  • proxy-renewal: this option is used to enable (true) or disable (false) the automatic registration of a long-term proxy into a MyProxy Server.
  • disable-voms-proxy: this option is used to generate plain (true) or VOMS proxy certificate (false).
  • rfc-proxy: this option is used to generate standard RFC proxies (true) or legacy proxies (false).
  • cn-label: this option is used to generate a PUSP for the given unique identifier.

below an example:

https://[eToken Server instance]:8443/eTokenServer/eToken/332576f78a4fe70a52048043e90cd11f?voms=vo.access.egi.eu:/vo.access.egi.eu&proxy-renewal=false&disable-voms-proxy=false&rfc-proxy=true&cn-label=user:test1

Connecting the Science Gateways/portals with the EGI monitoring system

To monitor a new Science Gateway (SG)/portal with the EGI monitoring system based on ARGO, the SG system administrator HAS TO:

  • Register the Science Gateway/portal in the GOCDB under EGI.eu NGI
    • For that, please contact the EGI Operations Team (<operations@egi.eu>)
  • Register the Science Gateway/portal in the GOCDB as service type: egi.Portal
    • For that, please contact the EGI Operations Team (<operations@egi.eu>)
  • Change in the GOCDB the Certification Status of the SG in Certified
    • For that, please contact the EGI Operations Team (<operations@egi.eu>);
    • By default ARGO will automatically fetches the Science Gateway endpoint from the GOCDB and implements simple http checks using standard NAGIOS probe.

ARGO uses a bundle of connectors/sync components to monitor different service components, but if necessary new ones can be easily developed and added.

How to join as a grid/cloud resource provider

Any EGI resource provider can join and offer capacity for members of the Infrastructure. The site needs to run one of the supported grid or cloud middleware software, enable per-user sub-proxies (for user authentication and authorisation), and join the vo.access.egi.eu Virtual Organisation. Please email applications-platform-support@mailman.egi.eu if you wish to join as a resource provider.

In order to provide authorization to the users of the VO, a couple of DNs (Distinghished Names) are required to be configured on the services to be enabled. For instance, for the CREAM CE the usual grid-mapfile is the place where to add them, for OpenStack it's /etc/keystone/voms.json. You can find below the instructions for each service.

The following Robot Certificate DNs must be configured:

/DC=EU/DC=EGI/C=HU/O=Robots/O=MTA SZTAKI/CN=Robot:zfarkas@sztaki.hu
/DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Robot - Giuseppe Platania platag@infn.it

Instructions for OpenStack providers

Keystone-VOMS has support for per-user subproxies in the latests releases of Keystone-VOMS for Mitaka onwards.

Configure and deploy as the normal Keystone-VOMS plugin following the Keystone-VOMS documentation. The

The code relies on gridsite for getting extra information about the proxies arriving to your Keystone installation. You can get packages from UMD or if your distribution is not supported you can compile packages following these instructions (for Ubuntu 16.04):

# Install dependencies
sudo apt-get install -y debhelper apache2-dev  gsoap  libcanl-c-dev \
                        libcurl4-openssl-dev  doxygen  libxml2 \
                        libxml2-dev  libssl-dev  libtool-bin  pkg-config

# get source code
curl -k https://emian.zcu.cz:8443/jenkins/view/release/job/gridsite-build-release/platform=ubuntu-14-x86_64/lastSuccessfulBuild/artifact/results/ubuntu/stable/gridsite_2.3.3.orig.tar.gz | tar -xzvf - -C /tmp

# make and make install
cd /tmp/gridsite-2.3.3/src/
make
sudo make install
sudo ldconfig

Once gridsite is installed, you just need to add into your apache keystone configuration the following line (location of the module may be different in your installation):

LoadModule gridsite_module /usr/local/lib/httpd/modules/mod_gridsite.so

Instructions for OpenNebula providers

OpenNebula sites are not yet supported in the platform, there is some unofficial instructions (the development is not 100% complete, hence no instructions and partial support is available). Here some hints, please use at your own risks!

For the PUSP support on an up-to-date rOCCI-server, CESGA can enable partial PUSP support by adding

SetEnv ROCCI_SERVER_AUTHN_VOMS_ROBOT_SUBPROXY_IDENTITY_ENABLE yes

to Apache2 VHost configuration for rOCCI-server. This enables PUSP support in accounting but does NOT provide yet isolation between different PUSP users of the same robot.

Instructions for EMI 3/gLite providers

There is an EGI manual that shows how to set up a per-user sub-proxy to allow identification of the individual users under a common robot certificate. You can find the guide here: https://wiki.egi.eu/wiki/MAN12

How to operate your service

  • To describe the Intervention in case of unscheduled failure of central operational tool, please refer to the documentation Tool Intervention Management


Notifying users about downtimes

Scheduled downtime

  • Scheduled interventions MUST be declared at least 24h in advance, specifying reason and duration.
  • Existing scheduled interventions CAN be extended, provided that it is done 24h in advance.

1. For Grid/Cloud Resource Providers:

  • Resource Provider (RP) has to register the scheduled downtime in the GOCDB by clicking on the 'Add a Downtime' link in the sidebar, specify the downtime information and the full list of impacted services.
    • In case there is more than one or select a site to select all the sites associated services.

2. For Science Gateway/portal providers:

  • Science Gateway provider has to register the scheduled downtime in the GOCDB by clicking on the 'Add a Downtime' link in the sidebar.
  • Display downtime information on the front-page of your Science Gateway.
  • The Science Gateway picks up the downtime information from GOCDB via API, and sends out a broadcast email message to active users of the platform. (under development)
  • The Science Gateway displays downtime information in a dedicated section of the front page (under development).
  • GOCDB Programming Interface documentation can be found here

Unscheduled (unexpected) downtime

  • Any intervention declared less than 24h in advance will be considered unscheduled.
  • Both grid/cloud resource providers AND science gateway/portal providers MUST declare unscheduled interventions as soon as they are detected to inform the users. Register the scheduled downtime in the GOCDB by clicking on the 'Add a Downtime' link in the sidebar, specify the downtime information and the full list of impacted services in case there is more than one or select a site to select all the sites associated services.
  • Unscheduled interventions CAN be declared up to 48h in the past (retroactive information to the user community).
  • Unscheduled downtimes can be fetched from the GOCDB via API as well and notified to the end-users.
  • GOCDB Programming Interface documentation can be found here

Display downtimes

  • There is a JSR-286 portlet on github that picks up the downtime information from GOCDB via API, and sends out a broadcast email message to active users of the platform.
  • The list of downtimes affecting the services of the platform (if any) is available here

EGI Applications on Demand service - Operational Level Agreement (OLAs)

OLAs supporting the EGI Applications on Demand service are available here:

https://documents.egi.eu/public/ShowDocument?docid=2773

https://documents.egi.eu/public/ShowDocument?docid=2782

How to join the user support team

If you wish to support platform users from your country, region or scientific disciplinary area, then please email applications-platform-support@egi.eu.

We can train you and then register you as a supporter in our team.

Policies

This EGI Applications on Demand service Security Policy is applicable to all and only the Participants involved in the provisioning of the EGI Applications on Demand service.

This Policy and the Implementation Guidelines aim to enable a low-barrier Service to be offered to a wide range of research users in Europe and their collaborators world-wide, by any Resource Centre organisation that elects to do so. In offering such services, the Resource Centre shall not negatively affect the security or change the security risk of any other Resource Centre or any other part of the e-Infrastructure. In particular, security incidents originating in the EGI Applications on Demand service should not impact the IT Infrastructure in ways that are incompatible with the operational model of other, more tightly controlled, parts of the infrastructure. This document also provides guidelines on the implementation of security procedures and controls to facilitate offering of the Service by Resource Centres and Science Gateways. The Guidelines contain normative information on how to implement the Policy.