Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Applications on Demand Service - information for developers

From EGIWiki
Jump to navigation Jump to search
Applications on Demand Service menu: Home Documentation for providers Documentation for developers Architecture




Applications on Demand (AoD) Service Information pages

The EGI Applications on Demand service (AoD) is the EGI’s response to the requirements of researchers who are interested in using applications in a on-demand fashion together with the compute and storage environment needed to compute and store data.

The Service can be accessed through the EGI Marketplace

Development guidelines and materials for application developers

In this section we collect some useful information for those interested to integrate new applications in one of the available Science Gateways and VRE.


How to integrate a new application in EC3

The process to integrate a new application in EC3 is described by the following process:

References

For further information about the EC3 Framework, please check the official web page.

How to integrate a new application in the FutureGateway Science Gateway

There are mainly two methods to add an application to the Science Software on Demand service (SSOD).

  • The first consists in integrating a new web application in the site 'Applications' section.
  • The second consists in get supported in hosting, installing, developing and configuring a new Science Gateway built on top of the FutureGateway Framework.

Setup a web application

To develop a new web application to be integrated in the SSOD, it is only necessary to know:

  • The FutureGateway system and its related set of REST APIs
  • Basic knowledge of web pages design, in particuar javascript and jQuery knowledge.

FutureGateway concepts

FutureGateway official documentation is available from its repository page.

The documentation exlpains how to install and then use REST APIs which specifications are available at: http://docs.fgapis.apiary.io/#.

The FutureGateway APIs are essentially managing three entities:

  • Infrastructures - Keep information about the access to the physical distributed infrastructure
  • Applications - Describe the activity to perform on top of a distributed infrastructure
  • Tasks - Keep application instances 'running' on top of distributed infrastructure

Beside these core set of API, other entities have been added:

  • Users groups and roles (UGR) - Manage FutureGateway users groups and roles
  • Auditing and accounting (AAA) - Retrieve usage statistics (Under construction)
Infrastructures

Infrastructures are defined as a set of key value pairs in the form of: (key_name, key_value).

The whole set of these values is then used by a FutureGateway component named 'Executor Interface', to physically access the distributed infrastructure.

Applications

An Application describe the activity to perform on top of an infrastructure. Applications may use one or more infrastructures and they can describe any kind of activity, from a batch execution till a PaaS creation.

Tasks

Any running Application is a Task, it holds information about its status and related input/output files. Tasks can also collect data at run-time.

Web page design

The ScienceSoftware on Demand service is implemented using a Liferay7 instance and each existing application has been developed using the Liferay SDK.

The source code of each application is publicly available in a dedicated GitHub repository.

The repository contains the necessary files to setup the application on the FutureGateway and also the web applciation source code available in the `portlet` subfolder.

Each web application has been generated with the Liferay' SDK command: `blade create -t npm-jquery-portlet` and it essentially uses the following files:

  • src/main/resources/META-INF/resources/init.jsp
  • src/main/resources/META-INF/resources/js/index.js

init.jsp

This is the most important file since it retrieves a 'super user' FutureGateway token used to automatically register a portal user into the FutureGateway system, associate this user to the Application group and if it necessary verify the EGI marketplace rights. At the end, a user level FutureGateway token will be generated. The whole operations above are performed in the back-end side of the service and the final user token will be passed to the client-side inside a variable accessible by javascript.

index.js

This is the code executed on the 'client' side and it is the responsible to handle any graphic component of the displayed web page. Then, collecting its values and properly using the FutureGateway REST APIs, will execute the application into the distributed infrastructure. During the execution, it is also possible to retrieve the status of the executing application and report this to the final user. At the bottom of this file there are several generic javascript functions used to call FutureGateway APIs (doGET, doPOST, etc.)

Create a new Science Gateway

This operation includes the considerations done in the previous chapter and it only requires as prerequisite to install a dedicated set of Future Gateway components. In particular these are:

  • The futuregateway DB
  • The APIServer
  • The APIServerDaemon and its ExecutorInterfaces

and optionally:

  • An SSH client
  • A Liferay7 instance

Any setup activity is collected in the FutureGateway repository.

There are mainly two options, the first using a script based installation, the second using docker containers. The script based installation can be used for phisical or virtual machines. The docker containers installation is capable to provide a FutureGateway system immediately, in particular:

  • Test script installation: this repository space is used to test intallation scripts (Ubuntu). However it can be used to create a compact FutureGateway instance, made only of its core components.

Once the docker containers have been generated, it is possible to test the generated service executing: `test_futuregateway.sh` .

  • Docker components: Under `docker/`folder exists a directory for each FutureGateway service. It is possible to create a customised docker image just modifying the Dockerfile and buiding the image. These steps can be easily managed by a dedicated Makefile. This file can be configured as well.

The futuregateway DB

Database generation scripts are included in the source code repository of the fgAPIServer component. The Database has a central role for the FutureGateway and it uses MySQL.

APIServer

This is the software component that manages FutureGateway REST APIs. It is a Python application using the Flask microframework and normally it runs as a WSGI process. This service may be configured in different ways by a dedicated configuration file, however configuration parameters may be also specified using environment variables or even the database.

APIServerDaemon

This is the component responsible to extract records from the FutureGateway Tasks queue and phisically access the distributed infrastructure to perform the activity described by the Applcation. The APIServerDaemon uses a modular set of sub-components named ExecutorInterfaces which uses the Infrastructure information to access teh distributed infrastructure.

General considerations

  • Developing an interface for a Science Gateway is always a non trivial task especially because each use case has different requirements. The FutureGateway system is very flexible and in most of the cases there is no need to customize the existing code. On the other hand the flexibility requires a bigger effort in terms of development.
  • The Science Software on Demand service provides the 'Propose' section which only contains a contact link.

It is highly suggested to describe the use case carefully specifying details about the distributed infrastructure to use.

  • Any FutureGateway repository contains a descriptive `README.md`file sometime containing very important usage instructions, especially for the `fgSetup` repository.