Applications on Demand Service - information for developers
|This article is Deprecated and has been moved to https://docs.egi.eu/users/applications-on-demand/.|
|Applications on Demand Service menu:||Home •||Documentation for providers •||Documentation for developers •||Architecture|
Applications on Demand (AoD) Service Information pages
The EGI Applications on Demand service (AoD) is the EGI’s response to the requirements of researchers who are interested in using applications in a on-demand fashion together with the compute and storage environment needed to compute and store data.
Development guidelines and materials for application developers
In this section we collect some useful information for those interested to integrate new applications in one of the available Science Gateways and VRE.
How to integrate a new application in EC3
The process to integrate a new application in EC3 is described by the following process:
- Describe the application to be integrated with Ansible, the open-source automation engine that automates software provisioning, configuration management, and application deployment.
- For more info about how to use Ansible, please refer to the official documentation http://docs.ansible.com/
- Use the Ansible receipt to create a new RADL template (for this contact Miguel Caballer at: micafer1 <at> upv <dot> es)
- For more info, please refer to the official documentation: http://ec3.readthedocs.io/en/devel/templates.html
- Example of RADL templates are available at: https://github.com/grycap/ec3/tree/master/templates
For further information about the EC3 Framework, please check the official web page.
How to integrate a new application in the FutureGateway Science Gateway
There are mainly two methods to add an application to the Science Software on Demand service (SSOD).
- The first consists in integrating a new web application in the site 'Applications' section.
- The second consists in get supported in hosting, installing, developing and configuring a new Science Gateway built on top of the FutureGateway Framework.
Setup a web application
To develop a new web application to be integrated in the SSOD, it is only necessary to know:
- The FutureGateway system and its related set of REST APIs
FutureGateway official documentation is available from its repository page.
The documentation exlpains how to install and then use REST APIs which specifications are available at: http://docs.fgapis.apiary.io/#.
The FutureGateway APIs are essentially managing three entities:
- Infrastructures - Keep information about the access to the physical distributed infrastructure
- Applications - Describe the activity to perform on top of a distributed infrastructure
- Tasks - Keep application instances 'running' on top of distributed infrastructure
Beside these core set of API, other entities have been added:
- Users groups and roles (UGR) - Manage FutureGateway users groups and roles
- Auditing and accounting (AAA) - Retrieve usage statistics (Under construction)
Infrastructures are defined as a set of key value pairs in the form of: (key_name, key_value).
The whole set of these values is then used by a FutureGateway component named 'Executor Interface', to physically access the distributed infrastructure.
An Application describe the activity to perform on top of an infrastructure. Applications may use one or more infrastructures and they can describe any kind of activity, from a batch execution till a PaaS creation.
Any running Application is a Task, it holds information about its status and related input/output files. Tasks can also collect data at run-time.
Web page design
The ScienceSoftware on Demand service is implemented using a Liferay7 instance and each existing application has been developed using the Liferay SDK.
The source code of each application is publicly available in a dedicated GitHub repository.
The repository contains the necessary files to setup the application on the FutureGateway and also the web applciation source code available in the `portlet` subfolder.
Each web application has been generated with the Liferay' SDK command: `blade create -t npm-jquery-portlet` and it essentially uses the following files:
Create a new Science Gateway
This operation includes the considerations done in the previous chapter and it only requires as prerequisite to install a dedicated set of Future Gateway components. In particular these are:
- The futuregateway DB
- The APIServer
- The APIServerDaemon and its ExecutorInterfaces
- An SSH client
- A Liferay7 instance
Any setup activity is collected in the FutureGateway repository.
There are mainly two options, the first using a script based installation, the second using docker containers. The script based installation can be used for phisical or virtual machines. The docker containers installation is capable to provide a FutureGateway system immediately, in particular:
This repository space is used to test intallation scripts (Ubuntu). However it can be used to create a compact FutureGateway instance, made only of its core components.
- The Docker compose solution.
The compose yaml file is generated by the script:
Once the docker containers have been generated, it is possible to test the generated service executing:
- Docker components: Under `docker/`folder exists a directory for each FutureGateway service. It is possible to create a customised docker image just modifying the Dockerfile and buiding the image. These steps can be easily managed by a dedicated Makefile. This file can be configured as well.
The futuregateway DB
Database generation scripts are included in the source code repository of the fgAPIServer component. The Database has a central role for the FutureGateway and it uses MySQL.
This is the software component that manages FutureGateway REST APIs. It is a Python application using the Flask microframework and normally it runs as a WSGI process. This service may be configured in different ways by a dedicated configuration file, however configuration parameters may be also specified using environment variables or even the database.
This is the component responsible to extract records from the FutureGateway Tasks queue and phisically access the distributed infrastructure to perform the activity described by the Applcation. The APIServerDaemon uses a modular set of sub-components named ExecutorInterfaces which uses the Infrastructure information to access teh distributed infrastructure.
Developing an interface for a Science Gateway is always a non trivial task especially because each use case has different requirements. The FutureGateway system is very flexible and in most of the cases there is no need to customize the existing code. On the other hand the flexibility requires a bigger effort in terms of development.
The Science Software on Demand service provides the 'Propose' section which only contains a contact link. It is highly suggested to describe the use case carefully specifying details about the distributed infrastructure to use.
Any FutureGateway repository contains a descriptive README.md file sometime containing very important usage instructions, especially for the fgSetup repository.