Difference between revisions of "AoD"

From EGIWiki
Jump to: navigation, search
 
(108 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
<!-- {{Template:Op menubar}} -->
 
<!-- {{Template:Op menubar}} -->
 +
 +
{{DeprecatedAndMovedTo|new_location=https://docs.egi.eu/users/applications-on-demand/}}
  
 
<b>Applications on Demand (AoD) Service Information pages</b>
 
<b>Applications on Demand (AoD) Service Information pages</b>
Line 5: Line 7:
 
{{Template:LTOS_menubar}} {{TOC_right}}  
 
{{Template:LTOS_menubar}} {{TOC_right}}  
  
The [http://access.egi.eu/ EGI Applications on Demand service] (AoD) is the EGI’s response to the requirements of researchers who are interested in using scientific applications in a on-demand fashion together with the compute and storage environment needed to compute and store data.
+
The [https://www.egi.eu/services/applications-on-demand/ EGI Applications on Demand service] (AoD) is the EGI’s response to the requirements of researchers who are interested in using applications in a on-demand fashion together with the compute and storage environment needed to compute and store data.
  
[http://access.egi.eu Go to the AoD service]
+
[https://marketplace.egi.eu The Service can be accessed through the EGI Marketplace]<br/>
  
 
= Description of service =
 
= Description of service =
  
 +
The service allows user-friendly access to a portfolio of applications and application hosting frameworks (e.g.: Science Gateways, VREs) that are configured to use the dedicated pool of cloud computing clusters from EGI. The service also allows users to run their own simulation/analysis models with custom input data.
 +
 +
The service operates as an open and extensible ‘hub’ for providers and e-infrastructure user support teams who wish to federated and share applications and services with individual researchers, or small, fragmented communities, typically referred to as ‘the long tail of science’.
 +
 +
The portfolio of applications is currently composed by a readily available set of applications relevant to different scientific and research areas. This portfolio is open to be extended thanks to the contributions of users of the service. If you are interested in this, please get in touch with us: support (at) egi.eu
 +
 +
 +
<!--
 
* One login node, plus a configurable number of compute nodes
 
* One login node, plus a configurable number of compute nodes
 
* An Enterprise Linux operating system
 
* An Enterprise Linux operating system
* A shared filesystem, mounted across all nodes
+
* A shared file-system, mounted across all nodes
 
* A batch job scheduler
 
* A batch job scheduler
 
* Access to a library of software applications
 
* Access to a library of software applications
Line 20: Line 30:
  
 
The portfolio of applications is currently composed by a readily available set of applications relevant to different scientific and research areas. This portfolio is open to be extended thanks to the contributions of users of the service. If you are interested in this, please get in touch with us: support (at) egi.eu
 
The portfolio of applications is currently composed by a readily available set of applications relevant to different scientific and research areas. This portfolio is open to be extended thanks to the contributions of users of the service. If you are interested in this, please get in touch with us: support (at) egi.eu
 +
-->
  
 
= Intended user groups =
 
= Intended user groups =
 
 
 
 
* Researchers, scientists and engineers
+
* Researchers and scientists.
* Application developers who want to share their app available to the community through AoD
+
* Application developers who want to share their applications available to the community through AoD.
* National e-Infrastructures who want to offer the EGI applications library to their national users by "plugging" compute and storage resources
+
* National e-Infrastructures who want to offer the EGI applications library to their national users by "plugging" compute and storage resources.
  
 
= Requirements and user registration =
 
= Requirements and user registration =
Line 31: Line 42:
 
The service is open for any researchers, scientists and engineers who need a simple and user-friendly access to compute, storage and applications services in order to carry out data/compute intensive science and innovation. The user needs to be affiliated with, or at least have a partner (for example a referee), a research institution in Europe to qualify for access.  
 
The service is open for any researchers, scientists and engineers who need a simple and user-friendly access to compute, storage and applications services in order to carry out data/compute intensive science and innovation. The user needs to be affiliated with, or at least have a partner (for example a referee), a research institution in Europe to qualify for access.  
 
 
 
 
Access <b><u>requires</u></b> acceptance of [https://documents.egi.eu/public/ShowDocument?docid=2635 Acceptable Use Policy (AUP) and Conditions of Use of the 'EGI Applications on Demand Infrastructure'].
+
Access <b><u>requires</u></b> acceptance of [https://documents.egi.eu/public/ShowDocument?docid=2635 Acceptable Use Policy (AUP) and Conditions of Use of the 'EGI Applications on Demand Service'].
  
 
Users of the service are asked to provide appropriate acknowledgement of the use in scientific publications. The following acknowledgement text can be used for this purpose:  
 
Users of the service are asked to provide appropriate acknowledgement of the use in scientific publications. The following acknowledgement text can be used for this purpose:  
  
 +
<!--
 
''This work used the EGI Applications On Demand Service, which is co-funded by the EGI-Engage project (Horizon 2020) under Grant number 654142.''
 
''This work used the EGI Applications On Demand Service, which is co-funded by the EGI-Engage project (Horizon 2020) under Grant number 654142.''
 +
-->
 +
''This work used the EGI Applications on Demand service, which is co-funded by the EOSC-hub project (grant number 777536). The HNSciCloud project (grant number 687614) is also sponsoring the service, allowing users to access the HNSciCloud services pilot for limited scale usage using the voucher schemes provided by the Exoscale contractor.''
  
 
When requesting access to AoD users are guided through a lightweight registration process.
 
When requesting access to AoD users are guided through a lightweight registration process.
Line 43: Line 57:
 
 
 
 
 
*<b>Scientific applications</b> that are offered ‘as services’ through graphical environments.
 
*<b>Scientific applications</b> that are offered ‘as services’ through graphical environments.
*<b>Science gateways</b> and <b>Virtual Research Environments</b> that offer integrated development environments to port custom applications with High throughput computing and cloud resources.
+
*<b>Science gateways</b> and <b>Virtual Research Environments</b> that offer integrated development environments to port custom applications with cloud resources.
*<b>Cloud</b> and <b>High-throughput</b> compute resources suited for both compute/data intensive applications and for the hosting of scientific services.
+
*<b>Cloud</b> compute resources suited for both compute/data intensive applications and for the hosting of scientific services.
*<b>Online Storage</b> resources for storing scientific data that serve as input and output for computational jobs.
+
*A network of <b>Consultants</b> and supporters who can provide guidance on the use of the service.
*A network of <b>Consultants</b> and supporters who can provide guidance on the use of the service
+
 
 +
= Service Grant =
 +
Each authenticated and authorized User is offered a grant, providing a pre-defined quota of resources, which can be used to run an application of choice.
  
Each authenticated and authorized user is offered a grant, providing a pre-defined quota of resources, which can be used to run an application of choice. The grant to run scientific applications can be renewed or increased upon request.
+
The EGI grant includes up to <b>4 vCPU cores, 8GB of RAM and 100GB of block storage</b>.
  
<!-- [[Image:AoD-architecture.png|thumb|center|600px|AoD service architecture]] -->
+
The grant to run applications is initially valid for 6 months and can be extended/renewed upon request.
  
 
= Benefits =
 
= Benefits =
 
Through this service:
 
Through this service:
* <b>Researchers</b> can access scientific applications underpinned by high-capacity compute and storage servers, to carry out compute and data-intensive simulations. Use of the service does not require advanced experience with distributed and/or cloud computing.  
+
* <b>Researchers</b> can access applications underpinned by high-capacity compute and storage servers, to carry out compute and data-intensive simulations. Use of the service does not require advanced experience with distributed and/or cloud computing.  
  
 
* <b>Developers</b> can integrate custom applications into the service and offer them as 'scalable, online services' for researchers and scientists worldwide.
 
* <b>Developers</b> can integrate custom applications into the service and offer them as 'scalable, online services' for researchers and scientists worldwide.
* <b>Providers</b> of compute and storage services can share their clusters and clouds to serve 'the long tail of science'.
+
* <b>Providers</b> of compute and storage services can share their clusters and clouds to serve 'the long tail of science'. For more information see [https://wiki.egi.eu/wiki/Applications_on_Demand_Service_-_information_for_providers AoD for infrastructure providers]
+
 
 
= Application library =
 
= Application library =
 
The service operates as an open modular environment where any application developer can integrate and share applications, and providers can plug their own data centre share. The following components are already integrated into the service and are available for users to access: <br>  
 
The service operates as an open modular environment where any application developer can integrate and share applications, and providers can plug their own data centre share. The following components are already integrated into the service and are available for users to access: <br>  
  
== Scientific Applications ==
+
<!--
[[Image:AoDs_applications.jpg|center|600px|AoDs_applications.jpg]] <br>
+
== Scientific Applications and Tools ==
 
+
[[Image:AoDs_applications_v2.jpg|center|600px|AoDs_applications_v2.jpg]] <br>
 +
-->
  
 
=== <u>Computer Science & Mathematics</u> ===
 
=== <u>Computer Science & Mathematics</u> ===
  
*<b>[https://www.r-project.org/ The R Project for Statistical Computing] (v3.2.2)</b> is a language and environment for statistical computing and graphics.  
+
*<b>[https://www.rstudio.com/ RStudio] (v3.2.2)</b> s a free and open-source integrated development environment (IDE) for R, a programming language for statistical computing and graphics..  
**The application is accessible through the Catania Science Gateway. The user manual is available [https://csgf.egi.eu/documents/620030/0/csg_manual_for_ltos.pdf/be218ad7-8a0d-42b7-b932-4b7667524c09 here].
+
**The application is accessible through the FutureGateway Science Gateway. The user manual is available [https://fgsg.ct.infn.it/egissod/web/ssod/r-studio1 here].
*<b>[https://www.gnu.org/software/octave/ GNU Octave]</b> is a high-level interpreted language, primarily intended for numerical computations. It provides capabilities for the numerical solution of linear and nonlinear problems, and for performing other numerical experiments.
 
** The application is accessible through the EC3 portal.
 
 
 
=== <u>Cultural Heritage</u> ===
 
 
 
*<b>The Semantic Search Engine (SSE)</b> is an application conceived to demonstrate the potential of information coupled with semantic web technologies to address the issues of data discovery and correlation.
 
**The application is accessible through the Catania Science Gateway. The user manual is available [https://csgf.egi.eu/documents/620030/0/csg_manual_for_ltos.pdf/be218ad7-8a0d-42b7-b932-4b7667524c09 here].
 
  
 
=== <u>Life Sciences</u> ===
 
=== <u>Life Sciences</u> ===
 
*<b>[http://chipster.csc.fi/ Chispter]</b> is a user-friendly analysis software for high-throughput data. It contains over 300 analysis tools for next generation sequencing (NGS), microarray, proteomics and sequence data. Users can save and share automatic analysis workflows, and visualize data interactively using a built-in genome browser and many other visualizations.  
 
*<b>[http://chipster.csc.fi/ Chispter]</b> is a user-friendly analysis software for high-throughput data. It contains over 300 analysis tools for next generation sequencing (NGS), microarray, proteomics and sequence data. Users can save and share automatic analysis workflows, and visualize data interactively using a built-in genome browser and many other visualizations.  
**The application is accessible through the Catania Science Gateway. The user manual is available [https://wiki.egi.eu/wiki/Access_Chipster_with_CSG#Access_Chipster_with_CSG here].
+
**The application is accessible through the FutureCatania Science Gateway. The user manual is available [https://fgsg.ct.infn.it/egissod/web/ssod/chipster-user-account-instructions here].
*<b>[http://www.clustal.org/clustal2/ ClustalW2]</b> is a multiple sequence alignment tool for the alignment of DNA or protein sequences.
 
**The application is accessible through the Catania Science Gateway. The user manual is available [https://csgf.egi.eu/documents/620030/0/csg_manual_for_ltos.pdf/be218ad7-8a0d-42b7-b932-4b7667524c09 here].
 
 
*<b>[https://galaxyproject.org/ Galaxy]</b> is an open, web-based platform for data intensive biomedical research.  
 
*<b>[https://galaxyproject.org/ Galaxy]</b> is an open, web-based platform for data intensive biomedical research.  
 
** The application is accessible through the EC3 portal. Instructions on how to use Galaxy workflows with EC3 are available [https://wiki.egi.eu/wiki/Galaxy_workflows_with_EC3 here].
 
** The application is accessible through the EC3 portal. Instructions on how to use Galaxy workflows with EC3 are available [https://wiki.egi.eu/wiki/Galaxy_workflows_with_EC3 here].
*<b>AutoDock Vina</b> is an open-source software for doing molecular docking.
 
** The application is available through the WS-PGRADE/gUSE portal. The user guide to run the AutoDock Vina application is available [https://ltos-gateway.lpds.sztaki.hu/web/wizard/help here]
 
 
*<b>[http://www.ks.uiuc.edu/Research/namd/ NAMD]</b> is a parallel molecular dynamics code designed for high-performance simulation of large bio-molecular systems.
 
*<b>[http://www.ks.uiuc.edu/Research/namd/ NAMD]</b> is a parallel molecular dynamics code designed for high-performance simulation of large bio-molecular systems.
 
** The application is available through the EC3 portal.
 
** The application is available through the EC3 portal.
  
=== <u>Others</u> ===
+
=== <u>Tools</u> ===
  
*<b>Hello World</b> is simple grid-based application that demonstrates the use of remote resources by printing the hostname where the job is executed.
 
**The application is accessible through the Catania Science Gateway. The user manual is available [https://csgf.egi.eu/documents/620030/0/csg_manual_for_ltos.pdf/be218ad7-8a0d-42b7-b932-4b7667524c09 here].
 
*<b>[http://tomcat.apache.org/ Apache Tomcat]</b> is an open source implementation of the Java Servlet, JavaServer Pages, Java Expression Language and Java WebSocket technologies. The Java Servlet, JavaServer Pages, Java Expression Language and Java WebSocket specifications are developed under the [http://jcp.org/en/introduction/overview Java Community Process].
 
** The application is accessible through the EC3 portal.
 
*<b>[http://gnuplot.sourceforge.net/ Gnuplot]</b> is a portable command-line driven graphing utility for Linux, OS/2, MS Windows, OSX, VMS, and many other platforms.
 
** The application is accessible through the EC3 portal.
 
 
*<b>[https://www.docker.com/ Docker]</b> is an open-source project that automates the deployment of Linux applications inside software containers.
 
*<b>[https://www.docker.com/ Docker]</b> is an open-source project that automates the deployment of Linux applications inside software containers.
 
** The application is accessible through the EC3 portal.  
 
** The application is accessible through the EC3 portal.  
Line 103: Line 103:
 
** The application is accessible through the EC3 portal.  
 
** The application is accessible through the EC3 portal.  
 
*<b>[https://mesosphere.github.io/marathon/ Marathon]</b> a production-grade container orchestration platform for Mesosphere's Data Center/Operating System (DC/OS) and Apache Mesos.
 
*<b>[https://mesosphere.github.io/marathon/ Marathon]</b> a production-grade container orchestration platform for Mesosphere's Data Center/Operating System (DC/OS) and Apache Mesos.
** The application is accessible through the EC3 portal.  
+
** The container orchestrator is accessible through the EC3 portal.  
 
*<b>[https://github.com/mesos/chronos Chronos]</b> is a distributed and fault-tolerant scheduler that runs on top of Apache Mesos that can be used for job orchestration.
 
*<b>[https://github.com/mesos/chronos Chronos]</b> is a distributed and fault-tolerant scheduler that runs on top of Apache Mesos that can be used for job orchestration.
** The application is accessible through the EC3 portal.
+
** The scheduler is accessible through the EC3 portal.
 +
*<b>[https://buildmedia.readthedocs.org/media/pdf/o-scar/latest/o-scar.pdf OSCAR]</b> is a framework for to efficiently support on-premises FaaS (Functions as a Service) for general-purpose fileprocessing computing applications.
 +
** The framework is accessible through the EC3 portal.
 +
*<b>[https://kubernetes.io/ Kubernetes]</b> is an open-source container-orchestration system for automating application deployment, scaling, and management. Through this framework users can create a containerized version of workflows and services that can be deployed in the EGI Infrastructure. 
 +
** The container-orchestration system is accessible through the EC3 portal.
 +
 
 +
== Science Gateways and Virtual Research Environments ==
 +
 
 +
*<b>The [http://www.catania-science-gateways.it/ FutureGateway Science Gateway (FGSG)]</b> is a programmable interface of a RESTful API Server, compliant with CSGF APIs specifications, able to provide an easy acces to the PaaS layer by leveraging on recent Web technologies.
 +
# The FGSG incorporates several applications and offers these "as services" for the user.
 +
# The FGSG provides users with an intuitive web interface to execute dockerized applications on the cloud resources of the platform.
 +
#* The FGSG take cares of starting the job in one of the available resources of the platform, of transferring the needed files (e.g. executable, input files, etc.) and downloading the output(s) in behalf of the users. FGSG is open source and released under the Apache 2.0 license.
 +
 
 +
The FGSG integrated in the AoD is available [https://fgsg.ct.infn.it/egissod/web/ssod here]
 +
 
 +
<!--
 +
*<b>The [http://guse.hu/about/architecture/ws-pgrade WS-PGRADE Portal] (Web Services Parallel Grid Runtime and Developer Environment Portal)</b> is the Liferay-based web portal (WS-PGRADE web application) of gUSE, wich also includes a graphical portal service.
 +
# WS-PGRADE is a Web portal hosted in a standard portal framework, using the client APIs of gUSE services to turn user requests into sequences of gUSE specific Web service calls.
 +
# WS-PGRADE is integrated with the cloud resources of the platform, and provides a 'job wizard' interface for the user.
 +
# Through the wizard one can define with a few clicks a computational job that WS-PGRADE will execute on the cloud resources.
 +
#* The environment takes care of instantiation of Virtual Machine image for the job, for sending input data for the job, and for the retrieval of job outputs.
 +
 
 +
The WS-PGRADE/gUSE integrated in the AoD is available [https://guse.egi.eu/c/portal/login?openIdLogin=true here]
 +
-->
  
*<b>[http://jupyter.org/ Jupyter]</b> is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text.
+
*<b>The [http://servproject.i3m.upv.es/ec3/ Elastic Cloud Computing Cluster (EC3)]</b> is a platform which allows to create elastic virtual clusters on top of Infrastructure as a Service (IaaS) providers.
** The application is accessible through the EC3 portal. Instructions on how to start Jupyter Notebook with EC3 are available [https://wiki.egi.eu/wiki/Jupyter_Notebook_with_EC3 here].
+
# The cluster, which is defined with a 'job wizard' interface for the user, is composed by a front node, where a batch job scheduler is running, and a number of compute nodes.  
 +
#* These compute nodes will be dynamically deployed and provisioned to fit increasing load, and undeployed when they are in idle status.
 +
# The list of application libraries/tools installed in the front-node can be exported via NFS in all the compute nodes.
 +
# The installation and configuration of the cluster is performed by means of the execution of Ansible receipts.
 +
# An Enterprise Linux OS is used for both he front-node and the compute nodes of the cluster.
 +
# The cluster provided with AoD is personal - users have root-access to the environment, and can setup and configure the system to their needs. It is meant to be shut down when no longer needed.  
  
== Science Gateways, Portals, Virtual Research Environments ==
+
The EC3 platform requires that ports <u>8800</u> and <u>8989</u> have to be accessible from outside.
  
*<b>The [http://www.catania-science-gateways.it/ Catania Science Gateway (CSG)]</b> is a new generation of Science Gateway based on standard that changes the way e-Infrastructures are used. The gateway incorporates several scientific applications and offers these "as services" for the user. The CSG provides users with an intuitive web interface to execute applications on the grid resources of the platform. The CSG take cares of starting the job in one of the available resources of the platform, of transferring the needed files (e.g. executable, input files, etc.) and downloading the output(s) in behalf of the users. CSG is open source and released under the Apache 2.0 license.
+
The EC3 platform integrated in the AoD is available [https://servproject.i3m.upv.es/ec3-ltos/index.php here]
** The implementation of the CSG tailored for supporting the users of the long tail is available [http://csgf.egi.eu/c/portal/login?openIdLogin=true here]
 
  
*<b>The [http://guse.hu/about/architecture/ws-pgrade WS-PGRADE Portal] (Web Services Parallel Grid Runtime and Developer Environment Portal)</b> is the Liferay-based web portal (WS-PGRADE web application) of gUSE, wich also includes a graphical portal service. WS-PGRADE is a Web portal hosted in a standard portal framework, using the client APIs of gUSE services to turn user requests into sequences of gUSE specific Web service calls. WS-PGRADE is integrated with the cloud resources of the platform, and provides a 'job wizard' interface for the user. Through the wizard one can define with a few clicks a computational job that WS-PGRADE will execute on the cloud resources. The environment takes care of instantiation of Virtual Machine image for the job, for sending input data for the job, and for the retrieval of job outputs.
 
** The implementation of the WS-PGRADE/gUSE tailored for supporting the users of the long tail is available [https://guse.egi.eu/c/portal/login?openIdLogin=true here]
 
  
*<b>[http://servproject.i3m.upv.es/ec3/ Elastic Cloud Computing Cluster (EC3)]</b> is a tool to create elastic virtual clusters on top of Infrastructure as a Service (IaaS) providers. EC3 uses 'recipes' to deploy TORQUE (optionally with MAUI) and SLURM clusters that can be self-managed with CLUES: it starts with a single-node cluster and working nodes will be dynamically deployed and provisioned to fit increasing load (number of jobs at the LRMS). Working nodes will be undeployed when they are idle. EC3 is integrated with the cloud resources of the platform, and provides a 'job wizard' interface for the user.
+
*<b> The [https://dashboard.appdb.egi.eu/ EGI VMOps dashboard]</b>, a framework allows users to perform Virtual Machine (VM) management operations on the EGI Federated Cloud. The graphical interfaces of the dashboard allows the user to choose VMs from the AppDB Catalogue, to define an execution topology for them, and then to instantiate them on infrastructure as a service clouds.  
** The implementation of the EC3 portal tailored for supporting the users of the long tail is available [http://servproject.i3m.upv.es/ec3-ltos/index.php here]
+
 
 +
The EGI VMOps dashboard is available [https://dashboard.appdb.egi.eu/ here]
  
 
= How can you access the service ? =
 
= How can you access the service ? =
  
 +
<!--
 
#Login to the [http://access.egi.eu User Registration Portal (URP)] with the EGI CheckIn service.
 
#Login to the [http://access.egi.eu User Registration Portal (URP)] with the EGI CheckIn service.
 
#Setup a profile, including details about your affiliation and role within a research institute/project/team.
 
#Setup a profile, including details about your affiliation and role within a research institute/project/team.
#Submit a resource request: indicate approximately how much computing and storage capacity you will underneath the scientific application of your choice.  The default allocation is of 10TB of on-line storage and 1000h of CPUs, but you can ask for more resources too.  
+
#Submit a resource request: indicate approximately how much computing and storage capacity you will underneath the scientific application of your choice.  The default allocation is of 10TB of on-line storage and 1000 CPU-hour, but you can ask for more resources (Such requests are evaluated separately).
 
#After your request is approved, login to the gateway/portal environment that hosts the application you would like to use. If you are a developer you can also integrate custom applications with the compute and storage resources you are granted access to.
 
#After your request is approved, login to the gateway/portal environment that hosts the application you would like to use. If you are a developer you can also integrate custom applications with the compute and storage resources you are granted access to.
 +
-->
 +
 +
#Login to the [https://marketplace.egi.eu/ EGI Marketplace] with the EGI AAI Check-In service.
 +
#Setup a profile, including details about your affiliation and role within a research institute/project/team.
 +
#Navigate the marketplace top-menu and click on the category: "Applications".
 +
#Click on the "Applications on Demand" service and submit an order to use one of the available applications.
 +
#When the request is approved, run the requested application(s) hosted in one of the gateway/portal environment.
  
= Help desk =
+
Please check the following [[HowToAccessTheEGIMarketPlace | guide]] for further details.
  
'''For generic questions or inquiries, please contact the EGI Support team at applications-platform-support@mailman.egi.eu'''.
+
= Help-desk =
 +
 
 +
'''For generic questions or inquiries, please contact the EGI Support team at: applications-platform-support@mailman.egi.eu'''.
  
 
For gateway-specific support please contact:  
 
For gateway-specific support please contact:  
* Catania Science Gateway (CSG): sg-licence@ct.infn.it
+
* FutureGateway Science Gateway (FGSG): fgsg@ct.infn.it
* WS-PGRAGE/gUSE portal: portalwide@lpds.sztaki.hu
 
 
* EC3 portal: micafer1@upv.es
 
* EC3 portal: micafer1@upv.es
  
 
= References =  
 
= References =  
 +
Scientific paper describing the service and its status at May 2017: https://documents.egi.eu/document/3132
 +
 
Presentations about the service:
 
Presentations about the service:
 
*Overview of the EGI Infrastructure for serving the long tail (EGI Community Forum, November 2015): [https://indico.egi.eu/indico/contributionDisplay.py?contribId=83&confId=2544]  
 
*Overview of the EGI Infrastructure for serving the long tail (EGI Community Forum, November 2015): [https://indico.egi.eu/indico/contributionDisplay.py?contribId=83&confId=2544]  
Line 145: Line 182:
 
*Slideset about the staus report of the platform at the EGI Conference 2016:  [https://indico.egi.eu/indico/event/2875/session/35/contribution/89]
 
*Slideset about the staus report of the platform at the EGI Conference 2016:  [https://indico.egi.eu/indico/event/2875/session/35/contribution/89]
 
*Slideset about the staus report of the EGI platform at the DI4R Conference 2016:  [http://digitalinfrastructures.eu/content/serving-long-tail]
 
*Slideset about the staus report of the EGI platform at the DI4R Conference 2016:  [http://digitalinfrastructures.eu/content/serving-long-tail]
 +
*Webinar to introduce the Applicatios on Demand (AoD) service to NGIs/USTs representatives, RI architects, resource providers and researchers [https://www.egi.eu/blog/webinar-egi-applications-on-demand-service/]
 +
*Slideset about the Applications on Demand (AoD) service introduced at IWSG 2017: [http://iwsg2017.psnc.pl/programme/]
 +
*IWSG2017 Proceeding [http://ceur-ws.org/Vol-2363/paper9.pdf]
 +
*The EGI applications on Demand service, Future Generation Computer Systems, Volume 98, September 2019, Pages 171-179 [https://doi.org/10.1016/j.future.2019.03.009]

Latest revision as of 12:02, 23 December 2020


Alert.png This article is Deprecated and has been moved to https://docs.egi.eu/users/applications-on-demand/.



Applications on Demand (AoD) Service Information pages

Applications on Demand Service menu: Home Documentation for providers Documentation for developers Architecture




The EGI Applications on Demand service (AoD) is the EGI’s response to the requirements of researchers who are interested in using applications in a on-demand fashion together with the compute and storage environment needed to compute and store data.

The Service can be accessed through the EGI Marketplace

Description of service

The service allows user-friendly access to a portfolio of applications and application hosting frameworks (e.g.: Science Gateways, VREs) that are configured to use the dedicated pool of cloud computing clusters from EGI. The service also allows users to run their own simulation/analysis models with custom input data.

The service operates as an open and extensible ‘hub’ for providers and e-infrastructure user support teams who wish to federated and share applications and services with individual researchers, or small, fragmented communities, typically referred to as ‘the long tail of science’.

The portfolio of applications is currently composed by a readily available set of applications relevant to different scientific and research areas. This portfolio is open to be extended thanks to the contributions of users of the service. If you are interested in this, please get in touch with us: support (at) egi.eu


Intended user groups

  • Researchers and scientists.
  • Application developers who want to share their applications available to the community through AoD.
  • National e-Infrastructures who want to offer the EGI applications library to their national users by "plugging" compute and storage resources.

Requirements and user registration

The service is open for any researchers, scientists and engineers who need a simple and user-friendly access to compute, storage and applications services in order to carry out data/compute intensive science and innovation. The user needs to be affiliated with, or at least have a partner (for example a referee), a research institution in Europe to qualify for access.

Access requires acceptance of Acceptable Use Policy (AUP) and Conditions of Use of the 'EGI Applications on Demand Service'.

Users of the service are asked to provide appropriate acknowledgement of the use in scientific publications. The following acknowledgement text can be used for this purpose:

This work used the EGI Applications on Demand service, which is co-funded by the EOSC-hub project (grant number 777536). The HNSciCloud project (grant number 687614) is also sponsoring the service, allowing users to access the HNSciCloud services pilot for limited scale usage using the voucher schemes provided by the Exoscale contractor.

When requesting access to AoD users are guided through a lightweight registration process. Through a lightweight vetting process, members of the EGI User Support team will perform the needed checks to validate the users' requests before to grant him/her the access to the large scale computing resources.

Service components

  • Scientific applications that are offered ‘as services’ through graphical environments.
  • Science gateways and Virtual Research Environments that offer integrated development environments to port custom applications with cloud resources.
  • Cloud compute resources suited for both compute/data intensive applications and for the hosting of scientific services.
  • A network of Consultants and supporters who can provide guidance on the use of the service.

Service Grant

Each authenticated and authorized User is offered a grant, providing a pre-defined quota of resources, which can be used to run an application of choice.

The EGI grant includes up to 4 vCPU cores, 8GB of RAM and 100GB of block storage.

The grant to run applications is initially valid for 6 months and can be extended/renewed upon request.

Benefits

Through this service:

  • Researchers can access applications underpinned by high-capacity compute and storage servers, to carry out compute and data-intensive simulations. Use of the service does not require advanced experience with distributed and/or cloud computing.
  • Developers can integrate custom applications into the service and offer them as 'scalable, online services' for researchers and scientists worldwide.
  • Providers of compute and storage services can share their clusters and clouds to serve 'the long tail of science'. For more information see AoD for infrastructure providers

Application library

The service operates as an open modular environment where any application developer can integrate and share applications, and providers can plug their own data centre share. The following components are already integrated into the service and are available for users to access:


Computer Science & Mathematics

  • RStudio (v3.2.2) s a free and open-source integrated development environment (IDE) for R, a programming language for statistical computing and graphics..
    • The application is accessible through the FutureGateway Science Gateway. The user manual is available here.

Life Sciences

  • Chispter is a user-friendly analysis software for high-throughput data. It contains over 300 analysis tools for next generation sequencing (NGS), microarray, proteomics and sequence data. Users can save and share automatic analysis workflows, and visualize data interactively using a built-in genome browser and many other visualizations.
    • The application is accessible through the FutureCatania Science Gateway. The user manual is available here.
  • Galaxy is an open, web-based platform for data intensive biomedical research.
    • The application is accessible through the EC3 portal. Instructions on how to use Galaxy workflows with EC3 are available here.
  • NAMD is a parallel molecular dynamics code designed for high-performance simulation of large bio-molecular systems.
    • The application is available through the EC3 portal.

Tools

  • Docker is an open-source project that automates the deployment of Linux applications inside software containers.
    • The application is accessible through the EC3 portal.
  • Hadoop is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.
    • The application is accessible through the EC3 portal.
  • Marathon a production-grade container orchestration platform for Mesosphere's Data Center/Operating System (DC/OS) and Apache Mesos.
    • The container orchestrator is accessible through the EC3 portal.
  • Chronos is a distributed and fault-tolerant scheduler that runs on top of Apache Mesos that can be used for job orchestration.
    • The scheduler is accessible through the EC3 portal.
  • OSCAR is a framework for to efficiently support on-premises FaaS (Functions as a Service) for general-purpose fileprocessing computing applications.
    • The framework is accessible through the EC3 portal.
  • Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management. Through this framework users can create a containerized version of workflows and services that can be deployed in the EGI Infrastructure.
    • The container-orchestration system is accessible through the EC3 portal.

Science Gateways and Virtual Research Environments

  • The FutureGateway Science Gateway (FGSG) is a programmable interface of a RESTful API Server, compliant with CSGF APIs specifications, able to provide an easy acces to the PaaS layer by leveraging on recent Web technologies.
  1. The FGSG incorporates several applications and offers these "as services" for the user.
  2. The FGSG provides users with an intuitive web interface to execute dockerized applications on the cloud resources of the platform.
    • The FGSG take cares of starting the job in one of the available resources of the platform, of transferring the needed files (e.g. executable, input files, etc.) and downloading the output(s) in behalf of the users. FGSG is open source and released under the Apache 2.0 license.

The FGSG integrated in the AoD is available here


  1. The cluster, which is defined with a 'job wizard' interface for the user, is composed by a front node, where a batch job scheduler is running, and a number of compute nodes.
    • These compute nodes will be dynamically deployed and provisioned to fit increasing load, and undeployed when they are in idle status.
  2. The list of application libraries/tools installed in the front-node can be exported via NFS in all the compute nodes.
  3. The installation and configuration of the cluster is performed by means of the execution of Ansible receipts.
  4. An Enterprise Linux OS is used for both he front-node and the compute nodes of the cluster.
  5. The cluster provided with AoD is personal - users have root-access to the environment, and can setup and configure the system to their needs. It is meant to be shut down when no longer needed.

The EC3 platform requires that ports 8800 and 8989 have to be accessible from outside.

The EC3 platform integrated in the AoD is available here


  • The EGI VMOps dashboard, a framework allows users to perform Virtual Machine (VM) management operations on the EGI Federated Cloud. The graphical interfaces of the dashboard allows the user to choose VMs from the AppDB Catalogue, to define an execution topology for them, and then to instantiate them on infrastructure as a service clouds.

The EGI VMOps dashboard is available here

How can you access the service ?

  1. Login to the EGI Marketplace with the EGI AAI Check-In service.
  2. Setup a profile, including details about your affiliation and role within a research institute/project/team.
  3. Navigate the marketplace top-menu and click on the category: "Applications".
  4. Click on the "Applications on Demand" service and submit an order to use one of the available applications.
  5. When the request is approved, run the requested application(s) hosted in one of the gateway/portal environment.

Please check the following guide for further details.

Help-desk

For generic questions or inquiries, please contact the EGI Support team at: applications-platform-support@mailman.egi.eu.

For gateway-specific support please contact:

  • FutureGateway Science Gateway (FGSG): fgsg@ct.infn.it
  • EC3 portal: micafer1@upv.es

References

Scientific paper describing the service and its status at May 2017: https://documents.egi.eu/document/3132

Presentations about the service:

  • Overview of the EGI Infrastructure for serving the long tail (EGI Community Forum, November 2015): [1]
  • Poster and animated slides from Demo at EGI Community Forum, November 2015 (Winner of best demo prize): [2]
  • Slideset about the concept of the EGI long-tail of science platform (from Nov. 2014): [3]
  • Slideset about the authentication and authorization model adopted (from Nov. 2015): [4]
  • Slideset about the staus report of the platform at the EGI Conference 2016: [5]
  • Slideset about the staus report of the EGI platform at the DI4R Conference 2016: [6]
  • Webinar to introduce the Applicatios on Demand (AoD) service to NGIs/USTs representatives, RI architects, resource providers and researchers [7]
  • Slideset about the Applications on Demand (AoD) service introduced at IWSG 2017: [8]
  • IWSG2017 Proceeding [9]
  • The EGI applications on Demand service, Future Generation Computer Systems, Volume 98, September 2019, Pages 171-179 [10]