Federated Cloud user support
|Overview||For users||For resource providers||Infrastructure status||Innovation roadmap|
Technical consultancy and support for user is available via the EGI.eu Support Team
The EGI Cloud federation is an hybrid cloud composed by public, community and private clouds, all supported by the EGI Core Infrastructure Platform services and focusing on the requirements of the scientific community. The result is a new type of research e-infrastructure based on the mature federated operations services that make EGI a reliable resource for science. When using EGI Federated Cloud resources, researchers and research communities can count on:
- Elastic computing infrastructure.
- Execute compute and data intensive workloads (both batch and interactive), host long-running services (e.g. web servers, databases or applications servers), or create disposable testing and development environments on VMs and scale your infrastructure needs over a federation of cloud providers. Select VM configurations (CPU, memory, disk) and application environments that better fit your requirements.
- VM image sharing and distribution
- Easily share and distribute VM images for your applications on a open library of Virtual Appliances. Community curated appliances are securely and automatically replicated accross the infrastructure. EGI provides a set of securely configured images with basic software ready to use on all providers that can be re-used and extended with contextualisation to execute your applications
- Unified view of federation
- The EGI Cloud provides: Single sing-on (SSO) for authentication and authorisation across all resource providers; Federated accounting with an integrated view of the the resource and service usage; Distributed information system for delivering a real-time view of the capabilities; and Federated monitoring to compute metrics for availability and reliability of the services.
- Beyond VMs
- Run docker applications on the EGI resources or use any of the supported PaaS and SaaS solutions that extend the IaaS capabilities of the EGI cloud and abstract the infrastructure to let you focus on your application development.
Current Users and Communities
EGI Federated Cloud already has a large use base, check the FedCloud Users Communities page for more details.
The EGI Federated Cloud considerably widens the usage models supported by EGI. Now, web services and interactive applications can be easily integrated in the infrastructure, the computing environments can be finely tuned to satisfy user’s needs in term of software (OSs and software packages) and hardware (number of cores, amount of RAM, etc.) and, many solutions are available to store, update and access big amount of data. These new opportunities offered by EGI hugely extended the potential user base of the infrastructure opening the doors to new research communities with minimal or none knowledge of the EGI ecosystem.
We classified the usage models enabled by the EGI Fededated Cloud as follows:
- Service hosting: the EGI Federated Cloud can be used to hosts any IT service as web servers, databases, etc. Cloud features, as elasticity, can help users to provide better performance and reliable services.
- Compute and data intensive: applications needing considerable amount of resources in term of computation and/or memory and/or intensive I/O. Ad-hoc computing environments can be created in the FedCloud sites also to satisfy very hard HW resource requirements.
- Datasets repository: the EGI Federated Cloud can be used to store and manage large datasets exploiting the big amount of disk storage available in the Federation.
- Disposable and testing environments: environments for training or testing new developments.
How to use the EGI Federated Cloud?
EGI provides users with a single sign-on mechanism to access the federated services with the use of X.509 certificates and Virtual Organisations (VOs). Before using the EGI Federated Cloud you will need to:
- Obtain a personal access certificate from a recognised Certification Authority (if you don't own one already).
- Join fedcloud.egi.eu Virtual Organisation. This incubator VO provides resources for application prototyping and validation. It can be used for up to 6 month by any new user.
- If you are part of an existing VO that has access to cloud resources, you may use that VO.
Creating your first VM - Instantiate an exiting image
The basic user workflow for starting a Virtual Machine (VM) looks like this:
- Browse in the Application Database Cloud Marketplace the available Virtual Appliances. Virtual Appliances are the templates for the root volume of the running instances (Operating System and applications). EGI offers a set of basic images with minimal configuration that can get you started easily, but you can also find complete application stacks. See for example these images:
- Read the description of the appliance for information on how to use it and connect to it once started.
- You may need to have a ssh key in order to login into the VM, check the FAQ on how to create one
- Get the IDs for starting the appliance at one of the sites supporting it in the "Availability and Usage" tab.
- Get the command line client to start the VM using the IDs from previous step
- Connect to VM and start using it
- You may need to allocate a public IP for the VM, check How can I assign a public IP to a VM in the Federated Cloud FAQ
Check out these tutorial slides for a practical overview of these steps.
API and SDKs access to Federated Cloud resources
Besides the command line client, there are several APIs and SDK ready to be used with the EGI Federated Cloud.Preferred API for EGI federated cloud is OCCI, which can be accesses using using a command-line client (rOCCI), high-level tools or directly implementing the ‘Open Cloud Computing Interface’ (OCCI) into your environment. OpenStack Nova API is also available for OpenStack sites belonging to the Federation.
A crash course on how to use programming interfaces of the EGI Federated Cloud, and how these APIs can be used to integrate high-level systems with it is available here. Please check the EGI Federated Cloud for developers guide for details on how to use them.
Every instantiated VM has some disk space provided with it, if you need more storage or need to share data, you can use a cloud storage solution. There are two kind of services: Block Storage and Object Storage. Check the EGI Federated Cloud Storage How To for more information.
Distributed data management with the EGI OpenData platform
The EGI OpenData platform is a solution allowing integration of various data repositories available in a distributed infrastructure, offering the capability to make data open, and link them to key open data catalogues following respective guidelines. The core enabling technology of OpenData platform is Onedata, a data management solution that allows a seamless and optimised access to data spread over a distributed infrastructure. Instructions on how to setup a OneData deployment in the EGI Federated Cloud are available here.
Docker containers, Docker Swarm
High level tools: Orchestrators, Platforms/Software as a Service
Read in this guidance about strategies of porting applications to the EGI Federated Cloud. The guide also includes references to high level user environments (orchestrators, Platform/Software as a Service) that can simplify the application integration and operation process for you. These environments offer high level abstractions and services on top of the baseline 'Infrastructure as a Service' cloud.
Running Hadoop applications
Using a WS-PGRADE gateway that is connected to the EGI Federated cloud, it is possible to deploy Hadoop clusters on EGI Federated Cloud resources, to execute Hadoop applications on those clusters and finally to release resources after application execution. The concept is outlined on these PPT slides. A user manual is available here (v1.3).
PRE-PRODUCTION GPGPUs-enabled cloud resources are available in selected sites of the EGI Federated Cloud. Check the GPGPU guide on FedCloud for details on how to access them.
Customized Virtual Appliances
You can prepare fully customised Virtual Appliances and deploy them to the sites:
- Prepare Virtual Machine Images (VMIs) that encapsulate your application. See the application porting tutorial for tips.
- Make the VMI available online, for example in the EGI Appliance Repository
- Register the VMIs as Virtual Appliances in the EGI Applications Database (for howto please click here)
- Once your VA is published (see howto here),inform the Manager of your VO through Applications Database about it (see howto here). He/she will include your images in the VO-wide image list, so these will be deployed on the Federated Cloud sites of your VO.
Once the 6-month testing period of fedcloud.egi.eu membership expires, you will need to move into a production VO:
- Several other VOs of EGI make resources available from the Federated Cloud. Find a suitable VO in the Operations Portal. (Search for Cloud as a middleware type.)
- Existing grid production VOs can be extended by VO manager to cloud by declaring cloud resources in VO ID card (in Operations Portal) and requesting via GGUS (assign to Perun Support Unit) Perun support.
- If no existing VO suits your case, a new VO can be created. Please follow VO Registration procedure. You can invite sites from the infrastructure to support them.
Running Galaxy Workflows with EC3
Using the EC3 open-source software platform, users can deploy elastic clusters on demand and dynamically deploy complex scientific virtual computing infrastructures on top of Infrastructure as a Service Clouds. More details on how to use Galaxy workflows on the platform are described in this wiki. Please check the Galaxy workflows in EGI with EC3 guide for further details.
How SMEs Can Use EGI's Cloud for Computer-Aided Engineering (CAE)
Please check how SMEs can use Computer-Aided Engineering (CAE) in the EGI Federated Cloud, with the example of OpenFOAM software containers.
- FedCloud FAQ page
- How to get a certificate (to access Federated Cloud resources)
- How to use the Applications Database Cloud Marketplace
- Porting your application/web service to the EGI Federated Cloud
- List and details about certified cloud resource providers
- Monitor tests performed by EGI on cloud resources
Users' technical support is provided via the EGI support contact.
Technical problems and questions relating to the use of the EGI Federated Cloud can be reported and dealt with through the EGI Helpdesk ticketing system.
Note: Please choose 'Federated cloud' in the 'Type of problem' field of the ticket submission form!
Feedback and open issues
A list of open-issue and feedbacks reported by the FedCloud users is available at this page.
Cloud providers in the EGI Federated Cloud use hardware virtualization technologies to host software on their resources. The cloud management platforms that make this possible can vary from site to site, but they all enable the provisioning of virtualized computing, storage and networking resources, thus they empower scientific groups to setup and operate domain specific services, applications and simulations on these resources. Read more about the technology that drives the Federated Cloud.