Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Federated Cloud user support"

From EGIWiki
Jump to navigation Jump to search
Line 106: Line 106:
|-
|-
| '''Web dashboard access'''
| '''Web dashboard access'''
| [[Federated_Cloud_APIs_and_SDKs#AppDB|AppDB VMOps Dashboard (in final test, take a preview)]]
| [[Federated_Cloud_AppDB_VMOps_Dashboard |AppDB VMOps Dashboard (in final test, take a preview)]]
| OpenStack Horizon (production instance under deployment)
| OpenStack Horizon (production instance under deployment)
|}
|}

Revision as of 10:08, 3 February 2017

Overview For users For resource providers Infrastructure status Site-specific configuration Architecture





Technical consultancy and support is available via the EGI.eu Support Team


Introduction

The EGI Cloud federation is a hybrid, Infrastructure as a Service cloud composed by public, community and private cloud providers. These providers are federated with the use of the 'EGI Core Infrastructure Platform', offering a scalable compute and storage infrastructure for scientific applications, services and data- and compute-intensive workloads.

Main features

Elastic computing infrastructure
Execute compute and data intensive workloads (both batch and interactive), host long-running services (e.g. web servers, databases or applications servers), or create disposable testing and development environments in VMs and containers. Scale your application or service within a single provider, or across multiple providers of the federation (within providers of your virtual organisation). Select VM configurations (CPU, memory, disk) and ready-to-deploy application VMs that best fit your needs.
VM image sharing and distribution
Easily share and distribute customised VM images to multiple clouds via the open 'Applications Database' library of Virtual Appliances. Community curated VMs and VM appliances are securely and automatically replicated across the infrastructure. The EGI User Community Support Team provides generic, baseline VM images, user communities can offer more specialised VMs and applications.
Unified view of federation
The EGI Cloud provides: Single sing-on (SSO) for authentication and authorisation across all resource providers; Federated accounting with an integrated view of the the resource and service usage; Distributed information system for delivering a real-time view of the capabilities; and Federated monitoring to compute metrics for availability and reliability of the services.
Beyond VMs
Run docker applications on the EGI resources; Use one of the already integrated PaaS and SaaS solution; Follow our user guides to deploy Hadoop, Docker Swarm, to access Object Storage and many more...

Usage models and examples

The flexibility of the Infrastructure as a Service EGI cloud can benefit various use cases and usage models. Besides serving compute/data intensive analysis workflows, Web services and interactive applications can be also integrated with and hosted on this infrastructure. Contextualisation and other deployment features can help application operators fine tune services in the cloud, meeting software (OS and software packages), hardware (number of cores, amount of RAM, etc.) and other types of needs (e.g. orchestration, scalability).

Since the opening of the EGI Federated Cloud, the following typical usage models have emerged:

  • Service hosting: the EGI Federated Cloud can be used to hosts any IT service as web servers, databases, etc. Cloud features, as elasticity, can help users to provide better performance and reliable services.
  • Compute and data intensive: applications needing considerable amount of resources in term of computation and/or memory and/or intensive I/O. Ad-hoc computing environments can be created in the FedCloud sites also to satisfy very hard HW resource requirements.
  • Datasets repository: the EGI Federated Cloud can be used to store and manage large datasets exploiting the big amount of disk storage available in the Federation.
  • Disposable and testing environments: environments for training or testing new developments.


Getting started

Access to the resources

EGI Infrastructure as a Service (IaaS) Cloud Resources can be accessed through Virtual Organizations (VOs). A VO is a grouping of IaaS cloud provider from the EGI federation, who allocate capacity for a specific user group. Users with similar interest/requirements can join or form a VO to gather resources from EGI cloud providers - typically for a given project, experiment or use case. There are generic VOs too, for example the fedcloud.egi.eu VO, which is open for any user who wants to experiment with the EGI Federated Cloud. You have to join a VO before you can interact with EGI IaaS cloud resources, while higher level services (PaaS, SaaS) do not always require VO membership.

VO membership is controlled in EGI by X.509 certificates. To access the EGI IaaS cloud you need to:

  1. Obtain a personal X.509 access certificate from a recognised Certification Authority (unless you have one already).
  2. Join an existing VO, or form a VO if none of the existing ones suit your purpose:
    1. The fedcloud.egi.eu Virtual Organisation serves as a test ground for users to try the EGI cloud and to prototype and validate applications. It can be used for up to 6 month by any new user.
    2. You can search for and join other established VOs. (Filter for cloud in the middleware column)

Remarks:

  • After the 6-month long membership in the fedcloud.egi.eu VO, you will need to move to a production VO, or establish a new VO.
  • Pre-existing VOs of EGI can be also extended with cloud resources. Consult with your VO manager.
  • If none of the existing VOs matches your use case, then a new VO can be created. Please follow VO Registration procedure. We can invite sites from the infrastructure to support your VO.

Basics

The basic user workflow in the EGI Federated Cloud looks like this:

Fedcloud.png


  • The IaaS Cloud user (or a user gateway that acts on behalf of the users) is responsible for managing the virtual infrastructure on top of the IaaS provider. He/she spins up Virtual Machines, Block Storage and Object Storage on the providers. The Federated Cloud is composed of a set of providers distributed all across Europe.
  • The VM instances are started using the images available as Virtual Appliances (VA) at EGI Applications Database (AppDB). Virtual Appliances are the templates for the root volume of the running instances (Operating System and applications). EGI offers a set of basic images with minimal configuration that can get you started easily, but you can also find complete application stacks. See for example these images:
  • New Virtual Appliances can be registered by Cloud Developers that have the knowledge to create and package images and make them available for a wider community. These VAs are then managed in the AppDB with the help of the special VO members that curate which appliances are available to their VO.
  • Virtual infrastructure managed by the IaaS cloud user will provide some services to be accessed by Service Users (could be the very same IaaS user but also other VO members or unrelated users). The method of accessing the services vary greatly from one service to another.

IaaS cloud resources on EGI can expose two types of interfaces (one or the other or both - depending on the cloud provider):

  • Open Standard interfaces: OCCI (Open Cloud Computing Interface) to manage compute, blocks storage and network resources. This interface set are currently exposed by all of the OpenNebula and Synnefo cloud providers, and some of the OpenStack providers.
  • OpenStack interfaces: The native OpenStack interfaces (with X.509 authentication). These interfaces are currently exposed by all of the OpenStack-based EGI cloud providers.

The user can interact with IaaS cloud resources via programming APIs, command line interfaces or Web dashboards. The different access modes are summarized in the following table:

Open Standards interface OpenStack interface
API level access OCCI OpenStack Compute & Openstack Object Storage
Command Line access rOCCI-cli OpenStack CLI with VOMS authentication plugin
Web dashboard access AppDB VMOps Dashboard (in final test, take a preview) OpenStack Horizon (production instance under deployment)

Creating your first VM instance

Follow this simple steps to start your first VM using the OCCI CLI:

  1. Get your environment ready to launch VMs. The command line client HOWTO describes how to install on your machine the client tool or how to get a docker image or VM image for VirtualBox that can be used to run the commands. You will need your certificate on your client.
  2. Browse the Application Database Cloud Marketplace the available Virtual Appliances. You can use a bare OS like this EGI Ubuntu 14 to get started.
  3. Get the IDs for starting the appliance at one of the sites supporting it in the "Availability and Usage" tab.
  4. Virtual Appliances in AppDB are ready to start on the EGI resources, but in order to use them, you will need to perform some contextualisation. Contextualisation is the process of customising the appliance when it is instantiated on the resources, e.g. entering some credentials to log into the VM.
    • Passwords are not allowed by default in EGI Federated Cloud for accessing VMs, you would likely need a ssh key to proceed. Check the FAQ on how to create one
    • Most VAs in AppDB support cloud-init, the de-facto standard for contextualisation. Check the documentation and examples.
  5. Create a VOMS proxy for getting access to the infrastructure
  6. Use the client to start the VM using the IDs obtained from AppDB.
  7. Now you can connect to the VM and start using it
    • You may need to allocate a public IP for the VM, check How can I assign a public IP to a VM in the Federated Cloud FAQ
    • Login using ssh: ssh -i <your private key> ubuntu@<your vm ip> (ubuntu is the default username for the Ubuntu images in AppDB)

Check out these tutorial slides for a practical overview of these steps.

API and SDKs access to Federated Cloud resources

Besides the command line client, there are several APIs and SDK ready to be used with the EGI Federated Cloud.Preferred API for EGI federated cloud is OCCI, which can be accesses using using a command-line client (rOCCI), high-level tools or directly implementing the ‘Open Cloud Computing Interface’ (OCCI) into your environment. OpenStack Nova API is also available for OpenStack sites belonging to the Federation.

A crash course on how to use programming interfaces of the EGI Federated Cloud, and how these APIs can be used to integrate high-level systems with it is available here. Please check the EGI Federated Cloud for developers guide for details on how to use them.

Creating custom appliances

You can prepare fully customised Virtual Appliances and make them available to the sites supporting your VO.

  1. First, prepare a Virtual Machine Image (VMI) that encapsulates your application.
  2. Make the VMI available online, for example in the EGI Appliance Repository
  3. Register the VMI as a new Virtual Appliance in the EGI Applications Database
  4. Once your VA is published, inform your VO through Applications Database about it.
  5. Once your appliance is in the VO-wide image list, it will be deployed on the Federated Cloud sites of your VO.

User guides

User roles in the EGI Federated Cloud

EGI Cloud resources are accessed through Virtual Organizations (VOs). Users that are members of a VO will have access to the providers supporting that VO: they will be able to manage VMs, block storage and object storage available to the VO. Resources (VMs and storage) in some providers may be shared across all members of the VO, please do not interfere with the VMs of other users (specially do not delete them).

Additionally, there are roles in the VO that have special consideration in the EGI Federated Cloud, listed below:

  • VO-wide image lists (the Virtual Appliances included in AppDB for the VO) can be managed by users that have the VO Manager, VO Expert or VO deputy roles. Information about the VO members with these roles is fetched from Operations Portal or for certain VOs from specialised attribute authorities. Check the AppDB guide on VO-wide image list management for more information.

Storage and Data Management

Every instantiated VM has some disk space provided with it, if you need more storage or need to share data, you can use a cloud storage solution. There are two kind of services: Block Storage and Object Storage. Check the EGI Federated Cloud Storage How To for more information.

The EGI OpenData platform is a solution allowing integration of various data repositories available in a distributed infrastructure, offering the capability to make data open, and link them to key open data catalogues following respective guidelines. The core enabling technology of OpenData platform is Onedata, a data management solution that allows a seamless and optimised access to data spread over a distributed infrastructure. Instructions on how to setup a OneData deployment in the EGI Federated Cloud are available here.

Containers

You can run your docker applications on EGI's Cloud. EGI also provides a docker image with the clients so you can test them easily. Docker Swarm, Kubernetes and Apache Mesos can be easily used on the resources provided by EGI.

High level tools: Orchestrators, Platforms/Software as a Service

Read this guidance about strategies of porting applications to the EGI Federated Cloud. The guide also includes references to high level user environments (orchestrators, Platform/Software as a Service) that can simplify the application integration and operation process for you. These environments offer high level abstractions and services on top of the baseline 'Infrastructure as a Service' cloud.

Community Specific Software

Running Galaxy Workflows with EC3

Using the EC3 open-source software platform, users can deploy elastic clusters on demand and dynamically deploy complex scientific virtual computing infrastructures on top of Infrastructure as a Service Clouds. More details on how to use Galaxy workflows on the platform are described in this wiki. Please check the Galaxy workflows in EGI with EC3 guide for further details.

Computer-Aided Engineering (CAE)

Please check how SMEs can use Computer-Aided Engineering (CAE) in the EGI Federated Cloud, with the example of OpenFOAM software containers.

Data Analytics

Running Hadoop applications

Using a WS-PGRADE gateway that is connected to the EGI Federated cloud, it is possible to deploy Hadoop clusters on EGI Federated Cloud resources, to execute Hadoop applications on those clusters and finally to release resources after application execution. The concept is outlined on these PPT slides. A user manual is available here (v1.3).

GPGPUs (pre-production!)

PRE-PRODUCTION GPGPUs-enabled cloud resources are available in selected sites of the EGI Federated Cloud. Check the GPGPU guide on FedCloud for details on how to access them.

Useful resources

User support

Technical support

Users' technical support is provided via the EGI support contact.

Helpdesk

Technical problems and questions relating to the use of the EGI Federated Cloud can be reported and dealt with through the EGI Helpdesk ticketing system.

Note: Please choose 'Federated cloud' in the 'Type of problem' field of the ticket submission form!

Feedback and open issues

A list of open-issue and feedbacks reported by the FedCloud users is available at this page.

Technical background

Cloud providers in the EGI Federated Cloud use hardware virtualization technologies to host software on their resources. The cloud management platforms that make this possible can vary from site to site, but they all enable the provisioning of virtualized computing, storage and networking resources, thus they empower scientific groups to setup and operate domain specific services, applications and simulations on these resources. Read more about the technology that drives the Federated Cloud.