Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Federated Cloud user support"

From EGIWiki
Jump to navigation Jump to search
(Use new template, update new page address)
 
(5 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Fedcloud_Menu}} {{TOC_right}}  
{{Fedcloud_Menu}} {{TOC_right}}  


<big>This page is the user guide and catalogue of technical manuals for users of the EGI Cloud service. Additional consultancy and support for users and cloud resource providers is available via the [mailto:support@egi.eu EGI.eu Support Team]</big>
{{DeprecatedAndMovedTo|new_location=https://docs.egi.eu/users/cloud-compute/}}
 
 
== What is the EGI cloud? ==
The EGI Cloud service is implemented in the form of a Federated Cloud. This EGI Federated Cloud is a standards-based, open cloud system that federates institutional clouds to offer a scalable computing platform for data and/or compute driven applications and services in research and science. The EGI Federated Cloud benefits researchers and innovators in two ways:
# The EGI cloud service: The EGI cloud federation includes 23 cloud sites from all across Europe. These clouds are available for users through community allocations, so called Virtual Organisations. Each Virtual Organisation includes a subset of the federated cloud sites, and makes those available for the given community through generic and/or community-specific policies and protocols. Members of a scientific community have to join the VO to access the cloud capabilities offered by the federated VO sites. The EGI federation model ensures single-sign on (i.e. after a user registers to the VO he/she is able to access every VO cloud); uniform interfaces (i.e. each VO cloud can be accessed via the same/harmonized interfaces) and application portability (i.e. every VO cloud uses the same Virtual Machine (VM) image and contextualization format). VO members can deploy new VMs on the cloud sites through the EGI AppDB VM marketplace, and can instantiate VMs and block storages via the graphical AppDB VMOps Dashboard or using the API and command line interfaces offered by the cloud sites. High level tools, such as orchestrators and application portals can offer additional, and science domain-specific capabilities for users. This user guide covers topics relating to getting access and using the EGI cloud service.
# The technology stack: The EGI Federated Cloud is built from open source software components, maintained by an open consortium, the Federated Cloud Task Force. The technology stack is currently capable of federating OpenStack, OpenNebula and Synnefo clouds. EGI promotes the federated cloud technology stack for scientific communities who want to establish community-specific cloud federations, and assists them through this process. The EGI security and operational policies and service management practices offer a baseline, but customizable framework for operating the community specific cloud federations. If you would like to build a community-specific cloud federation, then please get in touch with support@egi.eu and we help you in the process.
 
The EGI Federated Cloud, its technology stack and the related operational and security processes enable scientific communities to (1) share resources and applications across institutes and national borders; (2) develop portable, standard-based applications and services; (3) operate high-quality services for science; and ultimately to (4) establish sustainable e-infrastructures for large-scale, digital science.
 
== Introduction about using the EGI cloud service ==
 
The [https://www.egi.eu/services/cloud-compute/ EGI Cloud service] gives you the ability to deploy and scale virtual machines on-demand. It offers guaranteed computational resources in a secure and isolated environment without the overhead of managing physical servers. Cloud Compute offers the possibility to select pre-configured virtual appliances (e.g. CPU, memory, disk, operating system or software) from a catalogue replicated across all EGI cloud providers.
 
The EGI Cloud compute service is implemented as a hybrid, Infrastructure as a Service (IaaS) cloud composed by public, community and private cloud providers. These providers are federated with the use of the 'EGI Core Infrastructure Platform', offering a scalable compute and storage infrastructure for scientific applications, services and data- and compute-intensive workloads.
 
=== Main features ===
 
;Elastic computing infrastructure
:Execute compute and data intensive workloads (both batch and interactive), host long-running services (e.g. web servers, databases or applications servers), or create disposable testing and development environments in VMs and containers. Scale your application or service within a single provider, or across multiple providers of the federation (within providers of your virtual organisation). Select VM configurations (CPU, memory, disk) and ready-to-deploy application VMs that best fit your needs.
 
;VM image sharing and distribution
:Easily share and distribute customised VM images to multiple clouds via the open 'Applications Database' library of Virtual Appliances. Community curated VMs and VM appliances are securely and automatically replicated across the infrastructure. The EGI User Community Support Team provides generic, baseline VM images, user communities can offer more specialised VMs and applications.
 
;Unified view of federation
:The EGI Cloud provides: Single sing-on (SSO) for authentication and authorisation across all resource providers; Federated accounting with an integrated view of the the resource and service usage; Distributed information system for delivering a real-time view of the capabilities; and Federated monitoring to compute metrics for availability and reliability of the services.
 
;Beyond VMs
:Run docker applications on the EGI resources; Use one of the already integrated PaaS and SaaS solution; Follow our user guides to deploy Hadoop, Docker Swarm, to access Object Storage and many more...
 
=== Usage models and examples ===
 
The flexibility of the Infrastructure as a Service EGI cloud can benefit various use cases and usage models. Besides serving compute/data intensive analysis workflows, Web services and interactive applications can be also integrated with and hosted on this infrastructure. Contextualisation and other deployment features can help application operators fine tune services in the cloud, meeting software (OS and software packages), hardware (number of cores, amount of RAM, etc.) and other types of needs (e.g. orchestration, scalability).
 
Since the opening of the EGI Federated Cloud, the following typical usage models have emerged:
 
*'''Service hosting''': the EGI Federated Cloud can be used to hosts any IT service as web servers, databases, etc. Cloud features, as elasticity, can help users to provide better performance and reliable services.
** Example: [https://www.egi.eu/use-cases/scientific-applications-tools/nbis-toolkit/ NBIS Web services], [https://www.egi.eu/news/peachnote-in-unison-with-egi/ Peachnote analysis platform]
*'''Compute and data intensive''': applications needing considerable amount of resources in term of computation and/or memory and/or intensive I/O. Ad-hoc computing environments can be created in the FedCloud sites also to satisfy very hard HW resource requirements.
** Example: [https://www.egi.eu/news/new-egi-use-case-a-close-look-at-the-amatrice-earthquake/ VERCE platform], [https://www.egi.eu/use-cases/research-stories/the-genetics-of-salmonella-infections/ The Genetics of Salmonella Infections], [https://www.egi.eu/use-cases/research-stories/new-viruses-implicated-in-fatal-snake-disease/ The Chipster Platform]
*'''Datasets repository''': the EGI Federated Cloud can be used to store and manage large datasets exploiting the big amount of disk storage available in the Federation.
*'''Disposable and testing environments''': environments for training or testing new developments.
** Example: [[Training_infrastructure|Events conducted on the cloud-based EGI Training Infrastructure]]
 
<!--
=== Success Stories ===
Some links
 
-->
<!-- == Current Users and Communities  ==
 
EGI Federated Cloud already has a large use base, check the [[Federated Cloud Communities|FedCloud Users Communities]] page for more details.
 
[[Image:Fedclouduclogos.png|center|550px|Fedclouduclogos.png]]
-->
 
== Access to the resources ==
 
Cloud Compute service is accessed through '''Virtual Organisations (VOs)'''. A VO is a grouping of IaaS cloud providers from the federation, who allocate capacity for a specific user group. Users with similar interest/requirements can join or form a VO to gather resources from EGI cloud providers - typically for a given project, experiment or use case. There are generic VOs too, for example the fedcloud.egi.eu VO, which is open for any user who wants to experiment with the service. '''You have to join a VO before you can interact with the cloud resources at the IaaS level''', while interaction through higher level services (PaaS, SaaS) do not always require VO membership.
 
<!-- Rephrase -->
 
Most of the existing VOs in EGI rely on X.509 certificates for user authentication. While we are transitioning to a certificate-less experience for using the service, membership to the VO still require users to obtain a personal certificate from a recognised Certification Authoriy (unless you have one already). Check the following table for the available options to access the Cloud service:
 
<!-- Add a "user facing" description of VO, add info how to get a certificate link -->
 
{| cellspacing="5" cellpadding="5" border="0" class="wikitable"
|- style="background:lightgray;"
| '''VO'''
| '''fedcloud.egi.eu'''
| '''access.egi.eu (also called 'Applications on Demand Service', AoDS)'''
| '''Disciplinary VO'''
| '''Create your own VO'''
|-
| '''Membership requirements'''
| Valid IGTF certificate
| Member of European research institution.
| Depends on VO (most VOs require valid IGTF certificate). Check the available VOs at [https://operations-portal.egi.eu/vo/search Operations Portal]
| New VOs can use [https://aai.egi.eu EGI Check-in] accounts or IGTF certificates.
|-
| '''[[Federated_Cloud_AppDB_VMOps_Dashboard |AppDB VMOps access]]'''
| Yes
| Yes
| Depends on the VO. Access to OpenStack dashboard may be also available for OpenStack providers
| To be negotiated
|-
| '''[[Federated_Cloud_APIs_and_SDKs|CLI/API access]]'''
| Yes
| No
| Yes
| To be negotiated
|-
| '''Membership duration'''
| 6 month period, extendible up to 1 year
| 6 month period, extendible up to 1 year
| Defined by VO, most VOs have 1 year membership renewal mechanism
| To be negotiated
|-
| '''Resource limits'''
| Opportunistic access to resources, varying quota at each provider, typically VMs with up to 4 vCPUs and 2GB per core. Not guaranteed resources and limited lifetime of up to 3/6 months per VM ('''i.e. VMs may be killed automatically by the provider''').
| Opportunistic access to resources, varying quota at each provider, typically VMs with up to 4 vCPUs and 2GB per core. Not guaranteed resources and limited lifetime of up to 3/6 months per VM ('''i.e. VMs may be killed automatically by the provider''').
| Most VOs have guaranteed access to resources, details specified at VO SLA.
| To be negotiated.
|-
| '''Available providers'''
| [https://www.egi.eu/federation/egi-federated-cloud/ All providers of the federation]
| Providers of the Applications on Demand platform: INFN-Catania-Stack, CESGA, RECAS-Bari (See [https://documents.egi.eu/public/ShowDocument?docid=2773  SLA/OLA] for details)
| Depends on the VO
| To be negotiated
|-
| '''How to get access'''
| [https://perun.metacentrum.cz/cert/registrar/?vo=fedcloud.egi.eu Request membership to fedcloud.egi.eu] (requires valid IGTF certificate)
| Order at [https://marketplace.egi.eu/31-cloud-compute EGI Marketplace] with your [https://aai.egi.eu EGI Check-in] account
| Check the VO enrolment mechanism on [https://operations-portal.egi.eu/vo/search Operations portal]
| Request access via [https://marketplace.egi.eu/31-cloud-compute EGI Marketplace] with your [https://aai.egi.eu EGI Check-in] account
|}
 
==== Getting a user certificate ====
 
The easiest option is to get an ‘eScience Personal’ certificate online from the Terena Certificate Service CA. Check the countries where this is available, and follow the link to the respective CA page at the [https://www.terena.org/activities/tcs/participants.html TCS participants list] (See FAQs for details.)
 
If eScience Personal certificate is not available in your country, then obtain a certificate from a regular [https://www.eugridpma.org/members/worldmap/ IGTF CA] (this may require a personal visit at the CA).
 
== Basics ==
 
=== Worklflow ===
 
The basic user workflow in the EGI Federated Cloud is summarised in the following picture:
 
[[Image:Fedcloud.png|center|650px|Fedcloud.png]] <br>
 
* The '''IaaS Cloud user''' (or a user gateway that acts on behalf of the users) is responsible for managing the virtual infrastructure on top of the IaaS provider. He/she spins up Virtual Machines, Block Storage and Object Storage on the providers. The Federated Cloud is composed of a set of providers distributed all across Europe.
 
* The VM instances are started using the images available as Virtual Appliances (VA) at [http://appdb.egi.eu EGI Applications Database (AppDB)]. Virtual Appliances are the templates for the root volume of the running instances (Operating System and applications). EGI offers a set of basic images with minimal configuration that can get you started easily, but you can also find complete application stacks. See for example these images:
**[https://appdb.egi.eu/store/vappliance/egi.ubuntu.14.04 EGI Ubuntu 14], a basic [http://www.ubuntu.com/ Ubuntu 14.04] image
**[https://appdb.egi.eu/store/vappliance/egi.centos.6 EGI Centos 6], a basic [https://www.centos.org/ CentOS 6] image
**[https://appdb.egi.eu/store/software/compss.framework COMPSs], a VM for using [[HOWTO14_How_to_use_COMPSs|COMPSs]] programming framework
**[https://appdb.egi.eu/store/vappliance/biovel.portal BioVel Portal], for executing a http://www.biovel.eu/ biodiversity virtual e-Laboratory]
 
* New Virtual Appliances can be registered by '''Cloud Developers''' that have the knowledge to create and package images and make them available for a wider community. These VAs are then managed in the AppDB with the help of the special VO members that curate which appliances are available to their VO.
 
* Virtual infrastructure managed by the IaaS cloud user will provide some services to be accessed by '''Service Users''' (could be the very same IaaS user but also other VO members or unrelated users). The method of accessing the services vary greatly from one service to another.
 
=== Starting your first VM instance with VMOps dashboard ===
 
Follow these steps for starting your first VM instance using the web GUI of AppDB VMOps
 
# Log into the VMOps dashboard at [https://dashboard.appdb.egi.eu/vmops] using your EGI CheckIn credentials
# Click on "Create a new VM Topology" to start the topology builder, this will guide you through a set of steps:
## select the Virtual Appliance you want to start, these are the same shown in the [https://appdb.egi.eu/browse/cloud Application Database Cloud Marketplace], you can use the search field to find your VA;
## select the VO to use when instantiating the VA;
## select the site where to instantiate the VA; and finally
## select the template (flavour) of the instance that will determine the number of cores, memory and disk space used in your VM.
# Now you will be presented with a summary page where you can further customise your VM by:
#* Adding more VMs to the topology
#* Adding block storage devices to the VMs
#* Define contextualisation parameters (e.g. add new users, execute some script)
# Click on "Launch" and your deployment will be submitted to the infrastructure
# The topology you just created will appear on your "Topologies" with all the details about it, clicking on a VM of a topology will give you details about its status and IP. You can login into the VM with any user you created in the contextualisation parameters, even if you didn't specify any users, the AppDB creates one for you and provides the credentials for login via ssh to the new VM.
# If the VM has a default password, remember to replace it with a strong secret password immediately.
 
Check the [[Federated_Cloud_AppDB_VMOps_Dashboard| AppDB VMOps Dashboard guide]] for more detailed information and screenshots.
 
=== API and SDKs access to Federated Cloud resources ===
 
Besides the dashboard, there are several [[Federated Cloud APIs and SDKs|APIs and SDK ready to be used with the EGI Federated Cloud]]. IaaS cloud resources on EGI expose their interface of choice (one or the other or both - depending on the cloud provider):
* '''Open Standard interfaces''': OCCI ([http://occi-wg.org/ Open Cloud Computing Interface]) to manage compute, blocks storage and network resources. This interface set are currently exposed by all of the OpenNebula and Synnefo cloud providers, and some of the OpenStack providers.
* '''OpenStack interfaces''': The native OpenStack interfaces (with X.509 authentication/OpenID Connect). These interfaces are currently exposed by all of the OpenStack-based EGI cloud providers.
 
A crash course on how to use programming interfaces of the EGI Federated Cloud, and how these APIs can be used to integrate high-level systems with it is available [https://indico.egi.eu/indico/event/3113/ here].
 
== User guides ==
 
=== Authorization and User roles in the EGI Federated Cloud ===
 
EGI Cloud resources are accessed through '''''Virtual Organizations''''' (VOs). Users that are members of a VO will have access to the providers supporting that VO: they will be able to manage VMs, block storage and object storage available to the VO. '''Resources (VMs and storage) in some providers may be shared across all members of the VO''', please do not interfere with the VMs of other users (specially do not delete them).
 
Additionally, there are roles in the VO that have special consideration in the EGI Federated Cloud, listed below:
* VO-wide image lists (the Virtual Appliances included in AppDB for the VO) can be managed by users that have the ''VO Manager'', ''VO Expert'' or ''VO deputy'' roles. Information about the VO members with these roles is fetched from Operations Portal or for certain VOs from specialised attribute authorities. Check the [https://wiki.appdb.egi.eu/main:guides:manage_vo-wide_image_lists AppDB guide on VO-wide image list management] for more information.
 
=== Storage and Data Management ===
 
Every instantiated VM has some disk space provided with it, if you need more storage or need to share data, you can use a cloud storage solution. There are two kind of services: Block Storage and Object Storage. Check the [[HOWTO09|EGI Federated Cloud Storage How To]] for more information.
 
The EGI OpenData platform is a solution allowing integration of various data repositories available in a distributed infrastructure, offering the capability to make data open, and link them to key open data catalogues following respective guidelines. The core enabling technology of OpenData platform is Onedata, a data management solution that allows a seamless and optimised access to data spread over a distributed infrastructure. Instructions on how to setup a OneData deployment in the EGI Federated Cloud are available [[EGI_Opendata_platform|here]].
 
=== Creating custom appliances ===
 
You can prepare fully customised Virtual Appliances and make them available to the sites supporting your VO.
 
# First, prepare a Virtual Machine Image (VMI) that encapsulates your application.
#* There are several methods for preparing the image, check  the [[Federated_Cloud_Virtual_Machine_Image_Preparation|VMI guide]] for tips
#* We use [https://packer.io packer] with [https://www.virtualbox.org VirtualBox], which can run easily on your computer
#* Beware images should not contain any credentials, use [https://cloudinit.readthedocs.io/en/latest/ cloud-init] and follow [[Virtual_Machine_Image_Endorsement#Hardening_guidelines |hardening guidelines]].
#Make the VMI available online, for example in the [http://appliance-repo.egi.eu/images/ EGI Appliance Repository]
#* See [[FAQ10#How_can_I_upload_a_VM_image_to_the_EGI_FedCloud_repository | How can I upload a VM image to the EGI FedCloud repository]] entry in the FAQ
#* If you don't use the EGI Appliance Repository, please ensure that the server used has enough bandwidth to allow sites download the image.
# Register the VMI as a new Virtual Appliance in the [http://appdb.egi.eu EGI Applications Database]
#* See  [https://wiki.appdb.egi.eu/main:faq:how_to_register_a_virtual_appliance AppDB how to register a virtual appliance documentation]
# Once your VA is published, inform your VO through Applications Database about it.
#*  [https://wiki.appdb.egi.eu/main:guides:guide_for_managing_virtual_appliance_versions_using_the_portal Check the guide for managing VA versions]
#* VO-wide image lists can be managed by users that have the ''VO Manager'', ''VO Expert'' or ''VO deputy'' roles within the VO.
# Once your appliance is in the VO-wide image list, it will be deployed on the Federated Cloud sites of your VO.
 
=== Accessing EUDAT services from the EGI Cloud ===
From VMs instantiated in the EGI Federated Cloud, it is also possible interact with EUDAT services. Instructions on how to jointly use the EGI Federated Cloud and EUDAT services are available [[Jointly exploit EGI and EUDAT services|here]].
 
=== Kubernetes / Container based applications ===
 
EGI uses [https://kubernetes.io Kubernetes] deployed on VMs on the federated cloud sites to provide a complete container orchestration platform. Simpler use cases can run docker directly on a [https://appdb.egi.eu/store/vappliance/egi.docker.ubuntu.16.04 ready-to-use VM image at AppDB]. Learn more in the [[Federated Cloud Containers]] and start running your docker applications on EGI's Cloud.
 
=== High level tools: Orchestrators, Platforms/Software as a Service ===
 
Read '''[[Federated Cloud PaaS|this guidance]]''' about strategies of porting applications to the EGI Federated Cloud. The guide also includes references to high level user environments (orchestrators, Platform/Software as a Service) that can simplify the application integration and operation process for you. These environments offer high level abstractions and services on top of the baseline 'Infrastructure as a Service' cloud.
 
=== Specialised Software ===
 
==== Access the Chipster tested with CSG ====
 
<!--
With the [https://csgf.egi.eu CSG] members of the Infrastructure can access the Chipster tested to run bioinformatics applications on top of Infrastructure as a Service Clouds. For further details, please check this [[Access Chipster with CSG | Access Chipster with CSG ]] guide.
-->
 
With the [https://fgsg.egi.eu FGSG] members of the Infrastructure can access the Chipster tested to run bioinformatics applications on top of Infrastructure as a Service Clouds. For further details, please check this [[Access Chipster | Access Chipster ]] guide.
 
==== Running Galaxy workflows with EC3 ====
 
Using the [http://servproject.i3m.upv.es/ec3/ EC3] open-source software platform, users can deploy elastic clusters on demand and dynamically deploy complex scientific virtual computing infrastructures on top of Infrastructure as a Service Clouds. More details on how to use Galaxy workflows on the platform are described in this wiki. Please check the [[Galaxy workflows with EC3 | Galaxy workflows in EGI with EC3]] guide for further details.
 
==== Computer-Aided Engineering (CAE) ====
Please check how SMEs can use [[Computer-Aided Engineering (CAE) in the EGI Federated Cloud | Computer-Aided Engineering (CAE) in the EGI Federated Cloud]], with the example of OpenFOAM software containers.
 
==== Running Hadoop applications ====
 
Using a WS-PGRADE gateway that is connected to the EGI Federated cloud, it is possible to deploy Hadoop clusters on EGI Federated Cloud resources, to execute Hadoop applications on those clusters and finally to release resources after application execution. The concept is outlined on [https://indico.egi.eu/indico/event/2931/material/slides/ these PPT slides]. A user manual is [https://indico.egi.eu/indico/event/2931/material/0/ available here (v1.3)].
 
==== Running Jupyter Notebook with EC3 ====
 
Using the [http://servproject.i3m.upv.es/ec3/ EC3] open-source software platform, users can deploy a Jupyter Notebook on top of EGI Federated Cloud resources. For further details, please check the [[Jupyter Notebook with EC3 | Jupyter Notebook with EC3]] guide.
 
==== Running Beaker Notebooks in the EGI FedCloud ====
 
Check how to use [[Beaker | Beaker in the EGI Federated Cloud ]] Infrastructure.
 
=== GPGPUs ===
 
GPGPUs-enabled cloud resources are available in selected sites of the EGI Federated Cloud. Check the [[Federated_Cloud_GPGPU|GPGPU guide on FedCloud]] for details on how to access them.
 
== Useful resources  ==
 
*[[FAQ10|FedCloud FAQ page]]
*[http://www.egi.eu/how-to/get_a_certificate.html How to get a certificate (to access Federated Cloud resources)]
*[https://wiki.appdb.egi.eu/ How to use the Applications Database Cloud Marketplace]
*[[HOWTO10|Porting your application/web service to the EGI Federated Cloud]]
*[[Federated_Cloud_infrastructure_status|List and details about certified cloud resource providers]]
*[[Cloud_SAM_tests|Monitor tests performed by EGI on cloud resources]]
**[http://argo.egi.eu/lavoisier/status_report-site?report=Critical&Fedcloud=true&accept=html Status of cloud sites in the EGI federated cloud in the ARGO system]
 
== User support  ==
 
=== Technical support  ===
 
Users' technical support is provided via the [mailto:support@egi.eu EGI support contact].
 
=== Helpdesk  ===
 
Technical problems and questions relating to the use of the EGI Federated Cloud can be reported and dealt with through the [https://ggus.eu/ EGI Helpdesk ticketing system].
 
'''Note: '''Please choose 'Federated cloud' in the 'Type of problem' field of the ticket submission form!
 
== Technical background  ==
 
Cloud providers in the EGI Federated Cloud use hardware virtualization technologies to host software on their resources. The cloud management platforms that make this possible can [[Federated_Cloud_Operation#Current_Resource_Providers|vary from site to site]], but they all enable the provisioning of virtualized computing, storage and networking resources, thus they empower scientific groups to setup and operate domain specific services, applications and simulations on these resources. Read more about the [[Federated Cloud Technology|technology that drives the Federated Cloud]].

Latest revision as of 13:37, 13 October 2020