Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Deploy HTC clusters on top of IaaS infrastructure

From EGIWiki
Revision as of 17:35, 1 April 2019 by Larocca (talk | contribs)
Jump to navigation Jump to search
Engagement overview Community requirements Community events Training EGI Webinars Documentations



How to deploy virtual HTC clusters on EGI

About

Overall, this document describes how to use the Elastic Cloud Computing Cluster (EC3) platform to create elastic virtual clusters on the EGI Federated Cloud infrastructure.

Configuration of the environment

EC3 has an official Docker container image available in Docker Hub that can be used instead of installing the CLI. You can download it by typing:

]$ sudo docker pull grycap/ec3

You can exploit all the potential of EC3 as if you download the CLI and run it on your computer.

List available templates

To list the available templates, use the command:

]$ cd $HOME
]$ sudo docker run -v /tmp/.ec3/clusters:/root/.ec3/clusters grycap/ec3 templates

name             	kind                                         summary
---------------------------------------------------------------------------------------------------
blcr           	    component 	Tool for checkpoint the applications.
centos-ec2          images   	CentOS 6.5 amd64 on EC2.
ckptman             component 	Tool to automatically checkpoint applications running on Spot instances.
docker       	    component 	An open-source tool to deploy applications inside software containers.
gnuplot             component 	A program to generate two- and three-dimensional plots.
nfs            	    component 	Tool to configure shared directories inside a network.
octave       	    component 	A high-level programming language for numerical computations
openvpn             component 	Tool to create a VPN network.
sge           	    main    	Install and configure a cluster SGE from distribution repositories.
slurm         	    main    	Install and configure a cluster SLURM 14.11 from source code.
torque              main    	Install and configure a cluster TORQUE from distribution repositories.
ubuntu-azure        images   	Ubuntu 12.04 amd64 on Azure.
ubuntu-ec2          images   	Ubuntu 14.04 amd64 on EC2.

List virtual clusters

To list the available running clusters, use the command:

]$ cd $HOME
]$ sudo docker run -v /tmp/.ec3/clusters:/root/.ec3/clusters grycap/ec3 list
    name        state           IP         nodes 
-------------------------------------------------------
 cluster  configured  212.189.145.XXX    0

Create a cluster

To launch a cluster, you can use the recipes that you have locally by mounting the folder as a volume, or create your dedicated ones. Also, it is recommendable to maintain the data of active clusters locally, by mounting a volume. In the next example, we are going to deploy a new Torque/Maui cluster on one cloud provider of the EGI Federation (INFN-CATANIA-STACK). The cluster will be configured with the following templates:

torque (default template), configure_nfs (patched template), ubuntu-1604-occi-INFN-CATANIA-STACK

and

cluster_configure

. User’s templates are stored in

$HOME/ec3/templates