Deploy HTC clusters on top of IaaS infrastructure
Engagement overview | Community requirements | Community events | Training | EGI Webinars | Documentations |
How to deploy virtual HTC clusters on EGI
About
Overall, this document describes how to use the Elastic Cloud Computing Cluster (EC3) platform to create elastic virtual clusters on the EGI Federated Cloud infrastructure.
Configuration of the environment
EC3 has an official Docker container image available in Docker Hub that can be used instead of installing the CLI. You can download it by typing:
]$ sudo docker pull grycap/ec3
You can exploit all the potential of EC3 as if you download the CLI and run it on your computer.
List available templates
To list the available templates, use the command:
]$ cd $HOME ]$ sudo docker run -v /tmp/.ec3/clusters:/root/.ec3/clusters grycap/ec3 templates name kind summary --------------------------------------------------------------------------------------------------- blcr component Tool for checkpoint the applications. centos-ec2 images CentOS 6.5 amd64 on EC2. ckptman component Tool to automatically checkpoint applications running on Spot instances. docker component An open-source tool to deploy applications inside software containers. gnuplot component A program to generate two- and three-dimensional plots. nfs component Tool to configure shared directories inside a network. octave component A high-level programming language for numerical computations openvpn component Tool to create a VPN network. sge main Install and configure a cluster SGE from distribution repositories. slurm main Install and configure a cluster SLURM 14.11 from source code. torque main Install and configure a cluster TORQUE from distribution repositories. ubuntu-azure images Ubuntu 12.04 amd64 on Azure. ubuntu-ec2 images Ubuntu 14.04 amd64 on EC2.
List virtual clusters
To list the available running clusters, use the command:
]$ cd $HOME ]$ sudo docker run -v /tmp/.ec3/clusters:/root/.ec3/clusters grycap/ec3 list name state IP nodes ------------------------------------------------------- cluster configured 212.189.145.XXX 0
Create a cluster
To launch a cluster, you can use the recipes that you have locally by mounting the folder as a volume, or create your dedicated ones. Also, it is recommendable to maintain the data of active clusters locally, by mounting a volume. In the next example, we are going to deploy a new Torque/Maui cluster on one cloud provider of the EGI Federation (INFN-CATANIA-STACK). The cluster will be configured with the following templates:
torque (default template), configure_nfs (patched template), ubuntu-1604-occi-INFN-CATANIA-STACK, cluster_configure
. User’s templates are stored in
$HOME/ec3/templates