Difference between revisions of "Fedcloud-tf:CLI Environment"

From EGIWiki
Jump to: navigation, search
(List the contents of the CDMI storage)
 
(5 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{{Fedcloud-tf:Menu}} {{TOC_right}}
+
#REDIRECT[[HOWTO11_How_to_use_the_rOCCI_Client]]
 
 
This guide provides the information on how to setup a FedCloud command-line environment (using the [https://github.com/gwdg/rOCCI-cli Ruby OCCI Framework])
 
 
 
Setting up a command line structure basically consists of the following phases:
 
# Security setup
 
# Set up (r)OCCI tools
 
# Test
 
 
 
== Requirements ==
 
 
 
This guide has been tested on the following systems (64 bits):
 
* RedHat 6.x (x86_64)
 
* Scientific Linux 6.x (x86_64)
 
* CentOS 6.x (x86_64)
 
* Mac OS X > 10.6
 
* Debian >= 6 (amd64)
 
* Ubuntu 12.04 (amd64)
 
 
 
NOTE: 32 bits OSes are not supported
 
 
 
== Security setup ==
 
 
 
This section is the most fragile bit of the setup. It consists of setting up the following components:
 
# Obtain a Grid user certificate
 
# Configure the certificate based trust anchors
 
# Join the FedCloud Virtual Organisation
 
# Install VO client tools
 
# Configure VO membership information
 
 
 
=== Obtain a Grid user certificate ===
 
 
 
This step is fairly straight-forward in terms of processes. The goal is to obtain a digital certificate that is signed by a given certification authority, vouching for the validity of your real life identity encoded into the digital certificate issued to you.
 
 
 
Although each Certification Authority (including their affiliated Registration Authorities) follow individual procedures, you will eventually end up with a personal digital Grid certificate. More information on how to obtain a Grid certificate is available on the [http://www.egi.eu/how-do-I/get_a_certificate.html EGI website].
 
 
 
In any way, you now have (a) a digital certificate, and (b) a private key. Certificates use a cryptography technology called ''public key cryptography''. Without going too much into details, the ''public key'' is contained in your certificate, and can be used and obtained by everyone on the Internet (including Grids and Clouds). It is safe to publish the public key (that's why it is called "public key"). There is exactly one pendant complementing the public key, and that is the ''private key''. '''You must keep this private key safe at all circumstances''', as it is the only token with which you can ensure that the identity stored in your certificate is you. Anyone who possesses the private key can claim your identity!
 
 
 
However, once you have obtained your certificate (during the Grid certificate enrollment process, your computer has generated the public and the private key), you will have those two digital files. To conclude the first step of setting up your security environment, you need to ensure that both are safely stored in your home directory as follows:
 
 
 
$ ls -al ~/.globus
 
total 48
 
drwxr-xr-x    14 michel  staff    476 Mar  7 13:57 .
 
drwxr-xr-x+  73 michel  staff  2482 Apr  3 11:41 ..
 
-rw-r--r--    1 michel  staff  1765 Feb  5 20:07 usercert.pem
 
-r--------    1 michel  staff  1751 Feb  5 20:07 userkey.pem
 
-r--------    1 michel  staff  1751 Feb  5 20:07 usercred.pem
 
-rw-r--r--    1 michel  staff  3254 Oct 20  2011 userrequest.pem
 
 
 
Notes:
 
# File ''userkey.pem'' is your encoded and encrypted private key - you can access it '''only''' with a password that you have used during enrolling for a certificate.
 
# File ''userkey.pem'' must have the shown permissions, i.e. that only the current user can read its contents, and nothing else. Other tools will depend on this setting.
 
# File ''usercert.pem'' contains your user certificate (with the public key). Note the different permissions on this file compared to userkey.pem.
 
# File ''usercred.pem'' contains your user certificate with the private and public key. To create it, you can execute the command " ''cat ~/.globus/usercert.pem  ~/.globus/userkey.pem > ~/.globus/usercred.pem'' ". Note that this file shall have the same permissions as the userkey.pem.
 
# The directories shown in the listing contain archived previous versions of my own certificate (I maintain a Grid certificate for a number of years in the meantime).
 
 
 
If your system is set up this way, then the Grid certificate enrollment step is concluded.
 
 
 
=== Configure the certificate based trust anchors ===
 
 
 
The tools you are going to use will need to know which Certification Authorities you trust. For sure you will want to trust the CA that issued your own Grid certificate, but that is not enough. A number of services run in the EGI infrastructure use certificates issued by different CAs, and in order to access these services, you will have to trust those CAs as well.  This act of ''establishing trust in s CA'' is represented in your system by making the CA's certificate available to your tools. In a nutshell, if your tool can through some algorithms establish the fact that a given CA certificate was used to issue another certificate that is about to be verified, then the verified certificate is said to be trusted.
 
 
 
EGI adopts the list of endorsed Certificate Authorities that is maintained by the EUGridPMA initiative. This list is made available in the EGI Software Repository and is updated regularly. So, what is left to you is to
 
* Obtain these certificates, and
 
* install them into a default location
 
 
 
This is, again, different for different platforms that you may use. EGI directly supports Scientific Linux5 SL6 and Debian6 with a repository service available in the [http://repository.egi.eu/category/production/cas/ EGI Software Repository].
 
 
 
The paragraphs below will give a quick demo on how to install the Grid CA certificates on the most widely used operating systems
 
 
 
===== Debian >= 6 and Ubuntu 12.04 =====
 
 
 
Follow the [http://repository.egi.eu/category/umd_releases/distribution/umd-3/ EGI UMD repository installation guide]
 
 
 
Install the Grid certificates via
 
 
 
apt-get update
 
apt-get install ca-policy-egi-core
 
 
 
===== RedHat Linux 6 and Scientific Linux 6 =====
 
 
 
Follow the [http://repository.egi.eu/category/umd_releases/distribution/umd-3/ EGI UMD repository installation guide]
 
 
 
Install the Grid certificates via
 
 
 
yum install lcg-CA
 
 
 
===== Linux Generic and Mac OS X =====
 
 
 
Browse to the CA trust anchor's TGZ directory at http://repository.egi.eu/sw/production/cas/1/current/tgz/ and download all the certificate archives.( '''TODO''' - Add a simple script to automate that.)
 
 
 
Unpack all individual certificate archive into the same temporary directory. ('''TODO''' - provide a simple script for that.) Each certificate archive should expand into five individual files, plus a number of symbolic links to these, for example:
 
 
 
  44 Jan 21 11:00 ca_KEK-1.52.tar.gz
 
    7 Feb  1 13:55 2f2f573f.0 -> KEK.pem
 
  14 Feb  1 13:55 2f2f573f.namespaces -> KEK.namespaces
 
  18 Feb  1 13:55 2f2f573f.signing_policy -> KEK.signing_policy
 
    7 Feb  1 13:55 617ff41b.0 -> KEK.pem
 
  14 Feb  1 13:55 617ff41b.namespaces -> KEK.namespaces
 
  18 Feb  1 13:55 617ff41b.signing_policy -> KEK.signing_policy
 
  44 Jan 21 11:00 KEK.crl_url
 
  239 Jan 21 11:00 KEK.info
 
  464 Jan 21 11:00 KEK.namespaces
 
1273 Jan 21 11:00 KEK.pem
 
1299 Jan 21 11:00 KEK.signing_policy
 
 
 
Once completed, you need to make this directory available to your tools.
 
 
 
The agreed default location for these CA certificates is  ''/etc/grid-security/certificates''. It is possible to store the certificates in a different location, but that would require much of configuration hassle. So:
 
 
 
# sudo cp your_temporary_ca_dir /etc/grid-security/certificates
 
 
 
with ''your_temporary_ca_dir'' being the directory into which you have extracted the individual CA certificate archives in the previous step.
 
 
 
 
 
=== Install the fetch-crl utility ===
 
The fetch-crl utility will retrieve certificate revocation lists (CRLs) for a set of installed trust anchors.
 
 
 
===== Debian >= 6 and Ubuntu 12.04 =====
 
apt-get install fetch-crl
 
 
 
===== RedHat Linux 6 and Scientific Linux 6 =====
 
yum install fetch-crl
 
chkconfig fetch-crl on
 
 
 
=== Join the Virtual Organisation ===
 
 
 
In EGI, one need to be a member of a ''Virtual Organisation'' (VO) to be able to access EGI resources. Often, but not always, VOs are organised in ''Virtual Research Communities'' (VRCs). The EGI.eu website provides some information and guidance on [http://www.egi.eu/community/vrcs/ Virtual Research Communities] and [http://www.egi.eu/community/vos/ Virtual Organisations] including how to join or found VOs.
 
 
 
To tell resources (mainly for access and accounting purposes) which with which VO membership one would like to use the resources, the user needs to provide a ''proxy certificate'', that is signed using one's one certificate, and which carries the proper VO attributes with it. The user then uses this proxy certificate instead of her own real Grid Certificate to access the resources. This is the reason why users need to be member of a VO, and must create proxy certificates for EGI resource access.
 
 
 
For Cloud purposes, EGI currently maintains multiple VOs, mapping the different user communities and experiments. A generic VO, called ''fedcloud.egi.eu'', addresses the needs for testers, developers and generic users. EGI.eu provides a description of this VO on its [http://www.egi.eu/infrastructure/cloud/ own page].
 
 
 
You can apply for membership for the generic EGI Cloud Federation VO [https://perun.metacentrum.cz/perun-registrar-cert/?vo=fedcloud.egi.eu here]. '''Note: ''' This requires your Grid Certificate and private key be ''also'' stored in your Web Browser! Your membership should be granted fairly soon, so what's left is to set up VO information and ''VO trust'' for your command line environment.
 
 
 
=== Install VO client tools ===
 
 
 
To access the federated cloud you will need to install tools to create proxy certificates, and configure these so that they can create proper proxy certificates for you. The tools to be installed are the '''VOMS client tools''' that are developed as part of the EMI project.
 
 
 
These tools are available in the [http://repository.egi.eu/category/umd_releases/distribution/umd-3/ EGI UMD Repository] for Scientific Linux 5, SL6 and Debian 6.
 
 
 
===== Debian >= 6 and Ubuntu 12.04 =====
 
 
 
Follow the [http://repository.egi.eu/category/umd_releases/distribution/umd-3/ EGI UMD repository installation guide]
 
 
 
Install the VOMS client via
 
 
 
apt-get update
 
apt-get install voms-clients3
 
 
 
===== RedHat Linux 6 and Scientific Linux 6 =====
 
 
 
Follow the [http://repository.egi.eu/category/umd_releases/distribution/umd-3/ EGI UMD repository installation guide]
 
 
 
Install the VOMS client via
 
 
 
yum install voms-clients3
 
 
 
===== Linux generic and Mac OS X =====
 
 
 
* Obtain the latest snapshot (they are stable, though) from the  [http://radiohead.cnaf.infn.it:9999/view/VOMS/job/voms-clients_master/lastSuccessfulBuild/org.italiangrid%24voms-clients/artifact/org.italiangrid/voms-clients/3.0.4/voms-clients-3.0.4.tar.gz IGI repository].
 
 
 
* Unpack the archive into a directory of your choice (can be in your home directory):
 
$ tar xvzf voms-clients-3.0.0.tar.gz
 
 
 
* Update your shell's $PATH variable to include the ''bin'' subdirectory of the voms clients, e.g.
 
$ export PATH=$PATH:~/voms-clients/bin
 
Adapt the path to your local needs.
 
 
 
You should now have a working VOMS Clients environment. Try it out using:
 
$ voms-proxy-info
 
Proxy not found: /tmp/x509up_u501 (No such file or directory)
 
This error message is normal, certainly for the first time, when no proxy certificate could be found. The path to the certificate file is composed of a default prefix, and the user id in the system (in my case, I have the user id of 501).
 
 
 
=== Configure VO membership information ===
 
 
 
The VOMS client tools require some configuration information to be able to create proxy certificates for you. For this, it needs access to your Grid Certificate (by default it looks in ~/.globus/usercert.pem) and your certificate's private key (by default in ~/.globus/userkey.pem) and configuration information for each VO membership service (VOMS) you need to create proxy certificates for.
 
 
 
Provided that you kept your Grid Certificate and private key in ~/.globus, the only thing you need to do is to set up the VOMS information for your VO. Again, the accepted default location makes it easy to use to use the tools. You have two choices for providing the configuration information:
 
# local to you, within your home directory, or
 
# global on your system, for all users
 
 
 
This guide will set up global VOMS VO information for the VO. To do so, you need the VO name and the VO VOMS information (server name, server port, certificate and CA subject). You can find this information in the [http://operations-portal.egi.eu/vo VO ID card]. For the generic EGI Federated Cloud VO, the VO name is ''fedcloud.egi.eu'' and the VOMS information are ''voms1.egee.cesnet.cz'', 15002, ''/DC=org/DC=terena/DC=tcs/OU=Domain Control Validated/CN=voms2.grid.cesnet.cz'', ''/C=NL/O=TERENA/CN=TERENA eScience SSL CA'' . The following part of the guide will refer to the fedcloud.egi.eu VO, but the same applies to all the other VOs (using the [http://operations-portal.egi.eu/vo VO ID card] information)
 
 
 
VO information is kept in two pivotal locations :
 
* /etc/grid-security/vomsdir directlry, and
 
* /etc/vomses file.
 
 
 
Each file serves a specific purpose that can be read up on elsewhere. The guide will explain how to create or amend these two locations for the fedcloud VO as follows:
 
 
 
===== Creating the vomsdir information =====
 
 
 
Usually, this information is provided by the VO manager. In this case, we will create it by hand as follows:
 
 
 
# mkdir -p /etc/grid-security/vomsdir/fedcloud.egi.eu
 
# cd /etc/grid-security/vomsdir/fedcloud.egi.eu
 
# cat >voms1.egee.cesnet.cz.lsc <<EOF
 
/DC=org/DC=terena/DC=tcs/OU=Domain Control Validated/CN=voms1.egee.cesnet.cz
 
/C=NL/O=TERENA/CN=TERENA eScience SSL CA
 
EOF
 
# cat >voms2.grid.cesnet.cz <<EOF
 
/DC=org/DC=terena/DC=tcs/OU=Domain Control Validated/CN=voms2.grid.cesnet.cz
 
/C=NL/O=TERENA/CN=TERENA eScience SSL CA
 
EOF
 
 
 
===== Extending/creating vomses file =====
 
 
 
This file is a list of VO contact informations to obtain any VO specific attributes and roles for your proxy certificate:
 
 
 
# cd /etc
 
# cat >>vomses <<EOF
 
"fedcloud.egi.eu" "voms1.egee.cesnet.cz" "15002" "/DC=org/DC=terena/DC=tcs/OU=Domain Control Validated/CN=voms1.egee.cesnet.cz" "fedcloud.egi.eu" "24"
 
"fedcloud.egi.eu" "voms2.grid.cesnet.cz" "15002" "/DC=org/DC=terena/DC=tcs/OU=Domain Control Validated/CN=voms2.grid.cesnet.cz" "fedcloud.egi.eu" "24"
 
EOF
 
 
 
===== Testing the VO membership setup =====
 
 
 
$ voms-proxy-init -voms fedcloud.egi.eu --rfc --dont_verify_ac
 
Enter GRID pass phrase for this identity:
 
 
Created proxy in /tmp/x509up_u501.
 
 
Your proxy is valid until Thu Apr 04 04:02:38 CEST 2013
 
 
 
And finally list the info about your proxy certificate:
 
 
 
$ voms-proxy-info
 
subject  : /O=dutchgrid/O=users/O=egi/CN=Michel Drescher/CN=1746347155
 
issuer    : /O=dutchgrid/O=users/O=egi/CN=Michel Drescher
 
identity  : /O=dutchgrid/O=users/O=egi/CN=Michel Drescher
 
type      : RFC3820 compliant impersonation proxy
 
strength  : 1024
 
path      : /tmp/x509up_u501
 
timeleft  : 11:59:46
 
key usage : Digital Signature, Key Encipherment, Data Encipherment
 
 
 
== Set up (r)OCCI tools ==
 
 
 
This tutorial uses the rOCCI client provided mainly by GWDG and CESNET. It is maintained as a [https://github.com/EGI-FCTF/rOCCI-cli GitHub project].
 
 
 
===  Ubuntu 12.04 LTS ===
 
 
 
sudo -s
 
wget -O /etc/apt/sources.list.d/occi-cli-ubuntu-precise-amd64.list http://repository.egi.eu/community/software/rocci.cli/4.2.x/releases/repofiles/ubuntu-precise-amd64.list
 
wget -qO - http://repository.egi.eu/community/keys/APPDBCOMM-DEB-PGP-KEY.asc | apt-key add -
 
apt-get update
 
apt-get -y install occi-cli
 
 
ln -s /opt/occi-cli/bin/occi /usr/bin/occi
 
 
 
=== Debian 6 ===
 
 
 
sudo -s
 
wget -O /etc/apt/sources.list.d/occi-cli-debian-squeeze-amd64.list http://repository.egi.eu/community/software/rocci.cli/4.2.x/releases/repofiles/debian-squeeze-amd64.list
 
wget -qO - http://repository.egi.eu/community/keys/APPDBCOMM-DEB-PGP-KEY.asc | apt-key add -
 
apt-get update
 
apt-get -y install occi-cli
 
 
ln -s /opt/occi-cli/bin/occi /usr/bin/occi
 
 
 
=== Debian 7 ===
 
 
 
sudo -s
 
wget -O /etc/apt/sources.list.d/occi-cli-debian-wheezy-amd64.list http://repository.egi.eu/community/software/rocci.cli/4.2.x/releases/repofiles/debian-wheezy-amd64.list
 
wget -qO - http://repository.egi.eu/community/keys/APPDBCOMM-DEB-PGP-KEY.asc | apt-key add -
 
apt-get update
 
apt-get -y install occi-cli
 
 
ln -s /opt/occi-cli/bin/occi /usr/bin/occi
 
=== RedHat 6 or SL6 ===
 
 
 
sudo -s
 
wget -O /etc/yum.repos.d/rocci-cli.repo http://repository.egi.eu/community/software/rocci.cli/4.2.x/releases/repofiles/sl-6-x86_64.repo
 
yum install -y occi-cli
 
 
ln -s /opt/occi-cli/bin/occi /usr/bin/occi
 
 
 
=== Mac OS X ===
 
 
 
Detailed Mac OS X instruction are available [https://github.com/EGI-FCTF/rOCCI-cli/blob/master/doc/macosx.md here].
 
 
 
=== Other ===
 
On any system with '''Ruby >= 1.9.3''' and '''Rubygems''' present, you can install the client by running
 
 
 
gem install occi-cli
 
 
 
== Set up (b)CDMI tools ==
 
 
 
This tutorial uses the bCDMI client provided mainly by the Federated Cloud task force to access CDMI storage. It is maintained as a [https://github.com/EGI-FCTF/bCDMI GitHub project].
 
 
 
bCDMI client is still in an experimental status. Please, install it only if you are aware about the latest developments. Otherwise skip this section.
 
 
 
bCDMI is implemented in Bash, using cURL and other UNIX tools, thus it can run on almost any Linux OS. To install the client:
 
 
 
=== Install the script ===
 
 
 
===== Debian > 6 or Ubuntu > 11.04 =====
 
 
 
sudo apt-get install curl sed gawk
 
curl -o bcdmi https://raw.github.com/EGI-FCTF/bCDMI/master/bcdmi
 
curl -o bcdmi.auth https://raw.github.com/EGI-FCTF/bCDMI/master/bcdmi.auth
 
 
 
===== RedHat 6 or SL6 =====
 
 
sudo yum install -y curl sed gawk
 
curl -o ~/bcdmi https://raw.github.com/EGI-FCTF/bCDMI/master/bcdmi
 
curl -o ~/bcdmi.auth https://raw.github.com/EGI-FCTF/bCDMI/master/bcdmi
 
 
 
===== Mac OS X =====
 
curl -o ~/bcdmi https://raw.github.com/EGI-FCTF/bCDMI/master/bcdmi
 
curl -o ~/bcdmi.auth https://raw.github.com/EGI-FCTF/bCDMI/master/bcdmi
 
 
 
===== Others =====
 
A Windows version of the client is available [https://github.com/EGI-FCTF/bCDMI/win/ here].
 
 
 
=== Configure the bCDMI script ===
 
The default configuration of the bCDMI script should be is enough for most of the users. With this configuration, bCDMI automatically uses a X509 proxy for authentication and detects your VO from the proxy VOMS extensions.
 
 
 
Anyway, bCDMI has a wide set of authentication options who can be set into the ''bcdmi.auth'' file. Have a look to the contents of the default ''bcdmi.auth'' for more information.
 
 
 
== Test ==
 
 
 
To test your command-line environment, you can run try to list the resources in one of the [[Fedcloud-tf:testbed|FedCloud sites]]. To do so:
 
 
 
=== Create a proxy certificate ===
 
 
 
$ voms-proxy-init -voms fedcloud.egi.eu --rfc --dont_verify_ac
 
Enter GRID pass phrase for this identity:
 
 
Created proxy in /tmp/x509up_u501.
 
 
Your proxy is valid until Thu Apr 04 04:02:38 CEST 2013
 
 
 
=== Check that your proxy certificate is valid ===
 
 
 
$ voms-proxy-info
 
subject  : /O=dutchgrid/O=users/O=egi/CN=Michel Drescher/CN=1746347155
 
issuer    : /O=dutchgrid/O=users/O=egi/CN=Michel Drescher
 
identity  : /O=dutchgrid/O=users/O=egi/CN=Michel Drescher
 
type      : RFC3820 compliant impersonation proxy
 
strength  : 1024
 
path      : /tmp/x509up_u501
 
timeleft  : 11:59:46
 
key usage : Digital Signature, Key Encipherment, Data Encipherment
 
 
 
=== Select a FedCloud site ===
 
Get the URL of the OCCI interface of one of the [[Fedcloud-tf:testbed|FedCloud sites]], for example https://carach5.ics.muni.cz:11443/ and of the CDMI interface of one of the [[Fedcloud-tf:testbed|FedCloud sites]], for example http://prisma-swift.ba.infn.it:8080/
 
 
 
=== List the image and resource templates ===
 
 
 
$ occi --endpoint https://carach5.ics.muni.cz:11443/ --action list --resource os_tpl --auth x509 --user-cred /tmp/x509up_u501 --voms
 
 
https://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#monitoring
 
https://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#egi_sl6goldenimage_cesnet
 
https://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#generic_vm
 
https://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#octave
 
https://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#r
 
https://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#metacloud_debian_7_0_x86_64_0001_fedcloud_dukan
 
https://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#metacloud_ubuntu_12_04_lts_x86_64_0001_fedcloud_dukan
 
https://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#metacloud_scilinux_6_5_x86_64_0001_fedcloud_dukan
 
 
 
=== List the contents of the CDMI storage ===
 
 
 
$ ./bcdmi -e http://prisma-swift.ba.infn.it:8080/ list
 
test/
 
test2/
 
test3/
 
 
$ ./bcdmi -e http://prisma-swift.ba.infn.it:8080/ stat
 
{
 
  "objectName": "AUTH_113d9a9a671944648722e890ecb94d36/",
 
  "capabilitiesURI": "/cdmi/AUTH_113d9a9a671944648722e890ecb94d36/cdmi_capabilities/rootcontainer/",
 
  "parentURI": "/cdmi/",
 
  "childrenRange": "0-2",
 
  "objectType": "application/cdmi-container",
 
  "children": [
 
    "test/",
 
    "test2/",
 
    "test3/"
 
  ],
 
  "metadata": {}
 
}
 
 
 
[[Category:Fedcloud-tf]]
 

Latest revision as of 15:19, 11 May 2015