EGI Opendata platform
Overview | For users | For resource providers | Infrastructure status | Site-specific configuration | Architecture |
The EGI Open Data platform
Setup your ONEDATA deployment into the EGI Federated Cloud
This is a setep-by-step guide to setup a ONEDATA deployment into the EGI Federated Cloud. For further information about ONEDATA, please refer to its online documentation.
OneZone
- Create an instance using the EGI Docker (Ubuntu 14.04) image. Please, refer to the cli guide.
- Login into the new instance with your private key.
ssh -i [path-to-your-private-key] ubuntu@[ip-address-of-your-instance]
- Install docker-compose in the machine as root.
sudo -i curl -L https://github.com/docker/compose/releases/download/1.8.0-rc1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose
and verify its setup:
~# docker-compose -v docker-compose version 1.8.0-rc1, build 9bf6bc6
don't forget to exit from the root shell.
- In the home of the ubuntu user download the OneData software:
git clone https://github.com/onedata/getting-started
- Go to the scenario 3.0 folder.
cd getting-started/scenarios/3_0_oneprovider_onezone_multihost/
- Install OneZone using the run_onedata.sh script:
sudo ./run_onedata.sh --zone &
Wait for the message "Congratulations! onezone has been successfully started."
Now your OneZone instance is ready. You can access it with your browser:
https://[ip-address-of-your-onezone-instance]
Please, login with
login:admin password:password
OneProvider
- Create an instance using the EGI Docker (Ubuntu 14.04) image. Please, refer to the cli guide.
- Create your storage in the cloud and store its ID in an environment variable.
occi --endpoint $ENDPOINT --auth x509 --voms --user-cred $X509_USER_PROXY --action create --resource storage --attribute occi.storage.size="num([number-og-GB])" --attribute occi.core.title="[storage_name]" [ID] export STORAGE_ID=[ID]
- Link the block storage to your instance.
occi --endpoint $ENDPOINT --auth x509 --voms --user-cred $X509_USER_PROXY --action link --resource $COMPUTE_ID --link $STORAGE_ID
- Run an OCCI describe command to your instance to get the block storage path. You can find it in the occi.storagelink.deviceid attribute. See the example below.
occi --endpoint $ENDPOINT --auth x509 --voms --user-cred $X509_USER_PROXY --action describe --resource $COMPUTE_ID [[ http://schemas.ogf.org/occi/infrastructure#compute ]] >> location: /occi/compute/136618fb-4687-4a15-b144-5381d9aa0742 occi.core.id = 136618fb-4687-4a15-b144-5381d9aa0742 occi.core.title = onedata-provider100 occi.compute.cores = 2 occi.compute.hostname = onedata-provider100 occi.compute.memory = 4096 occi.compute.state = active Links: [[ http://schemas.ogf.org/occi/core#link ]] >> location: /occi/storagelink/136618fb-4687-4a15-b144-5381d9aa0742_fd8e7b72-0c1e-4f9c-a1a0-d4043c56b752 occi.storagelink.deviceid = /dev/vdb occi.core.source = http://cloud.recas.ba.infn.it:8787/occi/compute/136618fb-4687-4a15-b144-5381d9aa0742 occi.core.target = http://cloud.recas.ba.infn.it:8787/occi/storage/fd8e7b72-0c1e-4f9c-a1a0-d4043c56b752 occi.core.id = 136618fb-4687-4a15-b144-5381d9aa0742_fd8e7b72-0c1e-4f9c-a1a0-d4043c56b752 [[ http://schemas.ogf.org/occi/core#link ]] >> location: /occi/networklink/136618fb-4687-4a15-b144-5381d9aa0742_fe82ef7b-4bb7-4c1e-b4ec-ec5c1b0c7333_90.147.102.160 occi.networkinterface.mac = fa:16:3e:bf:26:94 occi.networkinterface.interface = eth0 occi.networkinterface.state = active occi.networkinterface.allocation = dynamic occi.networkinterface.address = 90.147.102.160 occi.core.source = http://cloud.recas.ba.infn.it:8787/occi/compute/136618fb-4687-4a15-b144-5381d9aa0742 occi.core.target = http://cloud.recas.ba.infn.it:8787/occi/network/fe82ef7b-4bb7-4c1e-b4ec-ec5c1b0c7333 occi.core.id = 136618fb-4687-4a15-b144-5381d9aa0742_fe82ef7b-4bb7-4c1e-b4ec-ec5c1b0c7333_90.147.102.160 Mixins: [[ http://schemas.openstack.org/template/os#b7765c28-6bc6-438a-8b7c-b6873103c5f5 ]] title: Image for Docker Ubuntu 14.04 [Ubuntu/14.04/VirtualBox]_EGI_fedcloud term: b7765c28-6bc6-438a-8b7c-b6873103c5f5 location: /occi/os_tpl/b7765c28-6bc6-438a-8b7c-b6873103c5f5 [[ http://schemas.openstack.org/template/resource#8 ]] title: Flavor: medium term: 8 location: /occi/resource_tpl/8 Actions: [[ http://schemas.ogf.org/occi/infrastructure/compute/action#start ]] [[ http://schemas.ogf.org/occi/infrastructure/compute/action#stop ]] [[ http://schemas.ogf.org/occi/infrastructure/compute/action#restart ]] [[ http://schemas.ogf.org/occi/infrastructure/compute/action#suspend ]]
- Login into the new instance with your private key.
ssh -i [path-to-your-private-key] ubuntu@[ip-address-of-your-instance]
- Format and mount the block storage attached to the instance
sudo mkfs.ext3 [block-storage-path-in-your-instance] sudo mount /dev/vdb [mount-point]
In the following an example with block storage identified by /dev/vdb and mount point /mnt/.
sudo mkfs.ext3 /dev/vdb sudo mount /dev/vdb /mnt/
- Install docker-compose in the machine as root.
sudo -i curl -L https://github.com/docker/compose/releases/download/1.8.0-rc1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose
and verify its setup:
~# docker-compose -v docker-compose version 1.8.0-rc1, build 9bf6bc6
don't forget to exit from the root shell.
- In the home of the ubuntu user download the OneData software:
git clone https://github.com/onedata/getting-started
- go to the scenario 3.0 folder.
cd getting-started/scenarios/3_0_oneprovider_onezone_multihost/
- edit file docker-compose-oneprovider.yml and replace ${ONEPROVIDER_DATA_DIR} with the folder where you mounted the block storage.
# data persistence - "[mount-point]:/volumes/storage"
- Install OneProvider using the run_onedata.sh script:
sudo ./run_onedata.sh --provider --provider-fqdn [OneZone-IP] --zone-fqdn [OneData-IP] &
Wait for the message "Congratulations! oneprovider has been successfully started."
See below an example of output.
IMPORTANT: After each start wait for a message: Congratulations! oneprovider has been successfully started. To ensure that the oneprovider is completely setup. Pulling node1.oneprovider.localhost (onedata/oneprovider:3.0.0-rc5)... 3.0.0-rc5: Pulling from onedata/oneprovider a3ed95caeb02: Pull complete 57c018ee1aa8: Pull complete ebd817e34e50: Pull complete c068f2d0906a: Pull complete b3bad60afaef: Pull complete 47175703b6bc: Pull complete 5cd40b9132e8: Pull complete 25c596923cc2: Pull complete e665a356e4f5: Pull complete 6585e20d8e6f: Pull complete 229064f88987: Pull complete debcc3142260: Pull complete 0056c9a8edc5: Pull complete Digest: sha256:88c0fd1fa61ebd4e548222beb14f9d29efacddcf0d8bea7f1f1185d4d453be64 Status: Downloaded newer image for onedata/oneprovider:3.0.0-rc5 Creating network "30oneprovideronezonemultihost_default" with the default driver Creating oneprovider-1 Attaching to oneprovider-1 oneprovider-1 | Starting op_panel: [ OK ] oneprovider-1 | oneprovider-1 | Configuring oneprovider: oneprovider-1 | * service_onepanel: set_cookie oneprovider-1 | * service_onepanel: purge_node oneprovider-1 | * service_onepanel: create_tables oneprovider-1 | * service_onepanel: add_default_users oneprovider-1 | * service_onepanel: add_nodes oneprovider-1 | * service: save oneprovider-1 | * service_couchbase: configure oneprovider-1 | * service_couchbase: start oneprovider-1 | * service_couchbase: wait_for_init oneprovider-1 | * service_couchbase: init_cluster oneprovider-1 | * service_couchbase: rebalance_cluster oneprovider-1 | * service_couchbase: status oneprovider-1 | * service: save oneprovider-1 | * service_cluster_manager: configure oneprovider-1 | * service_cluster_manager: start oneprovider-1 | * service_cluster_manager: status oneprovider-1 | * service: save oneprovider-1 | * service_op_worker: configure oneprovider-1 | * service_op_worker: setup_certs oneprovider-1 | * service_op_worker: start oneprovider-1 | * service_op_worker: wait_for_init oneprovider-1 | * service_op_worker: status oneprovider-1 | * service_op_worker: add_storages oneprovider-1 | * service_oneprovider: configure oneprovider-1 | * service_oneprovider: configure oneprovider-1 | * service_oneprovider: register oneprovider-1 | * service: save oneprovider-1 | * service_onepanel: add_users oneprovider-1 | oneprovider-1 | Container details: oneprovider-1 | * IP Address: 172.18.0.2 oneprovider-1 | * Ports: - oneprovider-1 | oneprovider-1 | Congratulations! oneprovider has been successfully started.
Space Management
This section describes how to create a new space in your OneZone instance and support it with your OneProvider instance.
- Access your OneZone instance with a browser
https://[ip-address-of-your-onezone-instance]
- Select Data Space Management and Create a new space
- Insert a name for the new space
- Click on the just create space and then on get support
- Copy the token
- Access the administrative panel of your OneProvider instance
https://[ip-address-of-your-onezone-instance]
- Select Space/Management/Support Space
- Fill in the form:
Storage: NFS
Token: token copied in OneZone
Support size: amount of storage you want to assign to this space (<= the size of the block storage you attached to the OneProvider instance)
- Go back to OneZone and reload the page in your browser
- Now, the provider is showed in the map. Select Data Space Management and the space you created. The provider will appear under the space.
- Select the provider and, then, Go to your files in the map.
- OneZone will redirect you in the Oneprovider web page to manage data. Upload some sample files.