Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "FedCloudWeNMR"

From EGIWiki
Jump to navigation Jump to search
 
(20 intermediate revisions by 4 users not shown)
Line 1: Line 1:
[[Category: Technology ]]
{{Fedcloud_Menu}}
[[Category: Fedcloud-tf]]
{{FedCloudUseCases |


FCUC_Status    = active |
{{FCloudCom_menubar}}
FCUC_StartDate = |
FCUC_EndDate  = | 
FCUC_EGIName  = Gergely Sipos / gergely.sipos@egi.eu  |
FCUC_EXTName  = |


ShortDesc =
{{TOC_right}}
== General Information  ==
*Status: {{{FCUC_Status |closed }}}
*Start Date: {{{FCUC_StartDate | 01/01/13 }}}
*End Date: {{{FCUC_EndDate | 30/04/14}}}
*EGI.eu contact: {{{FCUC_EGIName | Gergely Sipos / gergely.sipos@egi.eu}}}
*External contact: {{{FCUC_EXTName | Marco Verlato / Marco.Verlato@pd.infn.it}}}
== Short Description  ==
{{{FCUC_ShortDesc |
The objective of [http://www.wenmr.eu WeNMR] is to optimize and extend the use of the NMR and SAXS research infrastructures through the implementation of an e-infrastructure in order to provide the user community with a platform integrating and streamlining the computational approaches necessary for NMR and SAXS data analysis and structural modelling. Access to the e-NMR infrastructure is provided through a portal integrating commonly used software and GRID technology.
}}}
== Tasks  ==
{{{FCUC_Tasks |
Proof of concept demonstrating the validation and improvement of biomolecular NMR structures using VirtualCing, a Virtual Machine equipped with a complex suite of ~25 programs.
== Outcome ==
It was demonstrated at the EGI Technical Forum held in September 2012, at the Cloudscape V Workshop held in February 2013 and at the EGI CF 2013 held in April.
== Additional Files ==
* [https://prezi.com/hrzwekguwfkx/virtualcing-cloud-demo-on-wnodes/ Demonstration of VCIng over WNoDeS cloud]
* [https://documents.egi.eu/public/ShowDocument?docid=1593 Demostrations of VCing over the EGI Federated Cloud]
<!--
=====Use case 1 (in progress)=====
Using VMs prepared with Gromacs and some other software to run MD simulations for educational purpose, possibly on multi-core VMs.
These VMs are currently run on the SARA cloud for a computer practical of University courses, see [http://nmr.chem.uu.nl/~adrien/course/molmod/ here].<br/>
A more ambitious use case, always related to Gromacs, would be the submission over the cloud of the WeNMR/Gromacs jobs currently done over the grid as described
[http://www.egi.eu/blog/2011/10/31/running_multiprocessor_jobs_on_the_grid.html here].<br/>
The cloud version of WeNMR/Gromcas would have two main advantages:<br/>
:* the possibility to run on boxes of more than 6 cpu-cores, if the size of the VMs made available by the cloud provider allows it;<br/>
:* to avoid to hit the queue limits on cpu time that are reached very quickly on typical grid sites.<br/>
Of course that would require a not negligible effort in order to adapt the WeNMR/Gromacs portal to submit jobs to the cloud, for example a possible option could be to go through a JSAGA-OCCI adaptor as described [https://indico.egi.eu/indico/contributionDisplay.py?sessionId=33&contribId=189&confId=1417 here].


The objective of WeNMR is to optimize and extend the use of the NMR and SAXS research infrastructures through the implementation of an e-infrastructure in order to provide the user community with a platform integrating and streamlining the computational approaches necessary for NMR and SAXS data analysis and structural modelling. Access to the e-NMR infrastructure is provided through a portal integrating commonly used software and GRID technology.
=====Use case 2 (not more supported)=====
Validating and improving biomolecular NMR structures using VirtualCing (VCing), a Virtual Machine (VM) equipped with a complex suite of ~25 programs.<br/> A presentation of the current deployment at the Dutch National HPC Cloud is available [http://www.sara.nl/sites/default/files/Jurgen_Doreleijers_HPC_Cloud_Day_20111004_2.pdf here], and recently a paper has been published [http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3245154/?tool=pmcentrez here].<br/> The cloud usage framework is based on a pilot job mechanism making use of the [https://grid.sara.nl/wiki/index.php/Using_the_Grid/ToPoS ToPoS] tool. Therefore, such a framework would naturally allow for execution of VCing tasks across multiple cloud providers. Do notice that the framework is independent on the cloud access interface: it would work also with simple grid jobs, as far as the user-defined (or VO manager defined) VCing VM is available at the grid site e.g. in a SE (or in the VO software area mounted by the WNs) and the grid job is allowed to start the VM. Technical details about its current implementation are available [http://code.google.com/p/cing/w/list here]. <br/> A live [http://prezi.com/hrzwekguwfkx/virtualcing-cloud-demo-on-wnodes/ demonstration] about the deployment and use of VCing on the WNoDeS testbed of the INFN-CNAF computing centre has been shown at the EGI TF 2012 held in September. Further [https://documents.egi.eu/public/ShowDocument?docid=1593 demostrations of VCing over the EGI Federated Cloud] testbed were shown at the Cloudscape V Workshop held in February 2013 and at the EGI CF 2013 held in April.<br/>
VCing OS is Ubuntu 11.04 i386, and its image size has been recently shrinked to 8.1 GB in compressed qcow2 kvm format (from a 20 GB raw image). This demo version of the VCing image is in the [http://marketplace.egi.eu/metadata EGI marketplace] and is freely available for demonstration shows.<br/>
<br/>
1. Here below are the steps carried out to make the VCing demo image work with:<br/>
- the WNoDeS framework for the network and debian/ubuntu:
:* modifying the file /etc/network/interfaces as follows:
auto lo
iface lo inet loopback
auto eth0
allow-hotplug eth0
iface eth0 inet dhcp
auto eth1
allow-hotplug eth1
iface eth1 inet dhcp
:* deleting /etc/udev/rules.d/70-persistent-net.rules file
:* removing the hostaname in the file /etc/hostname
- the CESNET framework based on XEN:
:* copy the files provided by CESNET to:
/etc/grub.d/10_linux  # removed Grub sub-menus breaking pygrub
/etc/init/hvc0.conf      # added a new console, the file didn't exist before
/usr/sbin/update-grub # replacing KVM-specific disk names with universal hd0
:* then run the command update-grub
Other Cloud frameworks of course can have their own contextualisation recipes.<br/>
2. From any machine having the [https://grid.sara.nl/wiki/index.php/Using_the_Grid/ToPoS#Client_libraries Topos library] add the tokens to the Topos server VCing demo area https://topos.grid.sara.nl/4.1/pools/vCingDemo
$ ./topos createTokensFromLinesInFile vCingDemo [https://dl.dropbox.com/u/527460/Clouds/VCing/demo_token_list.txt demo_token_list.txt]
For testing purpose you can also try with one token by adding manually from the Topos web interface e.g. the following line:
refineEntryNrg 1iy6 9 http://nmr.cmbi.ru.nl/NRG-CING/data i@nmr.cmbi.ru.nl:/mnt/data/D/NMR_REDO . . BY_CH23_BY_ENTRY CING 0 auto 0 0 1
3. You can start now as many as possible VCing instances on the federated cloud. Job payloads will be retrieved automatically by the VCing from the Topos server pool until all the tokens are processed. For demo show purposes you can check live the token list [https://topos.grid.sara.nl/4.1/pools/vCingDemo/tokens/ here] and see that they get first locked and then disappear when processed. A graphical example of the final result can be shown by clicking [http://nmr.cmbi.ru.nl/NMR_REDO/demo/iy/1iy6/1iy6.cing/1iy6_redo/HTML/index.html here]. <br/>
4. Output data are copied at the end of the processing from VCing to the NMR server nmr.cmbi.ru.nl hosted at CMBI:
:* ssh traffic must be allowed from the VCing instance to the NMR server
:* the cloud subnet address must be included in the /etc/firewall.conf of the NMR server
:* a post-processing script does untar the zipped data output and makes it accessible through web e.g. from [http://nmr.cmbi.ru.nl/NMR_REDO/demo/iy/1iy6/1iy6.cing/1iy6_redo/HTML/index.html here]
:* in the last VCing image demo version (14 Feb 2013) the post-processing at the NMR server is issued remotely from the VM right after the output data copy, so no more need to log into the NMR server to perform post-processing as done in the EGI TF 2012 demo.<br/>


http://www.wenmr.eu/ |
In September 2013 a collaboration with the [https://dirac.ub.es/DIRAC/ DIRAC project] has started aiming at using the VMDIRAC portal as VM scheduler.<br/> The DIRAC broker is not involved in this use case because the job payload for the VCing VM is provided by the ToPoS server as described above. The DIRAC team used ssh contextualization to run in the VMs the VM Monitor Agent, that is in charge of the VM status update for the VM management in the DIRAC portal, and at the same time is checking CPU activity in the VM, if no activity in a certain window time, then it is stopping the VM automatically.<br/>
 
The goal of this effort is twofold:
Tasks = |
:* WeNMR can take advantage of the VM machinery in DIRAC and the web monitoring and management
Files = |
:* VMDIRAC can be prooved as a toool for VM scheduling without the use of DIRAC broker
 
-->
}}
}}}

Latest revision as of 15:33, 7 May 2015

Overview For users For resource providers Infrastructure status Site-specific configuration Architecture



Federated Cloud Communities menu: Home Production use cases Under development use cases Closed use cases High level tools use cases



General Information

  • Status: closed
  • Start Date: 01/01/13
  • End Date: 30/04/14
  • EGI.eu contact: Gergely Sipos / gergely.sipos@egi.eu
  • External contact: Marco Verlato / Marco.Verlato@pd.infn.it

Short Description

The objective of WeNMR is to optimize and extend the use of the NMR and SAXS research infrastructures through the implementation of an e-infrastructure in order to provide the user community with a platform integrating and streamlining the computational approaches necessary for NMR and SAXS data analysis and structural modelling. Access to the e-NMR infrastructure is provided through a portal integrating commonly used software and GRID technology.

Tasks

Proof of concept demonstrating the validation and improvement of biomolecular NMR structures using VirtualCing, a Virtual Machine equipped with a complex suite of ~25 programs.

Outcome

It was demonstrated at the EGI Technical Forum held in September 2012, at the Cloudscape V Workshop held in February 2013 and at the EGI CF 2013 held in April.

Additional Files