Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "GOCDB/Release4/Development/VSites/VOFeedXML"

From EGIWiki
< GOCDB‎ | Release4‎ | Development‎ | VSites
Jump to navigation Jump to search
 
(23 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Template:Op menubar}}
{{Template:GOCDB_menubar}}
{{TOC_right}}
[[Category:GOCDB]]
[[GOCDB/Release4/Development/VSites#VO_Feed_for_ATP|<< Back to Virtual Sites]]
[[GOCDB/Release4/Development/VSites#VO_Feed_for_ATP|<< Back to Virtual Sites]]
=VO Feed Example=
=VO Feed for ATP=
VO feed for the GOCDB EGI scoped data (ATP will recognize this gocdb data as the 'OPS' VO topology feed).  
<!--
[[GOCDB/Release4/Development/VSites#VO_Feed_for_ATP|<< First, please see VO Feed for ATP on VSites page for background info]]
-->
In order for ATP to use virtual sites, GOCDB have been asked to produce a VO feed for the GOCDB EGI scoped data (ATP will recognize this gocdb data as the 'OPS' VO topology feed). This feed should not be used by anyone except ATP and won't be supported outside of ATP's use (other VOs will not be supported). If there is a requirement for VO integration in GOCDB, it would need to be considered separately in a new proposal outside of VSites.
 
* Background information about VO feeds: https://tomtools.cern.ch/confluence/display/SAM/ATP+VO+Feeds
* Sample ATP feeds for different VO topologies:
** ms_topology: http://dashb-cms-vo-feed.cern.ch/dashboard/request.py/cmssitemapbdii
** atlas_topology: http://atlas-agis-api.cern.ch/request/atp/xml
** lhcb_topology: http://lhcb-web-dirac.cern.ch/topology/lhcb_topology.xml
** superbvo.org_topology: http://bbr-serv09.cr.cnaf.infn.it:8080/nagiosvo/superb-sites-for-nagios.xml
** alice_topology: http://sam-alice.cern.ch/alice_topology.xml
** ops_topology: see below
 
===ops_topology===
ATP will recognize the ops_topology VO feed as the 'OPS' VO topology feed:
* Applies to the GOCDB EGI scoped data
* The GOCDB physical sites are mapped to <atp_site> elements.  
* The GOCDB physical sites are mapped to <atp_site> elements.  
* If an endpoint is included in a virtual site, a corresponding <service> element must appear in this VO feed as a child of an <atp_site> element.
* If an endpoint is included in a virtual site, a corresponding <service> element must appear in this VO feed as a child of an <atp_site> element.
Line 14: Line 34:
* VSite == <group>
* VSite == <group>


Pseudo-code for the XML construction logic:
==Pseudo-code for building the ATP VO feed XML==
<pre>
<pre>


First query for all VSites and inner join member SEs  
Query for all VSites and inner join member SEs and iterate result set:
foreach VSite {


foreach VSite {
  - Create a new/empty in-memory <atp_site> hashmap where:  
  - Create a new in-memory <atp_site> element hashmap:  
  atpSiteElem[key='physicalSiteName', value='will hold the XML of this <atp_site> element']
    atpSiteElems[key='siteName', value='XML of this <atp_site> element']


   foreach VSite.SE {
   foreach VSites->ServiceEndpoint {
       - Get the hosting physical site name (key = ‘mySiteX’).  
       - Get the hosting physical site name (key = ‘myPhysicalSiteX’).  
       - Create a new atpSiteElem[‘mySiteX’] entry (if not already created).   
       - Create a new atpSiteElem[‘myPhysicalSiteX’] entry (if not already created).   
       + Add a new child <service> element to atpSiteElem[‘mySiteX’]  
       + For this SE, add a new child <service> element to atpSiteElem[‘myPhyscialSiteX’]  
   }
   }


Line 33: Line 53:
   }
   }


   - Write all atpSiteElem[] elements to the XML document.  
   - Write all atpSiteElem[] elements to the XML document.
   - Loop next VSite.  
   - Loop next VSite.  
}
}
</pre>
</pre>


Using this logic, <atp_site> elements will only ever have a <b>single nested <group> element</b> while <atp_site> elements with the same 'name' attribute are duplicated (which is permitted). However, for any single <atp_site>, it ensures that <b>ALL<b/> of the the listed <services> are members of <b>both</b> the specified VSite <group> <b>AND</b> the physical <atp_site>, which is correct.  
<b><u>Important:</u></b> Using this logic, <atp_site> elements will only ever have a single nested <group> element while <atp_site> elements with the same 'name' attribute are duplicated (which is permitted). However, for any single <atp_site>, this ensures that <b>ALL</b> of the the listed <services> are actually members of <b>both</b> the specified VSite <group> <b>AND</b> the physical <atp_site>, which is correct.
 
==Example==
<source lang="XML">
<?xml version="1.0" encoding="ISO-8859-1"?>
<root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="atp_vo_feed_schema.xsd">
  <title>Virtual Sites in GOCDB</title>
  <description>Groups of services defined under virtual sites in GOCDB</description>
  <feed_responsible name="GOCDB Administrators" dn="/C=UK/O=eScience/OU=CLRC/L=RAL/CN=goc.egi.eu/emailAddress=sct-certificates@stfc.ac.uk"/>
  <last_update>2012-02-01T09:46:10Z</last_update>
  <vo>OPS</vo>
 
  <!--
    note, the name attribute values on group elements are imaginary
    as VSites still have to be implemented
  --> 
 
  <atp_site name="RAL_LCG2" infrast="EGEE">
    <service hostname="lcgwms01.gridpp.rl.ac.uk" flavour="WMS" />
    <service hostname="lcgwms02.gridpp.rl.ac.uk" flavour="WMS" />
    <service hostname="lcgwms03.gridpp.rl.ac.uk" flavour="WMS" />
    <group name="All_RAL_LCG2_WMS_INSTANCES_VSITE">
  </atp_site>
 
  <atp_site name="RAL_LCG2" infrast="EGEE">
    <service hostname="lcgce03.gridpp.rl.ac.uk" flavour="CREAM-CE" />
    <service hostname="lcgce07.gridpp.rl.ac.uk" flavour="CREAM-CE" />
    <service hostname="lcgce08.gridpp.rl.ac.uk" flavour="CREAM-CE" />
    <service hostname="lcgce09.gridpp.rl.ac.uk" flavour="CREAM-CE" />
    <service hostname="lcgce05.gridpp.rl.ac.uk" flavour="CREAM-CE" />
    <group name="All_RAL_LCG2_CREAM_INSTANCES_VSITE">
  </atp_site>
 
  <atp_site name="GRIDOPS-GOCDB" infrast="EGEE">
    <service hostname="goc.egi.eu" flavour="egi.GOCDB" />
    <group name="OPS_TOOLS_VSITE">
  </atp_site>
 
  <atp_site name="GRIDOPS-OPSPORTAL" infrast="EGEE">
    <service hostname="operations-portal.egi.eu " flavour="egi.OpsPortal" />
    <group name="OPS_TOOLS_VSITE">
  </atp_site>
 
  <atp_site name="GRIDOPS-GGUS" infrast="EGEE">
    <service hostname="helpdesk.egi.eu" flavour="egi.GGUS" />
    <group name="OPS_TOOLS_VSITE">
  </atp_site>
 
</root>
</source>


== Monitoring Details ==
== Monitoring Details ==
Line 51: Line 120:


* Virtual sites can be integrated with SAM via VO feed, which would list all the virtual sites defined in the GOCDB (including all scopes; all such sites and their services would be considered supporting OPS VO)
* Virtual sites can be integrated with SAM via VO feed, which would list all the virtual sites defined in the GOCDB (including all scopes; all such sites and their services would be considered supporting OPS VO)
* With current implementation of ATP at least the following issues can be seen:
# virtual sites containing services outside of given region (no matter what scope) will be shown on the SAM regional instance (e.g. EGI_WMS virtual site would be visible on regional SAM showing only WMSes for that particular region). I'm not sure if this is actually an issue as this might be useful if properly explained in the docs.
# virtual sites (non-egi scoped) containing egi-scoped services will be shown on the SAM central instance
# virtual sites (egi scoped) containing mixture of egi-scoped and non-egi scoped services would be visible correctly on the SAM regional instance (cf. point 1), but on the central instance non-egi scoped services would be cut off from the virtual site (we need to clarify if such use case actually exists)
* In summary, none of those issues are critical, but we will need to discuss them during OTAG to see if there actual use cases behind.


==Questions for ATP:==
==Questions for ATP:==


* We've left the infrast="EGEE" attribute on the atp_site element. Is this correct?
* We've left the infrast="EGEE" attribute on the atp_site element. Is this correct?
<br />
* Please see [[GOCDB/Release4/Development/VSites#Monitoring_Limitations]]
<source lang="XML">
todo
</source>
<!--
<?xml version="1.0" encoding="ISO-8859-1"?>
<root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="atp_vo_feed_schema.xsd">
  <title>Virtual Sites in GOCDB</title>
  <description>Groups of services defined under virtual sites in GOCDB</description>
  <feed_responsible name="GOCDB Administrators" dn="/C=UK/O=eScience/OU=CLRC/L=RAL/CN=goc.egi.eu/emailAddress=sct-certificates@stfc.ac.uk"/>
  <last_update>2012-02-01T09:46:10Z</last_update>
  <vo>OPS</vo>
  <atp_site name="NGI_UK_CENTRAL_SERVICES" infrast="EGEE">
    <service hostname="lcgbdii.gridpp.rl.ac.uk" flavour="Top-BDII" />
    <service hostname="voms.gridpp.ac.uk" flavour="VOMS" />
    <service hostname="lcgwms01.gridpp.rl.ac.uk" flavour="WMS" />
  </atp_site>
  <atp_site name="EGI_CENTRAL_OPS" infrast="EGEE">
    <service hostname="goc.egi.eu" flavour="egi.GOCDB" />
    <service hostname="cic.gridops.org " flavour="egi.OpsPortal" />
    <service hostname="helpdesk.egi.eu" flavour="egi.GGUS" />
  </atp_site>
</root>
-->

Latest revision as of 12:32, 18 December 2012

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


GOC DB menu: Home Documentation Index


<< Back to Virtual Sites

VO Feed for ATP

In order for ATP to use virtual sites, GOCDB have been asked to produce a VO feed for the GOCDB EGI scoped data (ATP will recognize this gocdb data as the 'OPS' VO topology feed). This feed should not be used by anyone except ATP and won't be supported outside of ATP's use (other VOs will not be supported). If there is a requirement for VO integration in GOCDB, it would need to be considered separately in a new proposal outside of VSites.

ops_topology

ATP will recognize the ops_topology VO feed as the 'OPS' VO topology feed:

  • Applies to the GOCDB EGI scoped data
  • The GOCDB physical sites are mapped to <atp_site> elements.
  • If an endpoint is included in a virtual site, a corresponding <service> element must appear in this VO feed as a child of an <atp_site> element.
  • The VO feed should include all service endpoints, both visible and invisible to EGI.
  • For each physical site a <group> element will be used to indicate that all child endpoints belong to the virtual site named by the group.
    • It is permitted duplicate <atp_site> elements where necessary. e.g. RAL_LCG2 may appear twice with two different sets of endpoints linked to two different virtual sites.
  • The feed specifies that all sites are under the ops VO using the <vo>OPS</vo> element.

In summary, the GOCDB to ATP mappings are:

  • Site == <atp_site>
  • ServiceEndpoint == <service>
  • VSite == <group>

Pseudo-code for building the ATP VO feed XML


Query for all VSites and inner join member SEs and iterate result set: 
foreach VSite {

  - Create a new/empty in-memory <atp_site> hashmap where: 
  atpSiteElem[key='physicalSiteName', value='will hold the XML of this <atp_site> element']

   foreach VSites->ServiceEndpoint  {
      - Get the hosting physical site name (key = ‘myPhysicalSiteX’). 
      - Create a new atpSiteElem[‘myPhysicalSiteX’] entry (if not already created).   
      + For this SE, add a new child <service> element to atpSiteElem[‘myPhyscialSiteX’] 
   }

   foreach atpSiteElem[] {
      - Add a single child <group> element (VSite name defines the ‘name’ attribute) 
   }

   - Write all atpSiteElem[] elements to the XML document.
   - Loop next VSite. 
}

Important: Using this logic, <atp_site> elements will only ever have a single nested <group> element while <atp_site> elements with the same 'name' attribute are duplicated (which is permitted). However, for any single <atp_site>, this ensures that ALL of the the listed <services> are actually members of both the specified VSite <group> AND the physical <atp_site>, which is correct.

Example

<?xml version="1.0" encoding="ISO-8859-1"?>
<root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="atp_vo_feed_schema.xsd">
  <title>Virtual Sites in GOCDB</title>
  <description>Groups of services defined under virtual sites in GOCDB</description>
  <feed_responsible name="GOCDB Administrators" dn="/C=UK/O=eScience/OU=CLRC/L=RAL/CN=goc.egi.eu/emailAddress=sct-certificates@stfc.ac.uk"/>
  <last_update>2012-02-01T09:46:10Z</last_update>
  <vo>OPS</vo>

  <!-- 
    note, the name attribute values on group elements are imaginary 
    as VSites still have to be implemented 
  -->  

  <atp_site name="RAL_LCG2" infrast="EGEE">
    <service hostname="lcgwms01.gridpp.rl.ac.uk" flavour="WMS" />
    <service hostname="lcgwms02.gridpp.rl.ac.uk" flavour="WMS" />
    <service hostname="lcgwms03.gridpp.rl.ac.uk" flavour="WMS" />
    <group name="All_RAL_LCG2_WMS_INSTANCES_VSITE">
  </atp_site>

  <atp_site name="RAL_LCG2" infrast="EGEE">
    <service hostname="lcgce03.gridpp.rl.ac.uk" flavour="CREAM-CE" />
    <service hostname="lcgce07.gridpp.rl.ac.uk" flavour="CREAM-CE" />
    <service hostname="lcgce08.gridpp.rl.ac.uk" flavour="CREAM-CE" />
    <service hostname="lcgce09.gridpp.rl.ac.uk" flavour="CREAM-CE" />
    <service hostname="lcgce05.gridpp.rl.ac.uk" flavour="CREAM-CE" /> 
    <group name="All_RAL_LCG2_CREAM_INSTANCES_VSITE">
  </atp_site>

  <atp_site name="GRIDOPS-GOCDB" infrast="EGEE">
    <service hostname="goc.egi.eu" flavour="egi.GOCDB" />
    <group name="OPS_TOOLS_VSITE">
  </atp_site> 

  <atp_site name="GRIDOPS-OPSPORTAL" infrast="EGEE">
    <service hostname="operations-portal.egi.eu " flavour="egi.OpsPortal" />
    <group name="OPS_TOOLS_VSITE">
  </atp_site> 

  <atp_site name="GRIDOPS-GGUS" infrast="EGEE">
    <service hostname="helpdesk.egi.eu" flavour="egi.GGUS" />
    <group name="OPS_TOOLS_VSITE">
  </atp_site> 

</root>

Monitoring Details

The following details about how virtual sites will be monitored is provided by Marian Babik:

Data scoping

  • EGI/non-EGI scope is implemented in both regional and central GOCDB
  • SAM will fetch both scopes from the central GOCDB, which would make it possible to have non-EGI scoped services monitored by the regional SAM (non-egi services won't be distinguished from EGI ones in any way)
  • central SAM will show only EGI scoped services

Virtual Sites

  • SAM's vo feed concept currently enables creation of arbitrary groups of services that must be already defined in GOCDB (if a group is defined and none of its services are in GOCDB the entire group is rejected, all services/sites listed in the vo feed that are not in GOCDB are automatically rejected)
  • Virtual sites can be integrated with SAM via VO feed, which would list all the virtual sites defined in the GOCDB (including all scopes; all such sites and their services would be considered supporting OPS VO)

Questions for ATP: