Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "GOCDB/Release4/Development/MultipleEndpointsPerService"

From EGIWiki
Jump to navigation Jump to search
 
(40 intermediate revisions by the same user not shown)
Line 14: Line 14:
=Multiple Endpoints=
=Multiple Endpoints=
==Introduction==
==Introduction==
Define multiple Endpoints per Service. Original requirements:
Requirement for multiple Endpoints per (single) Service
* RT: https://rt.egi.eu/rt/Ticket/Display.html?id=3347
* To describe increasing heterogeneity of services (e.g. cloud).
* GGUS: https://ggus.eu/ws/ticket_info.php?ticket=93966  
* To put selected endpoints of a service into downtime, e.g. separate Tape and Disk endpoints in a single SRM service.
* To specify additional network locations for service functionalities that can't be described by the main Service-Endpoint URL alone. For example, a single service can define a main service URL <b>AND</b> a GRIS-Endpoint URL (Grid Resource Info Service). This will allow the Top-BDII to directly retrieve information about the service (e.g. for non-glite services) instead of via the Site-BDII via the GIIS URL.
 
 
Original Requirements
* Add GRIS URL per service, RT: https://rt.egi.eu/rt/Ticket/Display.html?id=3347
* Separate SRM endpoints, GGUS: https://ggus.eu/ws/ticket_info.php?ticket=93966  
* JRA1 discussion and detailed proposal: https://indico.egi.eu/indico/conferenceDisplay.py?confId=1922
* JRA1 discussion and detailed proposal: https://indico.egi.eu/indico/conferenceDisplay.py?confId=1922
* https://indico.cern.ch/getFile.py/access?resId=1&materialId=minutes&confId=278660   
* https://indico.cern.ch/getFile.py/access?resId=1&materialId=minutes&confId=278660   
GOCDB could be extended so that a single service could define multiple Endpoint objects to allow: 
* Multiple GRIS endpoint URLs per service. This will allow the Top-BDII to directly retrieve information about the endpoint.
* Separating tape from disk endpoints in a single SRM service. This is required to put selected endpoints of the service into downtime.


To address these requirements, we propose adding support for multiple <ENDPOINT> objects/elements per Service. Each Endpoint can define a different URL that is related to the service, for example, one Endpoint could define the actual service URL, a second could define the GRIS ldap URL, whilst a third could define an admin portal URL for the service (and so on). The <b>Endpoint.InterfaceName</b> (taken from GLUE2) would be used to distinguish between the different service endpoint types.  
To address these requirements, we propose adding support for multiple <ENDPOINT> objects/elements per Service. Each Endpoint can define a different URL that is related to the service, for example, one Endpoint could define the actual service URL, a second could define the GRIS ldap URL, whilst a third could define an admin portal URL for the service (and so on). The <b>Endpoint.InterfaceName</b> (taken from GLUE2) would be used to distinguish between the different service endpoint types.  
Line 30: Line 31:
==Implementation==
==Implementation==
* Add zero or more <ENDPOINT> objects per service (based on the GLUE2 Endpoint entity).
* Add zero or more <ENDPOINT> objects per service (based on the GLUE2 Endpoint entity).
* When creating a downtime, an extra step would be required to select *which* Endpoints from the selected Services should be put into downtime (all, some or zero). All the services' endpoints would be selected by default allowing specific endpoints to be deselected as required.
* When creating a downtime, an extra step would be required to select *which* Endpoints from the selected Services should be put into downtime (all, some or zero). All the services' endpoints would be selected by default allowing specific endpoints to be deselected as required. See presentation with sample screen captures: https://indico.egi.eu/indico/getFile.py/access?contribId=3&resId=1&materialId=slides&confId=2286
* The GOCDB-PI output would be extended for nesting multiple child <ENDPOINT> elements (see below).  
* The GOCDB-PI output would be extended for nesting multiple child <ENDPOINT> elements (see below).  
* Each <ENDPOINT> nests child <URL> and <InterfaceName> elements (and other GLUE2 attributes).
* Each <ENDPOINT> nests child <URL> and <InterfaceName> elements (and other GLUE2 attributes).
Line 46: Line 47:
* Endpoints can be deleted from a service, even the endpoints that are linked to (historic) downtimes. This removes the Endpoint-to-Downtime association. However, <b>the Service will always remain linked to the Downtime</b> because of the direct Service-to-Downtime association.   
* Endpoints can be deleted from a service, even the endpoints that are linked to (historic) downtimes. This removes the Endpoint-to-Downtime association. However, <b>the Service will always remain linked to the Downtime</b> because of the direct Service-to-Downtime association.   
* When a Service is deleted, associations with all linked downtimes are also deleted. In addition, downtimes that are orphaned are also deleted (downtimes that are linked to other services are not deleted).
* When a Service is deleted, associations with all linked downtimes are also deleted. In addition, downtimes that are orphaned are also deleted (downtimes that are linked to other services are not deleted).


[[File:DowntimeAssociations.jpg|400px|center|Downtime Site Service Endpoint Associations]]
[[File:DowntimeAssociations.jpg|400px|center|Downtime Site Service Endpoint Associations]]
* As shown above, a Service can define zero or more endpoints. A Service is always linked to its Downtimes (solid red line). Endpoints can be optionally linked to Downtimes (dashed red line) to indicate which parts of the Service are affected. If no endpoints are affected by a Downtime, the Service as a whole is considered in downtime.
* A sample downtime that mirrors the above diagram is at: https://gocdb-test.esc.rl.ac.uk/portal/index.php?Page_Type=Downtime&id=15713
* Presentation with sample screen captures: https://indico.egi.eu/indico/getFile.py/access?contribId=3&resId=1&materialId=slides&confId=2286
==Full GLUE2 Endpoint Rendering==
* The <ENDPOINT> elements below are incomplete compared to the GLUE2 <Endpoint> elements.
* In future we plan extend the data model to add all of the GLUE2 <Endpoint>, <StorageEndpoint>, <ComputingEndpoint> attributes.
* This constitutes further work but is certainly possible, especially as the core entity objects are now in place requiring only new attributes to be added to the existing endpoint objects.
* For more info on the GLUE2 XML rendering: http://www.ogf.org/documents/GFD.209.pdf


==Testing==
==Testing==
* Deployed on gocdb-test instance for testing (note, gocdt-test data is stale):  
* Deployed on gocdb-test instance for testing (note, gocdt-test data becomes stale):  
* PI: https://gocdb-test.esc.rl.ac.uk/gocdbpi/public/?method=get_service_group  (substitute PI method)
* Portal: https://gocdb-test.esc.rl.ac.uk/portal  
* Portal: https://gocdb-test.esc.rl.ac.uk/portal  
* Period of acceptance testing required first before deployment to production  
* PI: https://gocdb-test.esc.rl.ac.uk/gocdbpi/public/?method=get_service_group  (substitute required PI method)
** Sample PI queries with proposed changes:
** https://gocdb-test.esc.rl.ac.uk/gocdbpi/public/?method=get_service_group&service_group_name=OPSTOOLS
** https://gocdb-test.esc.rl.ac.uk/gocdbpi/public/?method=get_service&sitename=GRIDOPS_GOCDB
** https://gocdb-test.esc.rl.ac.uk/gocdbpi/public/?method=get_downtime&id=15713
** https://gocdb-test.esc.rl.ac.uk/gocdbpi/public/?method=get_downtime_nested_services&id=15713
* A period of acceptance testing is required before deployment to production


==PI Changes==
==PI Changes==
Line 61: Line 79:
* get_service is an alias to get_service_endpoint
* get_service is an alias to get_service_endpoint
* Current XML output: https://wiki.egi.eu/wiki/GOCDB/PI/get_service_endpoint_method
* Current XML output: https://wiki.egi.eu/wiki/GOCDB/PI/get_service_endpoint_method
* <u><b>Rename</b></u> the root <SERVICE_ENDPOINT> element to <SERVICE>.  
* <strike><u><b>Rename</b></u> the root <SERVICE_ENDPOINT> element to <SERVICE>.</strike>
* <u><b>Add new</b></u> <ENDPOINTS/> element
* <u><b>Add new</b></u> <ENDPOINTS/> element


Line 68: Line 86:
  <?xml version="1.0" encoding="UTF-8"?>
  <?xml version="1.0" encoding="UTF-8"?>
  <results>
  <results>
   <SERVICE PRIMARY_KEY="50257G0">  <!-- RENAME. <SERVICE_ENDPOINT> element to <SERVICE> -->
   <SERVICE_ENDPOINT PRIMARY_KEY="50257G0">  <!-- RENAME? <SERVICE_ENDPOINT> element to <SERVICE> -->
       <PRIMARY_KEY>50257G0</PRIMARY_KEY>
       <PRIMARY_KEY>50257G0</PRIMARY_KEY>
       <HOSTNAME>dgiref-globus.fzk.de</HOSTNAME>
       <HOSTNAME>dgiref-globus.fzk.de</HOSTNAME>
Line 119: Line 137:


  </results>
  </results>
</source>
===get_service_group===
* Current XML output: https://wiki.egi.eu/wiki/GOCDB/PI/get_service_group
* <strike><b><u>Rename</u></b> Service_Group's <SERVICE_ENDPOINT> element to <SERVICE></strike>
* <b><u>Add New</u></b> <ENDPOINTS> to Service
<source lang="XML">
<?xml version="1.0" encoding="UTF-8"?>
<results>
<SERVICE_GROUP PRIMARY_KEY="57654G0"> 
  <NAME>OPSTOOLS</NAME>
  <DESCRIPTION>All EGI Operational Tools</DESCRIPTION>
  <MONITORED>Y</MONITORED>
  <CONTACT_EMAIL>gocdb-admins@mailtalk.ac.uk</CONTACT_EMAIL>
  <GOCDB_PORTAL_URL> https://elided </GOCDB_PORTAL_URL>
  <SERVICE_ENDPOINT>                                <!-- RENAME? <SERVICE_ENDPOINT> to <SERVICE> -->
      <HOSTNAME>goc.egi.eu</HOSTNAME>
      <GOCDB_PORTAL_URL>https://elided</GOCDB_PORTAL_URL>
      <SERVICE_TYPE>egi.GOCDB</SERVICE_TYPE>
      <HOST_IP/>
      <HOSTDN>/C=UK/O=eScience/OU=CLRC/L=RAL/CN=goc.egi.eu</HOSTDN>
      <IN_PRODUCTION>Y</IN_PRODUCTION>
      <NODE_MONITORED>Y</NODE_MONITORED>
      <ENDPOINTS>                          <!-- NEW. <ENDPOINTS> wraps service's <ENDPOINT>s-->
        <ENDPOINT>
          <ID>1234</ID>
          <NAME>endpointname</NAME>
          <EXTENSIONS/>
          <URL>some url </URL>
          <InterfaceName>RIS</InterfaceName>
          <!-- To Add new GLUE2 type attributes here -->
        </ENDPOINT>
        <ENDPOINT>
          <ID>1234</ID>
          <NAME>endpointname</NAME>
          <EXTENSIONS/>
          <URL>some endpoint url</URL>
          <InterfaceName>eg.SRM.nearline</InterfaceName>
          <!-- To Add new GLUE2 type attributes here -->
        </ENDPOINT>
        <ENDPOINT>
          <ID>1234</ID>
          <URL>some endpoint url</URL>
          <NAME>endpointname</NAME>
          <EXTENSIONS/>
          <InterfaceName>eg.SRM.online</InterfaceName>
          <!-- To Add new GLUE2 type attributes here -->
        </ENDPOINT>     
      <ENDPOINTS>
      <EXTENSIONS/>
  </SERVICE> 
  <SERVICE> 
    ...elided...
  </SERVICE> 
  <EXTENSIONS/>
</SERVICE_GROUP>
</results>
</source>
</source>


===get_downtime===
===get_downtime===
* Current XML output: https://wiki.egi.eu/wiki/GOCDB/PI/get_downtime_method
* Current XML output: https://wiki.egi.eu/wiki/GOCDB/PI/get_downtime_method
* <strike><b><u>Rename</u></b> Downtime element's <ENDPOINT> element to <SE_ENDPOINT></strike>
* <b><u>Add New</u></b> <AFFECTED_ENDPOINTS/> element
* <b><u>Undecided:</u></b> Currently each ENDPOINT is partially rendered (no Endpoint-Extensions rendered). This makes the query more efficient and Endpoint-Extensions can be viewed in get_service, get_service_endpoint as above. We could render Endpoint-Extensions if necessary.
<source lang="XML">
<source lang="XML">
<?xml version="1.0" encoding="UTF-8"?>
<?xml version="1.0" encoding="UTF-8"?>
Line 130: Line 213:
   <HOSTNAME>somewhere.dl.ac.uk</HOSTNAME>
   <HOSTNAME>somewhere.dl.ac.uk</HOSTNAME>
   <SERVICE_TYPE>SE2</SERVICE_TYPE>
   <SERVICE_TYPE>SE2</SERVICE_TYPE>
   <SE_ENDPOINT>service.dl.ac.ukSE2</SE_ENDPOINT>  <!-- RENAME. <ENDPOINT> to <SE_ENDPOINT> -->
   <ENDPOINT>service.dl.ac.ukSE2</ENDPOINT>  <!-- RENAME? <ENDPOINT> to <SE_ENDPOINT> -->
   <HOSTED_BY>TestSite</HOSTED_BY>
   <HOSTED_BY>TestSite</HOSTED_BY>
   <GOCDB_PORTAL_URL>https://goc.egi.eu/portal/index.php?Page_Type=Downtimeampid=1</GOCDB_PORTAL_URL>
   <GOCDB_PORTAL_URL>https://goc.egi.eu/portal/index.php?Page_Type=Downtimeampid=1</GOCDB_PORTAL_URL>


   <AFFCTED_ENDPOINTS>  <!-- NEW. All SEs affected by DT (*may not* inc. all endpoints of a service) -->
   <AFFECTED_ENDPOINTS>  <!-- NEW. All SEs affected by DT (*may not* inc. all endpoints of a service) -->
     <ENDPOINT>
     <ENDPOINT>
        <ID>1234</ID>
        <NAME>endpointname</NAME>
        <!--<EXTENSIONS/> not rendered - see above -->
         <URL>some.srm.nearline.url</URL>
         <URL>some.srm.nearline.url</URL>
         <InterfaceName>SRM.nearline</InterfaceName>
         <InterfaceName>SRM.nearline</InterfaceName>
       </ENDPOINT>
       </ENDPOINT>
       <ENDPOINT>
       <ENDPOINT>
        <ID>1234</ID>
        <NAME>endpointname</NAME>
        <!--<EXTENSIONS/> not rendered - see above -->
         <URL>some.srm.online.url</URL>
         <URL>some.srm.online.url</URL>
         <InterfaceName>SRM.online</InterfaceName>
         <InterfaceName>SRM.online</InterfaceName>
Line 158: Line 247:
===get_downtime_to_broadcast===
===get_downtime_to_broadcast===
* Current XML output: https://wiki.egi.eu/wiki/GOCDB/PI/get_downtime_to_broadcast_method
* Current XML output: https://wiki.egi.eu/wiki/GOCDB/PI/get_downtime_to_broadcast_method
* <b><u>Add New</u></b> <AFFECTED_ENDPOINTS> element
* <b><u>Undecided:</u></b> Currently each ENDPOINT is partially rendered (no Endpoint-Extensions rendered). This makes the query more efficient and Endpoint-Extensions can be viewed in get_service, get_service_endpoint as above. We could render Endpoint-Extensions if necessary.
<source lang="XML">
<source lang="XML">
<?xml version="1.0"?>
<?xml version="1.0"?>
Line 173: Line 266:
   <AFFECTED_ENDPOINTS>  <!-- NEW. All SEs affected by DT (*may not* inc. all endpoints of a service) -->
   <AFFECTED_ENDPOINTS>  <!-- NEW. All SEs affected by DT (*may not* inc. all endpoints of a service) -->
     <ENDPOINT>
     <ENDPOINT>
        <ID>1234</ID>
        <NAME>endpointname</NAME>
        <!--<EXTENSIONS/> not rendered - see above -->
         <URL>some.srm.nearline.url</URL>
         <URL>some.srm.nearline.url</URL>
         <InterfaceName>SRM.nearline</InterfaceName>
         <InterfaceName>SRM.nearline</InterfaceName>
       </ENDPOINT>
       </ENDPOINT>
       <ENDPOINT>
       <ENDPOINT>
        <ID>1234</ID>
        <NAME>endpointname</NAME>
        <!--<EXTENSIONS/> not rendered - see above -->
         <URL>some.srm.online.url</URL>
         <URL>some.srm.online.url</URL>
         <InterfaceName>SRM.online</InterfaceName>
         <InterfaceName>SRM.online</InterfaceName>
Line 193: Line 292:
===get_downtime_nested_services ===
===get_downtime_nested_services ===
* Current XML output: https://wiki.egi.eu/wiki/GOCDB/PI/get_downtime_nested_services_method  
* Current XML output: https://wiki.egi.eu/wiki/GOCDB/PI/get_downtime_nested_services_method  
* <strike><b><u>Rename</u></b> Service's <ENDPOINT> element to <SE_ENDPOINT></strike>
* <b><u>Add New</u></b> <AFFECTED_ENDPOINTS> to Service
* <b><u>Undecided:</u></b> Currently each ENDPOINT is partially rendered (no Endpoint-Extensions rendered). This makes the query more efficient and Endpoint-Extensions can be viewed in get_service, get_service_endpoint as above. We could render Endpoint-Extensions if necessary.
<source lang="XML">
<source lang="XML">
<?xml version="1.0" encoding="UTF-8"?>
<?xml version="1.0" encoding="UTF-8"?>
Line 210: Line 314:
                 <HOSTNAME>somehost.dl.ac.uk</HOSTNAME>
                 <HOSTNAME>somehost.dl.ac.uk</HOSTNAME>
                 <SERVICE_TYPE>SE2</SERVICE_TYPE>
                 <SERVICE_TYPE>SE2</SERVICE_TYPE>
                 <SE_ENDPOINT>service.dl.ac.ukSE2</SE_ENDPOINT>  <!-- RENAME. <ENDPOINT> to <SE_ENDPOINT> -->
                 <ENDPOINT>service.dl.ac.ukSE2</ENDPOINT>  <!-- RENAME? <ENDPOINT> to <SE_ENDPOINT> -->
                 <HOSTED_BY>TestSite</HOSTED_BY>
                 <HOSTED_BY>TestSite</HOSTED_BY>
                 <AFFECTED_ENDPOINTS>  <!-- NEW. SEs affected by DT (*may not* inc. all endpoints of a service) -->
                 <AFFECTED_ENDPOINTS>  <!-- NEW. SEs affected by DT (*may not* inc. all endpoints of a service) -->
                     <ENDPOINT>
                     <ENDPOINT>
                      <ID>1234</ID>
                      <NAME>endpointname</NAME>
                      <!--<EXTENSIONS/> not rendered - see above -->
                       <URL>some.srm.nearline.url</URL>
                       <URL>some.srm.nearline.url</URL>
                       <InterfaceName>SRM.nearline</InterfaceName>
                       <InterfaceName>SRM.nearline</InterfaceName>
                     </ENDPOINT>
                     </ENDPOINT>
                     <ENDPOINT>
                     <ENDPOINT>
                      <ID>1234</ID>
                      <NAME>endpointname</NAME>
                      <!--<EXTENSIONS/> not rendered - see above -->
                       <URL>some.srm.online.url</URL>
                       <URL>some.srm.online.url</URL>
                       <InterfaceName>SRM.online</InterfaceName>
                       <InterfaceName>SRM.online</InterfaceName>
Line 227: Line 337:
                 <HOSTNAME>somehost2.dl.ac.uk</HOSTNAME>
                 <HOSTNAME>somehost2.dl.ac.uk</HOSTNAME>
                 <SERVICE_TYPE>SE2</SERVICE_TYPE>
                 <SERVICE_TYPE>SE2</SERVICE_TYPE>
                <ENDPOINT>somehost2.dl.ac.ukSE2</ENDPOINT>  <!-- RENAME? <ENDPOINT> to <SE_ENDPOINT> -->
                 <HOSTED_BY>TestSite2</HOSTED_BY>
                 <HOSTED_BY>TestSite2</HOSTED_BY>
                 <AFFECTED_ENDPOINTS>  <!-- NEW. All SEs affected by DT (*may not* inc. all endpoints of a service)-->
                 <AFFECTED_ENDPOINTS>  <!-- NEW. All SEs affected by DT (*may not* inc. all endpoints of a service)-->
                     <ENDPOINT>
                     <ENDPOINT>
                       <URL>some.srm.nearline.url</URL>
                      <ID>1234</ID>
                       <InterfaceName>SRM.nearline</InterfaceName>
                      <NAME>endpointname</NAME>
                      <!--<EXTENSIONS/> not rendered - see above -->
                       <URL>some.srm.online.url</URL>
                       <InterfaceName>SRM.online</InterfaceName>
                     </ENDPOINT>  
                     </ENDPOINT>  
                 </AFFECTED_ENDPOINTS>
                 </AFFECTED_ENDPOINTS>
Line 237: Line 351:
         </SERVICES>
         </SERVICES>
     </DOWNTIME>
     </DOWNTIME>
</results>
</source>
===get_service_group===
* Current XML output: https://wiki.egi.eu/wiki/GOCDB/PI/get_service_group
<source lang="XML">
<?xml version="1.0" encoding="UTF-8"?>
<results>
<SERVICE_GROUP PRIMARY_KEY="57654G0"> 
  <NAME>OPSTOOLS</NAME>
  <DESCRIPTION>All EGI Operational Tools</DESCRIPTION>
  <MONITORED>Y</MONITORED>
  <CONTACT_EMAIL>gocdb-admins@mailtalk.ac.uk</CONTACT_EMAIL>
  <GOCDB_PORTAL_URL> https://elided </GOCDB_PORTAL_URL>
  <SERVICE>                                <!-- RENAME. <SERVICE_ENDPOINT> to <SERVICE> -->
      <HOSTNAME>goc.egi.eu</HOSTNAME>
      <GOCDB_PORTAL_URL>https://elided</GOCDB_PORTAL_URL>
      <SERVICE_TYPE>egi.GOCDB</SERVICE_TYPE>
      <HOST_IP/>
      <HOSTDN>/C=UK/O=eScience/OU=CLRC/L=RAL/CN=goc.egi.eu</HOSTDN>
      <IN_PRODUCTION>Y</IN_PRODUCTION>
      <NODE_MONITORED>Y</NODE_MONITORED>
      <ENDPOINTS>                          <!-- NEW. <ENDPOINTS> wraps service's <ENDPOINT>s-->
        <ENDPOINT>
          <URL>some url </URL>
          <InterfaceName>RIS</InterfaceName>
          <EXTENSIONS/>
        </ENDPOINT>
        <ENDPOINT>
          <URL>some endpoint url</URL>
          <InterfaceName>eg.SRM.nearline</InterfaceName>
          <EXTENSIONS/>
        </ENDPOINT>
        <ENDPOINT>
          <URL>some endpoint url</URL>
          <InterfaceName>eg.SRM.online</InterfaceName>
          <EXTENSIONS/>
        </ENDPOINT>     
      <ENDPOINTS>
      <EXTENSIONS/>
  </SERVICE> 
  <SERVICE> 
    ...elided...
  </SERVICE> 
  <EXTENSIONS/>
</SERVICE_GROUP>
</results>
</results>
</source>
</source>


==Impact on other tools==
==Impact on other tools==
* Tools such as SAM and Ops portal would also need to support multiple endpoints, i.e. for monitoring and downtime notification (an implementation in GOCDB alone wouldn’t be much use).   
* Tools such as SAM and Ops portal would also need to support multiple endpoints, i.e. for monitoring and downtime notification (an implementation in GOCDB alone wouldn’t be much use for monitoring).   
* We have received a mail from Marian Babik (Wed 09/05/2012 17:09) explaining that SAM needs time to develop support for multiple endpoints before this can be introduced in GOCDB.  
* We understand that SAM needs time to develop support for multiple endpoints.  
* For detailed discussion see mins of JRA1 meeting: https://indico.egi.eu/indico/conferenceDisplay.py?confId=1922  
* For detailed discussion see mins of JRA1 meeting: https://indico.egi.eu/indico/conferenceDisplay.py?confId=1922  


Line 298: Line 365:
===Ops portal===
===Ops portal===
The new feature has an impact on the downtime notification module. The Ops portal product team stated that this a legacy module and that should be rewritten to be updated.  Considering that adding these activities in the product roadmaps is not easy at this stage (mainly for SAM that is involved in the migration of central services), we would like to know the EGI.eu position about that and the priority we should give to this requirement.
The new feature has an impact on the downtime notification module. The Ops portal product team stated that this a legacy module and that should be rewritten to be updated.  Considering that adding these activities in the product roadmaps is not easy at this stage (mainly for SAM that is involved in the migration of central services), we would like to know the EGI.eu position about that and the priority we should give to this requirement.
===AppDB===
AppDB queries GOCDB for its list of sites, services and the main service url. The MEP branch could affect how the AppDB queries for this type of information.

Latest revision as of 11:44, 1 December 2014

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


GOC DB menu: Home Documentation Index


<< Back to GOCDB/Release4/Development

Multiple Endpoints

Introduction

Requirement for multiple Endpoints per (single) Service:

  • To describe increasing heterogeneity of services (e.g. cloud).
  • To put selected endpoints of a service into downtime, e.g. separate Tape and Disk endpoints in a single SRM service.
  • To specify additional network locations for service functionalities that can't be described by the main Service-Endpoint URL alone. For example, a single service can define a main service URL AND a GRIS-Endpoint URL (Grid Resource Info Service). This will allow the Top-BDII to directly retrieve information about the service (e.g. for non-glite services) instead of via the Site-BDII via the GIIS URL.


Original Requirements

To address these requirements, we propose adding support for multiple <ENDPOINT> objects/elements per Service. Each Endpoint can define a different URL that is related to the service, for example, one Endpoint could define the actual service URL, a second could define the GRIS ldap URL, whilst a third could define an admin portal URL for the service (and so on). The Endpoint.InterfaceName (taken from GLUE2) would be used to distinguish between the different service endpoint types.

Implementation

  • Add zero or more <ENDPOINT> objects per service (based on the GLUE2 Endpoint entity).
  • When creating a downtime, an extra step would be required to select *which* Endpoints from the selected Services should be put into downtime (all, some or zero). All the services' endpoints would be selected by default allowing specific endpoints to be deselected as required. See presentation with sample screen captures: https://indico.egi.eu/indico/getFile.py/access?contribId=3&resId=1&materialId=slides&confId=2286
  • The GOCDB-PI output would be extended for nesting multiple child <ENDPOINT> elements (see below).
  • Each <ENDPOINT> nests child <URL> and <InterfaceName> elements (and other GLUE2 attributes).
  • Core InterfaceName values would be documented similar to service types.
  • It would be enforced that a Service always has at least one endpoint - deletion of a Service's last endpoint would be dissallowed.
  • We need to maintain backward compatibility for EGI/SAM by maintaining a single/root 'Service->URL' value that is rendered in the PI:
  • To maintain the single root Service->URL, we need a one time DB update:
    • Create a new one-to-many join between Service and Downtime and for each existing Service, populate this relationship for the existing data.
    • For each Service entity create a new URL field (Service->URL) and copy up the existing (single/hidden) endpoint->url value into this field.

* When a new service is created, a single endpoint is always created (the endpoint->interfaceName would be the same as the serviceType):

    • If the user provides a Service->URL value when creating the Service, this value is also used to seed the endpoint->url.
    • If the user doesn’t provide a Service->URL value(perfectly valid), we add an empty endpoint with no url (this empty endpoint is still required so we can link downtimes to the service).
    • At least one endpoint is required per Service (disallow deletion of Service's last endpoint)

  • Endpoints can be deleted from a service, even the endpoints that are linked to (historic) downtimes. This removes the Endpoint-to-Downtime association. However, the Service will always remain linked to the Downtime because of the direct Service-to-Downtime association.
  • When a Service is deleted, associations with all linked downtimes are also deleted. In addition, downtimes that are orphaned are also deleted (downtimes that are linked to other services are not deleted).


Downtime Site Service Endpoint Associations


Full GLUE2 Endpoint Rendering

  • The <ENDPOINT> elements below are incomplete compared to the GLUE2 <Endpoint> elements.
  • In future we plan extend the data model to add all of the GLUE2 <Endpoint>, <StorageEndpoint>, <ComputingEndpoint> attributes.
  • This constitutes further work but is certainly possible, especially as the core entity objects are now in place requiring only new attributes to be added to the existing endpoint objects.
  • For more info on the GLUE2 XML rendering: http://www.ogf.org/documents/GFD.209.pdf

Testing

PI Changes

  • Changes to PI methods and output are detailed below.

get_service_endpoint, get_service


 <?xml version="1.0" encoding="UTF-8"?>
 <results>
   <SERVICE_ENDPOINT PRIMARY_KEY="50257G0">  <!-- RENAME? <SERVICE_ENDPOINT> element to <SERVICE> -->
      <PRIMARY_KEY>50257G0</PRIMARY_KEY>
      <HOSTNAME>dgiref-globus.fzk.de</HOSTNAME>
      <GOCDB_PORTAL_URL>https://goc.egi.eu/portal/....</GOCDB_PORTAL_URL>
      <HOST_OS>SL5</HOST_OS>
      <BETA>N</BETA>
      <SERVICE_TYPE>SRM</SERVICE_TYPE>
      <CORE></CORE>
      <IN_PRODUCTION>Y</IN_PRODUCTION>
      <NODE_MONITORED>Y</NODE_MONITORED>
      <SITENAME>SomeSITE</SITENAME>
      <COUNTRY_NAME>SomeLand</COUNTRY_NAME>
      <COUNTRY_CODE>XX</COUNTRY_CODE>
      <ROC_NAME>NGI_XX</ROC_NAME>
      <URL>https://some.serviceurl.eu:8443/services/se<URL/> <!-- Maintain existing SE URL element for SAM/OpsPortal notifications -->
      <ENDPOINTS>   <!-- NEW. <ENDPOINTS> element wraps zero or more new <ENDPOINT> elements per Service -->
        <ENDPOINT>
           <ID>1234</ID>
           <NAME>endpointname</NAME> 
           <EXTENSIONS/>
           <URL>ldap://sBDII.grid.openu.ac.il:2170/mds-vo-name=LCG-IL-OU,o=grid</URL>
           <InterfaceName>RIS</InterfaceName>
           <!-- To Add new GLUE2 type attributes here -->  
        </ENDPOINT>
         <ENDPOINT>
           <ID>1234</ID>
           <NAME>endpointname</NAME> 
           <EXTENSIONS/>
           <URL>some.srm.nearline.url</URL>
           <InterfaceName>SRM.nearline</InterfaceName>
           <!-- To Add new GLUE2 type attributes here -->
        </ENDPOINT>
        <ENDPOINT>
           <ID>1234</ID>
           <NAME>endpointname</NAME>
           <EXTENSIONS/>
           <URL>some.srm.online.url</URL>
           <InterfaceName>SRM.online</InterfaceName>
           <!-- To Add new GLUE2 type attributes here -->  
        </ENDPOINT>      
      </ENDPOINTS>
      <EXTENSIONS>
        <EXTENSION>
           <LOCAL_ID>1</LOCAL_ID>
           <KEY>TEST_CHARGE</KEY>
           <VALUE>10</VALUE>
        </EXTENSION>
      </EXTENSIONS>
 </SERVICE> 

 </results>

get_service_group


<?xml version="1.0" encoding="UTF-8"?>
<results>
<SERVICE_GROUP PRIMARY_KEY="57654G0">  
   <NAME>OPSTOOLS</NAME>
   <DESCRIPTION>All EGI Operational Tools</DESCRIPTION>
   <MONITORED>Y</MONITORED>
   <CONTACT_EMAIL>gocdb-admins@mailtalk.ac.uk</CONTACT_EMAIL>
   <GOCDB_PORTAL_URL> https://elided </GOCDB_PORTAL_URL>
   <SERVICE_ENDPOINT>                                 <!-- RENAME? <SERVICE_ENDPOINT> to <SERVICE> -->
      <HOSTNAME>goc.egi.eu</HOSTNAME>
      <GOCDB_PORTAL_URL>https://elided</GOCDB_PORTAL_URL>
      <SERVICE_TYPE>egi.GOCDB</SERVICE_TYPE>
      <HOST_IP/>
      <HOSTDN>/C=UK/O=eScience/OU=CLRC/L=RAL/CN=goc.egi.eu</HOSTDN>
      <IN_PRODUCTION>Y</IN_PRODUCTION>
      <NODE_MONITORED>Y</NODE_MONITORED>
      <ENDPOINTS>                           <!-- NEW. <ENDPOINTS> wraps service's <ENDPOINT>s-->
        <ENDPOINT>
           <ID>1234</ID>
           <NAME>endpointname</NAME> 
           <EXTENSIONS/>
           <URL>some url </URL>
           <InterfaceName>RIS</InterfaceName>
           <!-- To Add new GLUE2 type attributes here -->
        </ENDPOINT>
         <ENDPOINT>
           <ID>1234</ID> 
           <NAME>endpointname</NAME> 
           <EXTENSIONS/>
           <URL>some endpoint url</URL>
           <InterfaceName>eg.SRM.nearline</InterfaceName>
           <!-- To Add new GLUE2 type attributes here -->
        </ENDPOINT>
        <ENDPOINT>
           <ID>1234</ID> 
           <URL>some endpoint url</URL>
           <NAME>endpointname</NAME> 
           <EXTENSIONS/>
           <InterfaceName>eg.SRM.online</InterfaceName>
           <!-- To Add new GLUE2 type attributes here -->
        </ENDPOINT>      
      <ENDPOINTS>
      <EXTENSIONS/>
   </SERVICE>   

   <SERVICE>   
    ...elided...
   </SERVICE>   
   <EXTENSIONS/>
</SERVICE_GROUP>
</results>

get_downtime

  • Current XML output: https://wiki.egi.eu/wiki/GOCDB/PI/get_downtime_method
  • Rename Downtime element's <ENDPOINT> element to <SE_ENDPOINT>
  • Add New <AFFECTED_ENDPOINTS/> element
  • Undecided: Currently each ENDPOINT is partially rendered (no Endpoint-Extensions rendered). This makes the query more efficient and Endpoint-Extensions can be viewed in get_service, get_service_endpoint as above. We could render Endpoint-Extensions if necessary.


<?xml version="1.0" encoding="UTF-8"?>
<results>
<DOWNTIME ID="1" PRIMARY_KEY="10G0" CLASSIFICATION="UNSCHEDULED">
  <PRIMARY_KEY>10G0</PRIMARY_KEY>
  <HOSTNAME>somewhere.dl.ac.uk</HOSTNAME>
  <SERVICE_TYPE>SE2</SERVICE_TYPE>
  <ENDPOINT>service.dl.ac.ukSE2</ENDPOINT>  <!-- RENAME? <ENDPOINT> to <SE_ENDPOINT> -->
  <HOSTED_BY>TestSite</HOSTED_BY>
  <GOCDB_PORTAL_URL>https://goc.egi.eu/portal/index.php?Page_Type=Downtimeampid=1</GOCDB_PORTAL_URL>

  <AFFECTED_ENDPOINTS>   <!-- NEW. All SEs affected by DT (*may not* inc. all endpoints of a service) -->
    <ENDPOINT>
         <ID>1234</ID>
         <NAME>endpointname</NAME> 
         <!--<EXTENSIONS/> not rendered - see above --> 
         <URL>some.srm.nearline.url</URL>
         <InterfaceName>SRM.nearline</InterfaceName>
      </ENDPOINT>
      <ENDPOINT>
         <ID>1234</ID>
         <NAME>endpointname</NAME> 
         <!--<EXTENSIONS/> not rendered - see above -->
         <URL>some.srm.online.url</URL>
         <InterfaceName>SRM.online</InterfaceName>
      </ENDPOINT>   
  </AFFECTED_ENDPOINTS>

  <SEVERITY>OUTAGE</SEVERITY>
  <DESCRIPTION>sample</DESCRIPTION>
  <INSERT_DATE>1384526939</INSERT_DATE>
  <START_DATE>1384531200</START_DATE>
  <END_DATE>1384621200</END_DATE>
  <FORMATED_START_DATE>2013-11-15 16:00</FORMATED_START_DATE>
  <FORMATED_END_DATE>2013-11-16 17:00</FORMATED_END_DATE>
</DOWNTIME>
</results>

get_downtime_to_broadcast

  • Current XML output: https://wiki.egi.eu/wiki/GOCDB/PI/get_downtime_to_broadcast_method
  • Add New <AFFECTED_ENDPOINTS> element
  • Undecided: Currently each ENDPOINT is partially rendered (no Endpoint-Extensions rendered). This makes the query more efficient and Endpoint-Extensions can be viewed in get_service, get_service_endpoint as above. We could render Endpoint-Extensions if necessary.


<?xml version="1.0"?>
<results>
 <DOWNTIME ID="57205437" PRIMARY_KEY="14101G0" CLASSIFICATION="SCHEDULED">
  <PRIMARY_KEY>14101G0</PRIMARY_KEY>
  <SITENAME>wuppertalprod</SITENAME>
  <HOSTNAME/>
  <SERVICE_TYPE/>
  <HOSTED_BY>TestSite</HOSTED_BY>
  <SEVERITY>OUTAGE</SEVERITY>
  <DESCRIPTION>dCache upgrade</DESCRIPTION>
  <GOCDB_PORTAL_URL>https://goc.egi.eu/portal/index.php?Page_Type=Downtimeampid=1</GOCDB_PORTAL_URL>

  <AFFECTED_ENDPOINTS>   <!-- NEW. All SEs affected by DT (*may not* inc. all endpoints of a service) -->
    <ENDPOINT>
         <ID>1234</ID>
         <NAME>endpointname</NAME> 
         <!--<EXTENSIONS/> not rendered - see above --> 
         <URL>some.srm.nearline.url</URL>
         <InterfaceName>SRM.nearline</InterfaceName>
      </ENDPOINT>
      <ENDPOINT>
         <ID>1234</ID>
         <NAME>endpointname</NAME> 
         <!--<EXTENSIONS/> not rendered - see above --> 
         <URL>some.srm.online.url</URL>
         <InterfaceName>SRM.online</InterfaceName>
      </ENDPOINT>   
  </AFFECTED_ENDPOINTS>

  <INSERT_DATE>1263908942</INSERT_DATE>
  <START_DATE>1264154400</START_DATE>
  <END_DATE>1264158000</END_DATE>
  <REMINDER_START_DOWNTIME>3155760000</REMINDER_START_DOWNTIME>
  <BROADCASTING_START_DOWNTIME/>
 </DOWNTIME>
</results>

get_downtime_nested_services

  • Current XML output: https://wiki.egi.eu/wiki/GOCDB/PI/get_downtime_nested_services_method
  • Rename Service's <ENDPOINT> element to <SE_ENDPOINT>
  • Add New <AFFECTED_ENDPOINTS> to Service
  • Undecided: Currently each ENDPOINT is partially rendered (no Endpoint-Extensions rendered). This makes the query more efficient and Endpoint-Extensions can be viewed in get_service, get_service_endpoint as above. We could render Endpoint-Extensions if necessary.


<?xml version="1.0" encoding="UTF-8"?>
<results>
    <DOWNTIME ID="1" PRIMARY_KEY="10G0" CLASSIFICATION="UNSCHEDULED">
        <SEVERITY>OUTAGE</SEVERITY>
        <DESCRIPTION>sample</DESCRIPTION>
        <INSERT_DATE>1384526939</INSERT_DATE>
        <START_DATE>1384531200</START_DATE>
        <END_DATE>1384621200</END_DATE>
        <FORMATED_START_DATE>2013-11-15 16:00</FORMATED_START_DATE>
        <FORMATED_END_DATE>2013-11-16 17:00</FORMATED_END_DATE>
        <GOCDB_PORTAL_URL>https://goc.egi.eu/portal/index.php?Page_Type=DowntimeAMPid=1</GOCDB_PORTAL_URL>
        <SERVICES>
            <SERVICE>
                <PRIMARY_KEY>1</PRIMARY_KEY>
                <HOSTNAME>somehost.dl.ac.uk</HOSTNAME>
                <SERVICE_TYPE>SE2</SERVICE_TYPE>
                <ENDPOINT>service.dl.ac.ukSE2</ENDPOINT>  <!-- RENAME? <ENDPOINT> to <SE_ENDPOINT> -->
                <HOSTED_BY>TestSite</HOSTED_BY>
                <AFFECTED_ENDPOINTS>   <!-- NEW. SEs affected by DT (*may not* inc. all endpoints of a service) -->
                     <ENDPOINT>
                       <ID>1234</ID>
                       <NAME>endpointname</NAME> 
                       <!--<EXTENSIONS/> not rendered - see above --> 
                       <URL>some.srm.nearline.url</URL>
                       <InterfaceName>SRM.nearline</InterfaceName>
                     </ENDPOINT>
                     <ENDPOINT>
                       <ID>1234</ID>
                       <NAME>endpointname</NAME> 
                       <!--<EXTENSIONS/> not rendered - see above --> 
                       <URL>some.srm.online.url</URL>
                       <InterfaceName>SRM.online</InterfaceName>
                     </ENDPOINT>
                </AFFECTED_ENDPOINTS>
            </SERVICE>
            <SERVICE>
                <PRIMARY_KEY>2</PRIMARY_KEY>
                <HOSTNAME>somehost2.dl.ac.uk</HOSTNAME>
                <SERVICE_TYPE>SE2</SERVICE_TYPE>
                <ENDPOINT>somehost2.dl.ac.ukSE2</ENDPOINT>  <!-- RENAME? <ENDPOINT> to <SE_ENDPOINT> -->
                <HOSTED_BY>TestSite2</HOSTED_BY>
                <AFFECTED_ENDPOINTS>  <!-- NEW. All SEs affected by DT (*may not* inc. all endpoints of a service)-->
                     <ENDPOINT>
                       <ID>1234</ID>
                       <NAME>endpointname</NAME> 
                       <!--<EXTENSIONS/> not rendered - see above --> 
                       <URL>some.srm.online.url</URL>
                       <InterfaceName>SRM.online</InterfaceName>
                     </ENDPOINT> 
                </AFFECTED_ENDPOINTS>
            </SERVICE>
        </SERVICES>
    </DOWNTIME>
</results>

Impact on other tools

  • Tools such as SAM and Ops portal would also need to support multiple endpoints, i.e. for monitoring and downtime notification (an implementation in GOCDB alone wouldn’t be much use for monitoring).
  • We understand that SAM needs time to develop support for multiple endpoints.
  • For detailed discussion see mins of JRA1 meeting: https://indico.egi.eu/indico/conferenceDisplay.py?confId=1922

Monitoring

SAM has to be updated to run tests against endpoint and not services. For each service SAM has to read all the endpoints registered in GOCDB and run the suitable tests according to the endpoint type (i.e. Endpoint.InterfaceName). It is not clear how we have to compute the service availability. How can we consider a service with, for example, one endpoint running and the other in downtime? Should we consider each endpoint as a service for the availability? Then, a discussion inside EGI.eu is needed to understand how to manage the multiple endpoints in a service for the availability computation.

Ops portal

The new feature has an impact on the downtime notification module. The Ops portal product team stated that this a legacy module and that should be rewritten to be updated. Considering that adding these activities in the product roadmaps is not easy at this stage (mainly for SAM that is involved in the migration of central services), we would like to know the EGI.eu position about that and the priority we should give to this requirement.

AppDB

AppDB queries GOCDB for its list of sites, services and the main service url. The MEP branch could affect how the AppDB queries for this type of information.