Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Message brokers"

From EGIWiki
Jump to navigation Jump to search
(Created page with '{{Template:Op menubar}} {{Template:Tools menubar}} <br> <center><font color="red">'''THIS IS A DRAFT DOCUMENT'''</font></center> __TOC__ <br> === Q1: What is the EGI Opera…')
 
 
(16 intermediate revisions by 6 users not shown)
Line 1: Line 1:
{{Template:Op menubar}}
{{Template:Op menubar}}
{{Template:Tools menubar}}
{{Template:Tools menubar}}
{{TOC_right}}
[[Category:SAM]]


<br>
The production EGI Operations Message Broker Network (PROD MSG) is used in order to facilitate the message exchange between the operational tools of EGI. This broker network consists of 2 geographically separated brokers which are operated by 2 institutes: AUTH and SRCE.
<center><font color="red">'''THIS IS A DRAFT DOCUMENT'''</font></center>
__TOC__
 
<br>
 
=== Q1: What is the EGI Operations Message Broker Network?  ===
 
The production EGI Operations Message Broker Network (PROD MSG) is used in order to facilitate the message exchange between the operational tools of EGI. This broker network consists of 4 geographically separated brokers which are operated by 3 institutes: AUTH, CERN and SRCE.
 
=== Q2: How can one find the list of the brokers that are part of the PROD MSG Network?  ===


= Find the list of brokers =
The list of the brokers that are connected to the PROD MSG Network is always available through the BDII information system. It is strongly advised that all producers and consumers that use the PROD MSG Network use the BDII in order to find the brokers that are part of the PROD MSG Network at any time.  
The list of the brokers that are connected to the PROD MSG Network is always available through the BDII information system. It is strongly advised that all producers and consumers that use the PROD MSG Network use the BDII in order to find the brokers that are part of the PROD MSG Network at any time.  


=== Q3: It seems a bit heavy weight to inquire the BDII every time that a consumer or producer wants to use the PROD MSG Network. Is it possible to cache the information?  ===
In order not to inquire the BDII every time that a consumer or producer wants to use the PROD MSG Network, information can be cached. The cache must refresh its information at least every day.


Yes it is possible to cache the information, but the cache must refresh its information at least every day
= Usage policy =
The PROD MSG Network is used in order to facilitate the message exchange between the operational tools of EGI. If the operators of an operational tool, that is not part of the existing set of approved operational tools for the the PROD MSG Network (a table of approved tools follows), want to use the PROD MSG Network, then they have to apply by sending a request via GGUS. Please indicate in the ticket that it must be handed to the "Messaging SU".


=== Q4: Who can use the PROD MSG Network?  ===
In their request they have to provide the following information:  
 
As stated in the answer to Q1, the PROD MSG Network is used in order to facilitate the message exchange between the operational tools of EGI. If the operators of an operational tool, that is not part of the existing set of approved operational tools for the the PROD MSG Network (a table of approved tools follows), want to use the PROD MSG Network, then they have to apply a request via (GGUS??) In their request they have to provide the following information:  


*name: a unique short name to identify the operational tool  
*name: a unique short name to identify the operational tool  
*contact information: contact information for notifications related to the messaging infrastructure (i.e. maintenance, upgrades)
*description: a multi-line description of what the operational tool does, including pointers (URLs) for more information  
*description: a multi-line description of what the operational tool does, including pointers (URLs) for more information  
*expected activity (if possible: average '''and''' peak numbers):  
*expected activity (if possible: average '''and''' peak numbers):  
Line 39: Line 32:
*security requirements in terms of ACLs: which accounts are allowed to do what on which destinations
*security requirements in terms of ACLs: which accounts are allowed to do what on which destinations


=== Q5: Is it possible to use the PROD MSG Network by other applications apart from operational tools of EGI?  ===
= Usage of PROD MSG by other applications =
The PROD MSG Network is a critical component for the operational requirements of the Infrastructure. Taking in mind that its capacity are not infinite, it is not advised that the PROD MSG Network is used for applications and tools outside of the operational tools of EGI. We envision that in the future, that the broker service will be yet another component of the UMD and that it will be possible to install it at the site, national and VO level opening the possibility for the creation of a number of service specific networks. Until then, application developers/operators can request for access to the PROD MSG Network and they will have to provide the same information as it is described [[Message_brokers#Usage_policy|above]]. Apart from this, we require the contact information of the operator of the application.


The PROD MSG Network is a critical component for the operational requirements of the Infrastructure. Taking in mind that its capacity are not infinite, it is not advised that the PROD MSG Network is used for applications and tools outside of the operational tools of EGI. We envision that in the future, that the broker service will be yet another component of the UMD and that it will be possible to install it at the site, national and VO level opening the possibility for the creation of a number of service specific networks. Until then, application developers/operators can request for access to the PROD MSG Network and they will have to provide the same information as it is described in the answer for Q4. Apart from this, we require the contact information of the operator of the application and that the application will run for a limited time in the DEVELOPER MSG Network
=Operational tools using PROD MSG=
 
Following is a list of operational tools, that are already using the PROD broker network:


<br>  
<br>  
Line 51: Line 43:
! Tool  
! Tool  
! Description or URL
! Description or URL
! Queues and Topics
|-
|-
| SAM
| ARGO
| https://wiki.egi.eu/wiki/SAM
| [[ARGO]]
|
|-
| APEL
| [[APEL]]
|  
|}
|}
</center>
</center>
= Maintenance windows =
The following maintenance windows are used for applying regular OS upgrades. All necessary precautions (switching off bdii services etc) are taken care of beforehand by the operations teams.
<br>
<center>
{| border="1" cellspacing="0" cellpadding="5"
|-
! Hostname (network endpoint)
! Window
|-
| mq.cro-ngi.hr
| first Wednesday of each month (10:00 - 14:00 CET)
|-
| broker-prod1.argo.grnet.gr
| first Thursday of each month (10:00 - 14:00 CET)
|}
</center>
= External links =
* [https://tomtools.cern.ch/confluence/display/MIG/Messaging.html Messaging Documentation]

Latest revision as of 13:56, 7 February 2017

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Tools menu: • Main page • Instructions for developers • AAI Proxy • Accounting Portal • Accounting Repository • AppDB • ARGO • GGUS • GOCDB
• Message brokers • Licenses • OTAGs • Operations Portal • Perun • EGI Collaboration tools • LToS • EGI Workload Manager


The production EGI Operations Message Broker Network (PROD MSG) is used in order to facilitate the message exchange between the operational tools of EGI. This broker network consists of 2 geographically separated brokers which are operated by 2 institutes: AUTH and SRCE.

Find the list of brokers

The list of the brokers that are connected to the PROD MSG Network is always available through the BDII information system. It is strongly advised that all producers and consumers that use the PROD MSG Network use the BDII in order to find the brokers that are part of the PROD MSG Network at any time.

In order not to inquire the BDII every time that a consumer or producer wants to use the PROD MSG Network, information can be cached. The cache must refresh its information at least every day.

Usage policy

The PROD MSG Network is used in order to facilitate the message exchange between the operational tools of EGI. If the operators of an operational tool, that is not part of the existing set of approved operational tools for the the PROD MSG Network (a table of approved tools follows), want to use the PROD MSG Network, then they have to apply by sending a request via GGUS. Please indicate in the ticket that it must be handed to the "Messaging SU".

In their request they have to provide the following information:

  • name: a unique short name to identify the operational tool
  • contact information: contact information for notifications related to the messaging infrastructure (i.e. maintenance, upgrades)
  • description: a multi-line description of what the operational tool does, including pointers (URLs) for more information
  • expected activity (if possible: average and peak numbers):
    • number of connections (= new sessions) per second
    • number of messages per second (received by the broker)
    • amplification factor (= number of messages sent by the broker divided by the number of messages received)
    • message sizes
    • number of clients connected at the same time
  • protocol(s) used as stomp|openwire x plain|ssl
  • exhaustive list of all destinations used, with wildcards if needed (e.g. /topic/grid.accounting.apel.{ngi} and /queue/Consumer.{role}.grid.accounting.apel.{ngi})
  • whether the operational tool is local (i.e. only using one node, messages should not be propagated) or global (network of broker wise); if local, we need the broker(s) that it is allowed to run on
  • credentials: a list of accounts/DNs used or a credential source (i.e. Nagios NGI instances from GOCDB)
  • security requirements in terms of ACLs: which accounts are allowed to do what on which destinations

Usage of PROD MSG by other applications

The PROD MSG Network is a critical component for the operational requirements of the Infrastructure. Taking in mind that its capacity are not infinite, it is not advised that the PROD MSG Network is used for applications and tools outside of the operational tools of EGI. We envision that in the future, that the broker service will be yet another component of the UMD and that it will be possible to install it at the site, national and VO level opening the possibility for the creation of a number of service specific networks. Until then, application developers/operators can request for access to the PROD MSG Network and they will have to provide the same information as it is described above. Apart from this, we require the contact information of the operator of the application.

Operational tools using PROD MSG


Tool Description or URL Queues and Topics
ARGO ARGO
APEL APEL

Maintenance windows

The following maintenance windows are used for applying regular OS upgrades. All necessary precautions (switching off bdii services etc) are taken care of beforehand by the operations teams.


Hostname (network endpoint) Window
mq.cro-ngi.hr first Wednesday of each month (10:00 - 14:00 CET)
broker-prod1.argo.grnet.gr first Thursday of each month (10:00 - 14:00 CET)

External links

Retrieved from "https://wiki.egi.eu/w/index.php?title=Message_brokers&oldid=93047"