Messaging Service improvements

Main

EGI.eu operations services

Support

Documentation

Tools

Activities

Performance

Technology

Catch-all Services

Resource Allocation

Security

Tools menu:	• Main page	• Instructions for developers	• AAI Proxy	• Accounting Portal	• Accounting Repository	• AppDB	• ARGO	• GGUS	• GOCDB
• Message brokers	• Licenses	• OTAGs	• Operations Portal	• Perun	• EGI Collaboration tools	• LToS	• EGI Workload Manager

Future plans and improvements for the Messaging Service

Security improvements

The current setup of the broker network service allows both authenticated and unauthenticated connections but the majority of the clients are using unauthenticated connections. This makes it impossible to implement proper ACLs for queues/topics.

In order to authenticate the clients there are 2 options, either use username/password credentials or x509 clientAuth. The possibility of using Grid proxy certificates is still investigated.

The infrastructure is ready to use authenticated connections but the following steps have to be followed:

• Identify and register the clients
• Setup credentials for each client
• Modify client configurations to use authenticated access

Reliability and availability improvements

In order to consume messages, clients need to specify the endpoint they need to connect to. Currently the majority of the clients choose one of the available brokers and use it without any failover options enabled. STOMP protocol provides failover functionality that can be used by clients either consuming or producing results. In most of the cases, given that the clients are using correctly the STOMP protocol, the only change that would be required on the client side is the connection endpoint.

After the upgrade of broker network software (to ActiveMQ 5.5) at the end of 2011, a new feature is available that allows dynamic failover of the servers. This allows message broker administrator to dynamically add or remove brokers on the network and request clients to move to another instance. This functionality will be tested on the TEST broker network and if the tests are successful, it will be implemented in the PROD network too.

Another feature that is available currently at the PROD messaging broker network is the Virtual Destinations. This feature minimizes the loss of messages due to disconnections for Topic destinations. Clients can consume from special Queues (Virtual Queues) instead of Topics. Queues keep the messages until the consumer receives them.

Scalability improvements

The list of the brokers that are connected to the PROD MSG Network is always available through the BDII information system. It is strongly advised that all producers and consumers that use the PROD MSG Network use the BDII in order to find the brokers that are part of the PROD MSG Network at any time. In order not to inquire the BDII every time that a consumer or producer wants to use the PROD MSG Network, information can be cached. The cache must refresh its information at least every day while it is highly recommended to do so in much more frequent periods (i.e. once per hour).

Although Virtual Queues improve reliability, they can have a negative impact on the scalability of the service. Each Virtual Queue that is created for a destination adds a multiplier to the messages that are produced for each message submission, so if one message is sent to a topic with 10 virtual queues, 11 messages will be send by the broker network. Given that Virtual Queues behave just like the normal Queues, if there is no subscriber to consume the messages, these messages will remain stored at the broker's store forever. In order to eliminate this issue, all messages with an expired TTL will be automatically moved to a special queue (Dead Letter Queue) in order to be cleaned by the system maintenance scripts.

Idle client eviction is another option that will be tested on the TEST Broker Network in order to be enabled at the PROD Broker Network. Currently brokers are keeping connections alive until the clients close them. This can lead to a large number of idle connections (either by producers that don't send anything or by consumers that listen on queues/topics where there is no message). Given that each connection to a broker is consuming resources of the service, idle connections will be evicted. Of course the client can reconnect at any time or even send anti-IDLE messages to avoid unnecessary disconnections. The maximum idle duration for the clients will be announced to EGI operations meeting.

Best practices

Given the number of programming languages and libraries that are available for connecting to the broker network it is impossible to cover all cases. The aim will be to cover the following functions:

• Broker discovery
• Message producer
• Message consumer

by:

• Setting the requirements of the function
• Having the function implemented in at least one programming language as a reference

Write access to the code repository could be enabled for others to contribute.

2012 Roadmap

2012/Q1

• Implementation of credential sharing procedure between brokers
• First round of client identification
• First round of communication with clients to setup credentials / request of clients to migrate to authenticated connections
• Test of (dynamic) failover functionality
• Usage of virtual queues by clients
• Announcement of the IDLE client duration to EGI operations meeting [DONE, Agenda-20-01-2012 ]
• Creation of the best-practices repository containing at least one of the described functions

2012/Q2

• Second round of client identification, check which instances are still using unauthenticated connections.
• Second round of communication with clients to setup credentials / request of clients to migrate to authenticated connections
• Release of all functions at the best-practices repository

2012/Q3

• Third round of client identification, check which instances are still using unauthenticated connections.
• Third round of communication with clients to setup credentials / request of clients to migrate to authenticated connections
• Request to EGI OMB to approve authenticated only connections to PROD message broker network

2012/Q4

• Enforce authentication to all clients if OMB approves it
• Enforce authorization rules if OMB approves it