Search This Blog

Monday, October 7, 2019

MQTT Cluster Retained topic issue and resolution

Problem Statement

The messages sent using retained flag , are identified through publisher node only to the broker . So the broker just stores the retained messages ,tagging publisher node information and only recognizes the publisher node when a subscriber connects to the same topic , connecting the same node.


Lets assume a scenario where you have a sensor publishing its status only when changed e.g. Door sensor. What happens if a new subscriber subscribes to this status?
Without retained messages the subscriber would have to wait for the status to change before it received a message. However with the retained message the subscriber would see the current state of the sensor. What is important to understand is that only one message is retained per topic. The next message published on that topic replaces the last retained message for that topic.

Processing Details

We will then examine how retained messages work in various cases-
The basic process is this:
  1. Publish message on a topic with retained message flag not set, and set
  2. Subscribe to message on topic
  3. Monitor messages received and analyse results

First case

From below screen we just want to show case that , If a publisher is publishing a message on a topic on which two subscribers (Subscriber 1 and Subscriber 2) are connected . Then they will receive the same message at the same time either with retained =true or retained =false(default case). Retained is required when we need last good value of a device.

Second Case

Suppose a publisher is publishing a message and one of the subscriber is in disconnect mode. Now what we want is to get the last good value of that client , since Subscriber 2 was in disconnect mode when publisher published the message so when its get connected again it will receive the last good value if and only if the node is same on which publisher published the message. 
-Mirroring is only for queues not for retained messages.
In case subscriber 2 get connected with Node 2 then in that case It wont receive the updated value which was published on Node1.

When to Use Retained Messages

Generally you will publish a message with the retained flag set when the message contains persistent data.
For example a sensor could publish information about itself like firmware version number,IP address, Current state.
This information is unlikely to change and so you only need to publish it once using the retain flag and any new clients can retrieve that information.

Solutions tried so for

There are few customized solutions as given below -
Either to store the retained messages in some consistent database like Cassandra or Riak as per Rabbitmq documentation.(Not tried yet as we need erlang knowledge as well to change or configure the conf script of rabbitmq to use cassandra or Riak).
These implementations are suitable for development but sometimes won't be for production needs. MQTT 3.1 specification does not define consistency or replication requirements for retained message stores, therefore RabbitMQ allows for custom ones to meet the consistency and availability needs of a particular environment. For example, stores based on Riak and Cassandra would be suitable for most production environments as those data stores provide tunable consistency.
We have tried to connect all the nodes available in clusters and published  the same message to all (Using HTTP APIs) but again as per broker rule its not publishing the message to all cluster nodes and only recognize the node which was connected and published the retained message.

We have also discussed in rabbitmq forum on this issue and they told , they don't support this feature as of now but in future versions they would support it.

Note: EMQ and Hive are already supporting this feature with cluster mode.


The retained message feature is useful feature for keeping the last state of an object, and is especially useful when the state doesn’t change frequently.
Quality of service settings don’t impact retained messages.