What is Retained Messages?

If you know MQTT even for just a little bit, you may already know that for each MQTT message, there is a topic name, and there is the payload. If you dig a little deeper, you’ll find that there are also message properties and flags. One of the flags is called Retain, which is what this post is about.

Upon receiving a message with the Retain flag set, the MQTT broker must store the message for the topic to which the message was published, and it must store only the latest message. So the subscribers which are interested in this topic can go offline, and reconnect at any time to receive the latest message instead of having to wait for the next message from the publisher after the subscription.

As illustrated below, when a client subscribes to a topic, if there is a retained message for this topic, the message is sent to the client immediately.

When to use MQTT Retained Messages?

While allowing publishers to decouple subscribers, the publish-subscribe pattern also has the disadvantage that subscribers cannot actively fetch messages from publishers. When a subscriber receives a message depends on when the publisher publishes it, which is inconvenient in some scenarios.

New subscribers can get the latest data immediately without waiting for unpredictable times with retained messages. Below are some examples:

• Smart home devices only send state data when the state changes, but the control APP needs to know the device's current state whenever the user opens the APP.
• The interval between sensors reporting data can be very long, but subscribers may need to get the latest data immediately after subscribing.
• Properties such as sensor version and serial number that do not change frequently can be published as a retained message for later subscribers to get the information.

How to use MQTT Retained Messages?

For MQTT client SDKs, there are typically APIs or parameters to set the Retain flag. For example the paho MQTT Java client library, and the the Erlang MQTT client emqtt.

For MQTT client applications, either with a command line or graphic interface, you should be able to find where to set the Retain flag.

In this post, we are not going to dig into the programming SDKs.
We will try to demonstrate MQTT retained messages using the open-source cross-platform MQTT 5.0 desktop client - MQTT X.

If you start the MQTT X application for the first time, you will see the main window below. Click the New Connection button to create an MQTT connection.

We only need to fill in a connection Name and leave the other parameters as default. The Host will default to the public MQTT Broker provided by EMQX Cloud. Finally, click the Connect button in the upper right corner to create an MQTT connection.

After the successful connection, publish a message to the topic sensor/t1 in the message input box.

Next, we check the Retain flag and publish two retained messages to the topic sensor/t2.

Then click the New Subscription button to create a subscription.

We subscribe to the wildcard topic sensor/+, which will match the topics sensor/t1 and sensor/t2.

Check out the blog Understanding MQTT Topics & Wildcards by Case for more details.

Finally, we will see that the subscription successfully receives the second retained message, neither the normal message for sensor/t1 nor the first retained message for sensor/t2. This shows that the MQTT Broker will only store the latest retained message for each topic.

Q & A about MQTT Retained Messages

How do I know a message is a retained message?

When a message is originated from the Retain storage in the broker, the Retain flag is set, so the subscriber knows that this is not a new message after its subscription.

That is, if a retained message is published after the subscription, the subscriber will receive it as a regular message (without the Retain flag). After a retained message is delivered, if the subscriber wishes to receive the retained message again, it needs to resubscribe.

In the example below, we subscribe to the topic sensor/t2 and then publish a retained message to the topic, the subscriber receives the message immediately, but without the ‘retain’ flag. Then we delete the subscription and re-subscribe to sensor/t2 to receive the message again with the ‘retain’ flag set.

How long are retained messages stored? How to delete it?

The broker will only store the latest retained message for each topic, and the validity of the retained message is related to the broker's settings. If the broker is set to store retained messages in memory, they are lost when the MQTT Broker is restarted; if they are stored on disk, they remain after the broker is restarted.

Retained messages are not part of session states, meaning retained messages are not deleted when the publishing session terminates. There are several ways to delete retained messages.

• When a client publishes a retained message with an empty payload to a topic, the broker deletes the retained message under that topic.
• Delete on the MQTT Broker, e.g., the EMQX MQTT Broker provides the ability to delete retained messages from management API or from the Dashboard.
• MQTT 5.0 protocol added Message Expiry Interval property, which can be used to set the expiration time of the message when publishing. The message will be automatically deleted after the expiration time, regardless of whether it is a retained message.

MQTT Retained Messages in EMQX

EMQX is the most popular MQTT Broker, with over 10 million downloads worldwide. Recently, EMQX released version 5.0, which achieved 100 million MQTT connections + 1 million message throughput per second through a 23-node cluster, making EMQX 5.0 the most scalable MQTT Broker worldwide so far.

EMQX 5.0 supports viewing and setting retained messages in the built-in Dashboard. You may use the following command to install EMQX 5.0 open-source version for trial.

docker run -d --name emqx -p 1883:1883 -p 8083:8083 -p 8084:8084 -p 8883:8883 -p 18083:18083 emqx/emqx:latest

After successful installation, use your browser to visit http://127.0.0.1:18083/ to experience the new EMQX 5.0 Dashboard.

The default Username is admin, and the Password is public.

After successful login, you can click the Configuration->MQTT menu to view the list of retained messages. You can also view the Payload of retained messages or delete a retained message.

Click on the Settings menu under Retainer, and you will see that EMQX supports setting the Storage (memory or disk), the Max Retained Messages, the Expire, and other parameters in the Dashboard.

Summary

This article introduces and demonstrates the use of MQTT Retained Messages. Referring to this article, readers can use MQTT Retained Messages to get data immediately after subscription.

In addition, the MQTT protocol has many more valuable features. Check out EMQ's MQTT Getting Started and Advanced series of articles for a deeper understanding, to explore more advanced applications of MQTT, and to start developing MQTT applications and services.

Originally published at https://www.emqx.com.