Why default setups struggle in plants
MQTT is a common choice for industrial IoT, but a default broker configuration assumes stable networks and well-behaved clients. Plants provide the opposite: high latency, links that flap, devices that publish irregular bursts at shift change or after maintenance, and clients that reconnect aggressively. Message loss is often not the broker’s fault at all: it comes from client reconnect behavior, or from queues that grow without bound when a link drops.
Broker architecture
- High availability. Decide what outage you can tolerate. Some brokers (for example EMQX or HiveMQ) support native clustering; others, such as Mosquitto, are typically deployed as single nodes, with bridging and external failover. Choose from your availability requirement and your operations skills, then test the failover.
- Persistence. If sessions and queued messages must survive a restart, store them on reliable, adequately fast disk and monitor free space.
- Limits. Configure maximum in-flight messages, maximum queued messages per client and maximum message size. Unbounded queues turn a link outage into a memory problem.
- Capacity. Load-test with realistic client counts, message rates and a reconnect storm. The data budget calculator helps estimate the steady-state rate.
Edge brokers and bridges
Placing a local broker in each production cell or site lets machines keep publishing during a wide-area outage. The local broker stores messages and bridges them to the central broker when the link returns. Plan for the catch-up burst: replay after a long outage is far above the steady-state rate.
Client behavior
- Use persistent sessions (clean start disabled) and QoS 1 for data you cannot afford to lose, accepting possible duplicates; the options are explained in MQTT QoS levels.
- Set keep-alive intervals that fit the network, and session expiry that matches how long an offline client should be remembered.
- Reconnect with exponential backoff and random jitter, so hundreds of devices do not hammer the broker at once after an outage.
- Give every client a unique, meaningful client ID, as two clients with the same ID will keep disconnecting each other.
Security
- Encrypt everything with TLS, and use mutual TLS with a certificate per client where devices support it.
- Plan certificate rotation before the first certificate expires.
- Disable anonymous access and apply access control lists per topic branch, so a sensor can publish only to its own topics (see topic design).
- Keep the broker in a segmented zone and allow outbound-only connections from OT devices.
Observability
Monitor connected clients, in-flight and queued messages, dropped messages, retained message count, CPU and memory, TLS handshake failures and disk usage. Alert on growing queues and on clients that disappear: it is much cheaper to see a backlog than to find a gap in the data later. The wider approach is in observability in hybrid IT/OT environments.
Test failure on purpose
Cut the wide-area link, restart a broker node, fill a queue. Then check what was delivered, what was duplicated and what was lost. The MQTT disconnect recovery test plan gives a structure for exactly this.