Embedded connectivity problems are rarely caused by a single obvious failure.

Recently, I worked on an MQTT reliability issue in a production embedded IoT system where the initial symptom looked relatively simple: after certain network interruptions and reconnect sequences, MQTT communication could stop recovering normally.

What made the problem interesting was that several different failure modes produced very similar external symptoms.

This is the debugging path we followed and some of the lessons that came from it.

The Initial Symptom

The device used an MQTT Agent architecture running on top of FreeRTOS. Under normal conditions, the system would:

1. Establish the network connection.
2. Connect to the MQTT broker.
3. Start the MQTT Agent command loop.
4. Subscribe to the required topics.
5. Resume the different application components that produce MQTT traffic.

During repeated connectivity testing, however, we observed cases where subscriptions stopped succeeding after a reconnect.

The system continued retrying, but the subscription operation failed immediately every time.

At first glance, this could easily be interpreted as another network or broker problem.

It wasn't.

First Lesson: Separate Similar Symptoms

One of the most important parts of the investigation was avoiding the assumption that every reconnect problem had the same root cause.

We had already identified another MQTT issue where a failed MQTT CONNECT operation was not correctly propagated to the caller.

The transport connection could therefore remain in an inconsistent state and subsequent reconnect attempts could eventually cause broker-side duplicate-client behavior.

That was one defect.

But the new subscription problem behaved differently.

The key observation was timing.

The subscription call failed almost immediately, while the application retried approximately every ten seconds.

That distinction was critical.

The ten-second interval was not the MQTT operation timing out. It was simply the application's retry cadence.

Once we separated those two behaviors, the investigation moved away from network timeout theories and toward the MQTT Agent itself.

Following the Error to the Agent

Instrumentation showed that the MQTT subscription API was returning an MQTT Agent resource error.

For operations such as SUBSCRIBE, the Agent needs internal resources to:

• queue the command,
• track the operation,
• and maintain state while waiting for the corresponding MQTT acknowledgment.

If those resources are exhausted or left occupied after an abnormal connection lifecycle, new Agent commands can fail before anything is even transmitted over the network.

That explained an important observation:

The network could already be healthy while the MQTT Agent was still unable to accept new work.

This is a very different failure from a TLS error, socket failure, broker rejection, or normal MQTT timeout.

Looking at the Reconnect Lifecycle

The next step was reviewing what happened around the MQTT Agent command loop during reconnect.

In an Agent-based architecture, reconnecting the TCP/TLS transport alone is not enough.

There are several pieces of state that have to remain synchronized:

• MQTT transport state
• Agent command queue
• pending acknowledgment entries
• application producers
• subscription state
• OTA or background MQTT users
• the Agent command loop itself

If application tasks continue producing commands while the Agent is being restarted, stale work can survive across what the application considers a new MQTT session.

That creates exactly the type of intermittent problem that is difficult to reproduce consistently.

Why Simply Reinitializing Wasn't Enough

A tempting recovery mechanism is to call the MQTT Agent initialization routine again whenever the connection is lost.

But initialization is not necessarily equivalent to performing a clean shutdown.

If commands are still pending, resetting structures without completing the expected Agent lifecycle can hide the underlying state-management problem rather than solve it.

Instead, we designed the recovery sequence around ownership of the MQTT Agent.

The application first prevents new MQTT producers from submitting work.

Then the Agent command loop is requested to terminate.

Once the command loop has actually returned, pending Agent commands can safely be canceled from the Agent-owned cleanup path.

Only after that cleanup does the transport disconnect and the system establish a new MQTT session.

Conceptually, the sequence becomes:

Gate producers ->terminate Agent loop -> cancel pending commands -> disconnect transport -> reconnect -> restart Agent loop -> resubscribe -> resume producers

The ordering matters.

Avoiding Overreaction

Another important design decision was not triggering a full MQTT recovery after a single resource error.

Transient failures can happen.

Instead, the recovery mechanism tracks consecutive Agent resource failures. Only after a small threshold is reached does the system request a controlled Agent restart.

This provides two benefits:

• Normal transient errors still use the lightweight retry path.
• A persistent internal Agent failure can no longer leave the device retrying forever without recovering.

Instrumentation Was Part of the Fix

For problems like this, adding the recovery logic is only half of the solution.

We also added visibility around the lifecycle:

• consecutive failure count,
• Agent termination result,
• command queue depth before cleanup,
• command queue depth after cleanup,
• reconnect result,
• and the result of the first subscription attempt after recovery.

This lets us answer a much more useful question than simply:

"Did MQTT disconnect?"

We can now determine which stage of the MQTT lifecycle failed and whether the recovery actually cleaned the Agent state.

What I Took Away From the Investigation

This issue reinforced several principles I use when debugging embedded connectivity systems.

First, identical external symptoms do not necessarily mean identical root causes.

Second, timing is evidence. An operation failing immediately is fundamentally different from one failing after a protocol timeout.

Third, reconnect logic should be treated as a state-machine problem, not simply as "disconnect and connect again."

Finally, internal observability is extremely valuable. Queue state, command-loop transitions, return codes, and lifecycle events often reveal problems that network traces alone cannot explain.

Reliable IoT systems are not built by assuming connections will remain healthy.

They are built by making every layer capable of returning to a known state when they don't.