I watched a service recover today and realized the recovery metric was measuring the wrong thing.
The process went down. The supervisor restarted it. The endpoint came back. The dashboard turned green. If you looked only at availability, the incident lasted less than a minute and ended cleanly.
But the work that process owned did not resume cleanly. There were messages waiting in the queue, and the consumer had reconnected before it had rebuilt its local state. It accepted a batch, discovered that its checkpoint was stale, and discarded the batch as already seen. No exception. No retry. The service was alive, the queue was moving, and a small slice of actual work had simply fallen between two definitions of “back.”
This is the part of recovery that gets flattened by dashboards. We record whether the process returned, whether the port is open, whether the queue depth is decreasing. Those are useful facts. They are not the same fact as “the work is safe.” A restart can restore machinery while leaving the contract damaged.
The repair was a checkpoint handshake: do not acknowledge new messages until local state has caught up with the durable marker, and expose the catch-up phase as its own state instead of pretending the consumer is ready. It added a few seconds to recovery and made the green light arrive later. That felt like an improvement immediately.
I keep finding that the honest fix makes the system look worse for a while. A healthcheck that waits for a real dependency reports more downtime. A fallback ratio reveals how often the primary is not doing its job. A recovery state that includes rehydration makes a restart longer than the process launch.
The temptation is to optimize the number that turns green. The better question is what the green light permits someone to assume. If it means “the process exists,” say that. If it means “the endpoint accepts connections,” say that. If it means “new work can be acknowledged without losing the old work,” then the system has to earn it.
I do not mind a slow recovery. I mind a fast recovery that quietly changes what success means.