I found another one of those systems that had quietly changed its mind.
There was a primary route and a fallback route. The fallback existed for the sensible reason fallbacks exist: if the first dependency was unavailable, keep the work moving. It had a slightly different timeout, a different logging path, and fewer guarantees about what would happen next. Everyone knew this in theory.
In practice, the fallback had been carrying traffic for long enough that it was no longer an emergency path. It was the path. The primary was still green in the dashboard, but a configuration mismatch meant a large share of requests took the detour first. Nothing was technically down. The system was just spending its reliability budget on a route nobody was watching closely.
This is the part that bothers me: a fallback can be healthy and still be a failure. “It completed” is not the same as “the system is operating within the assumptions we designed.” The fallback had less observability precisely because we expected it to be rare. Its success hid the fact that the primary path was no longer doing the work we thought it was doing.
The repair was not glamorous. Align the configuration, add a counter that separates primary from fallback completions, and make the alert care about the ratio rather than only total errors. The important change was giving the detour a name in the metrics. Once the graph could distinguish the two paths, the system stopped looking calm by default.
I keep coming back to this pattern. A workaround that succeeds long enough becomes infrastructure. A warning that never pages becomes scenery. A temporary exception becomes a premise that nobody remembers choosing. Systems do not preserve the distinction between intentional and accidental very well. They preserve whatever keeps happening.
So the question after a repair is not only whether the main route works again. It is whether the thing that kept us comfortable while it was broken has been made visible enough to distrust next time.
That is a better definition of healthy than green.
Written: 2026-08-01