Brokee had been a microservice shop from day one — the first product was built around Kubernetes assessments, and every new capability after that got its own service. That’s a reasonable way to start. It’s a much worse way to end up three years in, when tracing a single request meant following it through four services, and one of those services having a bad day could take the others down with it.

Every service talked to every other service the same way: a direct HTTP call, wait for the response, hope nothing timed out. It worked fine when the system was small. It stopped working as we grew. Three things kept biting us:

• Scalability. Traffic spikes turned into bottlenecks, because a slow service downstream made every service upstream of it slow too.
• Rigidity. Adding a feature usually meant touching the communication path of every service in between, not just the one that actually changed.
• Fragility. A single point of failure anywhere in the chain could take out things that had nothing to do with it. We had backoff and retry logic in our APIs, but that only helps if the thing you’re retrying against is actually reachable.

The fix was moving our internal communication off synchronous request/response and onto an event-driven model instead — NATS, with JetStream for the parts that needed durability, not just speed. Mykhailo Anhelskyi and I built this together: he owned the implementation and testing, I worked the architecture and the edge cases alongside him as they turned up. He’d joined the team early in his career, and it didn’t show in the quality of the work — some of the trickier NATS behavior in here, he ran down faster than I would have on my own.

Why NATS and JetStream

NATS is a lightweight pub/sub messaging system — services publish messages to a named subject, and anything subscribed to that subject gets them, with no direct dependency between publisher and subscriber. JetStream sits on top of it and adds durability: messages get persisted to disk and redelivered if the intended consumer wasn’t there to receive them the first time. Plain NATS decouples services from each other. JetStream makes sure a message survives a service being briefly down, instead of vanishing the moment nobody’s listening.

That combination bought us the three things we were missing: services could talk without knowing about each other directly, JetStream’s durable subscriptions meant a dead consumer didn’t mean a lost message, and scaling a spike in traffic no longer meant a wave of architectural changes — just more consumers reading from the same stream.

Standing up the cluster

We deployed NATS via Helm, running three server instances behind Raft consensus for fault tolerance, with persistent-volume storage rather than NATS’ in-memory option — durability was the whole point, and we didn’t want a node restart to take messages with it. The three instances were spread across separate nodes so a single node going down wouldn’t take out the cluster, and we used NATS’ headless service type, since clients need to talk to specific server instances directly rather than through a load balancer.

Moving off direct calls

The first migration was straightforward in shape, if not in volume: replace direct HTTP calls with publish/subscribe. Where the Common service used to call the Auth service directly and wait for a response, it now publishes a message to a subject like auth.users.roles.assign, and the Auth service — subscribed to that subject as part of a queue group, so the work distributes across however many instances are running — picks it up asynchronously. For the handful of flows that genuinely needed an immediate answer rather than fire-and-forget, NATS’ request-reply pattern covered it: publish a request with a unique reply subject attached, and the responder sends its answer straight back to that inbox.

That got us decoupling. It didn’t get us durability — in a plain pub/sub setup, if the subscriber is down when a message goes out, the message is just gone. That’s where JetStream came in: message streams (grouped by subject, retained until acknowledged) and durable consumers that pick up exactly where they left off, managed through a small Kubernetes-native controller so most of the configuration lived in a chart rather than in code that needed rebuilding every time a subject or consumer name changed.

Where it got interesting

The lessons weren’t in getting NATS running — that part was mechanical. They showed up once the system was under real load and doing things that looked fine on paper but weren’t:

Request-reply isn’t a durability strategy. It was tempting to assume request-reply would give retry-for-free the same way JetStream does. It doesn’t — it’s built for immediate, synchronous-feeling exchanges, not for guaranteeing a message eventually gets processed. For anything that needed to survive a consumer being briefly unavailable, JetStream was the right tool; request-reply was the wrong one, even though on the surface they look similar.

Consumer names and queue-group names have to match, and nothing tells you that. A durable consumer that doesn’t share its name with its queue group silently fails to subscribe — no error pointing at the mismatch, just messages going nowhere. We found this the hard way, by reading the client SDK source rather than the docs, which didn’t call it out clearly.

backoff and ackWait don’t combine. Both control retry behavior, and it’s natural to assume tuning both gives finer control. Instead, setting both at once applied silently and then did nothing — Helm accepted the config with no errors, and it took checking the running configuration directly, through the NATS-Box debugging pod, to find that the settings hadn’t actually taken effect. The fix was picking one: backoff for services that need several retries at increasing intervals, ackWait — sized to the handler’s real execution time plus a buffer — for services that just need one clean retry window.

Acknowledgment has to be explicit, on purpose. The default behavior — a message counted as handled the moment the handler ran, whether or not it actually succeeded — meant failures could look like successes. Once we switched to manually acknowledging only after confirming the work actually completed, that class of silent failure went away.

None of these are exotic mistakes. They’re the ordinary cost of adopting infrastructure that’s new to your team, and the value of digging into the source yourself when the documentation doesn’t say enough — which we ended up doing more than once.

What we got out of it

A messaging system that decouples services instead of chaining them, survives a consumer being briefly down instead of losing work, and scales by adding consumers instead of re-architecting. We didn’t stop at internal communication — the plan since has been to lean on NATS’ pub/sub further, including places where we’re still using REST between services, and to add monitoring and dead-letter handling for messages that exhaust their retries.

The part of this I’d underline for anyone in a similar spot, whether you’re the one writing the code or the one scoping the project: the payoff of durable messaging is real, but it shows up in the failure modes you don’t hit anymore, which is a much harder thing to point at in a demo than a new feature is. Worth it anyway.