Welcome to Grind Engineer , your guide to becoming a better engineer!
No fluff. Pure engineering insights.
Global hiring doesn't come with a playbook. Join Oyster's webinars and events to learn how leading companies are hiring, expanding, and staying compliant across borders.
Domain Names + Web and Email Hosting You Need
Still paying GoDaddy or Namecheap prices? Porkbun sells most domains at cost for low, transparent registration and renewal pricing with no nonsense. Get free features like WHOIS privacy and SSL certificates, plus real human support 24/7, 365 days a year. Save $1 on your next domain name now.
Have you launched your creator affiliate program this holiday season? Don't miss out on building demand and landing the best partnerships. Levanta's 90-Day Holiday Sprint breaks it all down. Download the Guide.
Netflix serves 250 million subscribers across 190 countries. On Christmas Eve 2012, a handful of AWS load balancers failed in US East. Instead of the entire streaming platform going dark, only a few services were affected. The rest kept serving. The reason? Circuit breakers.
TL;DR: A circuit breaker monitors calls to a downstream service and "trips open" when failures spike, blocking further requests so the failing service can recover and the rest of your system stays alive. It cycles through three states (Closed, Open, Half Open) and is the single most effective pattern for preventing one bad microservice from taking down your entire platform.
Same Principle as Your Electrical Panel
Your home's electrical circuit breaker does one thing: when it detects dangerous current, it cuts the circuit before your wiring melts. You flip it back on when the problem is fixed.
The software version works the same way. When calls to a downstream service start failing (timeouts, errors, 5xx responses), the circuit breaker stops sending requests to that service entirely. Your caller gets an instant failure instead of burning a thread for 30 seconds waiting on a response that won't come.
Michael Nygard introduced this pattern in his 2007 book Release It!, and Netflix made it mainstream with their open source library Hystrix. Most production microservices deployments use some form of circuit breaking today.
Three States, One Goal: Fail Fast
The circuit breaker is a state machine with three modes. Get these and you've got the whole pattern.
Closed is the default. Requests flow through to the downstream service normally. The breaker quietly counts failures inside a sliding window (say, the last 100 calls or the last 10 seconds). If the failure rate stays under a threshold (typically 50%), nothing happens. Life is good.
Open kicks in when failures cross that threshold. The breaker trips. Every incoming request gets an immediate failure response or a fallback value. Zero requests reach the downstream service. This does two things: it frees up threads in your service that would otherwise sit blocked, and it gives the downstream service breathing room to recover. The circuit stays open for a configured timeout, usually 30 seconds.
💡 Key Insight: The circuit breaker's real value is freeing threads. A service waiting on a 30 second timeout consumes a thread for 30 seconds. A tripped circuit breaker rejects that call in under 1 millisecond.
Half Open is the recovery test. When the timeout expires, the breaker lets a small number of trial requests through (say, 10 calls). If those succeed above a success threshold, the circuit closes again. Normal traffic resumes. If they fail, the circuit trips back open and resets the timer.
Cascading Failures Destroy Entire Platforms
Senior engineers lose sleep over this failure mode. And it's surprisingly easy to trigger.
Say Service B talks to a database. The database connection pool gets exhausted. Service B starts responding in 5 seconds instead of 50 milliseconds. Now Service A calls Service B, and threads in Service A sit waiting. 100 threads. 200 threads. Service A's entire thread pool fills up with requests waiting on Service B.
Now Service A can't handle ANY requests. Not even the ones that have nothing to do with Service B. Services C, D, and E that depend on Service A? They start failing too. One saturated database connection pool just took down five services.
I've watched this exact scenario play out at two different companies. Both times, the root cause was a single slow dependency. Both times, the blast radius was the entire platform.
With a circuit breaker in place, Service A trips open after detecting Service B's failures. It returns a fallback (cached data, a default response, or a graceful error) in under a millisecond. Its thread pool stays healthy. Services C, D, and E never feel the impact.
Netflix's 2012 Christmas Eve Proof
December 24, 2012. Peak streaming traffic. A handful of AWS Elastic Load Balancers in US East failed, cutting off access to some backend services.
Netflix had hundreds of ELBs running. Only the services behind the failed ELBs were affected because Hystrix circuit breakers isolated the blast radius. Services with open circuits served cached content or degraded gracefully. Subscribers on most devices kept watching.
This wasn't luck. Netflix had invested years in building Hystrix and testing it through Chaos Engineering (their Chaos Monkey intentionally kills services in production to validate resilience). The 2012 outage was the real world exam. They passed.
Hystrix is deprecated now (archived in 2018), but the vocabulary it established (circuit state, fallback, bulkhead isolation) became the industry standard.
Three Resilience Patterns Combined
These three resilience patterns solve different problems. You need all of them. They're not interchangeable.
Pattern | What it does | Protects against | Risk if used alone |
|---|---|---|---|
Retry | Repeats a failed call with backoff | Transient blips (network glitch, brief timeout) | Retry storms hammer a dying service harder |
Circuit Breaker | Stops all calls to a failing service | Sustained failures and slow responses | No help for one off transient errors |
Bulkhead | Isolates thread/connection pools per dependency | Resource exhaustion from one bad dependency | Doesn't stop retries or detect failure patterns |
The three work best together: Retry inside a Closed circuit breaker, backed by a Bulkhead that isolates each dependency's thread pool. When retries keep failing, the circuit breaker trips. The bulkhead keeps other dependencies on their own dedicated threads, unaffected.
Real world impact: circuit breakers cut error rates by 58%, bulkheads improved availability by 10%, and retries boosted success rates by 21% (across surveyed production systems).
The Thundering Herd Trap
When a circuit breaker transitions from Open to Half Open, every client instance that was blocked suddenly tries to send requests at the same time. If you have 50 service instances with open circuits, and they all hit Half Open within the same second, that's 500 trial requests slamming a service that's barely recovering.
This is the thundering herd problem, and it can force the circuit right back to Open.
Fix: limit trial requests to a small number (10 per instance). Add jitter to the timeout duration so instances don't all transition simultaneously. Resilience4j does this out of the box. Istio handles it at the mesh level by ejecting unhealthy pods and gradually readmitting them.
Per Instance vs Distributed: Pick Your Tradeoff
A per instance circuit breaker lives inside each service replica. Instance 1 might trip its circuit while Instance 2 still thinks everything is fine. Simple to build, no shared state needed. For fleets under 10 instances, this works.
A distributed circuit breaker shares state across all instances (through Redis, a shared cache, or a service mesh like Istio). When one instance detects failure, all instances stop sending traffic. Consistent behavior. But you've added a new dependency (the shared state store) that can itself become a point of failure.
For large fleets or serverless environments with hundreds of instances, distributed or mesh level circuit breaking is the practical choice. Istio does this through DestinationRule resources: connection pool limits and outlier detection, with zero application code changes.
When Circuit Breakers Are the Wrong Tool
Not every call needs one. Adding a circuit breaker where it doesn't belong just adds complexity.
Skip circuit breakers for in process method calls (there's nothing to isolate). Skip them for operations that must complete no matter what, like writing an audit log or processing a payment. For those, use a durable queue with retries.
Skip them when there's no fallback. A circuit breaker that trips open and returns an empty response is worse than a slow response. If you can't define what the fallback behavior should be, the circuit breaker will just convert a slow failure into a fast but confusing one.
And if your system has low traffic (a few requests per minute), the overhead of tracking failure rates and managing state transitions isn't worth it. A simple timeout and retry will do.
What Your Monitoring Dashboard Should Show
Circuit breaker state changes are some of the highest signal alerts you can set up.
Circuit trips to Open: A dependency is failing. Page the on call. This is a real incident in progress.
Circuit stays Open for more than 5 minutes: Recovery isn't happening. Escalate. Check the downstream service directly.
Rapid Open to Half Open to Open cycling: The service is "flapping." It recovers just enough to pass trial requests, then fails again under real traffic. This usually means the root cause hasn't been fixed, only the symptoms are temporarily clearing.
Track failure rate, circuit state, fallback invocation rate, and p99 latency per dependency. Resilience4j exports these as Micrometer metrics. Istio exposes them through Envoy's built in telemetry.
The Modern Toolkit
If you're building this today, here's the ecosystem:
Resilience4j (Java/Spring) is the standard for application level circuit breaking. Lightweight, composable, built for Java 8+. Version 2.2.0 is production ready.
Istio handles circuit breaking at the infrastructure level with zero code changes. Configure a DestinationRule and the sidecar proxy does the rest. Pair it with Resilience4j for application level fallbacks.
Polly (.NET) and gobreaker (Go) cover the other major ecosystems. Envoy proxy powers Istio's circuit breaking under the hood and can be used standalone.
Run both Istio (infrastructure) and Resilience4j (application) together. Istio ejects unhealthy instances. Resilience4j handles fallback logic and fine grained retry policies.
What This Means For Engineers
Start with the failure scenario, not the implementation. Before configuring a circuit breaker, answer: what happens when this dependency goes down? If you can't define a fallback (cached data, default response, graceful degradation), the circuit breaker won't save you.
Combine retry + circuit breaker + bulkhead as a unit. Retry handles transient blips. Circuit breaker handles sustained failure. Bulkhead prevents resource starvation. Using only one of the three leaves you exposed.
Treat circuit state changes as first class operational signals. An open circuit isn't a "warning." It's an active incident. Wire it to your alerting pipeline. If you're running microservices without circuit breaker dashboards, you're flying blind.
Sources
→ Find me on : Social Links
That’s it for today, keep learning!
Scortier, Signing Off!