Eventual Consistency: Why Your Database Lies to You

Creators lock in holiday calendars 90 days out. Structure commissions, recruit creators, and optimize your creator affiliate strategy before the rush with Levanta's 90-Day Holiday Sprint. Get the Free Guide.

Access up to $750* from your upcoming paycheck within minutes for a small fee, or wait 3 days and get it for free. No credit check. No late fees. 

Get Started Now

^{*Not all users will qualify. Advances range from $25–$750; average advance is $170. Express transfer fees may apply.}

Global hiring doesn't come with a playbook. Join Oyster's webinars and events to learn how leading companies are hiring, expanding, and staying compliant across borders.

Explore Upcoming Events

You add a book to your Amazon cart on your phone. You switch to your laptop. The book isn't there. You refresh. Still nothing. Three seconds later, it appears. You didn't lose it. Your laptop just read from a replica that hadn't caught up yet.

That three second gap? That's eventual consistency doing exactly what it's supposed to do.

Most engineers think consistency is a toggle. Strong or eventual. On or off.

It's not. It's a spectrum with at least three major stops.

[DIAGRAM 1: Consistency Spectrum — upload diagram-1-consistency-spectrum.png in Beehiiv editor]

Strong consistency means every read returns the most recent write. Period. If you write price = $50, every subsequent read on every node returns $50. No exceptions. The cost? Every write has to wait until all replicas acknowledge it. That's expensive when your replicas sit in Virginia, Frankfurt, and Singapore.

Causal consistency is the middle child nobody talks about. It guarantees that if event B was caused by event A, every node sees A before B. But unrelated events can arrive in any order. Google's Spanner uses a version of this with GPS clocks and atomic clocks to get both strong consistency and global distribution. Most companies don't have that budget.

Eventual consistency is the loosest guarantee. All replicas will converge to the same value... at some point. Could be milliseconds. Could be seconds. No upper bound is promised. And that's the whole point.

The CAP theorem (proven by Seth Gilbert and Nancy Lynch in 2002) says a distributed system can guarantee only two of three properties during a network partition: Consistency, Availability, and Partition tolerance.

Network partitions aren't theoretical. Cables get cut. Datacenters go dark. Cross region links drop. So partition tolerance isn't optional. You're always choosing between consistency and availability.

Amazon chose availability for its shopping cart. If the cart goes down during Black Friday, Amazon loses millions in revenue per minute. If a customer briefly sees a stale cart? Minor inconvenience. The math is obvious.

And it's not just availability. There are three forces pushing systems toward eventual consistency:

Latency. A strongly consistent write to 5 replicas across 3 continents means waiting for the slowest one. That might be 300ms on a good day. Eventually consistent writes return after hitting a single node. Sub millisecond.

Throughput. Coordinating consensus (think Paxos or Raft) on every single write limits your writes per second. Eventual consistency lets each node accept writes independently.

Fault tolerance. If 2 of 5 replicas are down, a strongly consistent system might refuse writes entirely. An eventually consistent system keeps serving. The downed replicas catch up when they recover.

Here's what happens under the hood when you update a price from $30 to $50 in an eventually consistent database.

Your write hits the leader node (or any node, in a leaderless system like Dynamo). The leader immediately acknowledges the write. Then it starts replicating the change to follower nodes asynchronously.

Follower 1 picks up the change in 5ms. Done.

Follower 2 is busy with a compaction job. It doesn't pick up the change for 200ms.

During those 200ms, any client reading from Follower 2 sees the old price ($30). That's a stale read. It's not a bug. It's the expected behavior of eventual consistency.

The catch: "eventually" has no SLA. In practice, replication lag in systems like DynamoDB or Cassandra is typically under 100ms. But under heavy load, network congestion, or partial failures, it can spike to seconds.

Stale reads are one thing. Conflicting writes are a different beast entirely.

Imagine two users update the same product price simultaneously, each hitting a different replica. Replica 1 says $50. Replica 2 says $45. When the replicas sync up, which value wins?

Last Write Wins (LWW) picks the write with the latest wall clock timestamp. It's simple. It's also dangerous. Wall clocks drift. Two servers might disagree on what time it is by tens of milliseconds. And even with perfect clocks, you're silently discarding one user's write. Cassandra uses LWW by default. It works for use cases where losing a write is tolerable (like updating a profile bio). For anything financial? Not a chance.

Vector clocks solve the clock drift problem by tracking causality instead of time. Each node maintains a counter. When you see conflicting vector clocks that aren't ancestors of each other, you know you have a genuine concurrent conflict. Amazon's Dynamo and Riak use vector clocks to detect conflicts, but they don't resolve them automatically. They return both versions and let the application decide.

Application level merge is the most flexible and the most work. Amazon's shopping cart famously takes the union of conflicting cart states. If one replica says {book, laptop} and another says {book, headphones}, the merged result is {book, laptop, headphones}. Additions are never lost because a lost addition = lost revenue. Deletions might reappear, but that's a minor annoyance compared to losing a sale.

From 24 hours of Amazon's production shopping cart traffic, 99.94% of requests saw exactly one version. Conflicts were astronomically rare (0.00057% saw 2 versions). The system optimizes for the common case while handling the rare case gracefully.

You probably use eventual consistency every day without thinking about it.

DNS is the largest eventually consistent system on the planet. When you update an A record to point your domain to a new IP, that change doesn't propagate instantly. Recursive resolvers around the world cache the old value based on the record's TTL (Time To Live).

Set a TTL of 3600 seconds (1 hour)? Resolvers might serve stale DNS records for up to 4 hours in practice. Set it to 60 seconds? Propagation happens in 5 to 10 minutes. The pro move: lower your TTL to 60 seconds two days before a planned migration, then raise it back after the cutover.

Netflix, Discord, and Apple all run on Apache Cassandra, which offers tunable consistency per query. Need strong consistency for a payment? Set consistency level to QUORUM (majority of replicas must agree). Serving a product catalog? Consistency level ONE is fast and good enough.

That tunability is the quiet superpower. You don't pick one consistency model for your entire system. You pick it per operation.

Pure eventual consistency has an embarrassing failure mode. You update your profile name to "Alex." You refresh the page. It still says "Alexander." You panic. You submit a bug report. Nothing was broken. You just read from a replica that hadn't caught up.

Read your writes consistency solves this. It guarantees that any read following your own write will reflect that write. Other users might still see stale data, and that's fine. But you always see your own changes.

How is it implemented? The simplest approach: sticky sessions. Route all of a user's requests to the same replica. If you wrote to Replica 3, you read from Replica 3. No lag, no surprise.

Azure Cosmos DB takes this further with 5 consistency levels: Strong, Bounded Staleness, Session, Consistent Prefix, and Eventual. Session consistency (read your writes + monotonic reads within a session) is their default, and it covers 90%+ of real world use cases without paying the latency tax of strong consistency.

What if your data structure could resolve conflicts automatically, with zero coordination between replicas?

That's what CRDTs (Conflict free Replicated Data Types) do. Formally defined by Marc Shapiro in 2011, CRDTs are data structures designed so that any two replicas can merge their states and always converge to the same result, regardless of the order operations arrive.

A simple example: a G Counter (grow only counter). Each replica tracks its own count. To get the total, you sum all replicas. Replica A has 5, Replica B has 3, Replica C has 7. Total = 15. No matter when they sync, the answer is always correct. No coordination needed.

Apple uses CRDTs in the Notes app. You edit a note on your iPhone while offline. You edit the same note on your Mac. When they reconnect, the changes merge automatically without either edit being lost.

Redis, Riak, and Cosmos DB all support CRDT data types. The trade off? Metadata overhead. Every operation needs to carry enough context for conflict free merging, which increases storage and network costs. For high throughput distributed counters, sets, and registers, CRDTs are the cleanest solution available today.

Not every system can tolerate stale reads. And pretending otherwise has caused real production incidents.

Banking. Two ATMs read a balance of $100 from different replicas. Both dispense $100. The account now has negative $100 and the bank ate the loss. Financial systems must use strong consistency. A ledger that's "probably correct" is worse than a system that's temporarily unavailable.

Inventory. Two customers read inventory = 1 from different replicas. Both purchase. The system converges to inventory = 0, but you shipped 2 units of a product you only had 1 of. This is why Stripe and other payment processors use serializable transactions for balance mutations.

Legal records. Court filings, regulatory submissions, anything with a compliance audit trail. Stale reads can mean regulatory violations.

The rule of thumb: ask yourself "what's the cost of a stale read?" If the answer is "a user sees slightly old data for a moment," eventual consistency is fine. If the answer is "we lose money, violate the law, or corrupt a critical business invariant," you need strong consistency. Full stop.

1. Most systems are a mix, not a monolith. The same application often uses strong consistency for payments and eventual consistency for product recommendations. Pick the right model per operation, not per system.

2. "Eventually" usually means milliseconds, not minutes. In production DynamoDB, Cassandra, and Cosmos DB clusters, replication lag is typically under 100ms. The scary edge cases (seconds of lag) happen under extreme load or partial failures, and that's exactly when you need availability most.

3. The question isn't "should I use eventual consistency?" It's "what's the cost of a stale read for this specific operation?" Answer that question and the architecture decision makes itself.

Sources:

→ Find me on : Social Links

That’s it for today, keep learning!
Scortier, Signing Off!