10x the context. Half the time.
Speak your prompts into ChatGPT or Claude and get detailed, paste-ready input that actually gives you useful output. Wispr Flow captures what you'd cut when typing. Free on Mac, Windows, and iPhone.
Welcome to Grind Engineer , your guide to becoming a better engineer!
No fluff. Pure engineering insights.
Notion ran on a single Postgres instance for five years. Then VACUUM started stalling, dead tuples piled up, and the team was staring at transaction ID wraparound, the failure mode where Postgres refuses every write to avoid clobbering your data. Sharding fixed it. It also meant rewriting how every query in the product finds its data.
What Sharding Buys That Replicas Cannot
Read replicas copy the whole dataset. Every replica still ingests every write, still holds the same working set, still runs the same VACUUM. You get more read capacity and zero write capacity.
Sharding is the only move that cuts the write load and the data volume at the same time. Split your rows across 32 machines and each machine carries roughly 1/32 of the writes and 1/32 of the data. Notion's second migration in 2023 took shard CPU from over 90% at peak down to around 20%.
The moment you split, though, your database stops being one database. It becomes a fleet of machines with a routing layer in front, and that layer has to work out which machine holds the row you asked for.
The Shard Key Decides Every Query
Pick a shard key and you've picked which queries stay fast forever.
Notion sharded on workspace ID, and their engineering post gives the reason plainly. Every block belongs to exactly one workspace, and people work inside one workspace at a time, so most queries land on one shard. Their block, space, discussion, comment and collection tables all follow the same key, so the rows a query needs sit together.
Get it wrong and a routine lookup starts touching every machine you own.
💡 Key Insight: Your shard key decides which future product features are cheap to build and which ones will need a second datastore to exist at all.
Slack ran out of road on their first choice. They started on workspace based sharding and moved to Vitess, the MySQL sharding layer YouTube built in 2010, which became a CNCF incubation project in February 2018. Flexibility was one of their stated reasons. With Vitess they could shard message data by channel ID instead of team, because single large customers had outgrown a single shard.
Four Things That Stop Working
Once related rows live on different machines, four things your database used to hand you for free become your code's problem.
Cross shard joins. A join needs both sides on the same machine. If orders is sharded by customer_id and products by product_id, joining them means pulling rows over the network and joining in the routing layer. Notion's team put it bluntly: distributed joins are expensive, and their whole key choice was built to avoid them.
Global secondary indexes. You shard users by user_id, then someone writes the login screen. Now every lookup by email fans out to all 32 shards, because the index that knows about emails is local to each shard. The usual fix is a separate lookup table mapping email to shard, which you now have to keep in sync.
Distributed transactions. Vitess ships three modes: SINGLE rejects any transaction that crosses a shard, and TWOPC gives you an atomic commit through two phase commit. The default is neither. It's MULTI, a best effort commit with no atomicity guarantee, which means your application is the thing that has to clean up a partial write.
Unique constraints. MongoDB's manual is direct about this. On a sharded collection only the shard key, a compound index prefixed by the shard key, or _id can be unique, and _id uniqueness is enforced per shard only when _id is not the shard key. So a globally unique username is no longer a UNIQUE constraint. It's an application problem, and of the four this is the one I see teams find out about last, usually after the schema is already live.
Shard key | Works well for | Where it hurts |
|---|---|---|
Workspace or tenant ID | Multi tenant SaaS, single tenant reads | One giant customer becomes one hot shard |
User ID | Per user timelines and settings | Any lookup by email, phone, or username fans out |
Hash of primary key | Even write distribution | Range scans and "recent items" queries hit everything |
Created date | Time series, archival | All of today's writes land on one shard |
Hot Shards Are a Distribution Problem
Even distribution of rows is not even distribution of traffic.
Slack described exactly this before Vitess: large enterprise customers concentrated load on specific shards while most of the fleet sat underutilized. Adding machines does nothing here. The busiest shard is busy because of what the key does, not because you're short on hardware.
Timestamp keys produce the same problem from the other direction. Shard by created_at and every write today lands on the same machine while the other 31 idle. You've built a distributed system with a single writer.
Resharding Is Possible and Still Expensive
Changing your mind is no longer impossible. Notion did it twice.
The 2021 migration went from one Postgres instance to 480 logical shards across 32 databases. It took audit log based double writes, a three day backfill on an m5.24xlarge, dark reads to verify the copy matched, and five minutes of scheduled downtime at the end.
Twenty one months later they went from 32 databases to 96 using Postgres logical replication, with no observable downtime. Deferring index creation cut the sync from three days to twelve hours. They also had to split PgBouncer into four groups so the connection count wouldn't flatten the old databases.
MongoDB added an online reshardCollection in 5.0. Read the requirements before you get excited: the docs ask for I/O below 50%, CPU below 80%, roughly double the collection and index size in free space on every recipient shard, and an application that tolerates a two second window where writes to that collection block.
Both are months of planning and custom tooling rather than a config change.
Most Teams Shard Two Years Too Early
I've watched teams reach for sharding at a data volume a laptop could serve. No survey backs this up, it's a judgment from watching enough of them, and I'll stand behind it: sharding early is the most expensive premature optimization in backend engineering. Unlike a bad cache, you cannot delete it on a Friday. It lives in every query your product will ever write.
Amazon RDS will rent you a db.r8g.48xlarge with 192 vCPUs and 1,536 GiB of memory. Before you split anything, spend a week on the boring list: fix the missing index, move analytics to a replica, add connection pooling, archive rows nobody reads, cache the top ten queries. Notion got five years out of one box.
If you're on Postgres and genuinely out of runway, look at Citus before you build routing yourself. It's a Postgres extension that distributes tables across nodes and keeps the SQL surface you already write against. Since Citus 11 in June 2022, the features that used to be enterprise only, including shard rebalancing over logical replication, are open source.
Shard when you have measured that a single writer is the bottleneck, not when the row count grows a comma.
What This Means For Engineers
Write the ten queries your product runs most, then pick the shard key that keeps all ten on one shard. Any query that isn't on that list is a query you're agreeing to make slow.
Before you shard, budget for the four features you're giving up. Cross shard joins, global uniqueness, multi shard atomicity, and secondary index lookups all become application code, and that code is where the bugs live.
Assume you'll reshard. Notion did it twice in under two years. Build the routing layer so the shard count is a lookup, never a modulo hardcoded in a repository class.
Sources:
→ Find me on : Social Links
That’s it for today, keep learning!
Scortier, Signing Off!