Welcome to Grind Engineer, your guide to becoming a better software engineer!
No fluff. Pure engineering insights.
In partnership with
AI help, without the trust tax.
Most AI tools ask you to trade your data for intelligence. Norton Neo doesn't. It's the first safe AI-native browser built by Norton, and it gives you powerful built-in AI without handing your privacy over to get it. Search, summarize, and write with AI built directly into your browser. Your data stays yours. Your context stays private.
Built-in VPN, anti-fingerprinting, and ad blocking come standard. No add-ons. No setup. No compromises.
Fast. Safe. Intelligent. That's Neo.
Catch takes real action, so it only acts when it's certain — checking with you when unsure. Always on, never drops the ball. Safe for what matters most. Trust Catch at catchagent.ai.
Netflix runs thousands of microservices. When one breaks at 3 AM, the on call engineer needs to answer three questions instantly: what depends on this service, what does it depend on, and is the problem local or upstream? For years, Netflix engineers answered those questions by pinging Slack channels and grepping through stale wikis. Not anymore.
Netflix built Service Topology, an internal system that merges three separate data sources into a single, live dependency graph. It updates in near real time as traffic patterns shift, processes millions of messages per second through Kafka, and stores relationships across eight billion nodes and 150+ billion edges.
TL;DR
Netflix's Service Topology system merges three data sources (eBPF network flows, IPC metrics, and distributed traces) into a unified dependency graph. It uses Apache Kafka for ingestion, Apache Flink for stream processing, and Cassandra for graph storage. The system handles 2 million reads/sec and 6 million writes/sec, giving engineers sub second query responses for dependency lookups, blast radius analysis, and root cause investigation.
Why Static Service Maps Fail at Scale
Here's the core problem. In a microservices architecture with hundreds of deploys per day, any static dependency map is outdated the moment you save it.
Netflix learned this the hard way over four years of supporting on call engineers. Three patterns kept repeating:
Engineers needed to know all upstream and downstream dependencies of a failing service
Teams needed blast radius estimates before rolling out changes
On call needed to distinguish local failures from upstream dependency issues
Manual approaches (wiki pages, spreadsheets, tribal knowledge) broke down completely. The dependency information was scattered across siloed tools, each showing a partial picture.
The diagram below shows how Netflix solved this by unifying three data sources into one queryable graph (see Fig 1: Netflix Service Topology Architecture).
The Three Data Sources (And Why You Need All Three)
Netflix didn't pick one data source and call it done. Each source has strengths and blind spots. The magic is in combining them.
Data Source | What It Captures | Blind Spot |
|---|---|---|
eBPF Network Flow Logs | Every network connection at kernel level | No application context (endpoints, protocols) |
IPC Metrics | Protocol details, endpoint specifics, latency | Only works for instrumented services |
Distributed Traces | Full request paths including conditional branches | Subject to sampling, incomplete coverage |
eBPF network flow logs capture connections at the kernel level. Every TCP connection, every UDP packet. This gives Netflix a complete view even for services that aren't instrumented. But eBPF sees IP addresses and ports. It doesn't know that port 8080 is serving the /recommendations API.
IPC (Inter Process Communication) metrics fill that gap. Instrumented services emit metrics with protocol and endpoint details. You get "Service A called Service B's /health endpoint over gRPC." But only instrumented services show up here.
Distributed traces reveal actual request paths, including conditional branches like "if the cache misses, call the database." But tracing uses sampling. At Netflix's scale, you can't trace every request.
💡 Key Insight: No single observability data source gives you a complete dependency graph. eBPF gives breadth (everything at network level), IPC metrics give depth (application level detail), and traces give context (actual request paths). The real engineering insight is building a system that reconciles all three into one coherent view.
The Three Stage Aggregation Pipeline
Raw data from these sources can't be stitched together directly. Network flow logs track individual hops through load balancers, NAT gateways, and proxies. An engineer asking "does Service A depend on Service B?" doesn't care about the three intermediaries in between.
Netflix built a three stage aggregation pipeline to solve this:
Stage 1: Raw Ingestion. Apache Kafka serves as the ingestion backbone. Topics handle up to one million messages per second. Data is encoded in Apache Avro with schemas stored centrally.
Stage 2: Intermediary Resolution. This is the clever part. The pipeline collapses multi hop paths into direct edges. Load balancer A forwarding to proxy B forwarding to Service C becomes a single edge: "caller → Service C." This runs on Apache Flink consuming from Kafka.
Stage 3: Graph Materialization. Processed edges are written into the graph store with time window metadata for historical queries.
Each Flink job handles one Kafka topic independently. Netflix tried a monolithic approach first (one giant Flink job processing everything) and found it was impossible to tune. Independent jobs per topic let teams scale and configure each pipeline separately.
# Simplified view of how Netflix processes topology edges
# Each Flink job follows this pattern per Kafka topic
def process_topology_event(event):
# Stage 1: Filter noise (health checks, internal probes)
if is_internal_probe(event):
return None
# Stage 2: Resolve intermediaries to direct edges
source = resolve_to_application(event.source_ip)
target = resolve_to_application(event.target_ip)
# Stage 3: Emit graph edge with time window
return GraphEdge(
source=source,
target=target,
edge_type=event.protocol,
timestamp=event.time,
window="5m" # aggregation window
)
Graph Storage: Cassandra Over Neo4j
This surprised me. Netflix chose Apache Cassandra (via their internal KVDAL abstraction layer) over traditional graph databases like Neo4j or AWS Neptune.
Why? Three reasons:
Requirement | Cassandra | Traditional Graph DB |
|---|---|---|
Availability | Multi region, tunable consistency | Typically single region primary |
Write throughput | 6M writes/sec at Netflix scale | Would struggle at this volume |
Operational maturity | Netflix runs Cassandra at massive scale already | New operational burden |
The data model is straightforward. Nodes are stored as records with properties. Edges are stored as adjacency lists for efficient traversal. Each node type and edge type gets its own namespace, enabling independent scaling.
The production numbers are staggering: 2 million reads/sec, 6 million writes/sec, eight billion nodes, 150+ billion edges across 2,400 EC2 instances in 12 Cassandra clusters.
The gRPC API: Sub Second or Bust
A dependency graph is useless if querying it takes 10 seconds during an incident. Netflix set sub second response time as a hard requirement for the gRPC API.
The API supports:
Multi hop queries: "Show me everything within 3 hops of Service X"
Availability tier filtering: "Only show me Tier 1 critical dependencies"
Business domain filtering: "Show me all services in the Streaming domain"
Historical time windows: "What did the topology look like at 2 AM last Tuesday?"
Historical queries use time window aggregation instead of storing separate graph snapshots. This is a smart tradeoff. Instead of keeping a full copy of the graph every 5 minutes (which would be absurdly expensive at 8 billion nodes), they store edges with time window metadata and reconstruct the graph for any point in time.
This is particularly valuable during incident retrospectives. Engineers can correlate dependency changes with incident onset: "A new edge appeared between Service A and Service B at 1:47 AM, and the incident started at 1:52 AM."
Real World Impact: Debugging at 3 AM
Before Service Topology, a typical incident investigation looked like this:
Get paged at 3 AM
Check dashboards for the failing service
Ask in Slack: "Does anyone know what depends on payment service?"
Wait 15 minutes for someone to respond
Manually check 4 different tools for partial dependency info
Hope nothing was missed
After Service Topology:
Get paged at 3 AM
Query the topology API for all dependencies of the failing service
See blast radius instantly
Check if the problem is local or caused by an upstream dependency
Start fixing the actual issue within minutes
The blast radius feature alone changed how Netflix approaches deployments. Before rolling out a change, teams can query: "If this service goes down, what's the blast radius?" The graph returns every downstream dependent, filtered by availability tier, in under a second.
Key Engineering Takeaways
1. No single data source is enough. eBPF, IPC metrics, and traces each have blind spots. The engineering value is in merging them and resolving conflicts between them.
2. Intermediary resolution is the hard problem. Raw network data shows hops through proxies and load balancers. Collapsing those into direct application to application edges is where most of the complexity lives.
3. Choose boring technology for the storage layer. Netflix picked Cassandra over purpose built graph databases because operational maturity at scale mattered more than query expressiveness. I've seen this pattern at three other companies too. The "right" database is often the one your team already knows how to operate at 3 AM.
4. Time window aggregation beats snapshot storage. Storing full graph snapshots is expensive. Storing edges with temporal metadata and reconstructing on query is cheaper and more flexible.
5. Sub second latency is non negotiable for incident tooling. If your dependency lookup takes 10 seconds, engineers won't use it during incidents. They'll fall back to Slack and tribal knowledge.
6. One Flink job per topic scales better than monoliths. Netflix tried the monolithic approach first. Independent jobs per data source are easier to tune, debug, and scale independently.
Sources
See you in the next one!
Scortier, Signing Off!