How and Why Netflix Built a Real-Time Distributed Graph: Part 1 — Ingesting and Processing Data Streams at Internet Scale

Key Actionable Insights

1

Map each Kafka source topic to its own dedicated Flink job rather than building a monolithic consumer. A 1:1 mapping from source topic to processing job allows independent tuning of CPU, memory, parallelism, and checkpointing intervals per data stream, which is critical when topics have varying throughput patterns throughout the day.

Netflix initially tried a single Flink job consuming all topics but found it impossible to find stable configurations. The tradeoff of more jobs to manage was worth the improved maintainability and operational stability.

2

Implement a deduplication stage using stateful process functions and timers in your stream processing pipeline. Buffering and merging overlapping updates to the same entities within a configurable time window significantly reduces downstream write volume, which is essential when operating at millions of records per second.

Netflix uses this approach before publishing to Data Mesh to reduce the 1M+ messages/second input down to manageable write volumes across their storage systems.

3

Persist streaming data to both real-time systems and batch-accessible data warehouse tables (such as Apache Iceberg) simultaneously. This dual-write strategy enables data backfill scenarios when older data is no longer available in Kafka topics due to retention policies, providing a safety net for data recovery and historical analysis.

Netflix tailors Kafka retention policies per topic based on throughput and record size to balance data availability with storage costs, making the Iceberg backup essential for completeness.

4

Choose a graph data model when your use case requires analyzing relationships across multiple microservices' data domains. Graphs excel at relationship-centric queries, flexible schema evolution, and pattern detection — capabilities that require expensive joins or manual denormalization in traditional table-based models.

Netflix's microservices architecture resulted in siloed data across hundreds of services, making cross-domain member interaction analysis extremely difficult until they adopted a graph representation.

5

Design your graph data model to be as generic and flexible as possible from the start, so that adding new node and edge types is an infrequent operation. Separating each node and edge type into its own Kafka topic allows bespoke tuning and scaling per type, even though it increases the number of topics to manage.

Netflix accepted the tradeoff of managing more Kafka topics in exchange for the ability to independently scale and tune each graph entity type as their business verticals expanded.