How Netflix Uses Apache

Netflix engineered a real-time recommendation delivery system for live events that can update over 100 million devices in under a minute.

Netflix built a Real-Time Distributed Graph (RDG) to connect member interaction data across their expanding business verticals including streaming, live events, and mobile games.

The article discusses how Netflix scales its Muse application to provide data-driven creative insights at a massive scale, focusing on the architectural evolution and optimizations made to handle t...

The article discusses the architectural evolution and technical decisions involved in building a robust Ads event processing pipeline at Netflix.

The article discusses the implementation of a system at Netflix for tracking 'impressions'—the visual elements users interact with while browsing content.

This article discusses Netflix's Distributed Counter Abstraction, a service designed to enable distributed counting at scale while maintaining low latency performance.

Netflix's TimeSeries Data Abstraction Layer is designed to efficiently store and query vast amounts of temporal event data with low latency.

Netflix's Key-Value Data Abstraction Layer (KV DAL) enhances data access across its distributed databases, addressing challenges in consistency, durability, and performance.

The article discusses how Netflix supports a diverse range of machine learning (ML) systems through its Machine Learning Platform (MLP) and the Metaflow framework.

The article announces the release of bpftop, a command-line tool aimed at optimizing and monitoring eBPF programs at Netflix.

Netflix recently hosted its first Data Engineering Summit, bringing together engineers to share insights on data processing patterns and building reliable data pipelines.

The article discusses the implementation of incremental processing at Netflix using Netflix Maestro and Apache Iceberg.

The article discusses the implementation of Streaming SQL within Netflix's Data Mesh framework, highlighting how it democratizes stream processing by allowing users to express complex data transfor...

The article discusses the challenges and strategies involved in migrating critical traffic at scale without downtime, focusing on replay traffic testing as a key technique.

The article discusses the development of a data reprocessing pipeline within Netflix's Asset Management Platform (AMP), designed to efficiently manage and update digital media assets' metadata.

The article discusses the implementation of sample data pipelines using Dataflow at Netflix, focusing on bootstrapping, standardization, and automation of batch data pipelines.

The article discusses how Netflix's Content Engineering team has transitioned to a federated GraphQL platform, enabling domain teams to build and operate their own Domain Graph Services (DGS).

The article discusses Netflix's auto-diagnosis and remediation system, Pensive, which addresses failures in their complex data platform.

The article features an interview with Pallavi Phadnis, a Senior Software Engineer at Netflix, discussing her journey into data engineering, her experiences with large-scale data challenges, and he...

The article discusses Netflix's approach to safely updating client applications, emphasizing the importance of quality and testing in a diverse device ecosystem.

The article discusses the evolution of data movement at Netflix Studio through the implementation of a Data Mesh architecture.

The article discusses the Elasticsearch indexing strategy implemented in Netflix's Asset Management Platform (AMP), focusing on how to efficiently manage and query large volumes of digital media as...

The Netflix Cosmos Platform is a computing framework that integrates microservices, asynchronous workflows, and serverless functions to handle resource-intensive algorithms and complex workflows.

This article discusses the optimization of data warehouse storage at Netflix, focusing on the AutoOptimize system designed to enhance performance and reduce costs.

The article discusses the advancements in the Video Multi-Method Assessment Fusion (VMAF) metric, which aims to improve video quality evaluation for the video community.

The article discusses Netflix's implementation of streaming telemetry through the open-source project gnmi-gateway, which leverages the OpenConfig data model and gRPC Network Management Interface (...

The article discusses how Netflix utilizes Apache Druid for real-time analytics to enhance user experience.

The article discusses the open-sourcing of Metaflow, a human-centric framework for data science developed by Netflix.

The article announces the open-source launch of Polynote, a polyglot notebook designed for data scientists and machine learning researchers.

The article discusses the infrastructure for Contextual Bandits and Reinforcement Learning, highlighting insights from a meetup hosted at Netflix.

Delta is a data synchronization and enrichment platform developed by Netflix to address the challenges of keeping multiple datastores in sync while allowing for data enrichment.

The article discusses Netflix's reimagined experimentation analysis infrastructure, focusing on how data scientists can now contribute more effectively to A/B testing through a modular architecture.

The article discusses the design principles for mathematical engineering in the Experimentation Platform at Netflix, highlighting the challenges and strategies for enhancing data science productivi...

The article discusses the implementation of the Netflix Media Database (NMDB), focusing on its architecture, system requirements, and key components that enable scalability, reliability, and effici...

The article discusses Netflix's experience with adopting GraphQL for their Marketing Technology team, highlighting the transition from REST APIs to GraphQL, the benefits gained, and the challenges ...

The Keystone Stream Processing Platform is Netflix's infrastructure for real-time data processing, enabling data-driven decision-making across the organization.

Netflix's Security Intelligence and Response Team (SIRT) introduces Diffy, a triage tool designed for digital forensics and incident response (DFIR) in cloud environments.

The article discusses Netflix's use of Apache Spark for enhancing its recommendation systems, detailing three key projects presented at the Spark+AI Summit 2018.

The article discusses Archer, a platform developed by Netflix to simplify media processing and innovation.

The article discusses Metacat, a metadata service developed by Netflix to enhance the discoverability and management of big data across various data sources.

The article discusses the lessons learned from building observability tools at Netflix, emphasizing the importance of log management, distributed request tracing, metrics analysis, and user interfa...

Titus, Netflix's container management platform, has been open-sourced to share its technology and insights gained from years of production use.

This article provides updates on Netflix's container management platform, Project Titus, detailing its evolution, integration with AWS, and future collaboration opportunities.

The article discusses Netflix's ongoing journey in platform engineering, highlighting the evolution of their cloud ecosystem, engineering challenges, and the adoption of new technologies.

The article discusses Simone, a distributed simulation service developed by Netflix that automates the testing and certification of its app on various devices.

This article discusses the evolution of container usage at Netflix, focusing on the development and implementation of Titus, Netflix's container management platform.

Netflix Conductor is a microservices orchestration engine designed to manage complex workflows in the content platform engineering team.

The article discusses Netflix's use of Apache Mesos for distributed resource scheduling, highlighting its evolution and various applications across the company's engineering projects.

The article discusses Netflix's approach to supporting global languages in their search functionality, detailing the challenges faced and solutions implemented for various languages.

The article discusses Netflix's development of the Photon project, a set of IMF validation tools aimed at enhancing the adoption of the Interoperable Master Format (IMF) standard.