Apache Kafka Programming Tutorials & Engineering Articles

The article discusses Pinterest's transition to a next-generation database ingestion framework designed to address the limitations of legacy systems.

This article introduces uForwarder, Uber's open-source push-based consumer proxy for Apache Kafka's async queuing system.

This article discusses how Uber utilizes a pull-based ingestion model in OpenSearch™ to effectively index streaming data.

This article discusses Uber's transition from batch to streaming data ingestion using Apache Flink, which significantly enhances data freshness and operational efficiency.

This article discusses Uber's implementation of Apache Pinot to manage and analyze its extensive inventory and catalog data efficiently.

This article discusses the implementation of zone failure resilience in Apache Pinot at Uber, detailing strategies to ensure uninterrupted service during zone failures.

Netflix engineered a real-time recommendation delivery system for live events that can update over 100 million devices in under a minute.

Netflix built a Real-Time Distributed Graph (RDG) to connect member interaction data across their expanding business verticals including streaming, live events, and mobile games.

The article discusses the implementation of a Policy Simulator at Uber to enhance the safety and determinism of Identity and Access Management (IAM) policy changes.

The article discusses the critical importance of maintaining consistent data across multiple systems as organizations grow.

The article discusses the challenges of maintaining consistent product data across systems and third-party platforms in digital commerce, focusing on reconciliation patterns that can enhance data i...

uReview is an AI code review platform developed by Uber to enhance the code review process by providing timely, high-quality feedback.

The article discusses how Uber processes early chargeback signals to mitigate payment fraud and enhance customer trust.

The article discusses the evolution of Uber's Search Platform, highlighting its transition from Elasticsearch to an in-house solution called Sia, and ultimately to the adoption of OpenSearch.

The article discusses how LinkedIn utilizes Hoptimator to enhance the ingestion process for Apache Pinot, a real-time distributed OLAP datastore.

The article discusses the implementation of a system at Netflix for tracking 'impressions'—the visual elements users interact with while browsing content.

The article discusses the MySQL fleet at Uber, which consists of over 2,300 independent clusters that support critical operations for the platform.

The article discusses Change Data Capture (CDC) at Pinterest, detailing its importance for real-time data processing and the implementation of a Generic CDC solution using Debezium.

This article discusses Netflix's Distributed Counter Abstraction, a service designed to enable distributed counting at scale while maintaining low latency performance.

The article discusses Uber's implementation of Presto Express, an enhancement to the Presto SQL query engine aimed at reducing the end-to-end Service Level Agreement (SLA) for short-running queries.

The article discusses Uber's advanced settlement accounting system, which is crucial for managing financial transactions involving payment service providers (PSPs).

The article discusses how Uber optimizes the training of Large Language Models (LLMs) using both open-source and in-house models.

This article discusses the significant improvements made to Uber's Experiment Evaluation Engine, achieving a 100x reduction in latency by transitioning from a remote evaluation architecture to a lo...

This article discusses the integration of ClickHouse, a high-performance columnar database, with Estuary Flow, a data integration platform, to enable real-time analytics on Salesforce data.

The article discusses Pinterest's implementation of Tiered Storage for Apache Kafka®️, highlighting a broker-decoupled approach that offloads data to cheaper remote storage.

The article provides an in-depth exploration of Riverbed, a framework within Airbnb's tech stack that optimizes data consumption from system-of-record data stores and updates secondary read-optimiz...

The article discusses how Uber utilizes Apache Pinot for low-latency offline table analytics, highlighting its capabilities in handling various use cases, including real-time and offline data inges...

The article discusses Uber's strategy for shifting end-to-end (E2E) testing left in their development process to improve efficiency and reduce operational costs.

The article discusses the Sparkle framework developed by Uber to standardize modular ETL processes, enhancing developer productivity and data quality.

Odin is Uber's stateful platform designed to manage various technologies for data storage efficiently.

This article introduces Kafka Tiered Storage at Uber, detailing its architecture and the motivation behind its implementation.

This article discusses how Uber counts job participation at scale, detailing the integration of Apache Pinot™ to address challenges in data processing and analysis.

The article delves into Uber's comprehensive accounting data testing strategies, emphasizing the importance of precision and integrity in financial processes.

The article discusses Uber's journey in scaling its AI/ML infrastructure, highlighting the transition from on-premise to cloud solutions, the implementation of new technologies, and the optimizatio...

The article discusses Uber's efforts to build a scalable, real-time chat system to enhance customer experience.

DataCentral is Uber's proprietary platform designed for Big Data observability, chargeback, and governance.

The article discusses a hybrid bulk data processing framework developed to improve recruiting efficiency during data ownership transfers, particularly in the context of company mergers and recruite...

This article discusses Uber's experience with garbage collection (GC) tuning to enhance the reliability of Presto, an open-source distributed SQL query engine.

uVitals is an anomaly detection and alerting system developed by Uber to enhance the reliability of its services by quickly identifying and addressing issues in multi-dimensional time series data.

This article discusses the CGW Stack, which combines ClickHouse, Grafana, and WarpStream to provide a cost-effective and efficient logging solution at scale.

The article discusses the implementation and operational benefits of the Unified PubSub Client (PSC) at Pinterest, highlighting improvements in developer velocity, stability, and scalability.

The article discusses how Uber utilizes Apache Pinot for real-time analytics of mobile app crashes, enhancing their ability to detect and resolve issues quickly.

This article discusses strategies for making large data loads into ClickHouse resilient and efficient, particularly when migrating from other systems.

The article discusses LinkedIn's innovative use of Apache Beam for real-time streaming processing, handling over 4 trillion events daily across more than 3,000 pipelines.

The article discusses the implementation of a Unified Session for analytical events at Uber, aimed at enhancing data consistency and analytics across various applications.

The article discusses the transformative impact of real-time data and data streaming technologies on business operations, emphasizing their role in enabling rapid decision-making.

The article discusses Uber's implementation of Attribute-Based Access Control (ABAC) to manage access across its microservices architecture.

This article discusses the integration of ClickHouse with Kafka Connect and Confluent Cloud to facilitate real-time event streaming, particularly for Ethereum Cryptocurrency events.

This article serves as an introductory guide to implementing Change Data Capture (CDC) between PostgreSQL and ClickHouse, utilizing native features and tools like Debezium and Kafka.

The article discusses Uber's Automated Audit Framework designed to manage and audit financial transactions at internet scale.