How LinkedIn Uses Apache Kafka

The article discusses how LinkedIn utilizes Hoptimator to enhance the ingestion process for Apache Pinot, a real-time distributed OLAP datastore.

The article discusses a hybrid bulk data processing framework developed to improve recruiting efficiency during data ownership transfers, particularly in the context of company mergers and recruite...

The article discusses LinkedIn's innovative use of Apache Beam for real-time streaming processing, handling over 4 trillion events daily across more than 3,000 pipelines.

The article discusses how LinkedIn scaled its Salt infrastructure to support its growing needs for remote execution jobs, achieving a tenfold increase in job capacity and improved reliability.

The article discusses LinkedIn's Hosted Search, a fully managed cloud-based search solution designed to simplify the integration of search functionalities for application teams.

The article discusses TopicGC, a service developed by LinkedIn to clean up unused metadata in Kafka clusters.

The article discusses how LinkedIn utilizes Apache Pinot for real-time analytics on network flow data, emphasizing the importance of observability in their infrastructure.

The article discusses the implementation of a load-balanced Brooklin Mirror Maker at LinkedIn, which efficiently replicates large-scale Kafka clusters.

The article discusses the implementation of near real-time personalization features at LinkedIn, focusing on how member actions can be leveraged to enhance recommendation systems without significan...

The article discusses Charles's journey of relocating his family to Silicon Valley for a role at LinkedIn, highlighting the support he received during the transition and his experiences in the tech...

The article discusses how LinkedIn implemented a Real-Time Feedback system to enhance developer tooling and improve productivity.

The article discusses the development of the Qualified Applicant (QA) AI model at LinkedIn, designed to enhance the job matching process by predicting the likelihood of positive recruiter actions b...

This article discusses the integration of batch and stream processing to create a near real-time dashboard for Recruiter usage statistics at LinkedIn.

The article highlights the top ten engineering blogs from LinkedIn in 2019, focusing on popular topics such as open source, artificial intelligence, and technical challenges at scale.

The article discusses how LinkedIn customizes Apache Kafka to handle an impressive scale of 7 trillion messages per day.

The article discusses the use of virtual private clusters for testing Apache Samza at LinkedIn, emphasizing the importance of stability and rigorous testing in managed stream processing services.

The article discusses how Apache Calcite can bridge the gap between offline and nearline computations in big data processing.

The article announces the release of Samza 1. 0, a distributed stream processing framework developed at LinkedIn, highlighting its significant features and improvements.

The article discusses the development of Samza Aeon, a tool designed to measure latency in asynchronous one-way flows within systems.

The article 'Getting to Know Todd Palino' features insights from Todd Palino, a Senior Staff Site Reliability Engineer at LinkedIn, highlighting his work with Apache Kafka and his contributions to ...

The article discusses the evolution of incremental data capture for Oracle databases at LinkedIn, highlighting the transition from a batch processing model to a near-real-time system.

The article discusses the importance of site speed monitoring during A/B testing and feature ramp-up at LinkedIn.

The article discusses the evolution of testing strategies for backend services at LinkedIn, specifically focusing on the transition from UI-driven tests to a more efficient testing framework using ...

This article discusses the challenges of data access in high-scale stream processing, particularly focusing on the read/write and read-only data access patterns.

The article discusses the open sourcing of Kafka Monitor, a framework designed to monitor and test Kafka deployments.

The article 'Kafkaesque Days at LinkedIn – Part 1' discusses the challenges and incidents faced by LinkedIn while using Apache Kafka for data pipelines and messaging.

The article discusses the Kafka ecosystem at LinkedIn, detailing its critical role as a messaging system and the various solutions developed to enhance its functionality.

This article features a Q&A with Jim Brikman discussing the challenges and strategies for splitting a codebase into microservices and artifacts.

The article discusses Gobblin, a unified data ingestion framework developed by LinkedIn, designed to bridge batch and streaming data ingestion.

The article discusses the development of Venice, a derived data serving platform designed to improve the handling of derived data at LinkedIn.

The article discusses Burrow, an open-source tool developed by LinkedIn for monitoring Kafka consumers.

The article discusses the implementation of multi-tier architectures using Apache Kafka at scale, highlighting key concepts such as Tiered Cluster Architecture, Kafka Mirror Maker, and performance ...

The article 'Running Kafka At Scale' discusses how LinkedIn utilizes Apache Kafka as a crucial messaging system for handling vast amounts of data.

The article discusses a series of technical talks focused on data science, featuring insights from various experts in the field.

The article discusses the graduation of Apache Samza from the Apache Incubator to a top-level Apache project, highlighting its significance in stream processing and the community growth during its ...

Apache Samza is LinkedIn's stream processing engine designed to handle real-time data processing needs.

The article provides a comprehensive guide to Web Components, detailing their significance in encapsulating code behind semantic tags.

The article discusses how LinkedIn operates Apache Samza at scale, focusing on its integration with Apache Kafka for processing high volumes of data.

The article discusses LinkedIn's Technical Talk focused on content relevance, highlighting the engineering challenges and solutions in recommending content to users.

The article announces the release of Voldemort 1. 9. 0, highlighting new features and enhancements aimed at improving the operability of Voldemort clusters.

The article discusses how LinkedIn utilizes Apache Samza to gain real-time insights into its performance by processing data from numerous services and machines.

The article discusses the annual Apache Kafka meetup hosted by LinkedIn during the Hadoop Summit, highlighting the significance of Kafka as a high throughput messaging system.

The article discusses benchmarking Apache Kafka's performance, achieving 2 million writes per second on a modest hardware setup.

Apache Samza is LinkedIn's open-source stream processing framework, designed to handle real-time data processing with fault tolerance and scalability.

The article discusses intra-cluster replication in Apache Kafka, highlighting its importance for increasing availability and durability within Kafka's messaging system.

The article announces the release of Apache Kafka 0. 7. 1, highlighting its new features and improvements over previous versions.

The article announces the first Apache release of Kafka, highlighting its features and significance in data handling.

Project Kafka, a distributed publish-subscribe messaging system, has reached version 0. 6, enhancing its capabilities for handling activity stream data at LinkedIn.