How LinkedIn Uses Kubernetes

The article discusses LinkedIn's modern Public Key Infrastructure (PKI) that integrates hardware-based security measures to ensure strong, verifiable identities for workloads.

The article discusses JUDE, LinkedIn's platform for generating high-quality embeddings for job recommendations using fine-tuned Large Language Models (LLMs).

The article discusses the development of domain-adapted foundation GenAI models at LinkedIn, focusing on their application within the Economic Opportunity Network (EON) project.

The article discusses Liger-Kernel, an open-source library designed to enhance GPU efficiency for training large language models (LLMs).

The article discusses LinkedIn's implementation of a Stateful Workload Operator for managing stateful systems on Kubernetes.

The article discusses the open sourcing of OpenHouse, a control plane designed for managing tables in a data lakehouse.

The article discusses the open sourcing of FlyteInteractive, a tool developed by LinkedIn to enhance machine learning (ML) development productivity.

The article discusses a hybrid bulk data processing framework developed to improve recruiting efficiency during data ownership transfers, particularly in the context of company mergers and recruite...

The article introduces OpenHouse, a control plane developed at LinkedIn for managing tables in open source data lakehouse deployments.

The article discusses the development of Hoptimator, a declarative data pipeline orchestrator designed to streamline the creation of end-to-end data pipelines at LinkedIn.

The article discusses the challenges and solutions related to operating system upgrades at LinkedIn's scale, focusing on the development of the Operating System Upgrade Automation (OSUA) system.

The article discusses DARWIN, LinkedIn's unified Data Science and Artificial Intelligence Workbench, designed to streamline the workflows of data scientists and AI engineers by centralizing various...

The article discusses LinkedIn's Remote Development initiative, which leverages cloud computing to enhance developer productivity by drastically reducing setup and build times.

Project Magnet introduces push-based shuffle in Apache Spark 3. 2, enhancing shuffle scalability and reliability.

This article discusses the challenges and solutions LinkedIn faced while scaling its Hadoop YARN cluster beyond 10,000 nodes.

The article discusses TonY's integration into the LF AI & Data Foundation, highlighting its role in facilitating distributed deep learning on Hadoop.

The article discusses the open-sourcing of Kube2Hadoop, a solution designed to provide secure access to HDFS from Kubernetes.

Apache Pinot 0. 3. 0 is an open-source, distributed OLAP data store developed at LinkedIn, designed for near-real-time analytics.

The article discusses LinkedIn's open sourcing of DataHub, a metadata search and discovery platform, detailing its development journey from WhereHows to DataHub.

The article discusses the 'skills genome methodology' developed by LinkedIn to identify unique skills associated with emerging jobs, which are rapidly growing but may lack a large workforce.

The article discusses the use of virtual private clusters for testing Apache Samza at LinkedIn, emphasizing the importance of stability and rigorous testing in managed stream processing services.

The article announces the release of Samza 1. 0, a distributed stream processing framework developed at LinkedIn, highlighting its significant features and improvements.

The article discusses Brooklin, a data ingestion service developed by LinkedIn to facilitate streaming data from various sources to multiple destinations.

The article discusses Gobblin, a unified data ingestion framework developed by LinkedIn, designed to bridge batch and streaming data ingestion.

The article discusses LinkedIn's participation in the Velocity conference in Santa Clara in 2014, highlighting the relevance of the event for Site Reliability Engineering and performance improvemen...

The article discusses the engineering behind LinkedIn Connected, a new iOS app that utilizes anticipatory computing to enhance user networking experiences.