#
Apache Spark Programming Tutorials & Engineering Articles
265 Apache Spark tutorials, guides, and engineering insights from Uber, NVIDIA, LinkedIn, and more
Companies Using This
Apache Spark Articles & Tutorials
Filter:
Connect ClickHouse to Amazon S3 Tables to query Iceberg data in place, transform retail analytics, and write reusable results back for dashboards and AI agents.
12 min read
Includes Code
--
Hitarth Trivedi, Jay Narale, Jianjian Xie, Aerial view of a busy multi-level highway interchange with cars and bright streetlights at night., Yunfei Chen, Sanah Imani
15 min read
--
Xinli Shang, Sivabalan Narayanan, Dharshan Tharumia, Sagar Sumit, Meenal Binwade, Vamshi Pasunuru, Kanan Patel, Suprit Acharya, Kumudini Kakwani, Arnav Balyan, Nimesh Khandelwal
12 min read
--
Pinterest Engineering
11 min read
Includes Code
--
Wesley Spencer See, Andrew Picard, Michelle Zhong, Dan McQuillan, Jianfeng Guo
16 min read
--
Aravind Patnam
16 min read
--
Enterprise applications have become increasingly demanding. Agentic AI has compounded the need for vast amounts of enterprise data with frequent…
Elizabeth Goodman
7 min read
Includes Code
--
Netflix Technology Blog
10 min read
--
Netflix Technology Blog
11 min read
Includes Code
--
For years, a Python developer who needed a GPU had two realistic choices: Learn NVIDIA CUDA C++ well enough to write an extension, set up a build toolchain…
Elizabeth Goodman
12 min read
Includes Code
--
Aravind Selvan, Calder Lund, and Gaurang Sadekar
13 min read
Includes Code
--
Prakhar Agarwal, Avinash Varma Sagi, Abhay Singh Chauhan, Kranthi Reddy, Divya Rai
11 min read
--
Pinterest Engineering
13 min read
--
Some microservices are difficult to test because their behavior depends on a long tail of inputs that are hard to model by hand. This post focuses on how to build a repeatable Apache Spark replay harness for testing without creating a second implementation of the service.
Vivek Yadav
8 min read
Includes Code
--
Some microservices are difficult to test because their behavior depends on a long tail of inputs that are hard to model by hand. For that class of problem, Apache Spark's massive parallelism and linear scalability can be leveraged to build a regression test harness. That makes it possible to compare old and new implementations, model upcoming rule changes, and quantify impact before a change reaches production.
Vivek Yadav
9 min read
--
13 min read
--
Justin Lee, Adam Hudson
10 min read
--
The article discusses Notion's journey in scaling its vector search infrastructure, achieving a 10x increase in scale while reducing costs by 90% over two years.
Preeti Gondi, Mickey Liu, Nathan Louie, Calder Lund, Jacob Sager
10 min read
Has Summary
--
Pinterest Engineering
15 min read
Includes Code
--
The article discusses Pinterest's transition to a next-generation database ingestion framework designed to address the limitations of legacy systems.
Pinterest Engineering
10 min read
Includes Code
Has Summary
--
This article details how Uber optimized Apache Hadoop's Distcp (Distributed Copy) tool to scale their data replication infrastructure from handling 250 TB to petabytes of daily data movement.
Abhay Yadav, Radhika Patwari, Sanjay Sundaresan
15 min read
Has Summary
--
This article details how Uber built and scaled Apache Hudi to power one of the world's largest data lakes, managing 19,500 datasets with trillions of records across a multi-hundred-petabyte reposit...
Prashant Wason, Balajee Nagasubramaniam, Surya Prasanna Kumar Yalla, Meenal Binwade, Xinli Shang, Jack Song
19 min read
Has Summary
--
PinLanding is a multimodal AI pipeline developed by Pinterest to generate shopping collections from billions of products.
Pinterest Engineering
8 min read
Has Summary
--
The article discusses Uber's transition from traditional keyword-based search using Apache Lucene to implementing semantic vector search with Amazon OpenSearch.
Hao Sun, Jiasen Xu, Smit Patel, Anand Kotriwal, Xu Zhang
11 min read
Has Summary
--
The article discusses Project Aether, a tool developed by NVIDIA to facilitate the migration of CPU-based Apache Spark workloads to GPU-accelerated environments on Amazon EMR.
Navin Kumar
6 min read
Includes Code
Has Summary
--
This article discusses how Uber utilizes a pull-based ingestion model in OpenSearch™ to effectively index streaming data.
Yupeng Fu, Varun Bharadwaj, Shuyi Zhang, Xu Xiong, Michael Froh
14 min read
Has Summary
--
This article discusses Uber's transition from batch to streaming data ingestion using Apache Flink, which significantly enhances data freshness and operational efficiency.
Xinli Shang, Peter Huang, Jing Li, Jing Zhao, Jack Song
6 min read
Has Summary
--
The article discusses Uber's implementation of I/O observability for its massive petabyte-scale data lake, focusing on the challenges and solutions in monitoring data access patterns across its hyb...
Arnav Balyan, Kartik Bommepally, Amruth Sampath, Jing Zhao, Akshayaprakash Sharma
10 min read
Has Summary
--
The article discusses the collaboration between IBM and NVIDIA to enhance large-scale data analytics through GPU-native Velox and NVIDIA cuDF, highlighting significant performance improvements over...
Gregory Kimball
7 min read
Has Summary
--
Uber's migration from Spark 2. 4 to Spark 3. 3 involved upgrading over 2 million Spark applications, utilizing innovative automation tools like Iron Dome.
Amruth Sampath, Arnav Balyan, Nimesh Khandelwal, Sumit Singh, Parth Halani, Suprit Acharya
8 min read
Has Summary
--
The article discusses how Netflix scales its Muse application to provide data-driven creative insights at a massive scale, focusing on the architectural evolution and optimizations made to handle t...
Netflix Technology Blog
10 min read
Includes Code
Has Summary
--
The article discusses the development of a real-time streaming analytics system for Stripe Billing, enabling customers to access subscription metrics with minimal latency.
Reed Trevelyan
8 min read
Has Summary
--
This article discusses the architecture and implementation of Uber's HiveSync, a critical service for data replication across its massive data lake.
Radhika Patwari, Trivedhi Talakola, Rajan Jaiswal, Chayanika Bhandary, Mukesh Verma, Sanjay Sundaresan
14 min read
Has Summary
--
The article discusses the critical importance of maintaining consistent data across multiple systems as organizations grow.
James Beswick
11 min read
Has Summary
--
This article discusses the development and implementation of forecasting models aimed at improving driver availability at airports, which are critical to Uber's ridesharing ecosystem.
Bob Zheng, Dhruv Ghulati, Manoj Panikkar, Michael (Yichuan) Cai
15 min read
Has Summary
--
The article discusses the deployment of a serverless, distributed data processing architecture using Apache Spark and NVIDIA AI on Azure.
Alexander Spiridonov
9 min read
Includes Code
Has Summary
--
The article discusses the evolution of Uber's Search Platform, highlighting its transition from Elasticsearch to an in-house solution called Sia, and ultimately to the adoption of OpenSearch.
Yupeng Fu, Shubham Gupta, Shanshan Song, Mingmin Chen
15 min read
Has Summary
--
This article details Uber's migration from Apache Hive to Apache Spark SQL for ETL workloads, highlighting the motivations behind the transition, the architecture involved, and the challenges faced...
Kumudini Kakwani, Akshayaprakash Sharma, Nimesh Khandelwal, Aayush Chaturvedi, Chintan Betrabet, Suprit Acharya
14 min read
Has Summary
--
The article discusses Uber's implementation of a configuration-driven archival and retrieval framework designed to manage vast amounts of regulatory data efficiently.
The article discusses the application of Graph Neural Networks (GNNs) in enhancing fraud detection within financial services.
Naim
10 min read
Includes Code
Has Summary
--
The article discusses how Atgenomix SeqsLab leverages NVIDIA technologies to enhance health omics analysis for precision medicine.
Yu-Ting Lin
9 min read
Has Summary
--
The article discusses how Pinterest enhances its machine learning feature iterations through an effective backfill process.
Pinterest Engineering
14 min read
Has Summary
--
The article discusses the use of GPU acceleration to enhance performance in Apache Spark applications, highlighting the challenges of migrating workloads from CPUs to GPUs.
Matt Ahrens
9 min read
Includes Code
Has Summary
--
The article discusses how to accelerate Deep Learning (DL) and Large Language Model (LLM) inference using Apache Spark in cloud environments.
ApacheApache SparkAWSAzureDeep LearningDockerJSONNumPyPythonPyTorchSemantic SearchTensorFlowTransformers
Rishi Chandra
9 min read
Includes Code
Has Summary
--
The article discusses Uber's migration of large-scale interactive compute workloads from Peloton to Kubernetes, focusing on minimizing disruption while enhancing resource management and cloud readi...
Sayan Pal, Rishabh Mishra
12 min read
Has Summary
--
This article discusses Uber's implementation of elastic resource management on Kubernetes, focusing on enhancements made to support Ray-based job management.
Bharat Joshi, Anant Vyas, Ben Wang, Axansh Sheth, Abhinav Dixit
10 min read
Has Summary
--
The article discusses how to accelerate Apache Parquet scans on Apache Spark using GPUs, specifically through the RAPIDS Accelerator for Apache Spark.
Matt Ahrens
7 min read
Includes Code
Has Summary
--
Uber's blog post discusses their migration of machine learning workloads to Kubernetes using Ray, detailing the challenges faced with their previous setup and the improvements achieved with the new...
Bharat Joshi, Anant Vyas, Ben Wang, Min Cai, Axansh Sheth, Abhinav Dixit
18 min read
Has Summary
--