Overview
The article discusses how LinkedIn reduced the upload of Apache Spark application dependencies by 99% through the implementation of a user-level caching mechanism. It highlights the challenges faced with dependency management and the significant performance improvements achieved by optimizing the dependency upload process.
What You'll Learn
1
How to implement a user-level cache for Apache Spark dependencies
2
Why reducing dependency upload time is crucial for Spark application performance
3
When to consider user-level versus cluster-level caching solutions
Prerequisites & Requirements
- Understanding of Apache Spark and its dependency management
- Familiarity with Hadoop and HDFS(optional)
Key Questions Answered
How did LinkedIn reduce Apache Spark application dependency uploads?
LinkedIn implemented a user-level caching mechanism that significantly reduced the number of repeated uploads of dependencies. This approach allowed for a 99.7% reduction in uploaded dependencies, improving application runtime and efficiency.
What percentage of runtime was previously spent on uploading dependencies?
On average, 20% of the runtime for Spark applications was spent on uploading dependencies, with some short-running jobs experiencing up to 80% of their runtime on this task.
What were the observed impacts of the caching feature on Spark applications?
The caching feature led to a median Spark job runtime reduction from 9.4 minutes to 5.5 minutes, representing a 40% decrease, which allowed for more frequent updates of anti-abuse models and improved overall performance.
What issues were encountered during the rollout of the caching feature?
During the initial rollout, some testing flows failed due to invalid paths in the spark.jars configuration, necessitating a rollback. Subsequent rollouts included extensive pattern analysis and gradual ramp-up to minimize risks.
Key Statistics & Figures
Reduction in uploaded dependencies
99.7%
Achieved through the implementation of a user-level cache.
Median Spark job runtime reduction
40%
Decreased from 9.4 minutes to 5.5 minutes after caching implementation.
Daily HDFS write operations reduction
25 million
This constituted approximately 10% of total HDFS write operations.
Time allocated to uploading JARs before optimization
2,000 hours per day
This time was reduced by a factor of 10 with the new caching feature.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Apache Spark
Primary compute engine for offline data analytics at LinkedIn.
Backend
Apache Hadoop Yarn
Used for managing Spark applications and their dependencies.
Storage
Apache Hadoop Distributed File System
Storage system for Spark application dependencies.
Orchestration
Azkaban
Used for scheduling batch workflows, including Spark jobs.
Key Actionable Insights
1Implement a user-level caching strategy to minimize redundant uploads of dependencies.This approach can significantly reduce the overhead associated with dependency management, especially in environments with high volumes of Spark applications.
2Conduct thorough testing and analysis before rolling out new features in production environments.Identifying potential issues early can prevent major disruptions and ensure a smoother transition when implementing new optimizations.
3Monitor and analyze dependency usage patterns to identify opportunities for optimization.Understanding how dependencies are used can inform decisions about caching strategies and help reduce unnecessary uploads.
Common Pitfalls
1
Failing to validate dependency paths can lead to application failures during rollout.
This occurs when users configure spark.jars incorrectly, which can disrupt workflows and necessitate rollbacks.
Related Concepts
Caching Strategies In Distributed Systems
Dependency Management In Apache Spark
Performance Optimization Techniques For Big Data Applications