•Amruth Sampath, Arnav Balyan, Nimesh Khandelwal, Sumit Singh, Parth Halani, Suprit Acharya•8 min read•advanced•
--
•View OriginalOverview
Uber's migration from Spark 2.4 to Spark 3.3 involved upgrading over 2 million Spark applications, utilizing innovative automation tools like Iron Dome. The article details the challenges faced during this transition and highlights the improvements in efficiency and cost savings achieved through this upgrade.
What You'll Learn
1
How to automate the migration of Spark applications using custom tools
2
Why upgrading to Spark 3.3 improves efficiency and security
3
How to implement shadow testing for Spark jobs
Prerequisites & Requirements
- Understanding of Apache Spark and its architecture
- Familiarity with Kubernetes and YARN for resource management(optional)
Key Questions Answered
What were the main motivations for Uber's migration to Spark 3.3?
Uber migrated to Spark 3.3 primarily for its out-of-the-box support for Kubernetes, improved efficiency through optimizations like adaptive query execution, enhanced security with vulnerability fixes, and to keep pace with the latest open-source contributions.
What challenges did Uber face during the Spark migration?
Uber faced challenges such as ensuring compatibility of existing applications, managing dependencies, and validating data across over 40,000 Spark applications without a staging environment, which complicated the migration process.
How did Uber automate the migration process for Spark jobs?
Uber utilized a tool called Iron Dome, which allowed for safe shadow testing of Spark jobs by intercepting Spark's Catalog interface and Hadoop's File Output Committer, ensuring that production paths were not affected during the testing phase.
What were the results of the migration to Spark 3.3?
The migration resulted in an 85% job migration rate within six months, with over 60% of jobs experiencing more than a 10% performance improvement, leading to substantial cost savings and increased developer productivity.
Key Statistics & Figures
Job migration rate
85%
This was achieved within six months due to automation efforts.
Performance improvement
Over 60%
More than 60% of jobs saw more than a 10% improvement in performance.
Resource usage reduction
50%
The migration resulted in a 50% reduction in runtime and resource usage overall.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Data Processing
Apache Spark
Used for data analytics and machine learning at Uber.
Orchestration
Kubernetes
Serves as a resource manager for Spark applications.
Resource Management
Yarn
Handles resource management alongside Kubernetes.
Programming Language
Python
Used for writing Spark applications.
Programming Language
Scala
Another language used for Spark applications.
Key Actionable Insights
1Implementing automation tools like Iron Dome can significantly streamline the migration process for large-scale applications.By using such tools, organizations can reduce manual effort and minimize risks associated with migration, ensuring a smoother transition.
2Conducting thorough data validation and shadow testing is crucial when migrating applications to a new version.This helps identify potential issues early and ensures that the new version behaves as expected without impacting production data.
3Staying updated with the latest open-source contributions can enhance application security and performance.Regularly upgrading to newer versions of frameworks like Spark can provide access to optimizations and security fixes that improve overall system efficiency.
Common Pitfalls
1
Neglecting to validate data during migration can lead to discrepancies in application behavior.
Without proper validation, organizations risk deploying applications that may not function as intended, leading to potential data integrity issues.
2
Failing to account for dependency compatibility can hinder the migration process.
If dependencies are not updated or compatible with the new version, it can result in application failures or degraded performance.
Related Concepts
Apache Spark Architecture
Kubernetes Resource Management
Data Validation Techniques
Automation In Software Migration