Visit the post for more.
Overview
The article discusses the development and implementation of Corona, a new scheduling framework at Facebook designed to improve the efficiency of MapReduce job scheduling. It highlights the limitations of the Hadoop MapReduce framework and how Corona addresses these challenges through better scalability, lower latency, and improved resource management.
What You'll Learn
1
How to implement a push-based scheduling system for job tracking
2
Why separating resource management from job coordination improves scalability
3
How to achieve no-downtime upgrades in a scheduling framework
4
When to use fair-share scheduling for multi-tenant environments
Prerequisites & Requirements
- Understanding of MapReduce and scheduling frameworks
- Experience with large-scale data processing systems(optional)
Key Questions Answered
What are the limitations of the Hadoop MapReduce scheduling framework?
The Hadoop MapReduce scheduling framework faces several limitations including scalability issues due to the job tracker managing both cluster resources and job scheduling, a pull-based scheduling model causing delays, and a static slot-based resource management model that leads to wasted resources. Additionally, it requires hard downtime during software upgrades.
How does Corona improve upon the limitations of Hadoop MapReduce?
Corona improves upon Hadoop MapReduce by introducing a dedicated cluster manager for resource tracking and separate job trackers for each job, allowing for push-based scheduling. This reduces scheduling latency, increases cluster utilization, and enables no-downtime upgrades, ultimately leading to better performance and resource management.
What deployment strategy was used for Corona at Facebook?
The deployment of Corona was staged in three phases: first, rolling out to 500 nodes for feedback; second, handling non-critical workloads to monitor performance; and finally, transitioning all mission-critical workloads to Corona, reducing the old MapReduce cluster to 60 nodes. This phased approach minimized disruption during the transition.
What benefits has Corona provided to Facebook's data infrastructure?
Corona has led to significant improvements in scalability, lower latency for job processing, no-downtime upgrades, and better resource management. Key metrics showed a reduction in average time to refill slots from 66 seconds with MapReduce to 55 seconds with Corona, and cluster utilization improved from 70% to over 95%.
Key Statistics & Figures
Average time to refill slot
55 seconds
This is an improvement from 66 seconds in the Hadoop MapReduce system.
Cluster utilization
over 95%
This is an increase from a maximum of 70% in the Hadoop MapReduce system during heavy workloads.
Job latency for a test job
25 seconds
This is a reduction from 50 seconds in the previous system.
Unfairness in resource allocation
3.6%
This represents a drop from an average of 14.3% in the Hadoop MapReduce system.
Technologies & Tools
Backend
Mapreduce
Initially used as the foundation for Facebook's data infrastructure before transitioning to Corona.
Backend
Corona
The new scheduling framework developed to improve job scheduling efficiency.
Data Processing
Hive
Used for executing queries that were transitioned from the old MapReduce system to Corona.
Key Actionable Insights
1Implementing a push-based scheduling system can significantly reduce latency in job processing.By moving away from a pull-based model, organizations can minimize delays in task scheduling, which is crucial for environments with high job turnover and small job sizes.
2Separating job coordination from resource management can enhance scalability and efficiency.This separation allows for more focused resource allocation and management, which is particularly beneficial in large-scale data processing environments.
3Adopting fair-share scheduling can improve resource allocation fairness in multi-tenant systems.This approach ensures that all teams have equitable access to resources, reducing conflicts and improving overall system performance.
4Staged deployment strategies can mitigate risks during major system transitions.By gradually moving workloads to a new system, organizations can identify and address issues without disrupting ongoing operations.
Common Pitfalls
1
Failing to account for the limitations of the existing scheduling framework can lead to performance bottlenecks.
Organizations may overlook the need for a more scalable solution until they experience significant slowdowns, which can disrupt operations and hinder growth.
Related Concepts
Mapreduce Scheduling Frameworks
Cluster Resource Management
Fair-share Scheduling
Push-based Vs Pull-based Scheduling