Overview
The article discusses a significant incident involving data loss in a Hadoop ecosystem at LinkedIn, highlighting the lessons learned from the event. It emphasizes the importance of thorough incident response and the challenges faced during recovery efforts.
What You'll Learn
1
How to establish a robust host life cycle management for Hadoop infrastructure
2
Why understanding network behaviors under load is critical for data recovery
3
How to prioritize data recovery based on data loss percentage and usage level
Prerequisites & Requirements
- Understanding of Hadoop infrastructure and data management principles
- Experience in incident response and data recovery processes(optional)
Key Questions Answered
What were the main causes of the Hadoop incident at LinkedIn?
The incident was caused by procedural gaps in the Hadoop infrastructure's host life cycle management, leading to the inadvertent reimaging of roughly 2% of machines across several racks. This failure occurred on a business-critical production cluster, resulting in data loss.
How did the data loss impact workflows at LinkedIn?
Despite only 0.02% of data being corrupted, the incident caused 10% of workflows to fail, particularly affecting revenue-impacting processes. This was due to the reliance on hot datasets that many workflows consumed, demonstrating the cascading effects of even minor data corruption.
What strategies were employed for data recovery during the incident?
The recovery strategy included prioritizing datasets based on the percentage of data lost, freshness, and usage level. Additionally, network resources were adjusted to speed up data recovery, and coordination with flow owners was critical to manage the recovery process effectively.
What were the key learnings from the Hadoop incident?
Key learnings included the need for improved host life cycle management, better understanding of network behaviors under load, and the importance of auditing workflows for disaster recovery protocols. These insights aim to enhance future incident response and data recovery efforts.
Key Statistics & Figures
Percentage of machines reimaged
2%
This figure represents the extent of the incident that led to data loss across the Hadoop infrastructure.
Percentage of data corrupted
0.02%
Despite the small percentage of data loss, it resulted in a significant failure rate of workflows.
Percentage of workflows that failed
10%
This failure rate highlights the impact of the data loss on business-critical processes.
Technologies & Tools
Backend
Hadoop
Used for managing large data sets and ensuring data redundancy through replication.
Workflow Management
Azkaban
Utilized for scheduling and managing workflows that were impacted by the data loss.
Key Actionable Insights
1Implement a robust host life cycle management system to prevent procedural gaps that can lead to incidents.This system is crucial for maintaining the integrity of the Hadoop infrastructure and ensuring that similar incidents do not occur in the future.
2Adjust network resources dynamically during recovery efforts to prioritize critical data transfers.By managing network bandwidth effectively, organizations can significantly reduce recovery times and minimize the impact of data loss on business operations.
3Establish clear communication channels among flow owners during incident recovery.Effective communication ensures that all stakeholders are aligned on recovery strategies, which is essential for minimizing downtime and operational disruptions.
Common Pitfalls
1
Assuming data recovery will be straightforward without considering the differences in data organization across clusters.
This can lead to underestimating the complexity of recovery efforts, as subtle differences in data layout can significantly complicate the identification and restoration of lost files.
Related Concepts
Data Management
Disaster Recovery
Incident Response
Hadoop Infrastructure