Overview
The article discusses Uber's approach to building a scalable observability and alerting ecosystem, highlighting the development of two key systems, uMonitor and Neris, which handle metrics-based and host-level alerting respectively. It emphasizes the importance of a robust alerting pipeline to maintain service reliability and mitigate issues promptly.
What You'll Learn
1
How to build a scalable alerting system using metrics and host-level checks
2
Why high cardinality is a challenge in alerting systems and how to address it
3
When to implement automated mitigation actions in response to alerts
Prerequisites & Requirements
- Understanding of microservices architecture and observability concepts
- Familiarity with metrics storage solutions like M3(optional)
Key Questions Answered
How does Uber handle alerting at scale?
Uber manages alerting at scale through two primary systems: uMonitor for metrics-based alerts and Neris for host-level checks. These systems are integrated into a common notification pipeline that ensures timely alerts and mitigations, addressing issues as they arise.
What is the role of Origami in Uber's alerting ecosystem?
Origami serves as the deduplication and notification engine for both uMonitor and Neris, helping to manage high cardinality by consolidating alerts and sending relevant notifications based on the criticality of issues detected across multiple services.
What challenges does high cardinality present in alerting systems?
High cardinality complicates alerting by generating numerous metrics that can overwhelm teams with notifications. Uber addresses this by using techniques like alert dependencies and the Origami system to consolidate alerts and reduce noise.
How does uMonitor improve alert management?
uMonitor enhances alert management by allowing teams to create alerts based on specific metrics, automate notifications, and implement mitigation actions, thus ensuring that alerts are relevant and actionable.
Key Statistics & Figures
Alert configurations in uMonitor
125,000
uMonitor checks 700 million data points over 1.4 million time series every second.
Checks run by Neris
1.5 million checks per minute
This is across the fleet of hosts in each data center.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Metrics Storage
M3
Used for storing metrics that uMonitor checks against.
Database
Cassandra
Maintains alert check state and stores alert configurations.
Key Actionable Insights
1Implement a metrics-based alerting system like uMonitor to enhance service reliability.By using a system that checks metrics against defined thresholds, teams can proactively identify and address issues before they escalate, improving overall service performance.
2Utilize a deduplication engine like Origami to manage high cardinality in alerts.This helps in consolidating alerts and reducing notification fatigue, allowing teams to focus on critical issues rather than being overwhelmed by noise.
3Incorporate automated mitigation actions in your alerting strategy.Automating responses to alerts can significantly reduce downtime and manual intervention, making your systems more resilient and responsive.
Common Pitfalls
1
Failing to manage high cardinality can lead to alert fatigue.
When too many alerts are generated, teams may ignore them, potentially missing critical issues. Implementing deduplication and consolidated notifications can help mitigate this problem.
Related Concepts
Microservices Architecture
Observability
Alerting Systems
Metrics Management