Building and operating systems that serve billions of people can present unprecedented and complex engineering challenges. The second Systems @Scale event of 2019 was held in New York, where engine…
Overview
The article recaps the Systems @Scale 2019 event held in New York, focusing on observability in complex distributed systems. It features insights from various speakers on topics such as deep systems, performance analysis tools, and innovative monitoring solutions.
What You'll Learn
1
How to implement distributed tracing as the backbone for observability in deep systems
2
Why aggregate analysis of distributed traces can simplify performance troubleshooting
3
How to develop meaningful Service Level Indicators (SLIs) that reflect user journeys
Key Questions Answered
What are deep systems and how do they affect observability?
Deep systems are complex, distributed, and layered architectures that require specialized observability techniques. Traditional metrics-first approaches often fail as systems scale, making it crucial to adopt distributed tracing to effectively monitor and troubleshoot these systems.
How does Uber utilize distributed tracing for performance analysis?
Uber employs distributed tracing to understand microservices architectures, using data mining and intuitive visualizations to simplify complex traces. This approach helps identify root causes of outages and reduces the time to mitigation, enhancing overall system reliability.
What challenges does Datadog face in scaling observability data?
Datadog encounters conflicting challenges as systems scale, needing high write throughput for data collection while maintaining low-latency queries for real-time operations. This requires careful architectural trade-offs to handle the increasing load and ensure effective observability.
What is the significance of meaningful SLIs in performance monitoring?
Meaningful Service Level Indicators (SLIs) are essential for accurately reflecting user experiences and guiding error budgets and Service Level Objectives (SLOs). Developing SLIs that go beyond basic metrics can significantly enhance the effectiveness of performance monitoring strategies.
Technologies & Tools
Data Transport System
Scribe
Used widely across Facebook for flexible data transport and monitoring.
Real-time Monitoring Platform
Scuba
Facilitates real-time ingestion, processing, and querying of structured logs.
Monitoring Infrastructure
Monarch
Google's planet-scale monitoring system that addresses scalability challenges.
Key Actionable Insights
1Implement distributed tracing to enhance observability in complex systems.As systems grow in complexity, traditional monitoring techniques may fail. Adopting distributed tracing allows engineers to visualize and understand the flow of requests across services, making it easier to identify and resolve issues.
2Utilize aggregate analysis of traces to streamline performance troubleshooting.By analyzing traces in aggregate, teams can uncover patterns and insights without the need for extensive hypothesis testing, leading to faster identification of performance bottlenecks and system inefficiencies.
3Focus on developing SLIs that accurately reflect user journeys.Effective SLIs should capture the true user experience rather than relying on superficial metrics. This approach ensures that performance goals align with actual user needs, leading to better service quality.
Common Pitfalls
1
Relying solely on conventional metrics for observability can lead to blind spots in deep systems.
As systems evolve into deep systems, traditional metrics may not capture the complexity of interactions. It's crucial to adopt more sophisticated methods, such as distributed tracing, to ensure comprehensive observability.