Overview
The article discusses Autometrics, a self-service metrics collection system developed at LinkedIn to streamline the process of metrics collection and visualization. It highlights the challenges faced with traditional monitoring systems and outlines the design and implementation of a more efficient, push-based metrics collection framework.
What You'll Learn
1
How to implement a push-based metrics collection system
2
Why self-service mechanisms improve metrics collection efficiency
3
When to use Apache Kafka for metrics transport
Prerequisites & Requirements
- Understanding of metrics collection and monitoring systems
- Familiarity with Apache Kafka and Avro(optional)
- Experience with Python programming
Key Questions Answered
What were the initial challenges with LinkedIn's monitoring system?
The initial monitoring system at LinkedIn faced several challenges including a clunky interface, slow metric addition processes, and inconsistent metric naming. These issues resulted in delays for developers and SREs, making it clear that the existing system would not scale effectively.
How does Autometrics improve metrics collection at LinkedIn?
Autometrics enhances metrics collection by implementing a push-based system that automatically collects standard metrics from services and allows users to tag custom metrics for automatic collection. This reduces manual effort and ensures consistency across metrics.
What technologies were used to build the Autometrics system?
The Autometrics system utilizes Apache Kafka for metrics transport, Python for the collector implementation, and RRDTool for time series data storage. This combination allows for efficient data handling and scalability in metrics collection.
What are the statistics on metrics collection in production?
In production, Autometrics collects over 500k metrics every minute, averaging about 8800 metrics per second. Each service maintains an average of 400 metrics, with some services having thousands, demonstrating the system's scalability.
Key Statistics & Figures
Metrics collected per minute
500k+
This is the rate at which metrics are collected in a production data center.
Average number of metrics per service
400
This average indicates the scale of metrics being managed across services.
Disk space used for RRD files
870G
This is the amount of disk space utilized for storing RRD files in the system.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Apache Kafka
Used for transporting metrics data in a scalable manner.
Backend
Python
Used for implementing the metrics collector.
Data Serialization
Avro
Used for serializing metrics data before sending it to the event bus.
Database
Rrdtool
Used for storing time series data collected from metrics.
Key Actionable Insights
1Implement a self-service metrics tagging system to empower developers.By allowing developers to tag their own metrics, organizations can reduce bottlenecks in the metrics collection process and encourage a culture of data-driven decision-making.
2Adopt a push-based metrics collection approach for real-time data.Transitioning from a poll-based to a push-based system can significantly improve the responsiveness and accuracy of metrics, allowing teams to react quickly to system changes.
3Utilize Apache Kafka for scalable metrics transport.Kafka's ability to handle large volumes of data efficiently makes it an ideal choice for transporting metrics, ensuring that the system can grow alongside increasing data demands.
Common Pitfalls
1
Failing to standardize metric naming conventions can lead to confusion and data inconsistency.
Without a clear naming strategy, different teams may use varied terms for similar metrics, making it difficult to aggregate and analyze data effectively.
Related Concepts
Metrics Collection Strategies
Self-service Tools In Software Engineering
Scalability In Monitoring Systems