Is it over for metrics?

Is it over for metrics in modern observability? As high-cardinality events and fast columnar rollups reshape how engineers investigate production issues, metrics may be shifting from the center of the stack to a simple optimization layer over richer data.

ClickHouse

•7 min read•advanced•

--

•View Original

KubernetesPrometheus

Overview

The article discusses the evolving role of metrics in observability, emphasizing that while they remain important, their function is shifting towards being an optimization layer rather than the core of monitoring systems. It highlights the limitations of traditional metrics and the growing importance of structured events and traces in modern observability practices.

What You'll Learn

1

How to effectively use metrics in observability while recognizing their limitations

2

Why structured events and traces are becoming more critical than traditional metrics

3

When to transition from metrics to a more flexible observability approach

Key Questions Answered

What are the limitations of using metrics in observability?

Metrics systems often require discarding high-cardinality dimensions, making it difficult to debug specific cases. This leads to a reliance on logs and traces for detailed investigations, as metrics alone may not provide sufficient context for understanding issues.

How are structured events changing observability practices?

Structured events allow for high-cardinality context and can be efficiently stored in columnar databases. This shift enables users to capture both logs and metrics in a single event, facilitating better analysis and issue resolution without predefined metrics.

Why is there a shift from metrics to structured events in observability?

As the limitations of metrics become evident, structured events provide richer context and allow for more flexible investigations. This evolution reflects the need for observability tools that adapt to unforeseen issues rather than relying solely on predefined metrics.

What role do LLMs play in modern observability workflows?

LLMs can assist in exploring raw data and identifying patterns, enabling automatic suggestions for useful aggregations. This capability allows teams to focus on investigating issues without needing to define metrics upfront, enhancing flexibility in observability.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Database

Clickhouse

Used for storing structured events and enabling efficient rollups of observability data.

Monitoring

Prometheus

Serves as an example of a metrics system that captures pre-aggregated statistics.

Key Actionable Insights

1
Transitioning to a more flexible observability model can enhance your team's ability to diagnose issues effectively.
By leveraging structured events and traces, teams can avoid the pitfalls of traditional metrics, allowing for more comprehensive investigations when issues arise.

2
Incorporate LLMs into your observability workflows to automate data exploration and pattern recognition.
This integration can streamline the process of identifying important metrics and dimensions, reducing the need for upfront decisions during development.

3
Consider the cost and complexity of metrics systems when designing your observability stack.
Understanding the limitations of metrics can guide you in choosing the right tools and approaches for effective monitoring and issue resolution.

Common Pitfalls

1

Relying solely on metrics can lead to incomplete understanding of system health.

Metrics often lack the granularity needed for detailed investigations, necessitating a fallback to logs and traces, which can complicate troubleshooting efforts.

2

Defining metrics too early in the development process can limit flexibility.

When teams commit to specific metrics upfront, they may miss important signals that only become apparent after issues arise, reducing the effectiveness of their observability strategy.

Related Concepts

Observability

Metrics Vs. Logs

Structured Events

Traces