Overview
The article discusses how Dash0 transitioned to using ClickHouse as a core database technology for their observability platform, leveraging its efficiency and scalability to handle OpenTelemetry data. It details the decision-making process, technical implementations, and operational optimizations that support their observability features.
What You'll Learn
1
How to optimize ClickHouse for high-cardinality data storage
2
Why using AggregatingMergeTree improves data deduplication
3
How to implement a hybrid storage approach with AWS S3 and local storage
Prerequisites & Requirements
- Understanding of OpenTelemetry and observability concepts
- Familiarity with ClickHouse and its client libraries(optional)
- Experience with database schema design and optimization
Key Questions Answered
What are the key features of Dash0's observability platform?
Dash0's observability platform offers cross-signal correlation, service health monitoring, and simplified troubleshooting. These features allow engineers to seamlessly navigate between metrics, traces, and logs, enhancing incident resolution times and dashboard generation.
How does Dash0 utilize ClickHouse for data storage?
Dash0 uses ClickHouse as its primary storage solution for OpenTelemetry data, leveraging its columnar architecture to efficiently handle high-cardinality data. The implementation includes a hybrid storage approach combining local storage for recent data and AWS S3 for older data, optimizing performance and cost.
What are the benefits of using AggregatingMergeTree in ClickHouse?
Using AggregatingMergeTree allows Dash0 to continuously deduplicate data despite constant writes, significantly reducing the data volume processed during queries. This is particularly effective for metrics metadata, where attributes rarely change, thus improving query efficiency.
What storage configuration does Dash0 use for ClickHouse?
Dash0 employs a multi-tiered storage configuration that includes 1-2 days of local storage for frequently accessed data, AWS S3 for older data, and a local cache for query acceleration. This setup balances performance with cost efficiency, particularly for large customer accounts.
Key Statistics & Figures
Retention period for metrics data
over a year
Dash0 aims to preserve metrics data for over a year to facilitate seasonal trend analysis.
Storage configuration for data
1-2 days local storage, AWS S3 for older data
This configuration supports efficient querying and cost management as data volumes increase.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Database
Clickhouse
Used as the primary storage solution for OpenTelemetry data.
Cloud Storage
AWS S3
Used for storing older observability data.
Observability Framework
Opentelemetry
Provides the signals and data that Dash0 processes.
Key Actionable Insights
1Implement a hybrid storage strategy to balance performance and cost in observability platforms.By combining local storage for recent data with cloud storage solutions like AWS S3, you can optimize query performance while managing storage costs effectively, especially as data volumes grow.
2Utilize AggregatingMergeTree for tables with high-frequency writes to improve data deduplication.This approach is particularly beneficial for metrics metadata, where attributes remain consistent across data points, allowing for efficient storage and faster query responses.
3Design table schemas with careful consideration of ORDER BY clauses to enhance query performance.Strategically ordering by resource attributes and timestamps can significantly improve filtering and compression, leading to faster query execution times.
Common Pitfalls
1
Failing to optimize ORDER BY clauses in ClickHouse can lead to poor query performance.
Without careful consideration of how data is ordered, queries may process more records than necessary, leading to slower execution times and increased resource usage.
2
Neglecting to use AggregatingMergeTree for high-frequency writes can result in inefficient data handling.
If data is not deduplicated effectively, it can lead to increased storage costs and slower query performance, especially for metrics that share similar attributes.
Related Concepts
Opentelemetry
Clickhouse Performance Optimization
Data Deduplication Strategies
Hybrid Storage Solutions