Overview
This article details the development of a ClickHouse-powered logging platform, named LogHouse, which efficiently manages over 19 PiB of log data while significantly reducing costs compared to traditional observability solutions like Datadog. It highlights the architectural decisions, performance metrics, and the advantages of using ClickHouse for large-scale log storage and analysis.
What You'll Learn
1
How to build a scalable logging platform using ClickHouse
2
Why ClickHouse is a cost-effective alternative to Datadog for log management
3
How to implement OpenTelemetry for log ingestion
Prerequisites & Requirements
- Familiarity with SQL and observability concepts
- Experience with ClickHouse and OpenTelemetry(optional)
Key Questions Answered
How does ClickHouse compare to Datadog in terms of cost for log management?
ClickHouse is at least 200x less expensive than Datadog for similar workloads. For instance, Datadog's projected cost for a 30-day retention period could reach $26 million, while ClickHouse's cost for managing 19 PiB of logs is approximately $150,000 per month.
What architectural decisions were made in building the LogHouse platform?
Key architectural decisions included avoiding cross-region data transfer to minimize costs, opting out of using Kafka for message buffering due to ClickHouse's high ingestion speed, and implementing structured logging in JSON format to enhance query performance.
What is the significance of using OpenTelemetry in this logging solution?
OpenTelemetry was chosen for log ingestion due to its growing community adoption and the ability to expand LogHouse to collect metrics and traces in the future. The decision was based on its potential to become a standard for observability data collection.
Key Statistics & Figures
Total log data stored
19 PiB uncompressed, 1.13 PiB compressed
This data volume is managed across AWS regions for ClickHouse Cloud.
Cost of ClickHouse infrastructure
$125,000 per month
This cost includes handling 5.4 PiB throughput per month.
Cost per TiB uncompressed
$23.76
This cost reflects the expenses associated with ClickHouse's infrastructure and S3 storage.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Database
Clickhouse
Used for storing and querying large volumes of log data.
Observability
Opentelemetry
Utilized for log ingestion and future expansion into metrics and traces.
Frontend
Grafana
Used for visualizing log data and enhancing user experience.
Key Actionable Insights
1Consider implementing ClickHouse for your logging needs to significantly reduce costs and improve scalability.With ClickHouse's ability to compress data effectively and handle high ingestion rates, it can be a game-changer for organizations facing high observability costs.
2Leverage OpenTelemetry for a standardized approach to log collection and observability.Using OpenTelemetry not only simplifies the ingestion process but also prepares your architecture for future expansions into metrics and tracing.
3Optimize your logging schema by using structured logging to enhance query performance.Structured logging allows for more efficient data retrieval and analysis, which is crucial when dealing with large volumes of log data.
Common Pitfalls
1
Relying on unstructured logging can lead to poor query performance.
Unstructured logs require linear scans during queries, which can be inefficient. Transitioning to structured logging improves performance significantly.
2
Overcomplicating the architecture by introducing unnecessary components like Kafka.
Kafka can add complexity and cost without significant benefits when ClickHouse can handle high ingestion rates directly.
Related Concepts
Observability Best Practices
Cost-effective Logging Solutions
Structured Logging Techniques