Overview
PinalyticsDB is Pinterest's proprietary time series database built on HBase, designed to store and visualize a large volume of time series reports. The article discusses the challenges faced in scalability and performance, and the improvements made in the new version of PinalyticsDB.
What You'll Learn
1
How to redesign a time series database schema to improve scalability
2
Why salting is important for distributing read and write loads in HBase
3
How to optimize coprocessor performance in HBase for better CPU utilization
Prerequisites & Requirements
- Understanding of HBase architecture and time series data management
- Experience with distributed systems and database optimization techniques(optional)
Key Questions Answered
What are the scalability challenges faced by PinalyticsDB?
PinalyticsDB faced scalability challenges due to hotspotting in HBase region servers, where certain reports received more traffic than others. This was exacerbated by the old schema design that created a new HBase table for every report, leading to uneven data distribution and performance issues.
How does the new row-key design improve performance?
The new row-key design incorporates a SALT value, report ID, metric ID, date-time, and segments, which helps distribute both reads and writes evenly across the HBase cluster. This design eliminates hotspotting by ensuring that popular metrics and recent data do not overload specific region servers.
What optimizations were made to the coprocessor in PinalyticsDB?
The coprocessor optimizations included modifying the request structure to reduce the number of HBase scans by coalescing aggregation requests. This resulted in improved CPU utilization, reduced RPC latency, and fewer blocked threads, enhancing overall performance.
What issues did the Thrift server face and how were they resolved?
The Thrift server experienced frequent out-of-memory (OOM) crashes due to large report metadata. This was mitigated by setting the JVM to exit on OOM and redistributing report metadata into smaller, manageable column families to prevent heap exhaustion.
Key Statistics & Figures
Improvement in Region Server CPU utilization
Significant improvements
After implementing the new coprocessor design and optimizations.
Reduction in RPC latency
Significant improvements
Following the changes made to the coprocessor request structure.
Report metadata size
60+ to 120+ MB
Some reports had metadata sizes that contributed to OOM issues.
Technologies & Tools
Database
Hbase
Used as the underlying storage system for PinalyticsDB.
Backend
Thrift
Used for communication between services in the PinalyticsDB architecture.
Key Actionable Insights
1Implement salting in your HBase schema design to avoid hotspotting issues.By incorporating a SALT value in the row-key, you can evenly distribute read and write operations across region servers, which is crucial for performance in high-traffic scenarios.
2Optimize coprocessor requests by coalescing similar requests to reduce the number of scans.This approach minimizes the load on HBase region servers and improves response times, making it essential for applications with high query volumes.
3Regularly monitor and manage report metadata size to avoid OOM issues in Thrift servers.Understanding the size and structure of report metadata can help prevent performance bottlenecks and ensure reliable service availability.
Common Pitfalls
1
Failing to account for the size of report metadata can lead to OOM crashes.
When report metadata is excessively large, it can overwhelm the JVM heap, causing performance issues and service downtime. It's crucial to manage metadata size effectively.
2
Not implementing salting can result in uneven load distribution.
Without salting, popular metrics can lead to hotspotting, which negatively impacts performance. This can be avoided by designing the schema to include a SALT value.
Related Concepts
Time Series Databases
Hbase Optimization Techniques
Distributed System Design Patterns