Overview
This article discusses how Pinterest scaled its ad corpus by 60 times through a partnership with Shopify and the implementation of a key-value store and memory optimizations in Go. The changes aimed to simplify infrastructure costs, improve system performance, and reduce maintenance overhead.
What You'll Learn
1
How to scale an ad-serving architecture using a key-value store
2
Why reducing the number of heap objects can improve system performance
3
How to implement real-time data updates in a distributed system
4
How to minimize latency in ad-serving systems through parallelization
Prerequisites & Requirements
- Understanding of distributed systems and caching concepts
- Familiarity with Go programming language(optional)
Key Questions Answered
How did Pinterest scale its ad corpus by 60x?
Pinterest scaled its ad corpus by leveraging a key-value store and memory optimizations in Go, which allowed for a significant increase in the number of shopping ads available for its recommendation engine. This approach simplified infrastructure management while maintaining performance.
What challenges did Pinterest face with its ad-serving architecture?
Pinterest encountered memory bottlenecks as the ad corpus grew, causing the candidate funnel service to hit memory limits. Additionally, garbage collection CPU pressure led to latency spikes, impacting ad delivery and performance.
What solutions were implemented to address garbage collection issues?
To mitigate garbage collection CPU pressure, Pinterest reduced the number of heap objects from approximately 280 million to around 60 million. This was achieved by cleaning up unused data fields and pooling objects, leading to improved system performance.
How did Pinterest improve data freshness in its ad-serving system?
Pinterest enabled real-time updates for its key-value store, reducing the time for data updates from hours to minutes. This allowed for more accurate ad delivery and reduced the number of missing candidates in the ad index.
Key Statistics & Figures
Increase in ad corpus size
60x
Achieved through the partnership with Shopify and architectural changes.
Reduction in startup time
< 2 minutes
Startup time decreased from 10 minutes after removing the in-memory index.
Reduction in heap objects
from ~280 million to ~60 million
This optimization led to improved system performance.
Reduction in latency
10%
Achieved by reducing the number of heap objects.
Reduction in QPS to KV Store
~94%
Implemented local caching to minimize traffic to the key-value store.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Programming Language
Go
Used for implementing memory optimizations and scaling the ad-serving architecture.
Data Streaming
Kafka
Used for real-time data updates in the key-value store.
Database
Key-value Store
External storage solution for ad indexing.
Key Actionable Insights
1Consider migrating from in-memory indexing to an external key-value store to simplify your architecture and reduce maintenance costs.This approach can help manage larger datasets without the complexity of sharding and can lead to improved system performance and reduced latency.
2Optimize your garbage collection strategy by reducing the number of long-lived objects and pooling allocations.This can significantly enhance performance by minimizing CPU usage during garbage collection, which is crucial for high-throughput applications.
3Implement parallel processing for data fetching and ranking to minimize latency in ad-serving systems.By running these processes concurrently, you can improve response times and enhance the user experience, especially in high-traffic scenarios.
4Adopt a lambda architecture for your data pipelines to facilitate both batch and real-time updates.This ensures that your data remains fresh and reduces the likelihood of serving outdated information, which is critical for ad performance.
Common Pitfalls
1
Over-relying on in-memory indexing can lead to scalability issues as data grows.
This can result in memory bottlenecks and increased infrastructure costs, making it difficult to maintain performance as the system scales.
2
Neglecting garbage collection optimization can lead to performance degradation.
If the system allocates more objects than the garbage collector can efficiently handle, it can cause CPU spikes and latency issues, impacting user experience.