Overview
The article discusses Pacer, Pinterest's new asynchronous computing platform designed to address the limitations of its predecessor, Pinlater. It highlights the architectural changes made to enhance scalability, performance, and reliability in job execution.
What You'll Learn
1
How to implement a stateful dequeue broker service using Helix and Zookeeper
2
Why job data sharding improves performance in asynchronous computing platforms
3
When to use dedicated worker pools for job execution in Kubernetes
Prerequisites & Requirements
- Understanding of asynchronous computing and job execution patterns
- Familiarity with Helix and Zookeeper(optional)
- Experience with Kubernetes for managing containerized applications
Key Questions Answered
What are the main limitations of the Pinlater platform?
Pinlater faced scalability bottlenecks, hardware efficiency issues, lack of isolation, and usability challenges as traffic increased. These limitations included lock contention, job execution interference, and shared service impacts on reliability, which necessitated a complete architectural revamp.
How does Pacer improve upon the architecture of Pinlater?
Pacer introduces a stateful dequeue broker service, dedicated worker pools, and utilizes Helix with Zookeeper for managing job queue partitions. This architecture eliminates lock contention, improves hardware utilization, and allows for job execution in isolation, enhancing overall performance.
What is the role of Helix and Zookeeper in Pacer?
Helix, in conjunction with Zookeeper, manages the assignment of job queue partitions to dequeue brokers. This setup ensures that each broker exclusively handles specific partitions, eliminating competition for job data and improving efficiency in job processing.
What improvements have been achieved by migrating to Pacer?
The migration to Pacer has resulted in the elimination of lock contention in the datastore, significant improvements in hardware utilization, and the ability for jobs to execute independently in customized environments, leading to better performance compared to Pinlater.
Key Statistics & Figures
Job executions handled daily by Pinlater
billions
This scale highlights the need for a robust asynchronous job execution platform.
Percentage of datastore calls that returned empty results in Pinlater
more than 50%
This inefficiency was a significant factor leading to the architectural changes in Pacer.
Dequeue request latency in Pacer
less than 1ms
This performance metric demonstrates the efficiency improvements achieved with the new architecture.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Helix
Used for managing the assignment of job queue partitions to dequeue brokers.
Backend
Zookeeper
Facilitates communication and resource configuration management between Helix and dequeue brokers.
Orchestration
Kubernetes
Used for deploying dedicated worker pools for job execution.
Key Actionable Insights
1Implementing a stateful dequeue broker can significantly reduce lock contention issues in your job processing architecture.This approach allows for exclusive access to job data partitions, enhancing throughput and reliability in high-traffic scenarios.
2Utilizing dedicated worker pools for each job queue can improve performance and resource allocation.By isolating job executions, you can customize resource configurations based on specific queue requirements, leading to better overall efficiency.
3Adopting Helix and Zookeeper for managing job queue partitions can streamline your operational overhead.This setup allows for dynamic assignment and monitoring of resources, ensuring optimal performance and reducing manual management efforts.
Common Pitfalls
1
Failing to isolate job queues can lead to performance degradation across the system.
When multiple job queues share the same worker hosts, a poorly performing job can negatively impact others. It's crucial to allocate dedicated resources for each queue to maintain optimal performance.
2
Not addressing lock contention can severely limit scalability.
Lock contention occurs when multiple threads compete for the same resource, leading to increased latency and reduced throughput. Implementing a stateful dequeue broker can mitigate this issue.
Related Concepts
Asynchronous Computing
Job Execution Patterns
Resource Management In Distributed Systems
Performance Optimization Techniques