Overview
The article discusses the open-sourcing of PinLater, an asynchronous job execution system developed by Pinterest. It highlights the system's architecture, its production usage, and the improvements made over time, emphasizing its scalability and reliability.
What You'll Learn
1
How to implement an asynchronous job execution system using PinLater
2
Why fault tolerance is crucial in job execution systems
3
When to use MySQL over Redis for job storage
Prerequisites & Requirements
- Understanding of asynchronous job execution concepts
- Familiarity with MySQL and Thrift(optional)
Key Questions Answered
What are the core components of PinLater?
PinLater consists of three core components: a stateless Thrift service for job submission and scheduling, a storage backend for job state management, and worker pools that execute the jobs. This architecture allows for efficient job processing and management.
How does PinLater achieve high throughput?
PinLater has achieved 5X more throughput on the MySQL backend, allowing over 2,000 enqueues per second with a single i2.2xl MySQL EC2 instance. This improvement enables the system to handle a significant volume of jobs efficiently.
What improvements have been made to PinLater recently?
Recent improvements to PinLater include enhancements that allow for 5X more throughput on the MySQL backend, the introduction of a checkpointing feature for long-running jobs, and a new dashboard for monitoring job status and debugging failures.
What types of jobs does PinLater handle?
PinLater handles a variety of jobs including core actions like Pinning, image thumbnail generation, and larger batch operations such as email delivery and push notifications. This versatility makes it a critical system for Pinterest.
Key Statistics & Figures
Job processing capacity
over six million jobs per minute
This metric highlights the high throughput capabilities of the PinLater system.
Throughput on MySQL backend
over 2,000 enqueues per second
This performance was achieved with a single i2.2xl MySQL EC2 instance.
Number of job queues
more than 500 job queues
This indicates the extensive scaling of the PinLater system across multiple clusters.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Database
Mysql
Used as the primary storage backend for job state management.
Backend
Thrift
Used for managing job submission and scheduling.
Cloud Infrastructure
Amazon EC2
Hosting the PinLater clusters.
Key Actionable Insights
1Implementing an asynchronous job execution system like PinLater can significantly improve the scalability of your applications.By decoupling job execution from scheduling, you can independently scale components based on demand, which is essential for high-traffic applications.
2Utilizing MySQL for job storage can enhance durability and reliability in job processing.With MySQL's reliable replication features, moving workloads from Redis to MySQL can provide better data integrity and performance.
3Creating a dashboard for monitoring job status can greatly aid in debugging and operational efficiency.Having visibility into job processing allows teams to quickly identify and resolve issues, improving overall system reliability.
Common Pitfalls
1
Overlooking the importance of fault tolerance in job execution systems can lead to data loss or job failures.
Without a reliable execution guarantee, jobs may not be processed correctly, resulting in a poor user experience and potential data integrity issues.
2
Failing to separate job execution from scheduling can hinder scalability.
If both processes are tightly coupled, it becomes challenging to scale them independently, which can lead to performance bottlenecks.
Related Concepts
Asynchronous Job Execution
Job Scheduling
Fault Tolerance In Distributed Systems
Scalability In Cloud Applications