Overview
This article discusses the redesign of Pinterest's ad serving systems, focusing on achieving zero downtime during the transition. It outlines the design principles, key decisions made, and the validation process that ensured the new system met performance and reliability standards.
What You'll Learn
1
How to design a graph execution framework for ad serving systems
2
Why implementing a write-once data model enhances data integrity
3
How to validate new ad serving systems against legacy systems
4
When to use type-safe nodes in a graph execution framework
Prerequisites & Requirements
- Understanding of graph execution frameworks and data integrity principles
- Experience with ad serving systems and concurrency issues(optional)
Key Questions Answered
What are the design principles for Pinterest's ad serving systems?
The design principles include extensibility, separation of concerns, safe-by-design, and development velocity. These principles guide the development of a flexible and maintainable ad serving framework that can adapt to future needs while ensuring data integrity and performance.
How does the Apex framework improve code organization?
Apex organizes code using a directed-acyclic graph (DAG) where each node represents a module owned by a team. This structure allows for easy addition or deprecation of functionalities by simply adding or removing nodes, thus enhancing extensibility and separation of concerns.
What validation methods were used to ensure the correctness of AdMixer?
Validation methods included value-based validation, metric-based validation, and live A/B testing on production traffic. These methods helped ensure that the new AdMixer system produced outputs consistent with the legacy system, Mohawk, under various conditions.
What challenges were faced during the migration from Mohawk to AdMixer?
Challenges included maintaining synchronization between both systems during the migration and ensuring that the new system produced consistent outputs despite differences in data models and external dependencies. A real-time validation framework was implemented to monitor discrepancies.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Apex
A graph execution framework used to organize code and manage execution dependencies in the ad serving system.
Data Serialization
Thrift
Used for defining data structures in the ad serving system, though it was identified as needing improvement to support immutability.
Messaging
Kafka
Utilized for logging inputs and outputs during validation processes to compare results between Mohawk and AdMixer.
Key Actionable Insights
1Implement a type-safe node structure in your graph execution framework to prevent runtime exceptions.Type safety helps catch errors during compile time, reducing the risk of failures in production environments. This is particularly important in systems where data integrity is critical.
2Adopt a write-once data model to enhance data integrity across your application.By ensuring that data can only be written once, you minimize the risk of concurrent modifications that could lead to inconsistencies, especially in multi-threaded environments.
3Utilize a component-wise validation framework to ensure new implementations match legacy systems.This approach allows for real-time monitoring and quick identification of discrepancies, which is crucial during migrations or major system updates.
Common Pitfalls
1
Failing to maintain synchronization between legacy and new systems during migration can lead to inconsistencies.
This often happens when teams rush to implement new features without ensuring that both systems produce the same outputs. Implementing a validation service can help monitor discrepancies in real-time.
Related Concepts
Graph Execution Frameworks
Data Integrity Principles
Concurrency Management In Software Systems