How we built a generic idempotency framework to achieve eventual consistency and correctness across our payments micro-service…
Overview
The article discusses the implementation of a generic idempotency framework at Airbnb to prevent double payments in a distributed payments system. It highlights the challenges of maintaining data integrity in a Service Oriented Architecture (SOA) and explains how the framework, named Orpheus, achieves eventual consistency and correctness.
What You'll Learn
1
How to implement a generic idempotency framework in a distributed payments system
2
Why maintaining data integrity is crucial in a Service Oriented Architecture
3
When to categorize exceptions as retryable or non-retryable
Prerequisites & Requirements
- Understanding of distributed systems and eventual consistency
- Familiarity with Java and its lambda expressions(optional)
Key Questions Answered
What is idempotency and why is it important in payments systems?
Idempotency allows clients to make the same API request multiple times without causing unintended effects, ensuring that a payment request is processed exactly once. This is critical in payments systems to avoid double charges or multiple payouts, maintaining financial integrity.
How does the Orpheus framework ensure eventual consistency?
The Orpheus framework uses a combination of write repair and idempotency keys to manage requests. It allows clients to retry requests without maintaining state, ensuring that operations are executed at most once, thus achieving eventual consistency across distributed services.
What are the phases of an API request in the Orpheus framework?
An API request in the Orpheus framework is divided into three phases: Pre-RPC, where payment details are recorded; RPC, where the request is sent to external services; and Post-RPC, where the response is recorded. This separation helps maintain data integrity and consistency.
What are the trade-offs of using a master database for idempotency information?
Using a master database for idempotency information ensures strong consistency and avoids issues like replica lag, which could lead to duplicate payments. However, it can create scaling challenges, necessitating sharding by idempotency key to manage load effectively.
Key Statistics & Figures
Consistency achieved
five nines
This level of consistency has been achieved for Airbnb Payments since the launch of the Orpheus framework.
Annual payment volume growth
doubled
The annual payment volume has doubled while maintaining high consistency in the payment system.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Programming Language
Java
Used for implementing the Orpheus framework and handling database transactions.
Database
Mysql
Used as the master database for storing idempotency information.
Key Actionable Insights
1Implement a generic idempotency solution across services to enhance data integrity.By using a framework like Orpheus, developers can ensure that payment requests are processed correctly without duplicating efforts across different services, allowing for faster iterations and improved reliability.
2Carefully categorize exceptions to avoid unintended consequences.Understanding which exceptions are retryable versus non-retryable is crucial. Misclassifying an exception can lead to either unnecessary retries or missed opportunities to recover from transient failures.
3Utilize Java lambdas to streamline database transactions.Leveraging Java lambda expressions can simplify the complexity of combining multiple database operations into a single transaction, enhancing code readability and maintainability.
Common Pitfalls
1
Misclassifying exceptions can lead to significant issues in payment processing.
If a transient error is labeled as non-retryable, it could result in failed transactions that never recover, while labeling a critical error as retryable could lead to duplicate payments.
Related Concepts
Eventual Consistency
Service Oriented Architecture
Idempotency In Distributed Systems