Overview
The article discusses the implementation of a queue proxy service at Notion to enhance observability and scalability of their Redis-based task queue. It details the challenges faced with the previous architecture and how the new system improves traffic management and resource allocation.
What You'll Learn
1
How to implement a queue proxy service for task management
2
Why decoupling application logic from Redis improves scalability
3
How to achieve 100% observability of queue traffic
Prerequisites & Requirements
- Understanding of Redis and task queue concepts
- Familiarity with Amazon ECS and load balancers(optional)
Key Questions Answered
What challenges did Notion face with their previous Redis queue architecture?
Notion's previous Redis queue architecture faced challenges such as tight coupling between application logic and Redis clients, which hindered deployment flexibility and security control. Additionally, there were observability blind spots that made it difficult to track the origins of tasks and diagnose resource issues.
How does the new queue proxy service improve observability?
The new queue proxy service ensures 100% observability coverage for all incoming and outgoing queue traffic. By routing all interactions through this proxy, Notion can log traffic comprehensively, allowing for better resource management and quicker issue resolution.
What is the significance of using feature flags during the migration process?
Feature flags allowed Notion to implement the queue proxy service gradually, enabling a percentage-based rollout and quick rollbacks if issues arose. This approach minimized disruption to users and maintained service continuity during the transition.
What is the enqueuing rate of tasks in Notion's Redis queue?
Notion's Redis queue has an enqueuing rate of up to 10,000 tasks per second, with approximately 100 million tasks in the queue at any given time. This high volume underscores the need for a robust task management solution.
Key Statistics & Figures
Enqueuing rate
10,000 tasks per second
This rate highlights the high demand and need for efficient task management within Notion's infrastructure.
Total tasks in queue
100 million tasks
The large volume of tasks emphasizes the importance of scalability and observability in task processing.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Database
Redis
Used to manage and execute asynchronous and background tasks.
Cloud Service
Amazon Ecs
Platform used to deploy the queue proxy service.
Key Actionable Insights
1Implement a queue proxy service to decouple application logic from your task queue.This decoupling allows for greater deployment flexibility and reduces the blast radius of changes, making it easier to manage infrastructure updates without affecting application performance.
2Utilize feature flags for gradual rollout of new infrastructure components.Feature flags help mitigate risks during deployment by allowing teams to test new features with a subset of users, ensuring stability before full-scale implementation.
3Establish comprehensive logging for all queue traffic.Having detailed logs enables quicker diagnosis of issues and better understanding of task origins, which is crucial for maintaining system health and performance.
Common Pitfalls
1
Tight coupling between application logic and Redis clients can lead to deployment challenges.
This coupling makes it difficult to change Redis configurations without redeploying all services, which can disrupt application performance and availability.