Facebook Ordered Queueing Service (FOQS) is a fully managed, distributed priority queueing service used for reliable message delivery among many services. FOQS has evolved from a regional deploymen…
Overview
The article discusses the evolution of the Facebook Ordered Queueing Service (FOQS) from a regional to a global deployment, emphasizing its disaster readiness and high availability. It details the architectural modifications made to ensure reliable message delivery across services, even during large-scale disasters.
What You'll Learn
1
How to implement a globally distributed priority queue system
2
Why migrating to a global architecture enhances disaster resilience
3
How to utilize intelligent routing to optimize message delivery
Prerequisites & Requirements
- Understanding of distributed systems and queueing mechanisms
- Experience with MySQL and message queuing services(optional)
Key Questions Answered
What challenges did FOQS face during regional installations?
FOQS faced significant challenges during regional installations, including increased cross-region latencies during failover and the complexity of managing traffic during outages. Clients had to explicitly balance their traffic away from impacted regions, leading to potential underutilization of resources and data being stuck on replicas until connectivity was restored.
How does FOQS ensure high availability during disasters?
FOQS ensures high availability during disasters by migrating to a global architecture that allows for the following of MySQL primaries upon failovers and the introduction of an intelligent routing service. This architecture enables seamless access to data across regions, even when one region is down, thus avoiding issues like stuck queue items.
What is the role of the routing service in FOQS?
The routing service acts as a proxy between clients and the queue service, abstracting routing concerns and optimizing data placement across a global queueing layer. It allows clients to specify logical region preferences, ensuring efficient handling of enqueue and dequeue requests, even during regional outages.
What strategies were used to migrate Async to a global installation?
To migrate Async to a global installation without downtime, the team employed strategies such as utilizing a shadow environment for testing, implementing a config-based rollout mechanism, and developing dashboards to monitor migration health. These strategies helped manage risks and ensure a smooth transition.
Key Statistics & Figures
Zero downtime during migration
0
The migration of Async to a global installation was completed in mid-2021 with zero downtime for clients.
Latency reduction in cache replenishment
nearly halved
Parallelizing cache replenishment aggregation reduced replenishment latency significantly, improving overall system performance.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Database
Mysql
Used for managing replicas and supporting the queueing service architecture.
Backend
Shard Manager
Supports colocating MySQL primaries with queue nodes in corresponding regions.
Key Actionable Insights
1Implementing a global architecture for your queueing service can significantly enhance disaster resilience and data availability.By migrating to a global setup, you can ensure that your service remains operational even during regional outages, thus providing a better user experience and reliability.
2Utilizing intelligent routing can optimize the performance of your distributed systems.By abstracting routing concerns and allowing for logical region preferences, you can reduce latency and improve the efficiency of message delivery across your services.
3Conducting thorough testing in a shadow environment can mitigate risks during major migrations.This approach allows you to identify potential issues without impacting live operations, ensuring a smoother transition to new architectures.
Common Pitfalls
1
Failing to account for cross-region latencies can lead to significant delays during failover.
When a primary replica in one region fails, the system must elect a secondary from another region, which can introduce latencies of up to hundreds of milliseconds.
2
Overloading top-priority queue nodes can lead to thundering herd problems.
As traffic increases, nodes at the top of the cache receive disproportionate requests, causing delays for other nodes and potentially starving them of traffic.
Related Concepts
Distributed Systems
Message Queuing
Disaster Recovery Strategies
Global Architecture In Cloud Services