Delos: Simple, flexible storage for the Facebook control plane

Mahesh Balakrishnan

At today’s Systems @Scale conference, we presented Delos, a fundamentally new architecture for building replicated storage systems. Its modular, layered design provides flexibility and simplicity w…

Overview

Delos is a new architecture for building replicated storage systems at Facebook, designed to provide flexibility and simplicity without compromising performance or reliability. Its modular design allows for rapid deployment and evolution of storage systems, enabling significant improvements in latency and operational efficiency.

What You'll Learn

1

How to implement a pluggable architecture for storage systems

2

Why separating materialization from ordering improves system flexibility

3

How to achieve 10x lower latency in storage systems without downtime

4

When to use different SimpleLog implementations for performance optimization

Key Questions Answered

What are the main benefits of the Delos architecture?

The Delos architecture provides flexibility, simplicity, and a modular design that allows for rapid deployment and evolution of storage systems. It enables low-dependency storage, supports diverse APIs, and allows for hot-swapping of components to optimize performance without downtime.

How does Delos achieve low-latency storage solutions?

Delos achieves low-latency storage solutions by implementing a pluggable architecture that separates API materialization from update ordering. This allows for the integration of new SimpleLog implementations that can be swapped in dynamically, resulting in performance improvements of up to 10x.

What challenges do existing monolithic storage systems face?

Existing monolithic storage systems are inflexible and often require complete rewrites to accommodate new use cases. They mix functionalities like ordering, materialization, and reconfiguration, making it difficult to adapt to varying performance and reliability requirements.

When was the initial version of Delos deployed in production?

The initial version of Delos was deployed in production within eight months, showcasing its rapid time-to-deployment for new storage systems.

Key Statistics & Figures

Latency improvement

10x lower latency

Achieved by swapping in a new SimpleLog implementation without service downtime.

Time to deploy initial version

8 months

Time taken to deploy the first version of Delos in production.

Technologies & Tools

Backend

Zookeeper

Initially used as the ordering mechanism before transitioning to a native SimpleLog.

Backend

Logdevice

Used for implementing an experimental disaggregated SimpleLog.

Key Actionable Insights

1
Utilize a pluggable architecture to enhance the flexibility of your storage systems.
By adopting a modular design similar to Delos, you can easily adapt to changing requirements and integrate new technologies without extensive rewrites.

2
Implement a VirtualLog abstraction to simplify distributed coordination in your systems.
This approach can help manage complexity and improve performance by decoupling state materialization from update ordering.

3
Consider the use of SimpleLogs for optimizing performance based on workload requirements.
Dynamic switching between different SimpleLog implementations can significantly enhance system performance and reliability.

Common Pitfalls

1

Relying on monolithic storage systems can lead to inflexibility and high maintenance costs.

These systems often require complete rewrites to accommodate new use cases, which can slow down development and increase technical debt.