Corrosion

Several times a second, as customer CI/CD pipelines tear up or bring down Fly Machines, our state synchronization system blasts updates across our internal mesh, so that edge proxies from Tokyo to Amsterdam can keep the accurate routing table that al

Fly.io logo
Fly.io
•
Thomas Ptacek, Peter Cai
•11 min read•advanced•
--
•View Original
ConsulDockerKubernetesRustSQLSQLiteVault

Overview

The article discusses Corrosion, a novel service discovery system developed by Fly.io that addresses the challenges of state synchronization in distributed systems. It highlights the lessons learned from past outages and the architectural decisions that led to the creation of Corrosion, which utilizes a gossip protocol for efficient state management without relying on distributed consensus.

What You'll Learn

1

How to implement a gossip protocol for state synchronization in distributed systems

2

Why avoiding distributed consensus can improve system performance

3

How to use CRDTs for managing state in distributed databases

Prerequisites & Requirements

  • Understanding of distributed systems and state synchronization concepts
  • Familiarity with SQLite and Rust programming language

Key Questions Answered

What is Corrosion and how does it function?
Corrosion is a Rust program that propagates a SQLite database using a gossip protocol. It allows for efficient state synchronization across distributed systems without relying on distributed consensus, thus avoiding common pitfalls associated with centralized databases.
What lessons were learned from the outages experienced with Corrosion?
The article emphasizes that distributed systems can propagate bugs rapidly. A significant outage occurred due to a deadlock caused by incorrect assumptions in the proxy code, highlighting the importance of understanding failure modes in distributed systems.
How does Corrosion differ from traditional service discovery systems?
Unlike traditional systems that rely on centralized databases for state management, Corrosion uses a decentralized approach where each node manages its own state and communicates changes through a gossip protocol, enhancing scalability and reducing bottlenecks.
What are the main architectural decisions behind Corrosion?
Corrosion was designed to avoid distributed consensus and instead takes inspiration from link-state routing protocols like OSPF. This allows it to efficiently manage state across a fully connected network of nodes, leveraging a gossip protocol for updates.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Programming Language
Rust
Used to implement the Corrosion service discovery system.
Database
Sqlite
Serves as the underlying database for Corrosion, allowing for local state management.

Key Actionable Insights

1
Implement watchdog mechanisms in your distributed applications to monitor for deadlocks and performance stalls.
By incorporating watchdogs, you can proactively detect and recover from issues that may lead to service outages, ensuring higher availability and reliability in your systems.
2
Consider using CRDTs for managing state in distributed databases to simplify conflict resolution.
CRDTs allow for eventual consistency without the need for locking or complex consensus algorithms, making them ideal for applications that require high availability and low latency.
3
Adopt a regionalized approach to state management in distributed systems to reduce the impact of state-related bugs.
By segmenting state management into regional clusters, you can limit the blast radius of potential issues, allowing for safer and more manageable deployments.

Common Pitfalls

1
Relying on distributed consensus protocols can lead to performance bottlenecks and increased complexity.
Many distributed systems struggle with the overhead of maintaining consensus across nodes, especially over long distances, which can hinder responsiveness and scalability.
2
Assuming that schema changes in distributed databases will not cause outages.
Even seemingly trivial changes can trigger widespread reconciliation messages in a gossip system, leading to unexpected performance issues and outages.

Related Concepts

Distributed Systems
Service Discovery
State Synchronization
Crdts
Gossip Protocols