How we completely rearchitected Mussel, our storage engine for derived data, and lessons learned from the migration from Mussel V1 to V2.
Overview
Airbnb completely rearchitected Mussel, their core key-value store for derived data, migrating from v1 to v2 with a NewSQL backend. The article details why the rearchitecture was needed, the new Kubernetes-native architecture with a stateless Dispatcher and Kafka-based write pipeline, and how they migrated over a petabyte of data across thousands of tables with zero downtime using a blue/green strategy with dual writes and shadow reads.
What You'll Learn
1
How to design a blue/green migration strategy for a petabyte-scale key-value store with zero downtime
2
Why replacing static hash partitioning with dynamic range sharding and presplitting improves performance at scale
3
How to use Kafka as a replication log for dual-write consistency during live database migrations
4
When to trade off data freshness versus cost by choosing between primary and secondary replicas
5
How to build a topology-aware TTL expiration service that scales across large data namespaces
Prerequisites & Requirements
- Understanding of distributed systems concepts including consistency models, partitioning, and replication
- Familiarity with key-value store architectures and their read/write patterns
- Knowledge of Kubernetes concepts including StatefulSets, manifests, and automated rollouts
- Understanding of Apache Kafka as a messaging and replication system
- Experience with large-scale data migration or database operations(optional)
Key Questions Answered
How did Airbnb migrate a petabyte of data with zero downtime?
Airbnb used a blue/green migration strategy with a custom pipeline: data was sampled from v1 backups, presplit tables were created on v2, data was bootstrapped using Kubernetes StatefulSets with checkpointing, then verified via checksums. Dual writes through Kafka maintained consistency, while shadow reads validated v2 correctness before gradually shifting traffic table-by-table with automatic circuit breakers for instant fallback.
What is Mussel and why did Airbnb rebuild it?
Mussel is Airbnb's core key-value store that bridges offline and online workloads, providing bulk load capabilities with single-digit millisecond reads. v1 had operational complexity with multi-step Chef scripts on EC2, static hash partitioning causing hotspots, limited consistency control, and opaque resource usage. New requirements like real-time fraud checks, instant personalization, and dynamic pricing demanded a modern, cloud-native platform.
How does Mussel v2's Dispatcher architecture work?
The Dispatcher is a stateless, horizontally-scalable Kubernetes service that translates client API calls into backend queries and mutations. It supports dual-write and shadow-read modes for migration, manages retries and rate limits, and integrates with Airbnb's service mesh. Reads use optimized point lookups, range/prefix queries, and stale reads from local replicas. Writes are persisted in Kafka first, then applied by the Replayer and Write Dispatcher.
What challenges arise when migrating from eventually consistent to strongly consistent storage?
Moving from eventual to strong consistency introduced write conflicts that required features like write deduplication, hotkey blocking, and lazy write repair. These solutions sometimes traded off storage cost or read performance. Additionally, shifting from hash-based to range-based partitioning required careful presplitting based on data sampling to prevent hotspots during migration.
How does Mussel v2 handle bulk data loading at scale?
Bulk load uses an Airflow-based pipeline that transforms warehouse data into a standardized format and uploads to S3. A stateless controller orchestrates jobs while a distributed StatefulSet worker fleet performs parallel ingestion from S3 into tables. Optimizations include deduplication for replace jobs, delta merges, and insert-on-duplicate-key-ignore to ensure high throughput at Airbnb scale.
What is the role of Kafka in Mussel v2's architecture?
Kafka serves as the durable write-ahead log for all mutations. Writes are persisted to Kafka first, then consumed and applied to the backend by the Replayer and Write Dispatcher. This event-driven model absorbs traffic bursts, ensures consistency, and removes operational overhead. During migration, Kafka maintained eventual consistency between v1 and v2 through dual consumption from the same topics.
How does Mussel v2 handle data expiration and TTL at scale?
Mussel v2 uses a topology-aware expiration service that shards data namespaces into range-based subtasks processed concurrently by multiple workers. Expired records are scanned and deleted in parallel to minimize sweep time. Subtasks are scheduled to limit impact on live queries, and write-heavy tables use max-version enforcement with targeted deletes to maintain performance and data hygiene.
What performance can Mussel v2 achieve simultaneously?
Mussel v2 can simultaneously ingest tens of terabytes in bulk data uploads, sustain over 100,000 streaming writes per second in the same cluster, and keep p99 reads under 25 milliseconds. It also provides per-namespace controls for toggling stale reads, combining the elasticity of object storage with the responsiveness of a low-latency cache.
Key Statistics & Figures
Data migrated
More than a petabyte
Total data migrated from v1 to v2 across thousands of tables with zero downtime
Tables migrated
Thousands
Number of tables migrated from v1 to v2
Use cases supported
100+
Existing user cases that needed feature parity during migration
Table size support
100TB+
V2 handles tables exceeding 100TB with p99 reads under 25ms
p99 read latency
< 25ms
Read performance maintained even for 100TB+ tables
Streaming write throughput
100,000+ writes per second
Sustained streaming writes per second in the same cluster alongside bulk loads
Bulk load capacity
Tens of terabytes
Simultaneous bulk data upload capability
Replication lag during dual writes
Tens of milliseconds
Typical lag between v1 and v2 during dual-write mode
v2 production runtime
1 year
Mussel v2 has been running successfully in production for a year
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Messaging
Kafka
Write-ahead log for durability, replication between v1 and v2, and event-driven write pipeline
Orchestration
Kubernetes
Container orchestration for stateless Dispatcher services and StatefulSet workers for bulk load and migration
Storage
S3
Intermediate storage for bulk load data uploaded from offline warehouses
Compute
EC2
Previous v1 infrastructure that required manual Chef scripts for scaling
Configuration Management
Chef
Used in v1 for multi-step node scaling and replacement scripts
Workflow Orchestration
Airflow
Scheduling and orchestrating data pipelines for bulk load onboarding from warehouses
Database
Newsql
Backend storage engine for Mussel v2 providing strong consistency and distributed capabilities
Key Actionable Insights
1Use a blue/green migration strategy with per-table granularity and automatic circuit breakers when migrating critical storage systems. This approach allows you to validate correctness through shadow reads, gradually shift traffic, and instantly revert to the old system if issues arise — all without impacting availability.Airbnb migrated over a petabyte of data across thousands of tables with zero downtime by making every migration step reversible and fine-tunable per table based on risk profile.
2When switching from hash-based to range-based partitioning, presplit target tables based on data sampling to prevent hotspots during ingestion. Inserting large consecutive data into range-based systems can overload specific nodes, so understanding data distribution upfront is critical.Airbnb sampled v1 backup data to create pre-defined shard layouts on v2, ensuring balanced ingestion traffic across backend nodes during migration.
3Persist writes to Kafka before applying them to the backend database to create an event-driven architecture that absorbs traffic bursts, ensures durability, and simplifies operational concerns. This pattern also enables dual-write capabilities during migrations and provides a natural replication mechanism.Kafka's proven stable p99 millisecond latency made it invaluable during migration, serving as the intermediary for write reliability throughout the entire process.
4Design your key-value store's consistency model to be configurable per namespace or use case. Some workloads prioritize data freshness using primary replicas, while others can tolerate staleness by reading from secondary replicas to save cost and improve performance.Mussel v2 gives callers a simple dial to toggle stale reads on a per-namespace basis, allowing teams to optimize for their specific SLA requirements rather than being locked into one consistency level.
5Implement topology-aware data expiration that shards cleanup work into range-based subtasks processed concurrently, rather than relying on storage engine compaction cycles. Schedule these tasks to limit impact on live queries while maintaining data hygiene at scale.V1's compaction-based TTL struggled at scale. V2's parallel expiration service with scheduled subtasks provides the same retention functionality with far greater efficiency and transparency.
6Use Kubernetes StatefulSets for bootstrap and migration workloads that need persistent local state and periodic checkpointing. This allows long-running data migration jobs (hours to days) to survive pod restarts and make incremental progress rather than restarting from scratch.The bootstrap step was the most time-consuming part of the migration pipeline, and StatefulSets with checkpointing were essential for efficiently handling large tables.
Common Pitfalls
1
Inserting large consecutive data ranges into a range-based partitioning system without presplitting can create hotspots that overload specific backend nodes. This is especially dangerous during migration when bulk ingestion traffic is high and unbalanced distribution can cascade into latency spikes or outages.
Airbnb solved this by sampling v1 data distribution and creating pre-defined shard layouts on v2 before migration, ensuring balanced ingestion across all nodes.
2
Migrating from an eventually consistent system to a strongly consistent one introduces unexpected write conflicts that the old system silently handled. Without features like write deduplication, hotkey blocking, and lazy write repair, these conflicts can cause data inconsistencies or performance degradation.
Resolving these conflicts sometimes required trading off storage cost or read performance, so teams should plan for these tradeoffs early in the migration design.
3
Assuming a new backend will push down query filters as effectively as the old one can lead to performance regressions. Mussel v2's range-based backend didn't push down range filters as well as v1's hash-based system, requiring client-side pagination for prefix and range queries.
Test query patterns thoroughly during the shadow read phase to identify performance differences before cutting over production traffic.
4
Attempting to migrate all tables simultaneously or in large batches increases risk and makes debugging failures extremely difficult. Rushing the migration can lead to cascading issues across dependent services.
Airbnb migrated one table at a time with per-table stage assignments and instant reversibility, allowing rapid iteration and safe rollbacks without impacting the business.
Related Concepts
Distributed Key-value Stores
Blue/Green Deployment Strategy
Dual-write Consistency Patterns
Range-based Vs Hash-based Partitioning
Eventually Consistent Vs Strongly Consistent Systems
Kafka As A Replication Log
Kubernetes Statefulsets
Bulk Data Ingestion Pipelines
Ttl And Data Expiration At Scale
Newsql Databases
Service Mesh Integration
Circuit Breaker Pattern
Data Presplitting Strategies
Shadow Read/Write Testing