Making Machines Move

At the heart of our platform is a systems design tradeoff about durable storage for applications. When we added storage three years ago, to support stateful apps, we built it on attached NVMe drives. A benefit: a Fly App accessing a file on a Fly Vo

Fly.io logo
Fly.io
•
Thomas Ptacek
•16 min read•advanced•
--
•View Original
AWSDockerNomadSQLite

Overview

The article discusses the complexities and solutions involved in migrating stateful applications on Fly.io's platform, focusing on the design and implementation of a new cloning operation for volumes. It highlights the challenges of maintaining data integrity and minimizing downtime during migrations, as well as the innovative use of technologies like dm-clone and iSCSI.

What You'll Learn

1

How to implement a cloning operation for stateful applications to minimize downtime during migrations

2

Why using dm-clone can improve data migration efficiency in cloud environments

3

When to use iSCSI over other network protocols for block storage

Prerequisites & Requirements

  • Understanding of cloud infrastructure and stateful applications
  • Familiarity with iSCSI and dm-clone(optional)

Key Questions Answered

How does Fly.io handle stateful application migrations without data loss?
Fly.io employs a cloning operation using dm-clone to create a new volume while the original remains active. This allows for data to be transferred in the background, minimizing downtime and preventing data loss during migrations. The process involves stopping the original machine, cloning the volume, and booting a new machine with the cloned volume attached.
What are the challenges of migrating applications with attached volumes?
Migrating applications with attached volumes is complex because traditional methods like copying and booting can lead to data loss. The original volume must remain active during the migration, requiring a new approach that ensures data integrity while minimizing downtime.
Why did Fly.io switch from nbd to iSCSI for network block devices?
Fly.io initially used nbd but faced issues with kernel threads getting stuck during network disruptions. Switching to iSCSI resolved these issues, providing a more robust solution for network block devices, especially in a global cloud environment where network reliability is crucial.

Technologies & Tools

Backend
Dm-clone
Used for creating clones of volumes to facilitate data migration without downtime.
Network
Iscsi
Implemented as a network protocol for transferring block storage data efficiently.

Key Actionable Insights

1
Implementing a cloning operation can significantly reduce downtime during migrations, allowing for seamless transitions between workloads.
This is particularly important for businesses that require high availability and cannot afford service interruptions during maintenance or scaling operations.
2
Utilizing dm-clone for volume management can enhance data integrity and reduce the risk of data loss during migrations.
This approach is beneficial for applications that handle critical data, ensuring that users experience minimal disruption while maintaining data consistency.
3
Choosing the right network protocol, such as iSCSI, can improve the reliability of data transfers in cloud environments.
Understanding the strengths and weaknesses of different protocols can help engineers make informed decisions that align with their infrastructure needs.

Common Pitfalls

1
Assuming that traditional volume migration methods will work for stateful applications can lead to data loss and extended downtime.
It's crucial to understand the specific requirements of stateful applications and to implement a migration strategy that accounts for data integrity and availability.

Related Concepts

Cloud Infrastructure Management
Stateful Vs. Stateless Applications
Volume Management Techniques