Each time we write about it, we get a little bit better at golfing down a description of what Litestream is. Here goes: Litestream is a Unix-y tool for keeping a SQLite database synchronized with S3-style object storage. It’s a way of getting the spe
Overview
This article by Ben Johnson explains two major new features added to Litestream's Virtual File System (VFS): writable mode and background hydration. These features were developed specifically to support Fly.io's Sprites product, enabling SQLite databases to serve both reads and writes directly from S3-compatible object storage during cold starts, then seamlessly transition to local storage through background hydration.
What You'll Learn
1
How Litestream VFS enables read-write SQLite queries directly against S3-compatible object storage without downloading the full database
2
Why single-writer mode is chosen over multi-writer distributed SQLite and what trade-offs that entails
3
How background hydration works to transition from remote object storage reads to local file reads without blocking queries
4
When to use the Litestream VFS writable mode versus standard Litestream sidecar configuration
5
How the many-SQLite-databases pattern can replace centralized PostgreSQL for per-tenant workloads
Prerequisites & Requirements
- Understanding of SQLite and its WAL (Write-Ahead Logging) mechanism
- Familiarity with S3-compatible object storage concepts
- Basic understanding of Virtual File Systems (VFS) and how SQLite uses them
- Understanding of cold start problems in ephemeral compute environments(optional)
- Litestream installed and basic familiarity with its backup/restore workflow
Key Questions Answered
How does Litestream VFS enable writing to SQLite databases stored in object storage?
Litestream VFS writable mode buffers writes to a local temporary write buffer instead of writing directly to object storage. Every second or so (or on clean shutdown), the write buffer syncs with object storage. This provides eventual durability — writes aren't truly durable until the sync happens. The VFS disables polling in write mode and assumes a single writer, avoiding the complexity of distributed multi-writer SQLite.
What is background hydration in Litestream and how does it improve performance?
Background hydration is a technique borrowed from systems like dm-clone where the VFS serves queries remotely from object storage while simultaneously running a background loop to download the full database to a local file. When you set the LITESTREAM_HYDRATION_PATH environment variable, Litestream hydrates to that file, writing only the latest versions of each page via LTX compaction. Reads switch over to the local file when hydration completes.
Why doesn't Litestream VFS support multiple writers?
The article explicitly states that multiple-writer distributed SQLite databases are avoided because of their extreme complexity. In write mode, Litestream VFS disables polling for remote changes and assumes a single writer with no additional backups to watch. This simplification is acceptable for the target use case of Sprite storage where each database has a single owner.
How does Fly.io Sprites use Litestream for sub-second boot times with 100GB storage?
Sprites use S3-compatible object storage as the root of storage, with a JuiceFS-based system that keeps a SQLite 'block map' database of in-use storage blocks. Litestream VFS allows this block map to serve reads and writes directly from object storage during cold starts, before the full database is downloaded. NVMe attached storage acts as a read-through cache, and background hydration pulls the complete database while queries are served remotely.
What is the many-SQLite-databases pattern and when should you use it?
The many-SQLite-databases pattern gives each tenant (in this case, each organization enrolled in Sprites) their own separate SQLite database, synchronized by Litestream to object storage. This pattern has nice scaling characteristics compared to a centralized PostgreSQL cluster, which the article notes has been a major engineering challenge to keep happy as Fly.io grew.
How does the Litestream VFS index work for serving reads from object storage?
The VFS maintains an index of (file, offset, size) tuples for every page of the database stored in object storage. This index data is stored in LTX files for efficient reconstitution when the VFS starts. When SQLite issues a read, the VFS library intercepts it, looks up the requested page in the index, fetches it from object storage, and caches it. Lookups are heavily cached for performance.
What happens to the hydration file when a Sprite bounces or exits?
The hydration file is written to a temporary file and discarded when the VFS exits. Because Sprites bounce frequently, the system cannot trust that a previously hydrated database reflects the latest state without performing a full restore. This behavior is currently baked into the VFS design specifically for Sprite-like environments with eventual durability requirements.
Key Statistics & Figures
Sprite durable storage per instance
100GB
Every Sprite boots up with 100GB of durable storage backed by S3-compatible object storage
Sprite launch time
Under 1 second
Sprites launch in under a second, requiring the storage stack to be available almost immediately
Block map worst-case size
Low tens of megabytes
The SQLite block map for Sprite storage can be up to low tens of megabytes worst case, which must be reconstituted during cold starts
Write buffer sync interval
Every second or so
The writable VFS syncs the local write buffer with object storage approximately every second or on clean shutdown
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Database Tooling
Litestream
Backup/restore and replication system for SQLite, providing continuous synchronization with S3-compatible object storage
Database
Sqlite
Core embedded database used for both the Sprites orchestrator and the block map storage system
Cloud Storage
S3
S3-compatible object storage serves as the root of Sprite storage and the backend for Litestream VFS
File Format
Ltx
Custom file format for storing database page index data efficiently, enabling fast VFS reconstitution and hydration compaction
Distributed Filesystem
Juicefs
Provides the storage stack for Sprites, using a rewritten metadata store based on Litestream SQLite
Programming Language
Elixir
Powers the global Sprites orchestrator that runs off S3-compatible object storage via Litestream
Database
Postgresql
Previously used centralized database for Fly Machines that the Sprites orchestrator replaced with per-tenant SQLite databases
Hardware
Nvme
Attached NVMe storage used as a read-through cache for the block map storage system
Linux Kernel
Dm-clone
Linux device-mapper target that inspired the background hydration design pattern used in Litestream VFS
Key Actionable Insights
1Consider the many-SQLite-databases pattern as an alternative to centralized PostgreSQL for per-tenant workloads. Each tenant gets their own SQLite database synchronized by Litestream to object storage, providing natural data isolation and simpler scaling characteristics.Fly.io's Sprites orchestrator uses this pattern to replace their centralized Postgres cluster, which had become a major engineering challenge to maintain at scale.
2For cold-start scenarios where you need to serve database queries before a full restore completes, use Litestream VFS to query directly from object storage. Set LITESTREAM_HYDRATION_PATH to enable background hydration that transitions to local storage without blocking reads.This is particularly valuable for ephemeral compute environments where instances need to serve requests within milliseconds of booting, such as serverless functions or Fly.io Sprites.
3Only enable Litestream VFS writable mode (LITESTREAM_WRITE_ENABLED=true) when your application can tolerate eventual durability — writes aren't truly durable until the periodic sync to object storage happens (approximately every second). For standard read/write workloads, use Litestream as a normal sidecar instead.The writable VFS was designed specifically for environments like Sprites where all storage already has an eventual durability property. Most applications need stronger durability guarantees.
4When designing storage systems that need to handle cold starts, borrow the background hydration pattern from dm-clone: serve queries from a remote source immediately while running a background loop to pull data locally. Switch over to local storage only when hydration is complete.This pattern allows you to decouple availability from data locality, providing immediate service while optimizing for steady-state performance in the background.
5Avoid implementing multi-writer distributed SQLite. The article strongly advises against this approach due to extreme complexity. Instead, design your system around single-writer semantics and use replication for read scalability.Litestream's VFS explicitly disables polling in write mode and assumes a single writer, treating multi-writer distributed SQLite as an intractably complex problem.
Common Pitfalls
1
Assuming the writable VFS provides the same durability as standard SQLite writes. Writes go to a local temporary buffer and are only synced to object storage every second or so. If the process crashes between syncs, those writes are lost.
This is acceptable for Sprite storage where all storage has eventual durability, but most applications require stronger guarantees. For standard workloads, use Litestream as a sidecar instead of the VFS.
2
Attempting to use the Litestream VFS with multiple concurrent writers. The writable VFS explicitly disables remote polling and assumes a single writer. Running multiple writers would lead to data corruption or loss.
Design your architecture around single-writer semantics. If you need multiple writers, use standard Litestream sidecar mode with a normal SQLite database file.
3
Relying on the hydration file persisting across process restarts. The VFS currently discards the hydration file on exit because it cannot guarantee the file reflects the latest state without a full restore.
This behavior is baked into the current VFS design for Sprite-like environments that bounce frequently. If your use case needs persistent hydration, you would need to manage database state verification separately.
4
Using VFS mode for steady-state production workloads. While the VFS is excellent for cold starts and point-in-time queries, running queries against object storage is significantly slower than local reads. The VFS should transition to local storage via hydration for steady-state performance.
The intended design is: serve from VFS during cold start, hydrate in the background, then switch to local file for normal operations.
Related Concepts
Sqlite Virtual File System (vfs)
S3-compatible Object Storage
Write-ahead Logging (wal)
Ltx File Format
Background Hydration Pattern
Dm-clone Device Mapper
Many-sqlite-databases Pattern
Eventual Durability
Read-through Caching
Point-in-time Recovery
Cold Start Optimization
Juicefs Metadata Store
Block Map Storage
Single-writer Replication