Supercharging the ML and AI Development Experience at Netflix with Metaflow

Key Actionable Insights

1

Adopt the incremental spin-based development workflow for building Metaflow flows: start with a skeleton, run once to persist initial artifacts, then use spin to iterate on individual steps. This approach mirrors notebook-style development but produces production-ready, deployable workflows.

This workflow is especially valuable when steps involve computationally expensive operations like model training, where re-running the entire pipeline for each code change wastes significant time and compute resources.

2

Use the metaflow-dev VSCode/Cursor extension to bind spin and run to keyboard shortcuts (Ctrl+Opt+S and Ctrl+Opt+R respectively), enabling a seamless code-edit-test cycle that rivals notebook interactivity while maintaining production-grade code structure.

The extension automatically saves files before spinning, and combined with the local card viewer in VSCode's built-in web-view, it creates a particularly powerful workflow for developing Metaflow Cards visualizations and dashboards.

3

Leverage spin's artifact injection capability (--artifacts-module) to test steps with different input configurations without modifying flow code. This enables testing inference steps with different models from separate runs or verifying behavior with edge-case data.

Create a Python module with an ARTIFACTS dictionary to override step inputs, then use the --persist flag and inspect_spin to examine outputs, enabling comprehensive testing scenarios without polluting the main Metaflow datastore.

4

Integrate spin into your unit testing strategy using the Runner API programmatically. This allows you to write assertions against individual step outputs, creating focused tests that verify step-level behavior in isolation from the rest of the workflow.

The Runner API's spin method with persist=True stores artifacts locally, and the task accessor lets you assert specific artifact values, making it straightforward to add step-level testing to CI/CD pipelines.

5

Configure AI coding agents (like Claude Code) with CLAUDE.md instructions that describe the spin-based incremental development workflow, including the exact command syntax for run and spin. This nudges agents toward faster iteration loops and helps them surface and fix errors more efficiently.

Agents don't naturally understand execution speed trade-offs, so explicit instructions to use spin for testing individual steps instead of running the entire flow can significantly reduce the agent's development loop time and improve error recovery.

6

Use Metaflow's three complementary execution modes strategically: run for full end-to-end execution and artifact persistence, resume for re-executing from a failure point or changed step onward, and spin for fast throw-away iterations during active development of individual steps.

Understanding the distinction between these modes is critical because run and resume create full versioned runs with metadata, while spin deliberately skips tracking for speed — making each mode appropriate for different stages of the development lifecycle.