An extraordinary talk on developing systems software in the large
Overview
This article explores the unique challenges of developing systems software at large scale, using Oxide Computer's software update system as a case study. It discusses how systems software—the foundational layer that abstracts hardware complexity—becomes exponentially harder when combined with 'programming in the large,' involving many components built by multiple people over extended periods. The piece introduces Dave Pacheco's OxCon talk on leading Oxide's multi-year software update project, which must handle distributed system updates across air-gapped environments.
What You'll Learn
1
Why systems software in the large represents the most challenging category of software development
2
How software update for distributed rack-scale systems constitutes a 'wicked problem'
3
Why air-gapped update delivery creates unique constraints compared to public cloud environments
4
How organizational procrastination and scope creep threaten large systems software projects
Prerequisites & Requirements
- Understanding of systems software concepts and the distinction between application and infrastructure software
- Familiarity with distributed systems and the challenges of updating software across multiple nodes
- Experience working on large-scale software projects with multiple contributors(optional)
Key Questions Answered
What is systems software in the large and why is it so difficult?
Systems software in the large is the intersection of two orthogonal challenges: building systems software (which abstracts hardware details and demands perfection) and programming in the large (software built by many people across many modules over extended time). The combination creates the most grueling software projects because the low-level correctness demands of systems software multiply with the coordination and complexity challenges of large-scale development.
Why is software update for the Oxide rack considered a wicked problem?
Software update for the Oxide rack qualifies as a wicked problem because it requires dynamically overhauling a distributed system while remaining operable in the liminal state between old and new software. It must work across air gaps without relying on operators or runbooks, and the problem wasn't fully understood until implementation was well underway. It combines distributed systems complexity with the constraint of self-service, autonomous operation.
What does air-gapped software update mean for rack-scale infrastructure?
Air-gapped software update means delivering updates to systems that have no connection to the public internet, which eliminates the public cloud's hidden crutch of operators and runbooks. Oxide's rack must be able to update all its software in the field without any external connectivity, requiring the update mechanism to be completely self-contained and autonomous. This is a constraint not faced by public cloud providers.
What is organizational procrastination in systems software projects?
Organizational procrastination, a term coined by Dave Pacheco in his OxCon talk, refers to the tendency of teams working on large systems software projects to defer the hardest problems in favor of easier or more tractable work. This is distinct from individual procrastination—it manifests at the team level when focus drifts toward less critical but more comfortable tasks, delaying progress on the core wicked problem.
What is the difference between functional and robust systems software?
A system that is merely functional can be deceptively distant from the robustness required of foundational software. Systems software demands perfection because it provides the abstraction layer that all other programs depend upon. The gap between 'it works' and 'it works reliably under all conditions' is enormous in systems software, where bugs in the foundation cascade upward through every layer that depends on it.
How did Oxide approach shipping their first software update mechanism?
When Oxide shipped their first rack two years ago, it included the minimum functionality necessary to update all software in the field. Their priority was making the update mechanism robust above all else, and they succeeded in achieving robustness. However, the experience was not yet the seamless, self-service facility they envisioned, requiring continued multi-year development to reach their goals.
Key Statistics & Figures
Software update project duration
2+ years
Dave Pacheco has been leading the software update project at Oxide for the past two years
Number of company-specific adapters
21
Oxide has 21 company-specific content adapters in their system
Technologies & Tools
Infrastructure
Oxide Rack
The rack-scale computer system that requires distributed software updates across air-gapped environments
Key Actionable Insights
1Prioritize robustness over user experience in the first version of critical infrastructure software. Oxide deliberately shipped a minimal but robust update mechanism rather than a polished but fragile one, accepting that the user experience would need improvement later.This applies when building foundational systems where failure is catastrophic. Getting correctness right first and improving ergonomics later is a proven strategy for infrastructure software.
2Guard against organizational procrastination by maintaining focus on the hardest problems. Teams naturally gravitate toward tractable work and away from wicked problems, which can delay the most critical deliverables without anyone consciously deciding to deprioritize them.This is especially relevant for multi-year projects with many contributors where it's easy for the team's attention to drift toward more comfortable tasks.
3Actively fight scope creep in systems software projects. Large systems software has a natural tendency to expand in scope because the problems are interconnected and not fully understood at the outset, making it tempting to solve adjacent problems simultaneously.Dave Pacheco specifically called out scope creep as a key challenge in leading the software update project at Oxide, suggesting that discipline around scope is critical for delivering on ambitious timelines.
4Design update systems to operate in the liminal state between old and new software versions. The hardest part of distributed software update is not deploying new code but maintaining operability while the system is partially updated.This applies to any distributed system that cannot tolerate downtime during updates. The transitional state between versions is where most update failures occur.
5Eliminate dependencies on human operators and runbooks for critical infrastructure operations. Oxide's air-gap constraint forced them to build a fully autonomous update system, which ultimately produces a more reliable system than one that relies on operator intervention.Even if your system isn't air-gapped, designing as if it were forces you to automate recovery paths and error handling that would otherwise be left to manual procedures.
Common Pitfalls
1
Assuming a functional system is the same as a robust system. In systems software, merely getting something to work is deceptively far from the level of reliability required for foundational infrastructure. The gap between 'works in testing' and 'works under all production conditions' is enormous.
This is especially dangerous because the distance between functional and robust is not visible until edge cases manifest in production, making it easy to underestimate remaining work.
2
Underestimating the complexity of the transitional state during software updates. The challenge isn't just deploying new software—it's maintaining system operability while parts of the distributed system run old software and parts run new software simultaneously.
Many update system designs focus on the end state (everything updated) and neglect the intermediate states, which is where the real complexity and failure modes live.
3
Relying on operator intervention as a hidden crutch for handling edge cases in distributed systems. Public clouds have large operations teams that can execute runbooks when automated systems fail, but this dependency is often invisible in system design.
Designing for air-gapped or self-service operation forces you to confront and solve problems that are otherwise masked by human operators.
4
Treating wicked problems as if they can be fully specified upfront. Wicked problems in software are not only never completely solved but also not even really understood until implementation is well underway, so waterfall-style planning will inevitably miss critical requirements.
Accept that your understanding of the problem will evolve significantly during implementation, and structure your project to accommodate this learning.
Related Concepts
Programming In The Large
Wicked Problems
The Mythical Man-month
No Silver Bullet
Distributed Systems Update
Air-gapped Deployment
Scope Creep Management
Software Abstraction Layers
Rack-scale Computing
Infrastructure Software Robustness