This is a technical report on three bugs that intermittently degraded responses from Claude. Below we explain what happened, why it took time to fix, and what we're changing.
Overview
The article provides a detailed postmortem of three infrastructure bugs that affected the response quality of Claude between August and early September. It outlines the nature of the issues, their impact on users, and the steps taken to resolve them, emphasizing the importance of maintaining model quality and the complexities involved in serving AI models at scale.
What You'll Learn
1
How to identify and resolve infrastructure bugs in AI systems
2
Why consistent model quality is crucial for user satisfaction
3
How to implement effective monitoring and evaluation processes for AI models
Key Questions Answered
What were the three main infrastructure bugs affecting Claude?
The three main bugs were a context window routing error, output corruption due to a misconfiguration, and an approximate top-k XLA:TPU miscompilation. Each bug had specific impacts on user requests and was resolved through targeted fixes.
How did the context window routing error affect users?
The context window routing error misrouted some Sonnet 4 requests to servers configured for a 1M token context window, affecting up to 16% of requests at its peak. This led to degraded responses for approximately 30% of Claude Code users during the affected period.
What changes are being made to prevent future issues?
To prevent future issues, the team is implementing more sensitive evaluations, continuous quality assessments on production systems, and developing faster debugging tools to better handle community feedback while maintaining user privacy.
When was the output corruption bug identified and resolved?
The output corruption bug was identified on August 25 and was resolved by rolling back the misconfiguration on September 2. This issue caused unexpected characters to appear in responses, affecting requests made to Opus 4.1 and Opus 4.
Key Statistics & Figures
Percentage of requests affected by context window routing error
16%
At the worst impacted hour on August 31, 16% of Sonnet 4 requests were affected due to the routing error.
Percentage of Claude Code users affected by routing issues
30%
Approximately 30% of Claude Code users who made requests during the routing error period experienced degraded responses.
Percentage of misrouted traffic on Amazon Bedrock
0.18%
Misrouted traffic peaked at 0.18% of all Sonnet 4 requests on Amazon Bedrock from August 12.
Percentage of requests affected by output corruption
Not specified
The output corruption bug affected requests made to Opus 4.1 and Opus 4 between August 25 and September 2, but specific percentages were not provided.
Technologies & Tools
AI Model
Claude
Serves responses to users via various platforms including APIs.
Hardware
AWS Trainium
One of the hardware platforms used to serve Claude.
Hardware
Nvidia Gpus
Another hardware platform utilized for serving Claude.
Hardware
Google Tpus
Used for deploying Claude across multiple configurations.
Compiler
Xla:tpu
The optimizing compiler that translates XLA High Level Optimizing language to TPU machine instructions.
Key Actionable Insights
1Implement continuous monitoring for AI systems to catch issues early.Continuous monitoring allows for real-time detection of anomalies and degradation in model performance, enabling quicker responses to user feedback and maintaining service quality.
2Enhance your evaluation processes to differentiate between working and broken implementations.Improved evaluation processes can help identify issues that standard checks might miss, ensuring that any changes made do not negatively impact user experience.
3Encourage user feedback to identify unexpected behavior in AI responses.User feedback is invaluable for diagnosing issues that may not be captured through internal monitoring, providing insights into real-world performance and user satisfaction.
Common Pitfalls
1
Relying too heavily on noisy evaluations can mask underlying issues.
This can lead to a failure in identifying bugs that affect user experience, as seen in the recent incidents where user reports did not correlate with internal evaluations.
2
Inadequate monitoring of user interactions can hinder bug diagnosis.
Privacy practices that limit access to user interactions may prevent engineers from effectively diagnosing and reproducing reported issues, complicating the resolution process.