Adventures in Production Rails Debugging

5 minute read At Shopify we frequently need to debug production Rails problems. Adding extra debugging code takes time to write and deploy, so we’ve learned how to use tools like gdb and rbtrace to quickly track down these issues. In this post, we’ll explain how to use gdb to retrieve a Ruby call stack, inspect environment variables, and debug a really odd warning message in production. We recently ran into an issue where we were seeing a large number of similar warning messages spamming our log files: /artifacts/ruby/2.1.0/gems/rack-1.6.4/lib/rack/utils.rb:92: warning: regexp match /.../n against to UTF-8 string This means we are trying to match an ASCII regular expression on a UTF-8 source string.

Shopify logo
Shopify
•
Scott Francis
•6 min read•intermediate•
--
•View Original
DockerJSONJuliaRailsRuby

Overview

The article discusses effective strategies for debugging production issues in Rails applications at Shopify, focusing on the use of tools like gdb and rbtrace. It highlights a specific case of handling UTF-8 warnings generated by the Rack library and the steps taken to resolve the issue without impacting production traffic.

What You'll Learn

1

How to use gdb to retrieve a Ruby call stack in production

2

Why using breakpoints in gdb can help debug Ruby applications effectively

3

How to suppress Ruby warnings in production without modifying the application code

Prerequisites & Requirements

  • Understanding of Ruby and Rails debugging techniques
  • Familiarity with gdb and rbtrace(optional)

Key Questions Answered

How can gdb be used to debug Ruby applications in production?
gdb can be attached to a running Ruby process to set breakpoints and inspect the call stack. This allows developers to analyze the state of the application without deploying new code, making it a valuable tool for debugging production issues.
What causes UTF-8 warnings in Ruby applications?
UTF-8 warnings occur when an ASCII regular expression is applied to a UTF-8 string. In the article, this was traced back to cookies being parsed that contained non-US-ASCII characters, which triggered the warnings in the log files.
What steps can be taken to suppress Ruby warnings in production?
One effective method is to monkey patch the Rack::Request#cookies method to silence warnings. This allows the application to continue functioning without the noise of excessive warning messages while still addressing the underlying issue later.

Key Statistics & Figures

Warning messages generated per minute
50k
This high volume of warnings was causing significant noise in the log files, prompting the need for effective debugging strategies.

Technologies & Tools

Debugging Tool
Gdb
Used to attach to Ruby processes for debugging and inspecting the call stack.
Debugging Tool
Rbtrace
Considered for tracing method calls but ultimately not used due to Docker configuration limitations.
Ruby Library
Rack
The source of the UTF-8 warning messages during cookie parsing.

Key Actionable Insights

1
Utilize gdb for debugging production issues without deploying new code.
By attaching gdb to an idle Unicorn worker, you can analyze the application state in real-time, which minimizes the risk of impacting user requests.
2
Implement breakpoints strategically to capture relevant stack traces.
Setting breakpoints at critical points in the code allows for detailed inspection of the call stack, helping to identify the source of issues effectively.
3
Consider monkey patching as a temporary solution for suppressing warnings.
While not a permanent fix, monkey patching can reduce log noise and allow developers to focus on critical issues without distraction.

Common Pitfalls

1
Failing to compile Ruby with debug symbols can hinder effective debugging.
Without debug symbols, setting breakpoints in the source code becomes impossible, limiting the ability to inspect the application state during runtime.

Related Concepts

Ruby Debugging Techniques
Production Monitoring Tools
Performance Analysis Strategies