In this deep-dive, we identified and worked through sporadic latency issues with services running on Kubernetes in our environment.
Overview
The article discusses the debugging of network stalls in Kubernetes environments at GitHub, highlighting the challenges faced with sporadic latency affecting service performance. It details the investigative process used to identify the root causes of these stalls, particularly focusing on the interaction between Kubernetes networking and system performance.
What You'll Learn
1
How to diagnose network latency issues in Kubernetes environments
2
Why understanding Linux kernel packet processing is crucial for performance
3
How to identify performance bottlenecks caused by specific processes like cadvisor
Prerequisites & Requirements
- Understanding of Kubernetes networking and Linux kernel operations
- Familiarity with tools like tcpdump, hping3, and bcc(optional)
Key Questions Answered
What are the causes of sporadic latency in Kubernetes services?
Sporadic latency in Kubernetes services can be caused by issues in the network stack, particularly related to how packets are processed by the Linux kernel. The article highlights that stalls can occur due to the interaction between network address translation (NAT) and the processing of packets by specific processes like cadvisor, leading to delays in response times.
How does cadvisor impact network performance in Kubernetes?
Cadvisor can negatively impact network performance by causing delays in processing packets due to excessive CPU usage from slow syscalls. The article details how cadvisor's polling of memory statistics can lead to stalls in packet processing, particularly when many 'zombie' cgroups are present, resulting in high latency.
What methods can be used to trace packet processing in Linux?
To trace packet processing in Linux, tools like bcc can be utilized to hook kernel functions and monitor the processing of packets in real-time. This allows for identifying which processes are causing delays and understanding the flow of packets through the system, particularly in relation to softirqs and ksoftirqd.
What role does the Linux kernel's NAPI play in packet processing?
The Linux kernel's NAPI (New API) improves packet processing efficiency by switching from interrupt-driven to polling methods when packet rates exceed a certain threshold. This helps manage high packet loads more effectively, but can also lead to stalls if the processing of packets is not balanced with user-space tasks.
Key Statistics & Figures
Maximum observed latency
100ms
Certain services were experiencing latency up to and over 100ms, which was unacceptable for the expected response times.
Round-trip time for ICMP packets
over 100ms
ICMP packets were observed with round-trip times exceeding 100ms, indicating significant delays in packet processing.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Orchestration
Kubernetes
Used for deploying and managing services at GitHub.
Monitoring
Cadvisor
Analyzes resource usage and performance characteristics of running containers.
Operating System
Linux Kernel
Handles packet processing and network management.
Networking Tool
Hping3
Used for sending custom TCP/IP packets to diagnose network issues.
Networking Tool
Tcpdump
Used for capturing and analyzing network packets.
Tracing Tool
Bcc
Used for live tracing of the Linux kernel to identify performance issues.
Key Actionable Insights
1Implement regular monitoring of network latency in Kubernetes clusters to identify potential stalls early.By actively monitoring latency, teams can catch issues before they escalate into larger performance problems, ensuring smoother service operation.
2Utilize tools like hping3 and tcpdump to perform targeted network diagnostics.These tools can help isolate network issues by allowing engineers to simulate traffic and analyze packet flow, providing insights into where latency occurs.
3Consider upgrading to newer kernel versions to benefit from performance enhancements.The article mentions that kernel version 4.19+ has improved the performance of memory.stat calls, which can mitigate latency issues related to cadvisor.
Common Pitfalls
1
Failing to monitor network performance can lead to undetected latency issues that affect service reliability.
Without regular monitoring, teams may not notice performance degradation until it impacts users, making proactive measures essential.
2
Over-reliance on tools without understanding underlying issues can result in misdiagnosis.
Tools like cadvisor provide valuable insights, but understanding the context of the data is crucial to avoid incorrect conclusions about performance bottlenecks.
Related Concepts
Kubernetes Networking
Linux Kernel Performance Tuning
Network Address Translation (nat)
Performance Monitoring Tools