Overview
This article discusses the open sourcing of kubenetmon, a tool developed by ClickHouse to monitor data transfer in ClickHouse Cloud. It outlines the challenges faced in understanding network behavior across multiple cloud providers and the specific goals achieved through the implementation of kubenetmon.
What You'll Learn
1
How to monitor data transfer in a multi-cloud environment using kubenetmon
2
Why understanding network behavior is crucial for cloud cost management
3
How to implement connection data collection using conntrack
Prerequisites & Requirements
- Understanding of Kubernetes and cloud networking concepts
- Familiarity with Go programming language(optional)
Key Questions Answered
What are the main goals of implementing kubenetmon?
The main goals of implementing kubenetmon include achieving forensics for individual L3/L4 connections, attributing connections to specific workloads and endpoints, and metering data transfer for cost-sensitive use cases. This structured approach helps in understanding network behavior and managing cloud costs effectively.
How does kubenetmon collect connection data?
Kubenetmon collects connection data using the conntrack feature of the Linux networking stack, which tracks the state of connections and can provide byte and packet counts for each connection. This data is scraped periodically to monitor network activity effectively.
What challenges did ClickHouse face in monitoring network behavior?
ClickHouse faced challenges in monitoring network behavior due to the complexity of data transfer charges across multiple cloud providers and regions. Understanding how different workloads interacted and the associated costs was difficult, necessitating a tailored solution like kubenetmon.
What were the limitations of existing solutions for network monitoring?
Existing solutions like Cilium Hubble and cloud provider flow logs were found lacking in providing both volumetric data and connection context. They either offered limited insights or were expensive, prompting ClickHouse to develop their own solution.
Key Statistics & Figures
Number of connection observations recorded per minute
up to a few million
This volume demonstrates the effectiveness of kubenetmon in high-traffic environments.
Compression ratio of stored data
0.0306
This indicates that nearly 400 billion rows of data were compressed from 86.69 TiB down to 2.66 TiB.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Orchestration
Kubernetes
Used for managing containerized applications in the cloud.
Database
Clickhouse
Used for storing and analyzing the connection data collected by kubenetmon.
Programming Language
Go
Used to implement the kubenetmon server and agent.
Networking
Conntrack
Used for tracking the state of network connections in Linux.
Key Actionable Insights
1Implementing a custom network monitoring tool like kubenetmon can provide deeper insights into data transfer costs and behaviors.By understanding individual connection behaviors, organizations can make informed decisions about resource allocation and cost management in cloud environments.
2Using conntrack for connection data collection can simplify the monitoring process while providing essential metrics.This approach leverages existing Linux capabilities, reducing the need for complex external solutions and allowing for efficient data tracking.
3Adopting a hub-and-spoke architecture for data processing can improve performance and scalability of monitoring solutions.By separating data collection and processing responsibilities, teams can optimize resource usage and maintain low memory footprints.
Common Pitfalls
1
Over-reliance on existing cloud provider tools can lead to incomplete data insights.
Many tools lack the granularity needed for effective monitoring, which can result in missed opportunities for cost savings and performance optimization.
2
Using polling instead of informers for Kubernetes API interactions can lead to performance issues.
Polling can overload the API and consume unnecessary resources, while informers provide a more efficient way to manage state.
Related Concepts
Cloud Networking
Data Transfer Monitoring
Cost Management In Cloud Environments