Overview
This article provides a comprehensive guide on how to send Windows Event Logs to ClickHouse using Fluent Bit, focusing on threat-hunting use cases. It covers the setup process, configuration of Sysmon for enhanced logging, and visualization of logs using Grafana.
What You'll Learn
1
How to set up Fluent Bit for collecting Windows Event Logs
2
How to configure Sysmon for advanced event tracing
3
How to visualize Windows Event Log data using Grafana
Prerequisites & Requirements
- Basic understanding of Windows Event Logs and Fluent Bit
- Fluent Bit v2.0.6 and Sysmon v14.13
- Familiarity with PowerShell and SQL(optional)
Key Questions Answered
How can I send Windows Event Logs to ClickHouse?
You can send Windows Event Logs to ClickHouse by using Fluent Bit to collect the logs and configure it to forward them to your ClickHouse instance. This involves setting up Fluent Bit with the appropriate input, filter, and output configurations.
What is Sysmon and how does it help in security monitoring?
Sysmon is a Windows system service that logs system activity to the Windows Event Log, providing detailed information about process creations, network connections, and more. It enhances security monitoring by allowing security engineers to detect and analyze malicious activities.
What are the steps to visualize Windows Event Log data in Grafana?
To visualize Windows Event Log data in Grafana, you need to configure Grafana to connect to your ClickHouse instance using the ClickHouse plugin. After setting up the data source, you can create dashboards to display various metrics from the logs.
What is the performance of ClickHouse when handling Windows Event Logs?
ClickHouse is capable of handling high volumes of logs efficiently. In testing, the setup achieved around 3000 events per second on a small Windows VM, indicating that ClickHouse and Fluent Bit can scale to millions of events per second under optimal conditions.
Key Statistics & Figures
Events processed per second
3000 eps
This was achieved during performance testing on a small Windows 2022 VM.
Compression ratio for Windows Event Logs
1/14
This indicates that ClickHouse is efficient in storing logs, requiring only 200 bytes for a Windows Event Log.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Data Collector
Fluent Bit
Used for collecting and forwarding Windows Event Logs to ClickHouse.
Security Monitoring Tool
Sysmon
Enhances Windows Event Logging by providing detailed system activity logs.
Database
Clickhouse
Acts as the backend for storing and querying the Windows Event Logs.
Data Visualization
Grafana
Used for creating dashboards to visualize the collected log data.
Key Actionable Insights
1Setting up Fluent Bit for Windows Event Log collection is straightforward and can significantly enhance your security monitoring capabilities.By configuring Fluent Bit to collect and forward logs to ClickHouse, you can leverage powerful analytics and visualization tools to detect anomalies and threats in your environment.
2Utilizing Sysmon with a community configuration can provide richer event data for security analysis.Applying a high-quality Sysmon configuration aligned with the MITRE ATT&CK matrix allows for better detection of sophisticated threats, making it a valuable addition to your security toolkit.
3Visualizing logs in Grafana can help you quickly identify trends and anomalies in your Windows Event Logs.Creating dashboards in Grafana allows for real-time monitoring and analysis of log data, which is crucial for proactive threat hunting and incident response.
Common Pitfalls
1
Failing to configure the ClickHouse database correctly can lead to data insertion issues.
Ensure that the JSON object type is enabled and that the database schema matches the expected log structure to avoid errors during data ingestion.
2
Not optimizing the schema for production can result in poor query performance.
It's important to define the expected columns and use appropriate data types to enhance performance and reduce I/O costs.
Related Concepts
Threat Hunting
Log Analysis
Cloud Native Observability
Data Visualization With Grafana