Overview
The article discusses Cloudflare's migration of its logging pipeline from syslog-ng to OpenTelemetry Collector, detailing the motivations behind the shift, the migration process, and the lessons learned. It highlights the benefits of using OpenTelemetry, such as improved telemetry data collection and easier contributions from the engineering team.
What You'll Learn
1
How to migrate a logging pipeline from syslog-ng to OpenTelemetry Collector
2
Why using OpenTelemetry improves telemetry data collection
3
How to implement custom components in OpenTelemetry Collector
Prerequisites & Requirements
- Understanding of logging systems and telemetry
- Familiarity with OpenTelemetry and its components(optional)
- Experience with Go programming language(optional)
Key Questions Answered
What were the reasons for migrating from syslog-ng to OpenTelemetry Collector?
The migration was motivated by the need for a more manageable codebase, improved telemetry data collection, and the ability for more team members to contribute due to OpenTelemetry being written in Go. Additionally, it allowed for unification of telemetry types under one daemon, reducing complexity.
What components were built for the OpenTelemetry Collector during the migration?
Four internal components were developed: cfjs1exporter for custom log formats, fileexporter for additional file formats, externaljsonprocessor for adding external data to logs, and a ratelimit processor to manage log message rates. These components addressed specific needs not met by existing solutions.
How did Cloudflare handle the migration process to minimize downtime?
Cloudflare implemented a systemd 'one shot' service to simultaneously stop syslog-ng and start the OpenTelemetry Collector, minimizing the gap during the transition. This approach helped avoid issues with log consumption during the migration.
What lessons were learned regarding failover during the migration?
During chaos testing, the cfjs1 exporter failed to switch to a secondary logging server when the primary server was unreachable, leading to log buffer overflow. This highlighted the need for tighter timeouts and better handling of partial failures in the logging system.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Opentelemetry Collector
Used for logging and telemetry data collection.
Backend
Syslog-ng
Previous logging system replaced by OpenTelemetry Collector.
Programming Language
Go
Language used for developing the OpenTelemetry Collector and custom components.
Key Actionable Insights
1Implement a unified logging system using OpenTelemetry to streamline telemetry data collection across services.This approach reduces cognitive load on engineering teams and allows for better insights into system performance, making it easier to manage and analyze logs.
2Develop custom components in OpenTelemetry to meet specific organizational needs, such as unique log formats or external data integration.Custom components can enhance the functionality of the OpenTelemetry Collector, allowing for tailored solutions that fit your infrastructure requirements.
3Use systemd services for coordinated service management during migrations to minimize downtime and avoid log loss.This method ensures that services are stopped and started in a controlled manner, preventing gaps in log collection during transitions.
Common Pitfalls
1
Failing to implement adequate failover mechanisms can lead to log loss during outages.
This issue arises when the logging system does not switch to a backup server in case of primary server failure, causing log buffers to fill up and services to block.
2
Not coordinating service stops and starts during migration can result in lost logs.
If services are stopped and started separately, there can be a gap where logs are not consumed, leading to potential data loss.
Related Concepts
Opentelemetry Best Practices
Logging Pipeline Architecture
Telemetry Data Collection Strategies