Shipping Logs

Nearly all of our apps are puking output. Sometimes, it’s intentional. Often this output is in the form of structured logs. Logs are helpful for a variety of use cases - debugging, tracking, collating, correlating, coalescing, and condensing the ha

Fly.io logo
Fly.io
•
Chris Fidao
•5 min read•beginner•
--
•View Original
DatadogDockerGolangNATSRust

Overview

The article discusses the importance of centralizing logs for applications running on Fly.io, detailing the process of shipping logs using the Fly Log Shipper and NATS. It emphasizes the benefits of log aggregation for correlation and retention, while providing a step-by-step guide for developers to implement logging in their applications.

What You'll Learn

1

How to centralize logs from multiple applications on Fly.io

2

Why log aggregation is essential for debugging and tracking events

3

How to configure the Fly Log Shipper to send logs to various sinks

4

When to use NATS for log streaming in distributed applications

Prerequisites & Requirements

  • Basic understanding of logging concepts and distributed systems
  • Familiarity with Fly.io and Vector(optional)

Key Questions Answered

How does Fly.io handle log aggregation for applications?
Fly.io aggregates logs by capturing stdout from applications running in Firecracker VMs. An init process gathers the output and sends it to a socket, where a Golang service forwards it to Vector, which then ships the logs to a NATS cluster for further processing.
What is the purpose of the Fly Log Shipper?
The Fly Log Shipper is an application that configures a Vector sink to send logs to various destinations like Loki, Datadog, or Cloudwatch. It allows developers to easily manage and ship logs from their applications to their preferred log aggregation service.
How can developers filter logs for specific applications?
Developers can filter logs by setting a SUBJECT environment variable in the fly.toml file, which allows them to specify which application's logs to ship. This can be done using wildcards to capture logs from specific applications across different regions and instances.
What are the benefits of centralizing logs?
Centralizing logs provides significant benefits such as correlation, allowing developers to track events across services, and retention, ensuring logs are stored for future reference. This is crucial for debugging and understanding application behavior.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Platform
Fly.io
Used to run applications globally and manage their logs.
Logging Tool
Vector
Handles the shipping of logs to various destinations.
Messaging System
Nats
Acts as a pub/sub service for log data.
Virtualization
Firecracker
Used to run lightweight VMs for applications.

Key Actionable Insights

1
Implementing centralized logging can significantly improve your debugging process by allowing you to correlate events across multiple services.
When applications generate a large volume of logs, having them in a centralized location makes it easier to identify issues and understand interactions between services.
2
Using the Fly Log Shipper simplifies the process of configuring log sinks for various services, enabling you to quickly adapt to different logging needs.
As your application evolves, you may need to change where your logs are sent. The Fly Log Shipper's flexibility allows for easy adjustments without significant overhead.
3
Setting the SUBJECT environment variable allows for targeted log shipping, which can reduce noise and focus on relevant logs.
In environments with multiple applications, filtering logs ensures that you only receive the information necessary for your current debugging or monitoring tasks.

Common Pitfalls

1
Failing to set the correct SUBJECT environment variable can lead to missing logs from specific applications.
Without the proper configuration, you might end up receiving logs from all applications, making it difficult to focus on the issues at hand.
2
Not retaining logs long enough can hinder debugging efforts when issues arise after a significant delay.
Logs that are not stored for an adequate period may be lost when you need them most, so it's crucial to implement a retention strategy.

Related Concepts

Log Aggregation
Centralized Logging
Distributed Systems
Event Correlation