Multiple Logs for Resiliency

Sam Ruby

You’ve done everything right. You are well aware of Murphy’s Law. You have multiple redundant machines. You’ve set up a regular back up schedule for your database, perhaps even are using LiteFS CLoud. You ship your logs to LogTail or perhaps some o

Fly.io

•

Sam Ruby

•5 min read•beginner•

--

•View Original

ElixirJavaScriptJSONLaravelNATSRailsRuby

Overview

The article discusses the importance of maintaining multiple logs for application resiliency, particularly in the face of unexpected network outages. It outlines strategies for implementing redundant logging systems to ensure critical log data is preserved and accessible during failures.

What You'll Learn

1

How to implement redundant logging in a Rails application

2

Why using a custom log shipper can improve log reliability

3

How to set up NATS for log subscription and processing

Prerequisites & Requirements

Familiarity with logging frameworks and practices
Access to NATS and a logging provider
Experience with application development and deployment

Key Questions Answered

What strategies can be used for redundant logging?

The article outlines two primary strategies for redundant logging: writing logs to separate files on each machine and creating a custom log shipper that directly subscribes to NATS. These methods ensure that logs remain accessible even during network failures.

How can I implement logging to multiple places in my application?

To implement logging to multiple places, you can configure your application to write logs to both STDOUT and a separate log file on a volume. This ensures that logs are preserved even if the application restarts, and you can use frameworks like Rails to manage this setup.

What are the benefits of using a custom log shipper?

Using a custom log shipper allows for more immediate access to log data and can be tailored to specific needs. By subscribing directly to NATS, you can process log entries in real-time, which can be more efficient than relying on third-party services.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Messaging

Nats

Used for capturing and processing log entries in real-time.

Logging

Activesupport Logger

Used in Rails applications for managing log output.

Logging

Vector

Used as part of the logging pipeline to ship logs to third-party providers.

Key Actionable Insights

1
Implementing redundant logging can significantly reduce the risk of losing critical log data during outages.
By ensuring that logs are stored in multiple locations, you can access necessary information even when primary logging mechanisms fail, which is crucial for troubleshooting and maintaining application health.

2
Using a custom log shipper can streamline your logging process and improve data reliability.
This approach allows you to directly manage log data flow, reducing dependencies on external services and enhancing the speed of log processing.

3
Regularly rotating log files is essential to prevent storage issues.
As logs grow in size, they can consume significant disk space. Implementing a log rotation strategy ensures that your logging system remains efficient and does not run out of storage.

Common Pitfalls

1

Failing to set up log rotation can lead to excessive disk usage.

Without log rotation, log files can grow indefinitely, potentially filling up disk space and causing application failures. It's crucial to implement a strategy for managing log file sizes.

2

Not having a backup logging system can result in data loss during outages.

If your primary logging mechanism fails and you lack a backup, critical log data may be lost forever. Implementing a redundant logging strategy mitigates this risk.

Related Concepts

Redundant Logging

Log Management

Nats Messaging

Application Resiliency