Kafka Producer Pipeline for Ruby on Rails

In the early fall our infrastructure team was considering Kafka, a highly available message bus. We were looking to solve several infrastructure problems that had come up around that time. We were looking for a reliable way to collect event data and send it to our data warehouse. We were considering a more service-oriented architecture, and needed a standardized way of message passing between the components. We were starting to evaluate containerization of Shopify, and were searching for a way to get logs out of containers. We were intrigued by Kafka due to its highly available design. However, Kafka runs on the JVM, and its primary user, LinkedIn, runs a full JVM stack. Shopify is mainly Ruby on Rails and Go, so we had to figure out how to integrate Kafka into our infrastructure.

Shopify logo
Shopify
•
Simon Eskildsen
•4 min read•advanced•
--
•View Original
DockerRailsRuby

Overview

The article discusses the integration of Kafka into Shopify's Ruby on Rails infrastructure to address various infrastructure challenges, including event data collection and service-oriented architecture. It outlines the decision-making process, the use of SysV message queues for reliable event delivery, and the transition to Docker containers while maintaining Kafka's operational integrity.

What You'll Learn

1

How to integrate Kafka with Ruby on Rails applications

2

Why using SysV message queues can enhance reliability in event-driven architectures

3

How to manage event data during Kafka downtime without data loss

Prerequisites & Requirements

  • Understanding of message queuing systems and event-driven architecture
  • Familiarity with Docker and containerization concepts(optional)

Key Questions Answered

How does Shopify integrate Kafka with Ruby on Rails?
Shopify integrates Kafka by using a Go producer that sits alongside its Ruby on Rails application. Events are delivered to Kafka through SysV message queues, which allow for reliable event delivery even during Kafka downtime, ensuring no data loss.
What are the advantages of using SysV message queues over POSIX message queues?
SysV message queues were chosen because they are supported on OS X, which does not implement POSIX message queues. They provide a system-level buffer that allows either end of the queue to restart at any time, simplifying deployment logic and enhancing reliability.
What challenges did Shopify face when adopting Kafka?
Shopify faced challenges such as a lack of experience with Kafka, concerns about data loss during downtime, and the complexity of directly linking Ruby with Kafka. They needed a solution that allowed for reliable message passing without overwhelming the system during Kafka outages.
How does Docker impact the Kafka producer setup at Shopify?
Docker isolates the IPC namespace, preventing containers from sharing SysV queues. This necessitated running a Kafka producer on the host and using a TCP to SysV MQ proxy to allow containers to send events over the network, maintaining operational efficiency.

Key Statistics & Figures

Event processing rate
thousands of events per second
This rate reflects the current performance of Shopify's Kafka producer setup.
Weekly event volume
billions of events per week
This statistic highlights the scale at which Shopify's infrastructure operates.
Queue capacity
sufficient to store two hours worth of events
This capacity ensures that even during Kafka downtimes, events can be buffered without loss.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Message Bus
Kafka
Used for collecting and processing event data within Shopify's infrastructure.
Inter-process Communication
Sysv Message Queues
Facilitates reliable event delivery between Ruby on Rails and the Go producer.
Containerization
Docker
Used to containerize Shopify's applications while managing IPC challenges.
Programming Language
Go
Used to build the Kafka producer that interfaces with the Ruby on Rails application.
Programming Language
Ruby
Main language used for Shopify's application development.

Key Actionable Insights

1
Consider using SysV message queues for reliable event delivery in your applications, especially when dealing with potential downtime.
SysV message queues allow for a robust buffer that can handle events even if the main processing system is temporarily unavailable, ensuring data integrity.
2
When containerizing applications, evaluate how IPC mechanisms like SysV queues will function in your architecture.
Understanding the limitations of IPC in containerized environments can help prevent operational issues and ensure seamless communication between services.
3
Explore the use of TCP proxies to facilitate communication between isolated services and centralized processing systems.
Using TCP proxies can simplify the architecture while maintaining the benefits of isolation provided by containers, allowing for efficient event handling.

Common Pitfalls

1
Directly linking Ruby applications to Kafka can lead to operational challenges, especially during Kafka downtimes.
This approach risks losing data if the application cannot handle Kafka's unavailability, making it crucial to implement a buffering mechanism.
2
Running a Kafka producer inside each container can create an overwhelming number of connections and complicate deployment.
This can lead to performance issues and operational complexity, especially in environments with many containers starting and stopping frequently.

Related Concepts

Event-driven Architecture
Message Queuing Systems
Containerization Best Practices
High Availability Systems