How we improved push processing on GitHub

Will Haltom

Pushing code to GitHub is one of the most fundamental interactions that developers have with GitHub every day. Read how we have significantly improved the ability of our monolith to correctly and fully process pushes from our users.

GitHub

•

Will Haltom

•8 min read•advanced•

--

•View Original

GitHub ActionsMySQLRailsRuby

Overview

The article discusses improvements made to GitHub's push processing system, which previously relied on a single, complex background job. By decoupling this process into multiple parallel jobs using Kafka, GitHub has enhanced reliability, reduced latency, and improved observability in handling push events.

What You'll Learn

1

How to implement a decoupled architecture using Kafka for event processing

2

Why isolating processing tasks reduces the risk of system-wide failures

3

How to improve latency in background job processing by parallelizing tasks

Key Questions Answered

What were the main issues with the previous push processing system on GitHub?

The previous system relied on a single, large background job, which was difficult to retry, tightly coupled, and caused latency issues. This led to failures in processing crucial tasks and increased the risk of errors impacting the entire job.

How did GitHub improve the push processing architecture?

GitHub improved the architecture by decoupling the push processing tasks into isolated, parallel jobs using Kafka. This allowed for better retry configurations, reduced latency, and improved reliability in processing pushes.

What are the results of the new push processing system?

The new system has a smaller blast radius for problems, lower latency in processing pushes, improved observability, and a higher reliability rate, with the new pipeline fully processing 99.999% of pushes compared to 99.897% previously.

Key Statistics & Figures

Percentage of pushes fully processed

99.999%

This is an improvement from the previous system, which fully processed 99.897% of pushes.

Daily push processing operations

300 million

This reflects the scale of operations handled by the new pipeline.

Pull request sync time

Dropped from around 3 seconds to around 2 seconds

This reduction in sync time demonstrates the improved latency of the new system.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Backend

Kafka

Used for publishing events in response to pushes and facilitating the decoupling of processing tasks.

Key Actionable Insights

1
Implementing a decoupled architecture can significantly enhance system reliability.
By isolating tasks, you reduce the risk of one failure affecting multiple processes, which is crucial for maintaining uptime in high-traffic applications.

2
Utilizing Kafka for event-driven architectures can streamline background job processing.
Kafka allows for efficient handling of events, enabling parallel processing that can drastically reduce latency and improve user experience.

3
Investing in observability tools is essential for monitoring complex systems.
With a clearer view of each job's performance, teams can quickly identify bottlenecks and address issues before they escalate.

Common Pitfalls

1

Relying on a single, monolithic job for processing can lead to significant system failures.

This happens because errors in one part of the job can cascade and affect all downstream processes, making it difficult to isolate and fix issues.

2

Not configuring appropriate retry strategies for different tasks can result in missed processing.

Tasks like sending webhooks are time-sensitive and require careful handling to avoid duplicate or delayed notifications.