Evolution of GitHub’s data centers

Sam Kottler

Over the past 18 months we’ve made a significant investment in GitHub’s physical infrastructure. The goal of this work is to improve the redundancy and global availability of our system.…

GitHub

•

Sam Kottler

•6 min read•beginner•

--

•View Original

Git

Overview

The article discusses GitHub's significant investments in physical infrastructure over the past 18 months, focusing on improving redundancy and global availability. It details the design and operation of their data centers and points of presence (POPs), highlighting the infrastructure's scalability and the processes involved in expanding compute and storage capacity.

What You'll Learn

1

How to design a hub and spoke network architecture for data centers

2

Why structured cabling is essential for data center scalability

3

How to efficiently manage hardware deployment using a hardware management system

Prerequisites & Requirements

Understanding of data center operations and networking concepts
Experience with hardware management systems(optional)

Key Questions Answered

What are the key features of GitHub's data center infrastructure?

GitHub operates four facilities, including two data centers and two points of presence (POPs). The data centers are designed for storing customer data and serving requests, while the POPs focus on internet connectivity. This setup allows for high redundancy and global availability, supporting thousands of requests per second.

How does GitHub ensure efficient expansion of compute and storage capacity?

GitHub collaborates with a cabinet integrator to manage the deployment of new cabinets. The process involves pre-generated network configurations and structured cabling, allowing for rapid setup and availability of hardware for engineers on the same day the cabinets are powered up.

What challenges does GitHub face with its points of presence (POPs)?

The POPs face challenges such as limited space and power availability due to high demand. This necessitates separate data centers for storing customer data, as the POPs are primarily focused on connectivity rather than data storage.

Key Statistics & Figures

Data storage capacity

Petabytes of Git data

This capacity supports users of GitHub.com.

Network throughput

100Gb/s

This throughput is achieved across transit, internet exchanges, and private network interfaces.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Cloud Computing

Amazon Web Services

Used for direct connect and private network interfaces.

Networking

Dense Wavelength Division Multiplexing (dwdm)

Used to connect POPs and data centers.

Hardware Management System

Gpanel

Used for mapping serial numbers to rack units and managing hardware configurations.

Key Actionable Insights

1
Implementing a hub and spoke design for your network can enhance redundancy and performance.
This design allows for more consistent latencies and throughput, which is crucial for high-demand applications.

2
Utilizing structured cabling can significantly streamline the process of adding new capacity in data centers.
Structured cabling simplifies connections and reduces setup time, enabling faster deployment of new hardware.

3
Regularly updating your hardware management system can improve operational efficiency.
Keeping accurate records of hardware configurations helps in quick troubleshooting and maintenance.

Common Pitfalls

1

Underestimating the demand for space and power in POP facilities can lead to operational challenges.

This can result in difficulties in deploying high-density storage and compute cabinets, impacting overall service availability.

Related Concepts

Data Center Design Principles

Network Architecture

Hardware Management Systems