GitHub uses MySQL to store vast amounts of relational data. This is the story of how we seamlessly upgraded our production fleet to MySQL 8.0.
Overview
The article discusses GitHub's upgrade of its MySQL infrastructure from version 5.7 to 8.0, detailing the motivations, preparation, upgrade plan, and challenges faced during the process. It emphasizes the importance of maintaining service level objectives (SLOs) and the collaborative effort required across multiple teams.
What You'll Learn
1
How to upgrade a large-scale MySQL infrastructure while maintaining service availability
2
Why it's crucial to have rollback capabilities during database upgrades
3
How to ensure application compatibility with new database versions through CI testing
Prerequisites & Requirements
- Understanding of MySQL database management and version compatibility
- Familiarity with GitHub Projects for tracking upgrades(optional)
Key Questions Answered
What motivated GitHub to upgrade to MySQL 8.0?
GitHub upgraded to MySQL 8.0 to ensure they receive the latest security patches, bug fixes, and performance enhancements, especially as MySQL 5.7 was nearing its end of life. New features in 8.0, such as Instant DDLs and invisible indexes, were also key motivations.
How did GitHub ensure application compatibility during the upgrade?
GitHub added MySQL 8.0 to their Continuous Integration (CI) environment to run alongside MySQL 5.7. This allowed them to identify regressions and incompatibilities early, ensuring that applications would function correctly with the new version before the upgrade.
What challenges did GitHub face during the MySQL upgrade?
Challenges included handling replication delays, ensuring backward compatibility for applications, and addressing bugs that were specific to MySQL 8.0. They also faced issues where queries that passed CI tests would fail in production due to real-world workloads.
What was the upgrade strategy for MySQL 8.0?
The upgrade strategy involved a gradual rollout starting with rolling replica upgrades, updating replication topology, promoting an 8.0 host to primary, upgrading internal instances, and finally cleaning up the old 5.7 servers after validation. This approach allowed for monitoring and rollback capabilities.
Key Statistics & Figures
Number of MySQL hosts upgraded
1200+
GitHub upgraded its entire fleet of over 1200 MySQL hosts to version 8.0.
Data stored
300+ TB
GitHub's MySQL infrastructure stores over 300 terabytes of data.
Queries served per second
5.5 million
The MySQL infrastructure serves approximately 5.5 million queries per second across its database clusters.
Number of database clusters
50+
GitHub operates over 50 database clusters as part of its MySQL infrastructure.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Database
Mysql
MySQL is the core relational database used for GitHub's infrastructure.
Database
Vitess
Vitess is used for horizontally sharding relational data within GitHub's MySQL clusters.
Tool
Orchestrator
Orchestrator is used to manage MySQL replication and facilitate failover during the upgrade process.
Tool
Percona Toolkit
Percona Toolkit is part of the ecosystem used to operate GitHub's MySQL fleet.
Tool
Gh-ost
gh-ost is used for online schema migrations in GitHub's MySQL infrastructure.
Tool
Freno
freno is used to throttle write workloads based on replication lag.
Key Actionable Insights
1Implement a gradual upgrade strategy for database systems to minimize risks.A gradual rollout allows for monitoring and quick rollbacks if issues arise, ensuring that service levels are maintained throughout the upgrade process.
2Utilize Continuous Integration (CI) to test application compatibility with new database versions.Running both old and new database versions in CI helps identify potential issues before they affect production, reducing the likelihood of downtime.
3Maintain sufficient replicas of the previous database version to facilitate rollbacks.Having enough 5.7 replicas online during the upgrade ensures that production traffic can be served without disruption if the new version encounters issues.
Common Pitfalls
1
Failing to maintain backward compatibility during a database upgrade can lead to service disruptions.
If the new version introduces changes that break compatibility with existing applications, it can result in downtime. Ensuring that applications are tested against both versions can help mitigate this risk.
2
Not having sufficient rollback mechanisms in place can complicate recovery from upgrade failures.
Without enough replicas of the previous version, rolling back can become difficult or impossible, leading to extended outages. It's crucial to plan for rollback capabilities ahead of time.
Related Concepts
Database Upgrade Strategies
Mysql Version Compatibility
High Availability In Database Systems
Continuous Integration For Database Testing