Overview
The article discusses the challenges Pinterest faced with their Apache Thrift schemas, which had become tightly coupled and complex, leading to inefficiencies in development cycles. The team refactored these schemas to create a broader and shallower dependency tree, resulting in significant improvements in build times, reduced unnecessary builds, and smaller binary sizes.
What You'll Learn
1
How to refactor Thrift schemas to improve build performance
2
Why a broad and shallow dependency tree enhances developer velocity
3
How to maintain backward compatibility during schema refactoring
Prerequisites & Requirements
- Understanding of Apache Thrift and its schema structure
- Experience with large codebases and dependency management(optional)
Key Questions Answered
What problems did Pinterest face with their Thrift schemas?
Pinterest's Thrift schemas were tightly coupled, causing slow build times, unnecessary builds, and a complex dependency graph that hindered developer velocity. This complexity led to inefficiencies in the development cycle and testing processes.
How did refactoring Thrift schemas improve build times?
Refactoring allowed services to depend on specific Thrift structs instead of entire files, resulting in a broader and shallower dependency tree. This change led to build time improvements of 15–35% across several projects due to reduced complexity.
What was the impact of reducing unnecessary builds?
After refactoring, Pinterest reduced unnecessary builds by 600x by ensuring services only depended on specific Thrift structs that rarely changed, rather than the entire schema file. This efficiency significantly improved developer productivity.
What strategies were used to maintain backward compatibility during refactoring?
The team focused on making namespace changes in languages like Python and Go that did not affect serialized structures, allowing for backward compatibility. They also ensured that changes had no runtime overhead and allowed for incremental rollouts.
Key Statistics & Figures
Build time improvement
15–35%
Observed across several projects after refactoring the Thrift schemas.
Reduction in unnecessary builds
600x
Achieved by changing dependencies from entire Thrift files to specific Thrift structs.
Binary size reduction
22–35%
Resulted from only including necessary schemas instead of all schemas in a file.
Technologies & Tools
Backend
Apache Thrift
Used for defining service interfaces and serialized data across Pinterest's tech stack.
Build Tool
Bazel
Installed to facilitate builds but was not fully utilized due to the complexity of the Thrift schemas.
Key Actionable Insights
1Refactoring your schemas can drastically reduce build times and improve developer efficiency.By breaking down large, tightly coupled Thrift files into smaller, more manageable schemas, teams can streamline their development processes and reduce the complexity of their dependency graphs.
2Maintaining backward compatibility is crucial during schema refactoring.Implementing changes that do not disrupt existing functionality allows for smoother transitions and minimizes the risk of introducing bugs in production environments.
3Utilizing automated tools for refactoring can enhance collaboration across teams.By automating the renaming and restructuring processes, teams can ensure consistency and reduce the manual effort required, enabling more developers to contribute to schema improvements.
Common Pitfalls
1
Failing to maintain backward compatibility during schema refactoring can lead to significant issues.
Changes that break existing functionality can disrupt multiple services and require extensive coordination to resolve, leading to delays and increased risk of errors.
2
Overlooking the impact of large dependencies can result in inefficient builds.
Including large Thrift files can trigger unnecessary builds across the entire codebase, slowing down development and increasing resource consumption.
Related Concepts
Dependency Management In Large Codebases
Refactoring Best Practices
Performance Optimization Techniques