Learn how fast YJIT really is. This posts walks through Shopify's new benchmarking harness for YJIT, plus graphs and benchmarks updated twice a day.
Overview
The article discusses the development of YJIT, a Just-In-Time Compiler for Ruby, and the implementation of a benchmarking harness to measure its performance. It highlights the advantages of using GitHub Actions and GitHub Pages to automate the reporting and visualization of benchmark results.
What You'll Learn
1
How to automate benchmark reporting using GitHub Actions and GitHub Pages
2
Why YJIT is faster than interpreted CRuby and MJIT
3
How to set up a Jekyll site for continuous reporting
4
When to use AWS for controlled benchmarking environments
Prerequisites & Requirements
- Basic understanding of Ruby and benchmarking concepts
- Familiarity with GitHub Actions and GitHub Pages(optional)
Key Questions Answered
How fast is YJIT compared to CRuby and MJIT?
YJIT is 20.6% faster than interpreted CRuby and 17.4% faster than MJIT. Specifically, on Railsbench, YJIT is 18.5% faster than CRuby and 21.0% faster than MJIT, showcasing its performance improvements.
What is the role of GitHub Actions in the benchmarking process?
GitHub Actions automates the running of benchmarks and updates the site without manual intervention. It allows for scheduled tasks similar to cron jobs, ensuring that the benchmark results are consistently up to date.
What challenges might arise when using AWS with GitHub Actions?
Challenges include SSH connectivity issues due to known host key management and the need to pass environment variables correctly. These issues can lead to failures in establishing a connection to the AWS instance from GitHub Actions.
How can graphs be generated for benchmark results?
Graphs can be generated using D3.js for dynamic visualizations or static SVGs for embedding. The choice depends on the desired interactivity and complexity of the visual representation.
Key Statistics & Figures
YJIT performance improvement over CRuby
20.6%
This statistic highlights the speed advantage of YJIT compared to the standard interpreted CRuby.
YJIT performance improvement over MJIT
17.4%
This shows how YJIT outperforms the existing MJIT implementation.
YJIT performance on Railsbench vs CRuby
18.5%
This specific improvement indicates YJIT's efficiency in a Rails environment.
YJIT performance on Railsbench vs MJIT
21.0%
This demonstrates YJIT's significant advantage in performance on Railsbench compared to MJIT.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Yjit
A Just-In-Time Compiler for Ruby that improves performance.
Devops
Github Actions
Automates the benchmarking and reporting processes.
Frontend
Github Pages
Hosts the benchmarking results site.
Frontend
Jekyll
Static site generator used for creating the benchmarking reports.
Frontend
D3.js
Library used for generating dynamic graphs from benchmark data.
Cloud
AWS
Provides controlled environments for running benchmarks.
Key Actionable Insights
1Automate your benchmark reporting process using GitHub Actions to save time and reduce manual errors.By setting up scheduled workflows, you can ensure that your benchmark results are always current without manual intervention, allowing you to focus on analysis rather than data collection.
2Utilize Jekyll for hosting your benchmarking reports to leverage GitHub Pages' capabilities.Jekyll simplifies the process of generating static sites from dynamic data, making it easier to present your findings in an organized manner.
3Consider using AWS for running benchmarks to maintain control over the testing environment.Running benchmarks on AWS allows you to configure the instance specifications, ensuring consistent performance metrics that are not influenced by shared environments.
Common Pitfalls
1
Failing to manage SSH keys properly can lead to connectivity issues when accessing AWS from GitHub Actions.
This can happen if the known host keys are not correctly set up or if the AWS instance's IP address changes, leading to failed SSH connections.
2
Not configuring Jekyll correctly can result in broken site builds.
If the _config.yml file is not set up properly, it can prevent Jekyll from running as expected, leading to issues in generating the site.
Related Concepts
Continuous Integration
Performance Optimization
Dynamic Site Generation
Benchmarking Methodologies