Overview
The article discusses Pinterest's transition from EC2 instances to Docker containers, detailing the migration process, benefits, and challenges faced by the engineering teams. It highlights the successful migration of over half of their stateless services, including the entire API fleet, and outlines the future direction towards adopting container orchestration technology.
What You'll Learn
1
How to migrate workloads from EC2 instances to Docker containers
2
Why adopting a phased approach can mitigate migration risks
3
How to leverage Docker networking for performance optimization
Prerequisites & Requirements
- Understanding of Docker and containerization concepts
- Familiarity with AWS services like EC2(optional)
Key Questions Answered
What benefits did Pinterest gain from migrating to Docker containers?
Pinterest experienced improved developer velocity, better reliability through immutable infrastructure, increased agility for infrastructure upgrades, and enhanced efficiency of their systems. These benefits were crucial in streamlining operations and reducing dependency on complex tools like Puppet.
How did Pinterest ensure performance during the API migration to Docker?
To maintain performance, Pinterest decided to use the host's network instead of Docker networking. They conducted tests to ensure the API fleet could run efficiently in containers, comparing metrics to identify any performance gaps during the migration process.
What challenges did Pinterest face during the migration process?
The migration was challenging due to the scale and complexity of Pinterest's infrastructure, including diverse programming languages, technology dependencies, and strict performance requirements. These factors necessitated careful planning and execution to avoid operational issues.
What is the current state of Pinterest's Docker platform after migration?
After migration, Pinterest operates with a single Amazon Machine Image (AMI) for all container services, using Docker containers for service dependencies and supporting components. They utilize Teletraan for deployments and have implemented a host networking model for optimal performance.
Key Statistics & Figures
Percentage of stateless services migrated
More than 50%
This includes the complete migration of Pinterest's API fleet.
Time taken to complete the API migration
Less than a month
The migration was monitored for regressions or issues, resulting in a smooth transition.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Containerization
Docker
Used for migrating services from EC2 instances and managing service dependencies.
Registry
Amazon Ecr
Primary Docker registry for hosting container images.
Deployment Tool
Teletraan
Used to deploy production service code and manage container deployments.
Key Actionable Insights
1Implement a phased migration strategy when transitioning to containerization to minimize risks and operational disruptions.By starting with non-critical applications, Pinterest was able to identify and resolve issues early, which facilitated a smoother transition for more complex systems like their API fleet.
2Utilize host networking for Docker containers to achieve better performance in production environments.Pinterest's decision to use the host's network allowed them to maintain native performance levels, which is crucial for high-availability services like APIs.
3Establish comprehensive monitoring metrics before and after migration to ensure performance consistency.This practice helped Pinterest identify gaps in monitoring and address issues that arose during the migration, ultimately leading to a successful deployment.
Common Pitfalls
1
Failing to conduct thorough testing before migrating critical applications can lead to performance issues post-migration.
Pinterest mitigated this risk by testing their API fleet in containers before full deployment, ensuring they could identify and resolve potential issues early.
2
Not having a comprehensive set of metrics can make it difficult to monitor performance and identify regressions during migration.
Pinterest established metrics to compare containerized and non-containerized environments, which helped them maintain oversight and confidence throughout the migration process.
Related Concepts
Container Orchestration
Microservices Architecture
Immutable Infrastructure
AWS EC2