Overview
The article discusses Pinterest's transition to Moka, a next-generation data processing platform built on AWS Elastic Kubernetes Service (EKS). It highlights the deployment infrastructure, logging and observability solutions, image management, and the user interface for Spark applications.
What You'll Learn
1
How to deploy EKS clusters using Terraform and Helm
2
Why logging and observability are crucial for Spark applications
3
How to manage container images for Spark applications in a Kubernetes environment
4
When to utilize Fluent Bit for log management in Kubernetes
Prerequisites & Requirements
- Understanding of Kubernetes and AWS services
- Familiarity with Terraform and Helm(optional)
Key Questions Answered
How does Pinterest deploy EKS clusters for Moka?
Pinterest standardizes on four cluster environments: test, dev, staging, and production. EKS clusters are deployed using Terraform and Helm, with a focus on different access levels and isolation for each environment.
What logging solution does Pinterest use for Spark applications?
Pinterest uses Fluent Bit to manage logs from Spark applications and system pods. Fluent Bit is deployed as a DaemonSet to ensure logs are collected and uploaded to Amazon S3, facilitating easier access and analysis.
What are the key components of Moka's observability framework?
Moka's observability framework uses Prometheus for metrics collection and OpenTelemetry for telemetry data processing. It integrates with Pinterest's custom time-series database, Goku, for metrics storage and visualization.
How does Moka manage container images for Spark applications?
Moka utilizes a structured image management system to build and maintain container images for Spark applications, ensuring compatibility with both Intel and ARM architectures. This includes generating multi-architecture Debian packages for Hadoop and Spark.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Cloud
AWS Elastic Kubernetes Service
Used for deploying and managing Kubernetes clusters for Moka.
Logging
Fluent Bit
Used for log management and forwarding logs to Amazon S3.
Infrastructure As Code
Terraform
Used for deploying EKS clusters.
Package Management
Helm
Used for managing Kubernetes applications.
Monitoring
Prometheus
Used for metrics collection in the observability framework.
Observability
Opentelemetry
Used for telemetry data processing.
Key Actionable Insights
1Implement a structured logging solution using Fluent Bit to enhance observability in Kubernetes environments.This approach allows for effective log management and easier troubleshooting of Spark applications, ensuring that logs are collected and stored in a centralized location.
2Standardize your deployment process using Terraform and Helm to manage EKS clusters efficiently.This practice not only streamlines the deployment process but also enhances the scalability and maintainability of your Kubernetes infrastructure.
3Utilize Prometheus and OpenTelemetry for comprehensive metrics collection and observability.These tools provide critical insights into system performance and stability, enabling proactive monitoring and optimization of resources.
Common Pitfalls
1
Neglecting the importance of logging and observability can lead to difficulties in troubleshooting and performance monitoring.
Without a robust logging solution, identifying issues in Spark applications becomes challenging, especially under heavy load or during failures.
2
Failing to standardize deployment processes can result in inconsistencies and inefficiencies.
Not using tools like Terraform and Helm for managing deployments may lead to configuration drift and complicate scaling efforts.