How Pinterest Uses AWS

The article discusses Pinterest's transition to a next-generation database ingestion framework designed to address the limitations of legacy systems.

The article discusses Pinterest's transition to Moka, a next-generation data processing platform built on AWS Elastic Kubernetes Service (EKS).

The article discusses Pinterest's journey in enhancing developer experience through the creation of PinConsole, an Internal Developer Platform built on Backstage.

This article discusses Pinterest's transition from a Hadoop-based platform to a Kubernetes-based data processing solution named Moka.

The article discusses the automated migration and scaling of Hadoop™ clusters at Pinterest, focusing on the challenges faced and the implementation of the Hadoop Control Center (HCC) to streamline ...

At AWS ReInvent 2024, Madhuri Racherla, VP of Infrastructure & SRE at Pinterest, discussed the company's infrastructure advancements aimed at optimizing performance and reducing costs.

The article discusses how Pinterest's Mobile Builds team utilizes Honeycomb(™) to enhance continuous integration (CI) observability and improve build stability.

The article discusses Change Data Capture (CDC) at Pinterest, detailing its importance for real-time data processing and the implementation of a Generic CDC solution using Debezium.

The article discusses the transition from Apache Hadoop YARN to Apache YuniKorn for resource management in Pinterest's batch processing platform, Monarch, now rebranded as Moka.

This article discusses the implementation of Ray Batch Inference at Pinterest, highlighting its advantages over previous solutions like Apache Spark and Torch Dataloader.

The article discusses Pinterest's implementation of feature caching in their recommender systems using Cachelib, an in-process caching engine developed by Meta Open Source.

The article discusses Pinterest's implementation of Tiered Storage for Apache Kafka®️, highlighting a broker-decoupled approach that offloads data to cheaper remote storage.

The article discusses Pinterest's migration from Druid to StarRocks for delivering faster analytics.

The article discusses Pinterest's adoption of TiDB as a replacement for HBase, detailing the motivations, selection methodology, and the journey of integrating TiDB into their infrastructure.

The article discusses the integration of Ray infrastructure at Pinterest, detailing the journey, challenges, and solutions implemented to optimize machine learning workflows.

This article discusses the complete redesign of Pinterest's ad serving system, known as Mohawk, which was rewritten to eliminate technical debt and improve performance, enabling the company to meet...

The article discusses Pinterest's transition from HBase, its first NoSQL datastore, to a new serving architecture with a unified storage service.

The article discusses the challenges of online-offline discrepancies in Pinterest's ads ranking system, emphasizing the importance of aligning offline model performance with online business metrics.

This article discusses the development of Pinterest's new wide column database, Rockstorewidecolumn, built on RocksDB.

This article discusses the improvements made to the Goku time series database at Pinterest, focusing on enhancing efficiency and user experience.

This article discusses Pinterest's implementation of a finer-grained access control (FGAC) framework to manage data access securely and efficiently within their data engineering platform.

The article discusses the optimization of Flink clusters at Pinterest to enhance stability and efficiency, detailing the strategies implemented to reduce costs and improve performance.

This article discusses the process of migrating data from HBase to TiDB at Pinterest with zero downtime.

This article details Pinterest's experience upgrading their Batch Processing Platform, Monarch, from Hadoop 2. 7. 1 to Hadoop 2. 10. 0.

The article discusses the development of a unified PubSub client library at Pinterest, aimed at improving the scalability, stability, and developer velocity of the Logging Platform.

The article discusses Pinterest's transition to executing end-to-end UI tests before code submissions, emphasizing the benefits of shifting testing left in the development process.

The article discusses Spinner, Pinterest's workflow platform, detailing its evolution from an in-house scheduler called Pinball to Apache Airflow.

The article discusses Pinterest's transition to running a large end-to-end UI test suite before every commit in their Android and iOS repositories.

This article discusses Pinterest's transition to using Druid as its next-generation analytics data store, detailing the architecture and optimization strategies for host types.

MemQ is a new, efficient, and scalable cloud-native PubSub system developed by Pinterest, designed to handle Near Real-Time data transportation while being up to 90% more cost-effective than Apache...

This article discusses Pinterest's Batch Processing Platform, Monarch, focusing on efficient resource management to ensure quality of service (QoS) while maintaining cost efficiency.

This article discusses Pinterest's transition to using Druid as their analytical data store, detailing the challenges faced with HBase, the architecture of their Druid implementation, and insights ...

The article discusses how Pinterest utilizes Apache Spark SQL for interactive querying, detailing the architecture, challenges faced, and solutions implemented to enhance user experience.

The article discusses the implementation of a near-real-time image similarity detection system at Pinterest using Apache Flink.

The article discusses Pinterest's Flink Deployment Framework, which is built on Bazel and integrates with various internal services to streamline the deployment of Flink jobs.

The article discusses how Pinterest has scaled its cache infrastructure to handle increasing demand from users.

The article discusses GokuL, an extension of Pinterest's time series database Goku, designed to efficiently store and query data beyond one day.

The article discusses Pinterest's implementation of Presto, an open-source distributed SQL query engine, detailing the challenges faced and solutions developed to manage large-scale data analysis.

Pinterest operates one of the largest Kafka deployments in the cloud, utilizing Apache Kafka as a message bus for data transport and real-time streaming services.

The article discusses the implementation of automated cluster management and recovery for Rocksplicator at Pinterest, highlighting the transition from manual management to using Apache Helix.

The article discusses the enhancements made to Knox, Pinterest's open-source secret management service, enabling it to support SPIFFE x509 identity documents for authentication in multi-tenant envi...

The article discusses Pinterest's implementation of geo-blocking APIs to manage media content visibility based on user location.

The article discusses Pinterest's transition from EC2 instances to Docker containers, detailing the migration process, benefits, and challenges faced by the engineering teams.

The article discusses Manas, a high-performing customized search system developed by Pinterest to handle billions of queries and improve search relevance among over 100 billion Pins.

The article discusses Pinterest's upgrade of HBase from version 0. 94. 26 to 1. 2, emphasizing the importance of maintaining high performance and availability during the transition.

Soundwave is an open-source configuration management database (CMDB) developed by Pinterest to manage and track AWS EC2 instances and their metadata.

Pixie is an advanced graph-based recommendation system developed by Pinterest to enhance user engagement by providing personalized recommendations in real-time.

The article discusses how Pinterest utilized machine learning to enhance user engagement globally, particularly through improvements in the home feed experience.

The article discusses Pinterest's implementation of Amazon Auto Scaling to improve infrastructure efficiency, particularly during varying traffic loads.

The article discusses the introduction of automatic object detection in visual search at Pinterest, highlighting its impact on user experience and the underlying deep learning technologies.