Overview
The article discusses the implementation and operational benefits of the Unified PubSub Client (PSC) at Pinterest, highlighting improvements in developer velocity, stability, and scalability. It also details the seamless migration process from native clients to PSC and outlines future enhancements planned for the platform.
What You'll Learn
1
How to automate PubSub service endpoint discovery using Resource Names
2
Why automated error handling can reduce application restarts by over 80%
3
How to migrate Java applications to the Unified PubSub Client with minimal changes
Prerequisites & Requirements
- Understanding of PubSub systems and client-server architectures
- Familiarity with Apache Kafka and Flink(optional)
Key Questions Answered
What improvements does the Unified PubSub Client offer over native clients?
The Unified PubSub Client (PSC) offers significant improvements such as automated service endpoint discovery, optimized client configurations, and automated error handling, which collectively enhance developer velocity and reduce application restarts by over 80%. This allows developers to focus more on application logic rather than troubleshooting connectivity issues.
How does automated error handling in PSC improve application stability?
Automated error handling in PSC captures and remediates known exceptions without disrupting the application layer. This has led to a reduction of over 80% in job failures caused by remediable client exceptions, significantly enhancing the stability of applications relying on the PubSub infrastructure.
What is the migration process from native clients to PSC?
Migrating from native clients to PSC involves replacing native client imports, updating configuration keys, and using Resource Names for service discovery. This process has achieved over 90% migration of Java applications with zero incidents reported, demonstrating its effectiveness and ease of adoption.
What are the future plans for the Unified PubSub Client?
Future enhancements for the Unified PubSub Client include adding capabilities for automatic resolution of more remediable errors, developing C++ support, and exploring Python integration. These improvements aim to further increase the technical capability and value of PSC within Pinterest's data infrastructure.
Key Statistics & Figures
Reduction in time spent setting up new PubSub producers and consumers
80%
This statistic highlights the efficiency gained through the automation features of PSC.
Reduction in Flink application restarts due to remediable client exceptions
>80%
This demonstrates the impact of automated error handling implemented in PSC.
FTE hours saved per year in KTLO work
275+
This reflects the operational efficiency gained by both application and platform teams after adopting PSC.
Percentage of Java applications migrated to PSC
>90%
This showcases the successful adoption of PSC among Java applications at Pinterest.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Apache Kafka
Used as part of the PubSub stack for data ingestion and transport.
Backend
Apache Flink
Integrated with PSC for data streaming applications.
Key Actionable Insights
1Implement automated service endpoint discovery to streamline client configurations.By utilizing Resource Names, developers can simplify the process of connecting to PubSub services, reducing the time spent on setup and configuration.
2Adopt automated error handling to minimize application downtime.Integrating automated error handling can significantly reduce the operational burden on teams by preventing unnecessary application restarts and allowing for smoother scaling of services.
3Leverage the full integration test suite provided with PSC for robust application testing.Using the built-in test suite ensures that applications are thoroughly vetted before deployment, reducing the risk of failures in production environments.
Common Pitfalls
1
Hardcoding invalid hostname/port combinations in client configurations.
This often leads to connectivity issues that require time-consuming investigations. Utilizing automated service discovery helps avoid these pitfalls by dynamically resolving endpoints.
2
Misconfiguring client settings that degrade performance.
Developers may lack a full understanding of the implications of each configuration. Standardized configurations provided by PSC mitigate this risk.
Related Concepts
Pubsub Systems
Client-server Architectures
Error Handling Strategies
Data Streaming Frameworks