Overview
The article discusses the development of a unified PubSub client library at Pinterest, aimed at improving the scalability, stability, and developer velocity of the Logging Platform. It highlights the challenges faced with existing PubSub systems and the benefits of adopting a unified client architecture.
What You'll Learn
1
How to implement a unified PubSub client library to enhance scalability
2
Why reducing dependencies between client applications and PubSub layers is crucial for reliability
3
When to use Topic URIs for efficient PubSub topic management
Prerequisites & Requirements
- Understanding of PubSub systems and client libraries
- Experience with Kafka and its client libraries(optional)
Key Questions Answered
What are the main challenges of using Kafka as a PubSub system?
The main challenges include reduced developer velocity due to tight dependencies between client applications and Kafka brokers, reliability issues from misconfigurations, and scalability limitations that arise from the direct connection to Kafka clusters. These issues can lead to cascading failures and hinder the platform team's ability to perform maintenance.
How does the unified PubSub client library improve service discovery?
The unified PubSub client library provides built-in service discovery, eliminating the need for each client application to manage this aspect. This streamlines the onboarding process for developers and reduces the risk of misconfiguration related to security protocols and cluster endpoints.
What is the architecture of the PSC?
The architecture of the PSC includes PSC Producer and PSC Consumer modules that manage backend producers and consumers for different PubSub systems. It leverages common functionalities like metrics collection, service discovery, and interceptors, allowing for a flexible and scalable design that can adapt to various backend systems.
What future features are planned for the PSC?
Future features for the PSC include auto error handling and configuration updates, message auditing, dynamic client reconfiguration, and seamless backend switching. These enhancements aim to improve the overall functionality and reliability of the PubSub system as it evolves.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Apache Kafka
Used as one of the primary PubSub systems at Pinterest.
Backend
Memq
Another PubSub system integrated into the unified client library.
Streaming
Apache Flink
Integrated with PSC to support streaming use cases at Pinterest.
Key Actionable Insights
1Implementing a unified PubSub client library can significantly reduce the interdependencies between client applications and the PubSub layer.By adopting a unified library, teams can ensure that changes in the PubSub layer do not adversely affect client applications, thus improving overall system reliability and developer productivity.
2Utilizing Topic URIs can streamline the management of PubSub topics across different environments.This approach allows developers to avoid repetitive configuration tasks and reduces the potential for misconfigurations, enhancing the onboarding experience for new applications.
3Regularly review and update configurations to optimize performance and reliability.As traffic patterns change, dynamically adjusting configurations can prevent issues related to throughput and latency, ensuring that the PubSub system remains efficient.
Common Pitfalls
1
Failing to properly configure client libraries can lead to performance issues and outages.
This often occurs when developers are not fully aware of the configurations required for optimal performance, leading to misconfigurations that can impact the entire PubSub platform.
2
Hardcoding cluster endpoints can result in application outages during maintenance.
When applications rely on static endpoints, any changes to the cluster can disrupt service. Implementing dynamic discovery methods can mitigate this risk.
Related Concepts
Pubsub Systems
Client Libraries
Service Discovery
Scalability In Distributed Systems