Overview
This article discusses Pinterest's implementation of a finer-grained access control (FGAC) framework to manage data access securely and efficiently within their data engineering platform. It highlights the challenges faced with traditional access control methods and details the design principles, architecture, and technologies used to enhance their Hadoop-based system, Monarch.
What You'll Learn
1
How to implement finer-grained access control in a data engineering platform
2
Why using dynamically generated Security Token Service (STS) tokens enhances data security
3
When to utilize LDAP groups for managing user permissions in a multi-tenant environment
Prerequisites & Requirements
- Understanding of access control mechanisms and AWS services
- Familiarity with AWS Security Token Service (STS) and Lightweight Directory Access Protocol (LDAP)(optional)
Key Questions Answered
What challenges did Pinterest face with traditional access control methods?
Pinterest encountered limitations with IAM roles, such as the maximum number of roles per account and the complexity of managing multiple roles for different datasets. This led to inefficiencies and increased costs due to the need for numerous clusters to manage access to various datasets.
How does the Credential Vending Service (CVS) work?
The Credential Vending Service (CVS) authenticates users and generates dynamic STS tokens that map users to specific permissions based on their LDAP group memberships. This allows for fine-grained access control without the need for creating multiple IAM roles, enhancing security and efficiency.
What is the role of Kerberos in Pinterest's FGAC implementation?
Kerberos is used for secure user authentication across Pinterest's systems. It enables services to impersonate users and manage identities securely, which is essential for maintaining multi-tenancy and ensuring that user permissions are correctly enforced within the Hadoop ecosystem.
How does Pinterest ensure user multi-tenancy in its Hadoop platform?
Pinterest implements user multi-tenancy by enforcing access protections for files in HDFS, encrypting data transfers, and requiring Kerberos authentication for Hadoop Web UIs. This ensures that multiple applications can run securely on the same cluster without interfering with each other.
Key Statistics & Figures
Number of Monarch clusters
More than 30
Monarch consists of over 30 Hadoop YARN clusters with 17,000+ nodes, processing petabytes of data daily.
Daily jobs processed by Monarch
Hundreds of thousands
Monarch is critical for processing both interactive queries and scheduled batch jobs.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Cloud Services
AWS
Used for hosting the data engineering platform and managing access control through IAM and STS.
Data Processing
Hadoop
Serves as the primary framework for batch processing and data management at Pinterest.
Directory Services
Ldap
Utilized for managing user identities and permissions in the access control framework.
Authentication
Kerberos
Provides secure authentication for users and services within the data infrastructure.
Key Actionable Insights
1Implement a Credential Vending Service (CVS) to streamline access control across data platforms.By centralizing access management through CVS, organizations can reduce the complexity and overhead associated with managing multiple IAM roles and clusters, leading to improved efficiency and security.
2Utilize LDAP groups to manage user permissions effectively in a multi-tenant environment.This approach allows for scalable permission management, enabling data custodians to easily authorize access without the need for extensive IAM role configurations.
3Adopt dynamic STS tokens for granting temporary access to AWS resources.Dynamic tokens enhance security by limiting access to only the necessary resources for a specific time, reducing the risk of unauthorized access.
Common Pitfalls
1
Over-reliance on IAM roles can lead to management complexity and inefficiencies.
Creating too many IAM roles for different datasets can overwhelm administrators and lead to increased costs and resource fragmentation. Instead, implementing a centralized access control mechanism like CVS can streamline management.
Related Concepts
Access Control Mechanisms
AWS Security Token Service (sts)
Lightweight Directory Access Protocol (ldap)
Kerberos Authentication
Hadoop Ecosystem