We run one of the largest microservices deployments in the world, with thousands of services that perform billions of requests per second. Keeping information secure as these services communicate g…
Overview
This article details Facebook's transition from the Kerberos authentication protocol to a TLS-based encryption infrastructure within its data centers. It discusses the complexities of securing microservices at scale, the design principles behind the new system, and the lessons learned during implementation.
What You'll Learn
1
How to migrate from Kerberos to TLS for service encryption
2
Why managing identity and access control is crucial in encryption systems
3
How to implement robust error handling in encryption systems
4
When to use symmetric vs asymmetric cryptography in service communications
Prerequisites & Requirements
- Understanding of encryption protocols and microservices architecture
- Experience with service orchestration and container management(optional)
Key Questions Answered
What were the main challenges in migrating to TLS from Kerberos?
The migration faced challenges such as managing service dependencies, ensuring backward compatibility, and maintaining performance during the transition. Clients needed to continue communicating with servers that did not yet support TLS, requiring a dual connection mechanism.
How does Facebook ensure the security of its encryption infrastructure?
Facebook employs a combination of TLS mutual-authentication with X.509 certificates, robust identity management, and error handling mechanisms to secure its encryption infrastructure. This approach allows for end-to-end security while maintaining operability and performance.
What is the role of the internal certificate authority (ICA) in Facebook's encryption system?
The ICA is responsible for signing certificates and supplying ticket keys to services, enabling secure identity verification during container setup. It plays a crucial role in managing the lifecycle of certificates and enforcing security policies.
What improvements were observed after implementing the new TLS design?
The new TLS design improved reliability for non-application errors from 99.99% to almost 100%, significantly reducing debugging overhead and allowing engineers to focus on performance enhancements.
Key Statistics & Figures
Reliability improvement
From 99.99% to almost 100%
This improvement was observed after the implementation of the new TLS design.
CPU usage reduction
10% to 15%
This reduction was achieved by implementing the Fizz library for TLS 1.3, enhancing performance for high-request rate services.
Technologies & Tools
Security Protocol
TLS
Used for mutual authentication and secure communication between services.
Certificate Standard
X.509
Utilized for authenticating endpoints in the TLS implementation.
Library
Fizz
Facebook's custom library for TLS 1.3, designed to optimize performance.
Rpc Framework
Thrift
Used for communication between services in Facebook's microservices architecture.
Orchestration System
Tupperware
Manages containerized services within Facebook's infrastructure.
Key Actionable Insights
1Implement a robust error handling mechanism in your encryption systems to prevent cascading failures.By anticipating potential errors, such as connection failures or service outages, you can design your system to maintain operability and avoid disruptions in service.
2Use TLS session tickets to optimize performance while maintaining security.Session tickets allow for efficient connection resumption, reducing the overhead of full TLS handshakes and improving service responsiveness, especially in high-load environments.
3Regularly rotate encryption keys and certificates to enhance security.Frequent key rotation minimizes the risk of compromised credentials being exploited, ensuring that even if a key is compromised, its usability window is limited.
Common Pitfalls
1
Failing to properly manage service dependencies during the migration to TLS can lead to communication breakdowns.
If clients are not updated to support TLS while servers are, it can result in service outages. A dual connection mechanism is essential to ensure compatibility during the transition.
2
Neglecting to implement robust error handling can lead to cascading failures in service connections.
Without proper error handling, a single failure can propagate through the system, causing widespread disruptions. Anticipating and managing potential errors is crucial.
Related Concepts
Encryption Protocols
Microservices Architecture
Identity Management
Error Handling In Distributed Systems