Starting the Avalanche

Application DDoS In Microservice Architectures

Netflix Technology Blog
11 min readadvanced
--
View Original

Overview

The article discusses the challenges and strategies associated with application layer DDoS attacks in microservice architectures, particularly focusing on Netflix's efforts to identify, test, and remediate these vulnerabilities. It highlights the unique risks posed by interconnected services and offers insights into tools and frameworks for testing and improving system resilience.

What You'll Learn

1

How to identify vulnerable API calls that could be exploited in a DDoS attack

2

Why understanding microservice dependencies is critical for preventing cascading failures

3

How to utilize the Repulsive Grizzly framework for testing application DDoS vulnerabilities

4

When to implement feedback loops between middle tier services and web application firewalls

Prerequisites & Requirements

  • Understanding of microservices architecture and DDoS attack vectors
  • Familiarity with web application firewalls and monitoring tools(optional)

Key Questions Answered

What are application layer DDoS attacks and how do they differ from traditional DDoS attacks?
Application layer DDoS attacks focus on exhausting resources at the application level, often using legitimate-looking traffic to bypass defenses like web application firewalls. Unlike traditional DDoS attacks that target network resources, application layer attacks exploit the intricate relationships between microservices, leading to cascading failures.
How can Netflix's Repulsive Grizzly framework assist in testing for DDoS vulnerabilities?
The Repulsive Grizzly framework is designed to facilitate testing for application DDoS vulnerabilities by simulating attacks on identified weak API calls. It operates using Python and supports high concurrency, making it easier to assess the resilience of microservices under attack conditions.
What indicators should be monitored to assess the impact of DDoS testing?
Key indicators include HTTP status codes, response latencies, and specific headers or stack traces. Monitoring these metrics helps determine if the test is effective and if adjustments are needed to the testing parameters to simulate realistic attack scenarios.
What best practices can help defend against application layer DDoS attacks?
Defending against application layer DDoS attacks involves understanding service dependencies, limiting request sizes, and implementing feedback loops between services and web application firewalls. Additionally, prioritizing authenticated traffic can mitigate the impact of potential attacks.

Key Statistics & Figures

API gateway error rate during testing
80%
This was observed during a DDoS simulation test in a production environment, highlighting the vulnerability of the system.
Percentage of DDoS attacks that are application layer
less than 1%
According to Akamai’s Q1 2017 State of the Internet Security report, this statistic underrepresents the potential impact of application layer attacks.

Technologies & Tools

Testing Framework
Repulsive Grizzly
Used for simulating application DDoS attacks to identify vulnerabilities.
AWS Orchestration Framework
Cloudy Kraken
Helps scale testing efforts across multiple regions in AWS.

Key Actionable Insights

1
Identify and monitor API calls that require significant resources from backend services to mitigate potential DDoS vulnerabilities.
This proactive approach allows teams to understand which parts of their system are most at risk and to implement safeguards before an attack occurs.
2
Utilize the Repulsive Grizzly framework to simulate application DDoS attacks in a controlled environment.
By testing under real-world conditions, teams can better prepare their systems for actual attack scenarios and improve resilience.
3
Implement feedback loops between middle tier services and web application firewalls to enhance detection and response capabilities.
This ensures that the WAF can react to unusual traffic patterns that may indicate a DDoS attack, allowing for quicker mitigation.

Common Pitfalls

1
Failing to understand the interdependencies between microservices can lead to cascading failures during an attack.
This happens because if one service becomes unstable, it can affect all dependent services, resulting in a total service outage. Teams should map out service dependencies to prevent this.
2
Relying solely on edge protections like web application firewalls without monitoring backend service health.
If a WAF is only positioned at the edge, it may miss critical requests that cause stress on backend services. Continuous monitoring and feedback mechanisms are essential.

Related Concepts

Microservices Architecture
Ddos Attack Vectors
Web Application Firewalls
Service Dependencies