Separating the signal from the noise

Bryan Sullivan

If a security tool catches a critical vulnerability, but also reports 99 other findings that turn out to be false positives, developers are going to ignore everything that the tool reports and then miss the important issues. Bryan Sullivan talks through how you can hone your tooling to separate the signal from the noise.

Microsoft

•

Bryan Sullivan

•4 min read•intermediate•

--

•View Original

Overview

The article discusses the importance of filtering out noise from automated tooling bugs to enhance developer experience. It emphasizes the need for engineering system owners to take responsibility for the tools they provide, ensuring that developers can trust the findings and focus on critical issues.

What You'll Learn

1

How to identify and eliminate noisy rules in automated tooling

2

Why reducing false positives in security tools is crucial for developer productivity

3

When to deprecate ineffective static analysis rules

Key Questions Answered

How can noisy automated tooling affect developer productivity?

Noisy automated tooling can lead developers to ignore important findings due to overwhelming false positives. When security tools report numerous false issues, developers may dismiss all alerts, risking missing critical vulnerabilities. This creates a cycle of distrust towards the tools, ultimately reducing their effectiveness.

What is the Pareto effect in the context of tooling bugs?

The Pareto effect refers to the observation that a small number of static analysis rules account for a large proportion of 'Won't Fix, False Positive' resolutions. This means that by focusing on a few problematic rules, organizations can significantly reduce the noise generated by their tooling, improving overall trust and usability.

Why was the CA2202 rule deprecated?

The CA2202 rule was deprecated because it generated excessive false positives, leading developers to disregard its findings. The decision to remove this rule resulted in a noticeable decrease in reported false positives, helping developers to trust the tooling more and focus on genuine issues.

Key Statistics & Figures

Percentage of Code Analysis rule detections fixed by developers

>99%

This statistic highlights the effectiveness of certain Code Analysis rules, such as cryptography detections, which are highly trusted by developers.

Key Actionable Insights

1
Identify and remove noisy rules from your automated tooling to improve developer trust.
By analyzing the telemetry of the rules, you can pinpoint which ones generate excessive false positives and take action to deprecate them, thus enhancing the overall developer experience.

2
Encourage a culture of feedback regarding tooling effectiveness among developers.
Regularly solicit input from developers about the tools they use, which can help identify problematic rules and improve the tools over time, fostering a more efficient development environment.

3
Focus on per-rule analysis rather than per-tool evaluations.
This approach allows for a more nuanced understanding of which specific rules are causing noise, enabling targeted improvements without losing valuable functionalities from the tools.

Common Pitfalls

1

Over-reliance on automated tooling without considering the quality of the rules.

This can lead to developers ignoring critical alerts due to overwhelming false positives, which diminishes the effectiveness of the tooling and can result in security vulnerabilities being overlooked.