Making the Internet observable: the evolution of Cloudflare Radar

David Belson
11 min readintermediate
--
View Original

Overview

The article discusses the evolution of Cloudflare Radar, a tool designed to enhance Internet observability by providing insights into security, performance, and usage trends. It highlights the continuous improvements made since its launch in 2020, including new features, data sets, and the importance of transparency in Internet health.

What You'll Learn

1

How to utilize Cloudflare Radar for monitoring Internet security trends

2

Why understanding AI bot and crawler activity is crucial for content creators

3

How to implement routing visibility to enhance network resilience

Key Questions Answered

What are the key components of Cloudflare Radar?
Cloudflare Radar consists of three key components: Internet Insights, Domain Insights, and IP Insights. These components provide aggregated data on Internet patterns, security threats, and technology adoption, enhancing the understanding of Internet health.
How does Cloudflare Radar help in measuring AI bot activity?
Cloudflare Radar provides an AI Insights page that aggregates data on AI bot crawling activity, including traffic trends by bot and crawl purpose. This allows content creators to understand how often their content is being scraped by AI platforms.
What improvements have been made to routing visibility in Cloudflare Radar?
Recent enhancements include route leak and origin hijack detection, which notify users of potential routing issues. Additionally, real-time BGP route visibility has been added to help network operators troubleshoot and maintain network health.
What role does the Radar API play in accessing data?
The Radar API allows users to programmatically access all data shown on Radar, enabling integration into their own tools and applications. This enhances usability by providing advanced filters for specific data retrieval.

Key Statistics & Figures

Post-quantum encryption adoption
Started at under 3% and grew to just over 47%
This growth was attributed to major browsers and code libraries activating post-quantum support by default.
Unexpected connection closures
About 20% of all connections to Cloudflare close unexpectedly
This indicates potential connection tampering, particularly in regions with restrictive Internet policies.

Technologies & Tools

Tool
Cloudflare Radar
Provides insights into Internet patterns, security, and performance.
Networking
Bgp (border Gateway Protocol)
Used for routing visibility and detecting routing anomalies.

Key Actionable Insights

1
Utilize Cloudflare Radar's AI Insights to track AI bot activity on your site.
This is crucial for content creators to understand how their content is being used and to negotiate better terms with AI platforms that scrape their sites.
2
Implement routing visibility features to monitor your network's health.
By using the real-time BGP route visibility, network operators can quickly identify and resolve routing issues, ensuring better uptime and reliability.
3
Leverage the Radar API for custom data integration.
This allows developers to create tailored applications that utilize Cloudflare's data, enhancing their services and providing deeper insights into Internet trends.

Common Pitfalls

1
Neglecting to monitor AI bot activity can lead to unrecognized content scraping.
Content creators who do not track AI crawlers may miss opportunities to protect their content and negotiate terms, leading to potential revenue loss.

Related Concepts

Internet Security Trends
AI Bot Activity Monitoring
Routing Visibility And Network Resilience