Analyze and control third-party AI crawlers in your website
Core platform•Official page ↗
Pay Per Crawl advanced configuration
You can now configure advanced Pay Per Crawl settings for your zone, including:
Introducing Redirects for AI Training
Cloudflare's network now supports redirecting verified AI training crawlers to canonical URLs when they request deprecated or duplicate pages.
Tools to prepare your site for the agentic Internet
AI Crawl Control now includes new tools to help you prepare your site for the agentic Internet—a web where AI agents are first-class citizens that discover and interact with content differently than human visitors.
Advanced WAF customization for AI Crawl Control blocks
AI Crawl Control now supports extending the underlying WAF rule with custom modifications. Any changes you make directly in the WAF custom rules editor — such as adding path-based exceptions, extra user agents, or additional expression clauses — are preserved when you update…
Analytics enhancements
AI Crawl Control metrics have been enhanced with new views, improved filtering, and better data visualization.
New reference documentation
is now available for AI Crawl Control:
AI Crawl Control Read Only role now available
Account administrators can now assign the AI Crawl Control Read Only role to provide read-only access to AI Crawl Control at the domain level.
