In-app reader
Blog AI BotsBot ManagementBots** +2 Show 2 more tags
** 5 Tags Show 5 tags
_**
Selected Tags
AI BotsBot ManagementBotsNetwork ServicesProduct News
All tags
Matching tags
No tags found
1.1.1.1
2FA
Abuse
Access
Access Control Lists (ACLs)
Accessibility
Account Takeover
Acquisitions
Addressing
Advanced Certificate Manager
Advanced DDoS
Advertising
Aegis
AEO
Africa
Afroflare
Agent Cloud
Agent Development Lifecycle
Agent Readiness
Agents
Agents Week
Agents Week 2026
AI
AI Bots
AI Gateway
AI Search
AI-SPM
AI WAF
AI Week
Alertmanager
Always Online
AMD
AMP
Analytics
Anonymous
Anti Malware
Anycast
API
API Gateway
API Security
API Shield
APJC
Apple
Application Security
Application Services
Area 1 Security
Argo Smart Routing
ASCII
Asia
Athenian Project
Atlassian
Attacks
Audit Logs
Austin
Australia
Authentication
Authy
Automatic HTTPS
Automatic Platform Optimization
Automation
AutoMinify
Auto Rag
Awards
AWS
Baidu
Bandwidth Alliance
Bandwidth Costs
Best Practices
Beta
Better Internet
BGP
Billing
Birthday Week
Blackbird
Black Friday
Bot Fight Mode
Bot Management
Botnet
Bots
BPF
Brand
Brand Protection
Brazil
Browser Insights
Browser Rendering
Browser Run
Bug Bounty
Bugs
BYOIP
Cache
Cache Purge
Cache Reserve
Cache Rules
California
Canada
Cap'n Proto
CAPTCHA
Careers
CASB
Categories
CDN
CDNJS
Certificate Authority
Certificate Pinning
Certificate Transparency
Certification
CFSSL
Challenge Page
ChatGPT
China
China Network
Christmas
Chrome
CIO Week
CISA
Claire
CLI
ClickHouse
Clientless
Clientless Web Isolation
Cloud Connector
Cloud Email Security
Cloudflare Access
Cloudflare Apps
Cloudflare Area 1
Cloudflare Calls
Cloudflare Email Service
Cloudflare for Campaigns
Cloudflare for SaaS
Cloudflare for Startups
Cloudflare Gateway
Cloudflare History
Cloudflare Images
Cloudflare Media Platform
Cloudflare Meetups
Cloudflare Network
Cloudflare One
Cloudflare One Client
Cloudflare One User Risk Score
Cloudflare One Week
Cloudflare OS
Cloudflare Pages
Cloudflare Polish
Cloudflare Queues
Cloudflare Realtime
Cloudflare Stream
Cloudflare Tunnel
Cloudflare TV
Cloudflare Workers
Cloudflare Workers KV
Cloudflare Workers KV (ES)
Cloudflare Workers (PT)
Cloudflare Zero Trust
Cloudforce One
Cloudy
Code Orange
Coinbase
Colombia
Community
Compliance
Compression
Config Rules
Configuration Management
Congestion Control
Connectivity
Connectivity Cloud
Consumer Services
Containers
Content Independence Day
Content Scanning
Context
Core
COVID-19
Crawler Hints
CrowdStrike
Cryptography
Crypto Week
CSAM Reporting
Customers
Customer Success
Customer Zero
CVE
CVE-2023-50387
Cyber Readiness
Cybersecurity
D1
Dashboard
Data
Database
Data Catalog
Data Center
Data Localization
Data Localization Suite
Data Loss
Data Loss Prevention
Data Platform
Data Privacy Day
Data Protection
Data Sovereignty
Data Transfer Bucket
DDoS
DDoS Alerts
DDoS Reports
Debugging
Deep Dive
Descaler
Design
Deskope
Developer Documentation
Developer Platform
Developers
Developer Spotlight
Developers Storage
Developer Week
Device Security
DevOps
DEX
Digital Experience Monitoring
Digital Forensics
Disrupt
Distributed
Distributed Systems
Distributed Web
Diversity
DLP
DMARC
DNS
DNS Filtering
DNS Flood
DNSSEC
DNS Security
Dogfooding
DoH
Domain Rankings
Domain Scoped Roles
dosd
Drupal
Due Process
Durable Execution
Durable Objects
Early Hints
Earth Day
eBPF
EC2
eCommerce
Edge
Edge Computing
Edge Database
Edge Rules
Education
Egress
Elastic
Elections
Election Security
Elliptic Curves
Email Routing
Email Security
Email Workers
EmDash
Emissions
Employee Resource Groups
Encrypted SNI
Encryption
Engineering
Enterprise
Entropy
EPYC
Ethereum
Europe
European Union
Events
Exploit
Fancy Bear
Fast Fonts
FCC
Feature Flags
FedRAMP
FedRAMP High
FedRAMP Moderate
Firefox
Firewall
Firmware
Florida
Football
Formal Methods
Forrester
Fortran
Foundation DNS
Founders' Letter
France
Fraud
Free
Freedom of Speech
Front End
Full Stack
Full Stack Week
Fun
Gartner
Gatebot
GA Week
GDPR
General Availability
Generative AI
Gen X
Geo Key Manager
Germany
GitHub
Go
Google Analytics
Google Cloud
Google Workspace
Government Innovation
Grace Hopper
Grafana
GraphQL
Green
Grinch
Growth
gRPC
Guest Post
Hackathon
Halloween
Hardware
HashiCorp
Hertzbleed
Heuristics
History
Holidays
Holocaust
Hong Kong
Hosting Con
Hostnames
HTTP2
HTTP3
HTTPS
Human Rights
Hurricane
Hybrid Cloud
Hyperdrive
IBM
ICANN
iCloud Private Relay
Identity
IETF
IETF Standards
IL4
Image Optimization
Image Recognition
Image Resizing
Image Storage
Impact
Impact Week
I'm Under Attack Mode
Incident Report
Incident Response
India
Indicators of Compromise
Indonesian
Inference
Infrastructure
Infrastructure as Code
Insights
Intel
Interconnection
Internal DNS
Internet Performance
Internet Quality
Internet Regulation
Internet Shutdown
Internet Summit
Internet Traffic
Internet Trends
Internship Experience
Intrusion Detection
Investors
IoCs
iOS
IoT
IPFS
IPsec
IPv4
IPv6
IRAP
Israel
Italy
IWD
JAMstack
Japan
JavaScript
Jengo
Jengo Policy
Joomla
Judeoflare
Kafka
Kernel
Keyless SSL
KeyTrap
Key Value
Killnet
Korea
Kubernetes
LangChain
Latency
Latin America
Latinflare
LavaRand
Lazarus group
Leaked Credential Checks
Legal
Legal Patents Sable
LGBTQIA+
Life at Cloudflare
Linux
Lisbon
Live Streaming
Llama
LLM
Load Balancing
Localization
Log4J
Log4Shell
Logging
Log Push
Logs
LUA
Machine Learning
Magecart
Magic Firewall
Magic Network Monitoring
Magic Transit
Magic WAN
Magic WAN Connector
Malicious JavaScript
Malware
Managed Components
Managed Rules
March of Cloudflare
MASQUE
MCP
Meerkat
MeetUp
Meris
Message Protocol
Mexico
Micro-frontends
Microsoft
Microsoft 365
Microsoft Azure
Middle East
Migration Hub
Milestones
Miniflare
Mirage
Mirai
Mitel
Mitigation
Mixed Content Errors
MLops
Mobile
Mobile SDK
Model Context Protocol
Moldova
Monitoring
Multi-Cloud
Multi-User
MySQL
NaaS
Net Neutrality
Network
Networking
Network Interconnect
Network Performance Update
Network Protection
Network Services
New Year
NGINX
Ninjas
NIST
Node.js
North America
Notebooks
Notifications
NSEC3
OAuth
Observability
Oceania
OCSP
Offices
Okta
Olympics
Onboarding
OpenAI
Open API
OpenBMC
OpenDNS
Open Source
OpenSSL
OpenTelemetry
Optimization
Origin Rules
Outage
Oxy
Pacific Northwest
Page Rules
Page Shield
Parallels
Partners
Partnership
Password-reuse
Passwords
Passwords (PT)
Patents
PAYGO
Payments
Pay Per Crawl
PCI Certified
Peering
Performance
Phishing
php
Phython
Pingora
Pipelines
PlanetScale
Plans
Platform Engineering
Platform Week
Plesk
Policy & Legal
Politics
Portugal
Postgres
Post Mortem
Post-Quantum
Precursor
Prepared Statements
Prisma
Privacy
Privacy Pass
Privacy Week
Private IP
Private Network
Product Design
Product News
Programming
Programming (PT)
Project Fair Shot
Project Galileo
Project Honey Pot
Project Pangea
Project Safekeeping
Project Turpentine
Prometheus
Protocols
Proudflare
Proxying
Public Sector
Python
Python Workers
Quantization
Queues
QUIC
QUICHE
Quicksilver
R2
R2 Super Slurper
Radar
Radar Alerts
Radar API
Radar Maps
Railgun
Randomness
Ransom Attacks
Rapid Reset
Raspberry Pi
Rate Limiting
RC4
RDDoS
React
Reading List
Real-time
Recruiting
Regional Services
Registrar
Reliability
Remote Browser Isolation
Remote Desktop Protocol
Remote Work
Replication
Research
Resolver
Restreaming
Retreat
Reverse Engineering
REvil
Risk Management
Road to Zero Trust
Rocket Loader
RocksDB
Routing
Routing Security
RPC
RPKI
RRDNS
RSA
Russia
Rust
Rust Workers
SaaS
SAAS Security
Sable
Salt
Sampling
Sandbox
SASE
Save The Web
SDK
Search Engine
Secrets Store
Secure Web Gateway
Security
Security Analytics
Security Center
Security Posture
Security Posture Management
Security Service Edge
security.txt
Security Week
SEO
Serverless
Serverless AI
Serverless (PT)
Serverless Week
Server Push
Servers
SIEM
Signed Exchanges (SXG)
SIM
Singapore
Single Sign On (SSO)
Smart Placement
Smart Shield
Snippets
SOC as a Service
South Africa
South America
Spain
spdy
Spectrum
Speed
Speed Brain
Speed & Reliability
Speed Week
Spoofing
Sports
SQL
SRE
SSE
SSH
SSL
Standards
Startup Enterprise Plan
Statistics
StopTheHacker
Storage
Sumo Logic
Super Bowl
Supercloud
Supply Chain Attacks
Support
Sustainability
SWAG
SWG
Swift
Switzerland
SXSW
SYN
SYN Flood
Syria
TCP
Team
Teams Dashboard
TechCrunch
Technical Writing
Tech Talks
Terraform
Testimonials
Testing
Texas
Thanksgiving
The Serverlist Newsletter
Threat Data
Threat Feeds
Threat Intelligence
Threat Operations
Threat Report
Threats
Tiered Cache
TikTok
TLS
TLS 1.3
Tools
Tor
Tracing
Traffic
Transform Rules
Transparency
Trends
Trust & Safety
TTFB
TTL
TURN
TURN Server
Turnstile
TypeScript
UDP
Ukraine
United Kingdom
Universal SSL
URL Scanner
USA
User Research
VDI
Vectorize
Vetflare
Video
Visibility
Vite
VoIP
VPC
VPN
Vulnerabilities
WAF
WAF Attack Score
WAF Rules
Waiting Room
WARP
WARP Connector
WASM
Web3
Web Application Firewall
WebAssembly
Web Asset Discovery
Webinars
WebMCP
WebP
WebRTC
WebSockets
Wildebeest
Womenflare
WordPress
Workers AI
Workers Launchpad
Workers Logs
Workers Observability
Workers Sites
Workers Unbound
Workers VPC
Workflows
World IPv6 Day
Wrangler
x402
Year in Review
Z3
Zaraz
Zero Day Threats
Zero Trust
Zero Trust Week
Zone Versioning
Network ServicesProduct News
AI BotsBot ManagementBotsNetwork ServicesProduct News
August 21, 2026
**Jin-Hee Lee
8 minute read
** COPY URL
We’re constantly building for the different goals of our customers. Some customers want to optimize for discovery, while others want to protect their content with the strictest security policy. Among these differing policies, there are multiple ways to mitigate bot traffic. Some mechanisms simply state your preference, assuming best intent from crawlers, and other approaches actually lock down content by outright blocking with a Bot Management solution.
We recognize that it's cumbersome to maintain multiple layers of protection on your website. For example, there are cases in which your robots.txt states that a crawler is Disallowed from accessing your website, while your enforcement rules actually don’t block that crawler. When your stated preferences and your enforced rules disagree, some crawlers treat it as a basis to disregard your preferences or try to bypass your enforced rules.
A couple of years ago, Cloudflare announced an easier way to disallow AI training on your website by tackling two of these layers: a managed value of robots.txt that told a fixed list of major Training crawlers not to train on your content, along with edge-enforced blocks to Training crawlers. On July 1, 2026, we launched easier options to manage different kinds of AI traffic use cases. You can say what you want to do about Search, Agent, and Training traffic on your website.
We're announcing Bot Preference Sync, available to all customers from the Free tier to Enterprise. Bot Preference Sync reflects what you've set in your AI bot configuration by updating corresponding preferences to your robots.txt, and it can be turned on or off at any time. No more static file for one use case: we'll help you tailor your robots.txt to reflect what you’ve already configured for different AI bot categories.
For years, the most pressing question in this space was: "Is my content being used to train AI models without my permission?" It's an important question, and it isn't going away. Alongside this, the questions we increasingly hear are about discoverability and engagement. How do I show up when someone asks an AI assistant something my site can answer? How much of my traffic is coming from AI crawlers versus real people? What content is actually driving referrals, and what is it worth?
The answers differ by business model. Discoverability and engagement are key, top-of-mind issues for any businesses trying to thrive on the modern web, but the funnels for these are different: an e-commerce store may want everything crawled and trained on, so its products surface when a shopper asks a chatbot for "the best sofa for a small apartment." A publisher that monetizes pages with ads may want the opposite: stay in the search index that sends readers to the page, but keep its articles out of model training and, crucially, be able to verify that its content really wasn't used without permission.
There's no single right answer, which is exactly the point. Your controls should reflect your strategy, which is why we've been building tools to give you visibility and choice at every layer. Bot Preference Sync ties these together, so the preference you set is the preference you publish.
On July 1, 2026, we made the case that mixed-use crawlers, or “bots that blend search, agent use, and training behind a single user agent,” put site owners at a disadvantage precisely because they make it hard to separate what you want from what you don't. That's still true, and our position on Transparency for site owners hasn't changed.
But there’s more than one way to approach Transparency. We want to reward the operators who are clear about their identity and how they are using the data they crawl. For purposes of bot Verification, the owners of bots that perform both Search and Training will need to provide additional information in order to not be blocked when “Disallow Training” is set. Those requirements are:
The bot must respect, via any mechanism, a “no training” preference in robots.txt
They give site owners a way to opt out of AI summaries.
They provide URL-level visibility into which pages were made available for training, as well metrics on search results, so you can see how your content was used for search and for training.
They can show publicly that Disallowing Training does not hurt your traditional search results.
Bots of leading AI models and service providers that meet these criteria are tracked publicly in the AI bot transparency section in Cloudflare Radar, which includes examples in which best practices are honored, as well as when they are not. Crawlers that don't provide Transparency will not get the benefit of the doubt — they're still blocked when you disallow training. In other words, this is a way of making Transparency the price of admission.
Bot Preference Sync is a new feature that keeps your robots.txt reflecting the AI bot preferences you've already set for Search, Agent, and Training on the Cloudflare zone-level dashboard. If a site owner already has a robots.txt file, the contents added by Bot Preference Sync will be prepended to the existing material, so any existing Disallow directives are maintained.
Instead of a site owner maintaining a separate static file, Cloudflare generates or updates your robots.txt based on your configuration, so what you say to the world and what you enforce at the edge are kept in sync.
For Search and Agent, the three options we announced on July 1 remain: Allow, Block on pages that serve ads, or Block everywhere. For Training, we’re refining the option to stop your content being used for training models with the Disallow option:
Disallow: a "no training" preference is written to your robots.txt, so that cooperating mixed-use crawlers who take the extra Transparency step can still access your content for search indexing, since they’re allowing site owners to directly verify how their data is used. Cooperating crawlers honor the preferences in robots.txt, and your Search visibility for cooperating crawlers is unaffected.
Let’s take the example below, in which someone has configured their AI bot policy to say “Allow Search, Allow Agents, Disallow Training.”
Since this example site has Bot Preference Sync on, their robots.txt would prepend something like the following (which has been shortened and anonymized for the sake of the example):
# BEGIN Cloudflare Bot Preference Sync
User-agent: TrainingBot1
User-agent: TrainingBot2
User-agent: TrainingBot3
User-agent: MixedUseBot-Extended
Disallow: /
...
# END Cloudflare Bot Preference Sync
**
We’ll use bots that we track in BotBase to periodically update the list of bots that is added to robots.txt when you choose to Block or Disallow a given category. The Verified bots that are classified as Search, Agent, and Training can be viewed at any time in our public bots directory.
For all new customers, Bot Preference Sync will be on by default, to make it easier to manage blocks and preferences that reflect the same policy. For existing customers who are using the legacy managed robots.txt feature, we'll prompt you to review and confirm your preferences to transition to the new Bot Preference Sync upon its upcoming launch.
Some customers may want or need to be more hands-on in stating their preferences, for example, if they have a special arrangement with a given company to which they want to grant an exception. Because Bot Preference Sync is designed to tackle policy decisions made category-wide rather than case-by-case, it will not directly read from individual custom rules with more complex logic. Customers with a more fine-tuned security policy always have the option to turn off the sync that sets group policies, and tailor their file to match their custom policy.
We’re also making a change that allows publishers or ad-supported sites to have a different default from other site owners. We’ve created a default to make it easier for publishing sites that rely on ads and expect them to be reserved for human visitors. At the time of onboarding, such customers can select the option, “I monetize from pages with ads on this domain", which will set Training to Disallow as the default. (Customers have the choice to change this setting at any time.) This way, you stay in search while keeping your content out of model training.
For the non-publisher case, new customers will not have any blocks or disallows added by default when they onboard a domain: the choice is up to the customer. You can choose if you want to block Search or Agent or Training at any point, but the starting point will not add any blocks on your behalf.
Bot Preference Sync will be available to all customers, on every plan, in the coming week. Keep an eye on our changelog for availability, and watch your dashboard (and inbox) for the prompt to confirm your preferences!
This is one step in a longer effort. We'll keep working with the large bot operators to make sure we're not compromising on familiar challenges (like training without consent) nor emerging questions (like discoverability and engagement). Beneath it all is our effort to promote greater Transparency and control for site owners.
**
**
On this page
Discuss Online
**
AI BotsBot ManagementBotsNetwork ServicesProduct News
Follow on Social Media
Email address _
We’ll never share your email address.
**Subscribe
Thanks for subscribing! Check your inbox to confirm.
Discussion
Sign in to join the discussion.
Keep reading
Optional: create a free account to save items, track programs, and sync across web + app. Reading stays free.