Cloudflare has extended its 'block AI training' option to all customers, the company said in a blog post. The feature lets sites remain indexed by search engines while preventing their content from being used to train AI models, addressing the overlap between search crawling and AI training.

The company also introduced a 'responsible' crawler operator designation. To qualify, operators must offer an opt-out for training and AI summaries, disclose URL-level indexing status, and commit that opting out will not affect search rankings. Apple, Google, Microsoft, Amazon, Anthropic, Meta, and OpenAI were among the entities granted the designation.

Cloudflare noted that roughly 17% of sites have enabled some form of training-blocking mechanism, while fewer than 1% currently block search crawlers.