The company has also conducted a survey of AI bot activity across its network, identifying Bytespider, Amazonbot, ClaudeBot, and GPTBot as the top four AI crawlers in terms of request volume. Despite the popularity of these bots, many customers are not aware of their presence and few have taken measures to block or challenge their requests. Cloudflare's new feature aims to address this issue and help protect original content creators.
Key takeaways:
- Cloudflare has launched a new feature to block all AI bots with a single click, available for all customers, in response to the rising demand for content used to train AI models and the lack of transparency from some AI companies.
- The most popular AI bots seen on Cloudflare’s network are Bytespider, Amazonbot, ClaudeBot, and GPTBot, with Bytespider leading in terms of number of requests, extent of Internet property crawling, and frequency of being blocked.
- Cloudflare's machine learning models can identify bot activity even when operators attempt to disguise their bots as real browsers by using a spoofed user agent.
- Cloudflare customers can report misbehaving AI crawlers through the Enterprise Bot Management system or a dedicated reporting tool, and Cloudflare will continue to evolve its bot detection and blocking capabilities to protect content creators.