The company's machine-learning scoring system can identify bots, even those that attempt to disguise themselves as real browsers. Cloudflare's network, which sees an average of 57 million requests per second, provides ample data to determine which digital fingerprints can be trusted. The new feature is available to all customers, who can activate it by clicking the "Block AI Scrapers and Crawlers" toggle button in the Security -> Bots menu for their website.
Key takeaways:
- Cloudflare has introduced a new feature for web hosting customers to block AI bots from scraping website content without permission, in response to customer concerns and to protect content creators.
- Despite existing methods like the robots.txt file, some AI bots ignore these directives and continue to scrape content, leading to legal issues and accusations of content theft.
- Cloudflare's machine-learning scoring system can identify and block these bots, even when they try to disguise themselves as real browsers.
- The company has warned that some AI companies may continue to try and evade bot detection, but it will continue to evolve its machine learning models and add more bot blocks to protect content creators and their control over their content.