The issue is further complicated by AI companies not properly identifying their web crawlers, making it difficult for publishers to manage access to their content. Some publishers have taken legal action against AI companies like Perplexity for scraping and republishing content without proper attribution. To address these challenges, some publishers have struck content licensing deals with AI companies, while others, like TollBit, have developed models to charge AI companies for scraping content. The ongoing tension between AI companies and publishers underscores the need for new economic models in the age of AI.
Key takeaways:
- AI-powered search engines like OpenAI and Perplexity send significantly less referral traffic to news sites compared to traditional Google search, with a reported 96% decrease.
- AI companies have increased their web scraping activities, with OpenAI, Perplexity, and others scraping websites 2 million times on average in the last quarter of 2024.
- Legal actions have been taken against AI companies like OpenAI and Perplexity for allegedly infringing on intellectual property and scraping content without proper attribution.
- AI scraping is causing increased server costs for publishers, and companies like TollBit are offering solutions by charging AI companies for content scraping.