Sign up to save tools and stay up to date with the latest in AI
bg
bg
1

GPTBot: OpenAI releases new web crawler

Aug 07, 2023 - news.bensbites.co
OpenAI has introduced a new web crawler named GPTBot, which is used to crawl the web and gather knowledge for its AI features, such as ChatGPT. It also provides AI-generated responses to queries. The user agent token for GPTBot is "GPTBot" and its full user-agent string is provided. OpenAI has also shared the IP ranges used by GPTBot.

Website owners can control GPTBot's access to their site using the robots.txt file. To block GPTBot completely, they can add "GPTBot" to their site's robots.txt. They can also specify which parts of their site GPTBot can access by adding the GPTBot token to their site's robots.txt. This is similar to how one would block other web crawlers like GoogleBot or BingBot.

Key takeaways:

  • GPTBot is OpenAI’s web crawler used to consume knowledge for its AI features and provide AI-generated answers to questions.
  • The user-agent token for GPTBot is 'GPTBot' and it can be blocked from accessing a website or parts of it using robots.txt.
  • OpenAI has published the IP ranges that GPTBot uses, which can be used for more specific control over its access.
  • Website owners can disallow GPTBot from crawling their site if they don’t want OpenAI using their content, similar to blocking other web crawlers like GoogleBot or BingBot.
View Full Article

Comments (0)

Be the first to comment!