Definition
GPTBot is OpenAI's web crawler that fetches publicly accessible web pages to collect training data for future models such as those behind ChatGPT. The bot identifies itself with the user agent 'GPTBot' and has been active since August 2023. OpenAI publishes the IP ranges GPTBot crawls from, so you can distinguish real visits from bots impersonating it.
GPTBot is not OpenAI's only bot, and that distinction is crucial for your visibility. OAI-SearchBot builds the search index ChatGPT uses to retrieve and cite sources when it consults the web. ChatGPT-User fetches a specific page when a user shares a link in a conversation or asks for current information. GPTBot itself serves training only. So if you block GPTBot, you remain citable in ChatGPT search, as long as OAI-SearchBot is allowed.
You steer GPTBot through robots.txt. 'User-agent: GPTBot' followed by 'Disallow: /' blocks the whole site; with 'Disallow: /private/' and 'Allow: /blog/' you exclude sections. OpenAI states that GPTBot respects robots.txt and skips pages behind a paywall, pages collecting personal data, or pages that violate its policies. Changes to robots.txt are picked up on the next crawl visit; data already collected is not removed retroactively.
Whether to allow GPTBot is a strategic choice. Advocates of blocking point to copyright and the lack of compensation for training data; many publishers and media companies block it for that reason. Advocates of allowing argue that models that know your brand, products and expertise from training mention you more often and more accurately for questions without web access, and that the influence of training on brand knowledge only becomes visible at a next model version. For most businesses that want to be found, visibility outweighs protection.
Either way, check what happens now: many CDNs and security plugins block GPTBot and sometimes OAI-SearchBot too by default, without the site owner knowing. Traze checks the robots.txt and your server's actual response to OpenAI's crawlers in the technical scan and reports it when the search crawler is being blocked unintentionally.