Definition
An AI crawler is a bot that fetches web pages on behalf of an AI system: to collect training data, to build a search index for an answer engine, or to read a page live when a user asks for it. Well-known examples are GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Google's opt-out for AI training), Applebot-Extended, Meta-ExternalAgent, Amazonbot, Bytespider (ByteDance) and CCBot (Common Crawl).
The distinction between those three purposes matters. Training crawlers (GPTBot, ClaudeBot, CCBot) collect text used to train future models; blocking them only takes effect at a next model version. Search index crawlers (OAI-SearchBot, PerplexityBot) fill the index answer engines use to cite sources; blocking them means you no longer appear as a source. User agents (ChatGPT-User, Perplexity-User) fetch a page as soon as a user asks for it in a conversation and do not always respect robots.txt, because the request comes from a person.
AI crawlers behave differently from Googlebot. They almost never execute JavaScript, so client-side rendered content is empty to them. They use shorter timeouts and fetch fewer pages per visit, so a slow server or deep navigation leads to missed pages. And there are many of them: on many sites AI crawlers now make up a large share of bot traffic, which creates server load and sometimes costs.
You steer AI crawlers through robots.txt, per user agent. Many sites block training crawlers on principle or to protect their content, but accidentally block the search crawlers too, which makes them disappear from AI answers. If you want to be visible in ChatGPT search and Perplexity, you must at least allow OAI-SearchBot and PerplexityBot; if you do not want to contribute to training, block GPTBot, ClaudeBot and Google-Extended separately. Some CDNs and firewalls (such as Cloudflare) block AI crawlers by default, so check that setting.
You measure the effect in server logs: which AI user agents visit, how often, which pages they fetch and which status codes they receive. Traze checks in the technical scan whether AI crawlers are allowed, whether published pages have actually been fetched by the search crawlers and whether the content is readable without JavaScript.