An AI crawler is a bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches website content specifically for AI model training or for live retrieval to power AI-generated search and answer responses.
Unlike traditional search engine crawlers focused purely on indexing for search results, AI crawlers may be gathering content for entirely different purposes, including training future model versions rather than immediate search indexing.
Businesses can choose whether to allow or block specific AI crawlers through robots.txt, weighing the potential visibility benefits of AI citation against concerns about content being used for AI training without direct compensation.
We help clients make informed AI crawler access decisions within our Technical SEO service.