AI Crawlers
22 bots documented
Complete directory of AI crawlers and LLM training bots. Track and manage access from ChatGPT, Claude, Gemini, Perplexity, and 50+ AI crawlers.
AI crawlers are specialized web scrapers used by AI companies to train their language models. Understanding and managing these crawlers is essential for modern web management. Our AI crawler directory provides detailed documentation on: - Which AI models are training on your content - How to detect AI crawler traffic - robots.txt configuration for each crawler - Block/allow recommendations for AI training Popular AI crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, and PerplexityBot. Each has different purposes and policies regarding content usage.
All ai crawlers bots
Amazon Kendra
Amazon's ai & llm crawlers
Anthropic-AI
Anthropic's AI crawler
Bard-AI
Google Bard AI crawler
ChatGLM Spider
Zhipu's ai & llm crawlers
ChatGPT Operator
OpenAI's ai & llm crawlers
ChatGPT-User
AIUser-initiated ChatGPT browsing
ClaudeBot
Anthropic's Claude web crawler
Cohere-AI
Cohere's AI crawler
Cohere-Command
Cohere Command crawler
Cohere-Training
Cohere training data crawler
Bing Copilot
AIMicrosoft Bing Copilot AI assistant
DeepseekBot
AIDeepSeek AI crawler
Gemini-AI
AIGoogle Gemini AI crawler
GPTBot
OpenAI's web crawler for training ChatGPT
Grok
AIxAI Grok crawler
MistralAI-User
User-initiated Mistral AI browsing
OAI-SearchBot
OpenAI's ai & llm crawlers
owlin bot
Owlin's ai & llm crawlers
Perplexity-User
User-initiated Perplexity search
PerplexityBot
Perplexity's AI search crawler
sider.ai bot
Vidline Inc.'s ai & llm crawlers
The Knowledge AI
Unknown Author's ai & llm crawlers