Free score
Get a practical score instead of waiting for a paid demo or enterprise onboarding.
Generative AI Crawler Ecosystem
AI User-Agent Crawlers to Allow for Citations
Blocking these agents in robots.txt or via Cloudflare WAF prevents your brand from appearing in AI-generated answers and search citations.
| Crawler / User-Agent | Organization | Role in AI Citations | GEO Status |
|---|---|---|---|
GPTBot | OpenAI | Powers SearchGPT & ChatGPT web search indexing. | ✓ Recommended |
ChatGPT-User | OpenAI | Performs live web browsing on behalf of active ChatGPT users. | ✓ Recommended |
OAI-SearchBot | OpenAI | Dedicated crawler for SearchGPT AI search citation links. | ✓ Recommended |
ClaudeBot | Anthropic | Powers web retrieval and citations for Claude 3.7 & Claude Sonnet. | ✓ Recommended |
PerplexityBot | Perplexity AI | Real-time conversational citations and answer grounding. | ✓ Recommended |
Google-Extended | Training control token for Gemini models and AI Overviews. | ✓ Allow | |
Applebot-Extended | Apple | Feeds Apple Intelligence on iOS 18+ and macOS Sequoia. | ✓ Recommended |
Bytespider | ByteDance | Retrieval crawler for ByteDance generative systems. | ✓ Allow |
cohere-ai | Cohere | Enterprise search RAG and knowledge grounding. | ✓ Allow |
Standard Snippet
Recommended robots.txt Template for AI Search
Add this to the root of your domain (e.g. https://yourdomain.com/robots.txt) to guarantee uninterrupted AI citation access.
# FoxyGEO Recommended AI Search Robots.txt Policy # Allows generative search engines to cite your content while protecting private directories User-agent: * Allow: / Disallow: /admin/ Disallow: /private/ Disallow: /api/private/ # Conversational AI Search & Citation Engines (Keep Allowed for GEO) User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Applebot-Extended Allow: / # LLM Discovery Standards # https://yourdomain.com/llms.txt # https://yourdomain.com/sitemap.xml Sitemap: https://yourdomain.com/sitemap.xml
Actionable report
Every scan provides a comprehensive report with recommendations and prompt opportunities.
AI crawler ready
Designed around what AI crawlers actually need: canonical pages, schema, sitemap, robots, and llms.txt.