AI Optimization
Free Robots.txt Generator
Generate a customized robots.txt file to control which AI crawlers and search engines can access your website. Choose from 20+ AI bots including ChatGPT, Claude, and Perplexity. Essential for AI SEO and Generative Engine Optimization (GEO).
Basic Configuration
/admin//private//tmp/Paths must start and end with /
Crawler Control
AI Crawlers
GPTBot(OpenAI)
Trains ChatGPT models
Allow
OAI-SearchBot(OpenAI)
Powers ChatGPT web search
Allow
ChatGPT-User(OpenAI)
Fetches shared links
Allow
ClaudeBot(Anthropic)
Claude AI crawler
Allow
anthropic-ai(Anthropic)
Claude training data
Allow
claude-web(Anthropic)
Fresh web content
Allow
PerplexityBot(Perplexity)
AI search index
Allow
Google-Extended(Google)
Gemini AI
Allow
Amazonbot(Amazon)
Alexa & recommendations
Allow
Applebot-Extended(Apple)
Apple AI training
Allow
Bytespider(ByteDance)
TikTok's AI
Allow
DuckAssistBot(DuckDuckGo)
Private AI answers
Allow
cohere-ai(Cohere)
Enterprise LLMs
Allow
meta-externalagent(Meta)
Meta AI crawler
Allow
MistralAI-User(Mistral)
French AI company
Allow
Search Engines
Googlebot(Google)
Google Search
Allow
Googlebot-Image(Google)
Google Images
Allow
Googlebot-Mobile(Google)
Google Mobile Search
Allow
Bingbot(Microsoft)
Microsoft Bing
Allow
YandexBot(Yandex)
Russian search engine
Allow
Protected by reCAPTCHA
Quick Tips for Robots.txt Best Practices
1. Test Before Deploying
Always test your robots.txt in Google Search Console before going live
2. Monitor AI Crawler Activity
Track which AI bots visit your site and how often
3. Update Regularly
Review your robots.txt quarterly as new AI crawlers emerge
4. Balance Access and Protection
Allow AI crawlers for visibility while protecting sensitive content
5. Consider Crawl Delay
Set appropriate delays to manage server resources
6. Include Your Sitemap
Help crawlers discover all your important content
💡 Pro tip: Combine your robots.txt with an llms.txt file for complete AI optimization. While robots.txt controls access, llms.txt provides context about your business for AI systems.
Why Control AI Crawler Access?
As AI becomes the primary way users discover information, controlling which AI systems can access your content is crucial. While allowing AI crawlers can increase your visibility in AI-generated responses, you may want to block certain crawlers to protect proprietary content, reduce server load, or maintain control over how your content is used in AI training.
🎯
Content Control
Decide which AI systems can use your content for training or real-time responses
👁️
AI Visibility
Allow helpful AI crawlers to increase your brand mentions in AI responses
⚡
Server Resources
Manage crawler traffic to optimize server performance and reduce costs
Important Notes & Resources
- •Some crawlers (like Perplexity-User) may ignore robots.txt when fetching user-requested pages
- •Robots.txt is publicly visible - don't include sensitive paths that reveal hidden content
- •Not all bots respect robots.txt - it's a request, not enforcement
- •Changes may take days or weeks to be recognized by all crawlers
Frequently Asked Questions
Explore more free tools