Free tool, runs in your browser
Robots.txt generator with AI crawler controls
This robots.txt generator writes a valid file from a few choices: a default allow or block for all crawlers, paths to disallow, separate decisions for common AI crawlers such as GPTBot, ClaudeBot and Google-Extended, and your sitemap URL. Copy the result or download robots.txt and upload it to your domain root.
Ignored by Google.
How to use the Robots.txt Generator
- 1Choose the defaultAllow all is right for most public sites. Block all is for staging sites.
- 2List private pathsAdd paths such as /admin/ or /cart/ that crawlers should skip.
- 3Decide on AI crawlersToggle each AI crawler on or off. Blocking training crawlers does not remove you from search results.
- 4Add your sitemapEnter the absolute sitemap URL so crawlers can find it.
- 5Publish and testUpload to /robots.txt and check it in Search Console's robots.txt report.
How robots.txt works
Crawlers read /robots.txt before fetching pages and follow the group that matches their user agent most specifically. The format is standardised in RFC 9309. It is a request, not a lock: well-behaved crawlers obey it, others may not. Never use it to hide private data.
AI crawler user agents
| User agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data collection |
| OAI-SearchBot | OpenAI | Search results in ChatGPT |
| ChatGPT-User | OpenAI | Fetches pages a user asks about |
| ClaudeBot | Anthropic | Training data collection |
| Google-Extended | Controls use in Gemini models, not Search | |
| PerplexityBot | Perplexity | Search index |
| CCBot | Common Crawl | Open web dataset |
Operators document their crawlers, for example OpenAI's bots page. If you want to be cited by AI assistants, keep the search and user-triggered agents allowed. See how to get cited by AI assistants.
Frequently asked questions
Will blocking GPTBot remove me from ChatGPT answers?
GPTBot is the training crawler. ChatGPT search uses OAI-SearchBot and user requests use ChatGPT-User, which are separate user agents. Block or allow each one on purpose.
Does robots.txt stop a page from being indexed?
Not reliably. A blocked URL can still be indexed if other sites link to it. Use a noindex meta tag on a crawlable page to keep it out of search results.
Where does robots.txt go?
At the root of each host, for example https://example.com/robots.txt. Subdomains need their own file.