Robots.txt Generator
Build a robots.txt file with allow and disallow rules for any crawler, add your sitemap, and block AI training or AI search bots with ready-made presets. Then test any URL to see whether a bot may crawl it.
1. All crawlers User-agent: *
2. Specific crawlers
3. AI crawlers
4. Sitemaps
Test a URL
How to use the Robots.txt Generator
- Choose the default for all crawlers: allow everything, block everything or add your own Allow and Disallow rules.
- Add groups for specific bots if one crawler, such as Googlebot-Image, needs different rules.
- Pick AI crawlers to block. Use Block all AI bots, Block AI training only, or tick individual bots.
- Add your sitemap URL so search engines can find all your pages.
- Test a URL for any bot, then copy or download robots.txt and upload it to the root of your site.
Robots.txt syntax reference
| Directive | Example | Meaning |
|---|---|---|
User-agent | User-agent: GPTBot | Starts a group of rules for the named crawler; * means every crawler without its own group |
Disallow | Disallow: /admin/ | Do not crawl URLs whose path starts with /admin/ |
Allow | Allow: /admin/help/ | Exception inside a disallowed folder |
* wildcard | Disallow: /*?sort= | Any characters: blocks every URL containing ?sort= |
$ end anchor | Disallow: /*.pdf$ | Only URLs that end in .pdf |
Sitemap | Sitemap: https://example.com/sitemap.xml | Location of a sitemap; must be a full URL, can appear anywhere |
Crawl-delay | Crawl-delay: 5 | Seconds between requests (Bing, Yandex); Google ignores it |
Which rule wins?
Under the Robots Exclusion Protocol (RFC 9309), which Google and Bing follow, a crawler first picks the most specific group that names it. Googlebot-Image uses a Googlebot-Image group if there is one, otherwise a Googlebot group, otherwise *. It does not combine its own group with the * group. Inside the chosen group, the rule with the longest matching path wins, and if an Allow and a Disallow rule are equally long, Allow wins.
Worked example with these rules for User-agent: *:
| URL path | Matching rules | Result |
|---|---|---|
/shop/ | none | Allowed |
/admin/users | Disallow: /admin/ (7 characters) | Blocked |
/admin/help/faq | Disallow: /admin/ (7) and Allow: /admin/help/ (12) | Allowed (longer rule) |
/files/report.pdf | Disallow: /*.pdf$ | Blocked |
/files/report.pdf?v=2 | none (the URL does not end in .pdf) | Allowed |
Paste the same rules into the tester above to check any URL against your own file.
Blocking AI crawlers: what to know
- Training versus search. Bots such as GPTBot, ClaudeBot, CCBot and Google-Extended gather content for training models. Bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot power AI search answers that can link back to you. Block only the first kind if you want to stay visible in AI search.
- User-requested fetchers (ChatGPT-User, Claude-User, Perplexity-User) visit a page because a person asked for it. Companies treat these differently, and some state that they may not apply robots.txt to them.
- Google-Extended and Applebot-Extended are tokens, not separate crawlers. Blocking them does not affect Google Search or Apple search results.
- Robots.txt is voluntary. Reputable crawlers obey it, but it is not access control. Use server rules or authentication for content that must stay private.
- Google reads at most 500 KiB of a robots.txt file, and new rules usually take effect within a day, when the cached copy is refreshed.
Frequently asked questions
Where do I put robots.txt?
In the root folder of your site, so it loads at https://www.example.com/robots.txt. Crawlers only look there. Each subdomain and each protocol (http and https) needs its own file.
Does robots.txt stop AI companies from using my content?
It works for crawlers that respect it. OpenAI (GPTBot), Anthropic (ClaudeBot), Google (Google-Extended), Apple (Applebot-Extended), Common Crawl (CCBot) and others publicly say they follow robots.txt. It is a request, not a lock: it does not remove content already collected and cannot stop crawlers that ignore it.
Will blocking AI bots hurt my Google ranking?
No. Google-Extended and Applebot-Extended only control AI training and grounding; Googlebot and Applebot keep crawling for search. Blocking AI search bots such as OAI-SearchBot or PerplexityBot can remove your pages from those AI search products, so block only the training bots if you still want to appear there.
Can robots.txt hide a page from Google?
Not reliably. Disallow stops crawling, but a blocked URL can still be indexed if other sites link to it, usually without a description. To keep a page out of search results, allow crawling and use a noindex meta tag or X-Robots-Tag header instead.
What do * and $ mean in robots.txt?
An asterisk matches any sequence of characters and a dollar sign marks the end of the URL. For example, Disallow: /*.pdf$ blocks every URL ending in .pdf, and Disallow: /*? blocks every URL that contains a query string.
Does Google support Crawl-delay?
No. Googlebot ignores Crawl-delay; Google adjusts its crawl rate automatically. Bing and Yandex read it as the number of seconds to wait between requests.