Robots.txt Tester
Paste a robots.txt file, enter one or more URLs and choose a crawler. The tester applies Google's matching rules (RFC 9309): it picks the right user-agent group, finds the longest matching rule and highlights it, and flags syntax problems in the file.
Last updated:
Results
File check
Compare crawlers
How to use the Robots.txt Tester
- Paste your robots.txt into the editor (open yoursite.com/robots.txt in a new tab and copy it), or edit the sample.
- Enter URLs or paths to test, one per line. Full URLs and paths like /admin/ both work.
- Choose a crawler such as Googlebot, or type a custom user agent.
- Click Check URLs. Each URL shows Allowed or Blocked with the exact rule and line number that decided it, highlighted in the editor.
- Fix and retest: results update as you type, and the Compare crawlers table shows how every bot treats the first URL.
How robots.txt rules are matched
The tester follows the Robots Exclusion Protocol as standardized in RFC 9309 and implemented by Google:
- Pick one group. The crawler looks for groups whose
User-agentname matches its own (case-insensitive). If none exists it uses theUser-agent: *group. Several groups for the same name are merged. Only one group ever applies. - Find matching rules. Each
AllowandDisallowpath is compared with the start of the URL path plus query string. Matching is case-sensitive. - Longest rule wins. The rule with the longest path (in characters) decides. If an Allow and a Disallow rule have the same length, Allow wins.
- No match means allowed.
/robots.txtitself is always allowed.
Pattern examples
| Rule | Blocks | Does not block |
|---|---|---|
Disallow: /admin | /admin, /admin/, /admin.php, /administrator | /Admin, /blog/admin |
Disallow: /admin/ | /admin/, /admin/users | /admin, /administrator |
Disallow: /*.pdf$ | /guide.pdf, /files/a.pdf | /guide.pdf?v=2, /pdf-tools/ |
Disallow: /*? | /search?q=a, /page?id=1 | /search, /page |
Disallow: /*sessionid= | /cart?sessionid=9, /a?b=1&sessionid=2 | /cart |
Disallow: / | Every URL | Nothing (except /robots.txt) |
Disallow: (empty) | Nothing | Every URL |
Crawler fallbacks used by the tester
Some crawlers obey another group when there is none for their own name. Googlebot-Image, Googlebot-News and Googlebot-Video fall back to Googlebot; Applebot follows Googlebot rules when it has no group of its own; Bingbot falls back to msnbot; YandexBot obeys Yandex groups. AdsBot-Google ignores the * group entirely and only follows rules that name it. AI crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended follow normal group selection, so to opt out of them you need a group that names them, as in the sample file. To create a new file, use the Robots.txt Generator, and check the URLs your sitemap lists with the Sitemap URL Extractor.
Frequently asked questions
How does Google decide whether a URL is blocked?
Google first picks the single group whose user-agent line matches the crawler most specifically, falling back to the * group. Within that group, the Allow or Disallow rule with the longest matching path wins. If an Allow and a Disallow rule match with the same length, Allow wins. If no rule matches, the URL is allowed.
How do * and $ work in robots.txt?
An asterisk matches any sequence of characters, including none, so Disallow: /*.pdf blocks every URL containing .pdf. A dollar sign at the end anchors the rule to the end of the URL, so Disallow: /*.pdf$ only blocks URLs that end in .pdf, not /file.pdf?download=1.
Why does Googlebot ignore my User-agent: * rules?
A crawler obeys only one group. If your file has a group for Googlebot, Googlebot follows only that group and ignores everything under User-agent: *. Copy the general rules into the specific group if they should apply to both.
Does robots.txt remove pages from Google?
No. Disallow stops crawling, not indexing: a blocked URL can still appear in results if other pages link to it, usually without a description. To keep a page out of search results, allow crawling and use a noindex meta tag or X-Robots-Tag header instead.
Are Crawl-delay and Noindex supported?
Google ignores Crawl-delay and stopped supporting Noindex in robots.txt in 2019. Bing and Yandex honor Crawl-delay. The tester lists these lines as notes so you know they will not affect Googlebot.
Is the path matching case-sensitive?
Yes. Paths in Allow and Disallow rules are case-sensitive, so Disallow: /Admin does not block /admin. Directive names and user-agent names are not case-sensitive.
Do other search engines use the same rules?
Bing, Yandex and most major crawlers also use longest-match precedence and support * and $, as standardized in RFC 9309. Smaller or older bots may behave differently or ignore robots.txt entirely, since it is a voluntary standard.
Are all the tools on FreeToolCabin tested for quality and accuracy?
Yes! Every tool on our site undergoes automated Q/A testing in real browser environments to ensure high precision, correct output, and zero errors. If you happen to spot a bug or have a suggestion, please contact us at [email protected] and our team will review it right away.