How to use the Robots.txt Generator
- Pick the default for all crawlers: allow everything, block everything or list custom paths to disallow and allow.
- Add groups for individual crawlers, or click Block AI training crawlers to opt out of the best-known AI bots in one step.
- Paste the full URL of your XML sitemap so crawlers can find every page.
- Fix anything flagged in the warnings, and type a few paths into the tester to confirm the rules do what you expect.
- Download the file and upload it to the root of your site so it is reachable at /robots.txt.
Features
- Allow-all, block-all or custom path rules for every crawler
- Separate groups for Googlebot, Bingbot, SEO tools and AI crawlers
- One-click opt-out of AI training crawlers such as GPTBot and CCBot
- Automatic clean-up of full URLs and paths missing a leading slash
- Path tester that applies the longest-match rule from RFC 9309
- Merges identical groups into one block for a tidy file
What robots.txt does
Robots.txt is a plain text file at the root of a website that tells automated crawlers which URLs they may request. Search engines read it before crawling, and most reputable bots respect it. The format was used informally from 1994 and was finally published as an internet standard, RFC 9309, in 2022. It is a request, not a lock: any bot can ignore it, so never rely on robots.txt to hide private content. Use passwords or server rules for that.
How the rules are read
A file is made of groups. Each group starts with one or more User-agent lines naming crawlers, followed by Disallow and Allow rules. A crawler uses the group that names it most specifically and ignores the rest, so Googlebot follows a User-agent: Googlebot group instead of the * group, not as well as it. Within a group, the rule with the longest matching path wins, and if an Allow and a Disallow rule are equally long, Allow wins.
| Rule | Effect |
|---|---|
Disallow: / | Blocks the whole site |
Disallow: (empty) | Allows everything |
Disallow: /admin/ | Blocks the folder and everything inside it |
Disallow: /*.pdf$ | Blocks URLs ending in .pdf |
Disallow: /*?sort= | Blocks URLs containing a sort parameter |
Allow: /admin/help/ | Re-opens a sub-folder inside a blocked folder |
Paths are case-sensitive and match from the start of the URL path, so /Admin/ and /admin/ are different rules.
Blocking AI crawlers
Several AI companies publish the user-agent tokens their crawlers obey. GPTBot collects data for OpenAI’s models, CCBot builds the Common Crawl dataset that many models are trained on, ClaudeBot is Anthropic’s crawler, and Bytespider belongs to ByteDance. Google-Extended and Applebot-Extended are not separate crawlers but control tokens: blocking them tells Google and Apple not to use your content for their AI models while normal search crawling continues. PerplexityBot gathers pages for Perplexity’s answer engine, so blocking it also means Perplexity is less likely to cite you. Decide whether visibility in AI answers matters to you before blocking everything.
Crawl-delay and sitemaps
Crawl-delay asks a crawler to wait a number of seconds between requests. Bing and Yandex honor it; Google ignores it and adjusts its crawl rate automatically based on how quickly your server responds. A Sitemap line can appear anywhere in the file, must be a full URL, and helps every search engine discover your pages. Create one with the XML Sitemap Generator.
Mistakes that hurt rankings
- Blocking pages you want removed. If a page is disallowed, Google cannot see its noindex tag, and the URL can stay in results. Allow crawling and add noindex with the Meta Tag Generator instead.
- Blocking CSS and JavaScript. Google renders pages like a browser; hiding scripts and styles can make your pages look broken to it.
- Leaving a staging file live. A forgotten
Disallow: /from a test site can remove a whole site from search over time. - Wrong location. The file must live at the root of each host, for example https://shop.example.com/robots.txt; a file in a sub-folder is ignored.
Frequently asked questions
Where do I put the robots.txt file?
Upload it to the top-level folder of your website so it opens at https://yourdomain.com/robots.txt. Each subdomain and protocol needs its own file.
Does blocking a page in robots.txt remove it from Google?
Not reliably. A blocked URL can still appear in results if other sites link to it. To remove a page, let Google crawl it and add a noindex robots meta tag.
Will blocking AI crawlers affect my Google rankings?
No. Blocking Google-Extended, GPTBot or CCBot does not change how Googlebot crawls or ranks your site. Only blocking Googlebot itself would affect Google Search.
Why does Google ignore Crawl-delay?
Google sets its crawl rate automatically based on how your server responds. If Googlebot is overloading your server, return 503 or 429 responses temporarily; Google slows down on its own.
Can I use wildcards in robots.txt?
Yes. Major search engines support * to match any sequence of characters and $ to mark the end of a URL. For example, Disallow: /*.pdf$ blocks every URL that ends in .pdf.