Robots.txt Generator
Build a valid robots.txt file - set user-agents, allow/disallow paths, crawl-delay, and sitemap URL - then copy or download it instantly.
- Free, no account
- No watermark
- No usage limit
About the Robots.txt Generator
Most robots.txt generators still act like it's 2015. They'll wave Googlebot through and block your /admin/ folder, and that's about the extent of it. But the reason a lot of people go hunting for one now is different. They want the AI scrapers off their content. GPTBot, Google's training crawler, Common Crawl, ClaudeBot, all quietly pulling in whatever you publish. This generator has a one-click Block AI crawlers preset that drops those in for you, so you're not chasing the exact user-agent strings one forum post at a time. Pick a preset, tweak the rules, watch the file build live, then copy or download it. It's free, runs entirely in your browser, and nothing you type gets sent anywhere.
How to use
- Pick a preset (optional). Allow all bots, block all bots, block AI crawlers, or WordPress defaults. Each one fills the form with a working setup you can edit. The WordPress button writes what WordPress core itself serves, which is a
Disallowon/wp-admin/withadmin-ajax.phpallowed back through, and nothing else. - Set the User-agent. A specific bot name like
GooglebotorGPTBot, or leave the*to hit every crawler at once. - List Disallow paths, one per line, for what you want kept out (
/admin/,/cart/). Allow paths carve out exceptions inside the same block. - Add a Crawl-delay if a bot is hammering your server (seconds between requests). Google ignores it, plenty of others don't.
- Drop in your Sitemap URL. It lands on its own line at the bottom, which is exactly where it belongs.
- Need different rules for different bots? Hit + Add rule block and stack as many as you want.
- Copy or download a ready-to-upload
robots.txt.
Blocking the AI crawlers, which most generators skip
The Block AI crawlers preset fills the form with the four bots people ask about most: GPTBot (OpenAI), Google-Extended (Google's AI training fetch), CCBot (Common Crawl, the dataset half the models train on), and ClaudeBot (Anthropic's training crawler). Each gets its own block with Disallow: /. One click and the AI opt-out is written for you.
A user-agent token is a name a company publishes, and names get withdrawn. Anthropic retired anthropic-ai and Claude-Web, so a file still carrying either one blocks nothing of Anthropic's while looking to a reader like it does. A lot of generators and forum answers have not caught up, and this one had not either until 2026-08-25. If you copied a robots.txt from anywhere, open it and search it for those two strings before you trust it.
Google-Extended is a completely separate switch from Googlebot. Blocking Google-Extended tells Google "don't use my pages to train Gemini," and it costs you nothing in normal search. Your pages stay fully crawlable and rankable. Block plain Googlebot instead and you'd wipe yourself off Google entirely.
The four in the preset are a starting point rather than a complete list. New AI crawlers show up constantly. The ones people usually add next are PerplexityBot, Bytespider (ByteDance, the TikTok company), Amazonbot, and Applebot-Extended. Give each its own rule block with Disallow: / and you've built a solid wall. That's the whole point of the multi-block builder, stacking bot after bot without copy-pasting yourself into a syntax error.
Sort the training crawlers from the answer crawlers before you block everything with a familiar name on it. Anthropic runs Claude-SearchBot for the answers that cite sources and Claude-User for a page a person asked Claude to open, and neither of those feeds a training run. Blocking them costs you the citation without buying back any training data.
This only stops crawlers that read robots.txt and choose to obey it. The named ones publicly say they do, and for a reputable AI company that's a real commitment. But a scraper built to ignore the rules will read straight past every line you write. And blocking GPTBot today does nothing about a model that already trained on your site last year, it only affects what gets collected from here on. If you genuinely need content walled off, you need a login or a firewall rule.
The one gotcha worth keeping straight
Disallow controls crawling, whether a bot fetches a page. It does not hide the page, and it doesn't reliably pull it out of Google either. The file is public, anyone can open yourdomain.com/robots.txt and read it, so listing Disallow: /secret-login/ just points a spotlight at the exact thing you meant to bury. And if another site links to a page you disallowed, Google can still show that URL in results, minus the title and description it never got to crawl.
So if your actual goal is "keep this out of search," the tool for that is a noindex tag on the page itself, not a line in here. If you both noindex a page AND block it in robots.txt, Google can't crawl the page to see the noindex, so it can sit in the index indefinitely. The fix is to unblock it, let Google read the noindex, wait for the recrawl, then block it again later if you care about crawl budget. And if plain privacy is what you're after, neither line gives you that. Anything genuinely sensitive belongs behind a real login, since a page stays reachable by anyone who types or guesses its URL.
Before you hit upload
The single scariest line in any robots.txt is Disallow: /. It blocks your entire site. It's what staging environments use to stay out of Google, and it rides along to production more often than anyone likes to admit, someone copies the file up and a week later the traffic's just gone. If your rankings ever drop suddenly for no obvious reason, open your live robots.txt and check this line first.
A quieter mistake: blocking your CSS or JavaScript folders. Google renders the page like a browser to judge it, and a page it sees as broken and unstyled ranks worse than the real thing, so leave those assets crawlable.
On WordPress that mistake has two specific addresses, and an old recipe that still circulates hits both. /wp-includes/ is where core keeps jQuery, and /wp-admin/admin-ajax.php is the endpoint a lot of themes and plugins render front-end content through, so a file that disallows /wp-includes/ or blocks /wp-admin/ without allowing admin-ajax.php back through is hiding your own scripts from the renderer. WordPress core serves neither of those lines. The preset here matches core rather than the recipe.
Then, after you upload, actually open yourdomain.com/robots.txt in a browser. If plain text rules are staring back, you're all set. Your homepage HTML or a 404 means the file isn't being served right and no bot will ever read it. It also has to sit at the domain root, a file at /blog/robots.txt does nothing at all.
Frequently asked questions
If I have a rule for `*` and a rule for Googlebot, does Googlebot follow both?
No. A crawler reads the single user-agent group that matches its name most specifically and obeys only that one. Googlebot sees its own block and ignores the * block completely, including the rules you assumed covered everybody. So a path you want Googlebot kept out of has to be repeated inside the Googlebot block, not left sitting in the catch-all. That's the most common reason a Disallow someone swears they wrote ends up doing nothing.
Can I block by pattern, like every PDF or any URL with a query string?
Yes, though it depends on which crawler is reading. Google, Bing, and the other major bots support two wildcards that never made it into the original spec: * stands in for any run of characters, and $ pins the end of the URL. So Disallow: /*.pdf$ blocks every PDF, and Disallow: /*? catches anything with a ? in it, which is a fast way to keep faceted-search and tracking-parameter URLs out of the crawl. Smaller or older crawlers may treat those symbols as plain text and match nothing, so don't count on wildcards for a bot you can't verify.
Why isn't my Disallow rule matching the page I meant?
Usually one of two reasons. Paths are case-sensitive, so Disallow: /Admin/ leaves /admin/ wide open. And the match runs as a prefix from the root rather than a whole word, so Disallow: /admin with no trailing slash blocks /admin, /admin/, and /administrator in one shot, which may be wider than you wanted. Match the exact capitalization, and add or drop the trailing slash on purpose instead of by habit.
Does Crawl-delay do anything for Google?
No, Googlebot ignores it outright. If Google is crawling too aggressively, that gets managed from Search Console or simply by how fast your server responds. Bing, Yandex, and several smaller crawlers do respect Crawl-delay, so it's still useful against a non-Google bot that's straining your server. This generator only writes the line when you enter a number above zero, so you won't ship a meaningless Crawl-delay: 0.
How do I know a crawler is actually obeying my robots.txt?
Check your server access logs after you publish. A crawler that respects the file stops requesting the paths you disallowed, and you'll see it back off within a day or two of its next visit. The catch is that robots.txt matches on the user-agent name a bot announces, and any scraper can call itself Googlebot to walk through a block meant for something else. That's why Google publishes its crawler IP ranges, so you can reverse-DNS a suspicious visitor and confirm it really came from Google rather than someone borrowing the name, so go by the verified IP address instead of the name a bot announces.
Do I need a separate robots.txt for each subdomain?
Yes. Every host reads its own file at its own root. example.com/robots.txt has zero authority over blog.example.com or shop.example.com, and even http versus https counts as a different host. If you run subdomains, ship a file to each one of them.
I already submitted my sitemap in Search Console. Do I still add it here?
It's worth doing. The Sitemap: line gets read by every engine that fetches your robots.txt, not just the one webmaster tool you pasted the URL into, so it helps Bing and the rest discover new pages too. Think of it as a backstop to the direct submission rather than a swap for it. This tool writes the line on its own at the bottom, outside every rule block, which is where crawlers expect to find it.