Tools / robots.txt Generator
robots.txt Generator
Build a robots.txt file group by group. Add allow and disallow rules per crawler, point at your sitemap, block AI bots in one click, and get a warning before you accidentally close the whole site.
Save this as robots.txt in your site root, so it answers at /robots.txt. It guides crawling, not indexing — use a noindex meta tag to keep a page out of search results.
Frequently asked questions
Does robots.txt keep a page out of Google?
No. It asks crawlers not to fetch a URL, but a blocked page can still be indexed from links pointing at it. To keep a page out of results, let it be crawled and serve a noindex meta tag instead.
Where does the file go?
In the site root, so it answers at https://yourdomain.com/robots.txt. Crawlers do not look anywhere else, and a robots.txt in a subfolder is ignored.
How do I block AI crawlers?
Use the AI crawlers button. It adds groups for GPTBot, CCBot, ClaudeBot, PerplexityBot and Google-Extended, each disallowing the whole site. Google-Extended controls AI training only and does not affect Search.
Is Crawl-delay respected?
Bing and Yandex honor it. Google ignores it — use the crawl rate settings in Search Console instead.
Why do I need a group for *?
Crawlers only follow the most specific group that matches their name. Any bot you did not name explicitly falls back to the * group, and without one it gets no rules at all.
Let's build something you're proud of
I help businesses launch, optimize and maintain their online presence, from code to SEO.
Let's talkRelated tools
htaccess Redirect Generator
Free .htaccess redirect generator for Apache and cPanel. Build 301 and 302 mod_rewrite rules, force HTTPS and pick a canonical www host.
CSS Gradient and Shadow Generator
Free CSS gradient and box-shadow generator. Build linear or radial gradients and layered shadows with a live preview, then copy the CSS.
UUID / ULID Generator
Free online UUID and ULID generator. Create one or many UUIDv4 or ULID values in your browser with optional uppercase output and no signup.
QR Code Generator
Free online QR code generator. Create a PNG QR code from a URL or text in your browser, then download it. No signup and no upload.
Password Generator
Free online password generator. Create strong random passwords with custom length, letters, numbers, symbols and optional ambiguous-character exclusion.
Random Number Generator, Picker & Shuffler
Free online random number generator, list picker and shuffler. Generate a random integer, pick one item from a list, or shuffle lines.