Tools/Seo Tools/Robots.txt Generator

Robots.txt Generator Free - Google Compliant

Generate a clean and Google-compliant robots.txt file for your website. Easily create allow or disallow rules, add sitemap URLs, and download the file instantly free with no login required.

If this tool isn’t working as expected, please take a screenshot of the error and report the problem here so we can investigate and improve it.

About this tool

A misconfigured robots.txt file can accidentally block search engines from crawling your entire website - or fail to protect admin pages and staging environments from indexing. This tool generates a syntactically correct file using the directives search engines recognize (User-agent, Allow, Disallow, Sitemap), eliminating manual writing errors.

One caveat worth knowing: this tool also lets you add a Crawl-delay value, but Google does not support or read the Crawl-delay directive at all - some other search engines (like Bing) may honor it, so only add it if you're specifically trying to slow down a non-Google crawler.

A robots.txt generator creates a properly formatted robots.txt file for your website - controlling which pages search engine crawlers can access, which to ignore, and where the sitemap is located. This tool builds one User-agent group per file using Allow, Disallow, Crawl-delay, and Sitemap directives - the core directives search engines actually read.

How to Use Robots.txt Generator

Choose Rules

Select allow or disallow rules for specific paths under one user-agent at a time.

→

Customize Options

Add crawl delay and sitemap URL if required.

→

Preview Robots.txt

Instantly preview the generated robots.txt file in real format.

→

Copy or Download

Copy the robots.txt content or download it as a file.

Common Workflows

Start From a Preset

Click Allow All, Block All, or Block Admin to load a ready-made rule set for user-agent "*", then adjust from there.

Add Rules for a Specific Bot

Type a user-agent name (e.g., "Googlebot", "GPTBot", or "*" for all bots), then add Allow or Disallow rules with a path - each rule set applies to the one user-agent you've entered.

Add a Sitemap and Crawl-Delay

Optionally add your Sitemap URL and a Crawl-delay value in the Global Settings section - both are appended to the output.

Preview, Copy, and Download

Check the live preview on the right, then click Copy Code or Download File to get your robots.txt.

Best For

Generates a robots.txt file for one user-agent group at a time - type any bot name (e.g., *, Googlebot, GPTBot) and add Allow/Disallow rules for it, so you can target AI crawlers by name, one at a time.
Includes a Sitemap URL line - helps search engines find and index your sitemap automatically.
Four one-click presets - Allow All, Block All, Block Admin, and Custom/Reset - as starting points instead of a blank form.

Examples

A basic robots.txt with an admin block and sitemap

Input

User-agent: *, Rules: Allow /, Disallow /admin, Disallow /wp-admin, Sitemap: https://example.com/sitemap.xml

Output

User-agent: *
Allow: /
Disallow: /admin
Disallow: /wp-admin

Sitemap: https://example.com/sitemap.xml

Rules are automatically sorted alphabetically by path in the output, regardless of the order you added them in.

Use Cases

Blocking admin or staging paths

Use the Block Admin preset or add your own Disallow rules for paths like /admin or /staging.

Targeting a specific AI crawler

Type the bot's exact user-agent name (e.g., GPTBot) and add Disallow rules for it - build and copy one bot's block at a time.

Publishing your sitemap location

Add your Sitemap URL in Global Settings so it's included as a Sitemap line in the file.

Common Mistakes

Problem

Assuming multiple bots can be configured in one generation

Solution

The tool builds one User-agent group at a time - to give different bots different rules, generate and copy each group separately, then combine them yourself in the final file.

Problem

Adding Crawl-delay and expecting Google to honor it

Solution

Google does not read the Crawl-delay directive at all - it only has an effect on crawlers that support it, such as Bing's.

Problem

Entering 0 or a negative Crawl-delay value

Solution

This clears the entire preview, not just the Crawl-delay line - use a positive number, or leave the field blank.

Problem

Entering an invalid Sitemap URL

Solution

An invalid URL is flagged with a red border and also clears the entire preview until it's fixed or removed.

Problem

Assuming Disallow removes a page from Google's index

Solution

Disallowing a URL only blocks crawling - it doesn't guarantee the page stays out of the index. Use a noindex meta tag on the page itself to control indexing.

Tips & Best Practices

Start from a preset, then customize

The Allow All, Block All, and Block Admin presets cover the most common starting points - load one and adjust the rules from there instead of starting blank.

Type any bot name to target it specifically

The User-agent field accepts any text, so you can target a specific crawler like Googlebot, Bingbot, or an AI crawler like GPTBot or CCBot by typing its exact name.

Only add Crawl-delay if you have a specific non-Google crawler in mind

Since Google ignores this directive entirely, only set it if you're trying to slow down a different bot that's overloading your server.

Test your finished file with Google Search Console

After publishing your robots.txt at your site's root, use Search Console's own tools to confirm Google can read it as expected - this generator doesn't validate the file against a live crawler itself.

Limitations

One user-agent group per generation

All rules you add apply to a single User-agent value - there's no way to configure multiple bots with different rules in one generated file. Build and combine multiple groups manually if you need that.

Crawl-delay isn't read by Google

The Crawl-delay directive is included if you set it, but Google's crawlers ignore it entirely - only some other search engines may honor it.

No rule testing or validation against a live site

The tool generates syntactically correct output but doesn't test it against your actual site or verify how any specific crawler will interpret it.

No dedicated AI-crawler presets

There's no built-in button or list for AI crawlers - you target them by typing their exact user-agent name yourself, the same way as any other bot.

Doesn't control search-index removal

This generates a robots.txt file only - it doesn't prevent an already-indexed page from appearing in search results (that requires a noindex tag, not a robots.txt rule).

0 or negative Crawl-delay, or an invalid Sitemap URL, blanks the whole preview

Both of these input states clear the entire output, not just the affected line - double-check these two fields if the preview unexpectedly goes empty.

Comparisons

Disallow Rule vs. Noindex Meta Tag

Both control search engine behavior, but they do fundamentally different jobs.

	Disallow (robots.txt)	Noindex (meta tag on the page)
What it controls	Whether crawlers are allowed to request the URL	Whether the page is allowed to appear in search results
Can the page still be indexed?	Yes - if other sites link to it, it can still be indexed without being crawled	No - noindex reliably keeps it out, but only if the page can still be crawled to see the tag
Where it lives	One shared file at your site's root	A meta tag on each individual page

Which should you use?

Use Disallow to manage crawler traffic and server load; use noindex when your actual goal is keeping a specific page out of search results.

FAQs

The most common question is what robots.txt can and can't do. It controls which pages crawlers are allowed to request - it does not remove a page from Google's index by itself (a blocked page can still appear in search results if other sites link to it; use a noindex meta tag to actually keep a page out of the index). The FAQ below also covers crawl-delay, AI crawlers, and where to place the file.

Does Google support the Crawl-delay directive?

No. Google's crawlers ignore Crawl-delay entirely, even though this tool lets you add it. Some other search engines, like Bing, may honor it - only set it if you're targeting a non-Google crawler.

Does blocking a page in robots.txt remove it from Google's index?

No. Disallowing a URL only tells crawlers not to request it - a blocked page can still appear in search results if other sites link to it. To keep a specific page out of the index, use a noindex meta tag on that page instead.

Can I block AI crawlers like GPTBot?

Yes. Type the bot's exact user-agent name (for example, GPTBot, CCBot, or Google-Extended) into the User-agent field and add Disallow rules for it. There's no dedicated AI-crawler preset - you configure one bot's rules at a time, the same way as any other user-agent.

Can I set different rules for multiple bots in one file?

Not in a single generation. This tool builds one User-agent group at a time - generate and copy the rules for each bot separately, then combine the groups yourself into one final robots.txt file.