Skip to content

Robots.txt Generator - Build Robots.txt Online

Presets
Import existing robots.txt
Replaces the rules and sitemaps below.
Rules
Rule 1
Sitemaps
Output
User-agent: *
Allow: /
Disallow: /admin
Disallow: /api
Disallow: /private
Disallow: /tmp
Test a URL
Blocked by Disallow: /adminUsing the "User-agent: *" group.

Follows Google's rules: the most specific user-agent group applies, the longest matching path wins, and Allow wins a tie. Supports * and $.

Build a robots.txt file

Start from a preset, then edit the rule groups: each has a user agent, Allow and Disallow paths, and an optional Crawl-delay. Add your sitemap URLs at the bottom. The preview updates as you type. Save the result as robots.txt at the root of the host, such as https://example.com/robots.txt. Each subdomain needs its own file; shop.example.com does not read the one on example.com.

Presets

  • Allow all. Every crawler can fetch every path.
  • Block all. Disallow: / for everyone. For staging sites, but password-protect them as well, since a blocked staging URL can still be indexed from links.
  • Block AI crawlers. Keeps search engines in and asks 12 AI crawlers to stay out.
  • Standard SEO. Allows everything except /admin, /api, /private and /tmp.

Example

User-agent: *
Disallow: /admin
Allow: /admin/public/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Here /admin/settings is blocked for every crawler, but /admin/public/logo.png is allowed, because /admin/public/ is the longer match. GPTBot has its own group, so it ignores the * rules and is blocked everywhere. A crawler only ever follows one group, which catches people out: rules in * do not also apply to a bot that has a group of its own.

Common mistakes

  • Blocking CSS and JavaScript folders. Google renders pages, and a page it cannot render can rank worse.
  • Using robots.txt to hide private URLs. The file is public, so it lists exactly where to look.
  • Paths are case-sensitive: Disallow: /Admin does not block /admin.

Related Tools

Frequently Asked Questions

Can robots.txt block AI crawlers?
It can ask them to stay away, and the large ones say they honour it. The Block AI crawlers preset adds a Disallow: / group for 12 user agents: GPTBot, CCBot, ClaudeBot, Claude-Web, anthropic-ai, Google-Extended, Applebot-Extended, Bytespider, cohere-ai, PerplexityBot, Amazonbot and meta-externalagent. robots.txt is a request, not access control. A crawler that ignores it has to be blocked at the server or CDN.
Will blocking Google-Extended remove me from Google Search?
No. Google-Extended is a control token, not a separate crawler. Googlebot still crawls and indexes the site; the rule only tells Google not to use the content for training Gemini models. Blocking PerplexityBot is different: it is Perplexity's search crawler, so your pages stop appearing as sources there.
Does robots.txt stop a page being indexed?
No. It controls crawling, not indexing. A blocked URL can still show up in Google, usually with no description, if other sites link to it. To keep a page out, allow crawling and add a noindex meta tag or X-Robots-Tag header. Google has to fetch the page to see the noindex.
How does the tester decide if a URL is blocked?
It follows Google's rules. The crawler uses the group whose User-agent matches its name exactly, or the * group if none does. Within that group the longest matching rule wins, and if an Allow and a Disallow are the same length, Allow wins. * matches any characters and $ anchors the end, so Disallow: /*.pdf$ blocks /files/report.pdf but not /report.pdf?v=2.
Can I test my existing robots.txt?
Yes. Open Import existing robots.txt, paste the file and click Import. The rules load into the builder, where you can test URLs, edit them and copy the result.
Does Google support Crawl-delay?
No. Googlebot ignores Crawl-delay; Bing and Yandex honour it. If Googlebot is overloading your server, return 503 or 429 responses for a while and it slows down.