sprytools

robots.txt Generator

Generate a valid robots.txt file visually. Add User-agent blocks for Googlebot, Bingbot, or all bots, set Disallow and Allow paths, add a Sitemap URL, and configure Crawl-delay. Live preview updates instantly. Copy or download your robots.txt in one click. 100% free, client-side.

 
  
 
  
 
 
Did we solve your problem today?
 
   
 
 
                            
  
What is a robots.txt Generator?


A robots.txt generator creates the robots.txt file that tells search engine crawlers which pages on your website they can and cannot access. The file lives at the root of your domain — https://example.com/robots.txt — and is one of the first things any crawler checks before indexing your site.


How to Use This Tool


    

  1. Add a User-agent block — click “Add Block” and choose a bot from the dropdown (or type a custom name). Start with * to set rules for all bots.
    2. 

  2. Add Disallow paths — enter paths you want to block (e.g. /admin/, /private/). Paths must start with /.
    3. 

  3. Add Allow paths — use these to explicitly allow subpaths within a blocked directory (e.g. allow /private/public/ inside a blocked /private/).
    4. 

  4. Set Crawl-delay (optional) — enter the number of seconds the bot should wait between requests. Note: Googlebot ignores this; use Google Search Console to control its crawl rate.
    5. 

  5. Enter your Sitemap URL — helps crawlers find your XML sitemap (e.g. https://example.com/sitemap.xml).
    6. 

  6. Copy or Download — click “Copy” to copy the generated file, or “Download” to save it as robots.txt.
    7. 



robots.txt Syntax





































DirectiveExampleMeaning
User-agentUser-agent: *Which bot this block applies to
DisallowDisallow: /admin/Paths the bot must not crawl
AllowAllow: /admin/public/Exceptions within a Disallow
Crawl-delayCrawl-delay: 10Seconds to wait between requests
SitemapSitemap: https://…/sitemap.xmlLocation of your XML sitemap


Common Mistakes


Empty Disallow


Disallow: (with no value) means allow everything — the opposite of what most people intend. To block all crawling, use Disallow: /.


Paths Without a Leading Slash


All paths in Disallow and Allow must start with /. Disallow: admin/ is invalid; the correct form is Disallow: /admin/.


Expecting robots.txt to Hide Pages


robots.txt prevents crawling, not indexing. If another site links to a blocked URL, Google may still show that URL in search results without visiting it. Use <meta name="robots" content="noindex"> on the page itself to prevent indexing.


Googlebot Ignoring Crawl-delay


Googlebot does not respect Crawl-delay. To slow down Google’s crawl rate, use the crawl rate setting in Google Search Console.


Example: Standard Website


User-agent: *
Disallow: /admin/
Disallow: /checkout/
Allow: /admin/public/

Sitemap: https://example.com/sitemap.xml


Example: Block All Bots


User-agent: *
Disallow: /


Example: Block AI Training Bots


User-agent: *
Disallow: /

User-agent: Googlebot
Disallow: /admin/

User-agent: GPTBot
Disallow: /

User-agent: Claude-Web
Disallow: /


Privacy


Everything runs in your browser. No paths, URLs, or configuration data you enter are ever sent to a server.
 
 
FAQ
 
 What is a robots.txt file? 
A robots.txt file tells search engine crawlers which pages or directories on your website they are allowed or not allowed to crawl. It sits at the root of your domain (e.g. https://example.com/robots.txt) and is checked by crawlers before they index your site.
 
 What does Disallow: (empty) mean? 
An empty Disallow: directive — written as "Disallow:" with nothing after the colon — means "allow everything". It is commonly added after a User-agent line as a way to explicitly allow all bots. Omitting Disallow entirely has the same effect.
 
 How do I block all search engines from my site? 
Add a User-agent block for * (all bots) with "Disallow: /". This tells every crawler not to crawl any page. Note: robots.txt is an advisory standard — malicious bots may ignore it.
 
 What is Crawl-delay? 
Crawl-delay tells a bot how many seconds to wait between requests to your server. For example, "Crawl-delay: 10" means the bot should wait 10 seconds between each request. Note: Googlebot ignores Crawl-delay — use Google Search Console to control Googlebot's crawl rate instead.
 
 Does robots.txt prevent pages from being indexed? 
No. Blocking a page in robots.txt prevents it from being crawled, but if another site links to that page, Google may still index the URL without visiting it. To prevent indexing, use a "noindex" meta robots tag on the page itself.
 
 Can I have multiple User-agent blocks? 
Yes. You can have as many User-agent blocks as you need. Each block applies only to the specified bot. Start with a "*" block for all bots, then add specific blocks for bots that need different rules.