Robots.txt Generator

Build a valid robots.txt with allow/disallow rules, crawl-delay and sitemap. Free, in your browser.

robots.txt

Place the file at your site root (/robots.txt). Remember: robots.txt guides crawlers but doesn't enforce privacy.

Free to use — premium coming soon

FREE
  • Allow/disallow rules
  • Sitemap & crawl-delay
  • Copy & download
PREMIUM
  • Remove ads
  • Per-bot rules & templates

About the Robots.txt Generator

A robots.txt file is a plain-text instruction sheet that lives at the root of your domain (for example example.com/robots.txt) and tells web crawlers which parts of your site they may or may not request. This generator builds that file for you: pick the user-agents, list the paths you want to block or permit, add your sitemap URL, and copy the finished output. It is the implementation of the long-standing Robots Exclusion Protocol, the same convention search engines like Google and Bing have followed for decades, now formalised as an internet standard (RFC 9309).

Use this tool when you are launching a site, restructuring URLs, or want to keep crawlers out of low-value areas such as internal search results, faceted-filter URLs, cart and checkout paths, staging directories, or admin folders. A clean robots.txt helps search engines spend their crawl budget on pages that matter and stops thin or duplicate URLs from clogging the crawl queue. It is not an SEO magic button, but on large sites it can meaningfully reduce wasted crawling. For a brand-new small site you may need only a Sitemap line and an open policy.

Each rule group starts with a User-agent line, followed by one or more Disallow and Allow directives whose values are URL paths matched from the left. Disallow: /private/ blocks that folder and everything under it; Allow can carve out an exception inside a blocked path; an empty Disallow value (Disallow:) blocks nothing. Field names are case-insensitive but the path values are case-sensitive, and the longest, most specific matching rule wins. The Sitemap directive takes a full absolute URL and can appear multiple times. Note that Google ignores crawl-delay, while Bing and Yandex still honour it, so this generator labels that directive as optional and non-universal.

Important: robots.txt is a request, not a lock. Compliant crawlers obey it, but the file is publicly readable by anyone and badly behaved bots can ignore it entirely. Because it merely lists paths, it can actually advertise the sensitive folders you hoped to hide. It is therefore not a security control and not a reliable way to keep a page out of search results, since a disallowed URL can still be indexed if other sites link to it. To truly remove a page from search use a noindex tag or X-Robots-Tag header (the page must stay crawlable for that to be seen), and protect private data with passwords or server-side access controls. This tool runs entirely in your browser, so nothing you type is uploaded or stored.

Frequently asked questions

Where does the robots.txt file need to be placed?

It must live in the root directory of the host it applies to and be reachable at https://yourdomain.com/robots.txt. A file in a subfolder or with a different name is ignored, and each subdomain (and the http vs https version) needs its own file.

Does Disallow stop a page from showing up in Google?

Not reliably. Disallow only asks crawlers not to fetch the URL, but Google can still index that URL if other pages link to it, sometimes showing it with little or no description. To actually keep a page out of results, use a noindex meta tag or X-Robots-Tag and leave the page crawlable, or require a password.

Is crawl-delay supported by search engines?

It is supported by some, not all. Google ignores crawl-delay entirely (adjust crawl rate in Search Console instead), while Bing and Yandex still respect it. Because it was never part of the original standard, treat it as optional and crawler-specific.

Can I use robots.txt to protect private or admin pages?

No. robots.txt is publicly viewable, so listing /admin/ or /api/ in it can broadcast exactly which paths exist. Polite bots obey it but malicious ones do not. Use authentication, IP restrictions, or server-side access control for anything that must stay private.

What happens if my site has no robots.txt file at all?

A missing file (returning a 404) is treated as fully allowed, so crawlers may access every public URL. That is fine for many small sites. Add a robots.txt when you want to block specific paths, declare your sitemap, or manage crawl traffic on a larger site.

From our blog

Keyword Density in 2026: Why the Number Matters Less Than You Think

By the Super Simple Digital Tools Team · Updated June 2026

Keyword density was once treated as a dial you could turn to improve rankings: hit 2%, the theory went, and search engines would reward you. That era is over. Google has confirmed it does not treat keyword density as a direct ranking factor, and its systems now evaluate meaning, user intent, and overall page quality rather than counting how many times a phrase appears. A density report still tells you something useful, but what it measures is the texture of your writing, not your future position in search results.

The most practical way to use a density check is as an editing pass. Run your draft through it and look at the top of the list. If your intended topic appears prominently and reads naturally in context, good. If an unrelated filler word dominates, or your main phrase barely registers, that is a writing problem worth fixing. The goal is balance: copy that signals its subject clearly without hammering the same words into every sentence. The percentages are a mirror, helping you see repetition you stopped noticing while drafting.

It helps to understand what the number actually represents. Density is a fraction: occurrences divided by total words, times 100. That means the same keyword count produces a very different percentage in a 300-word note versus a 2,000-word guide. Chasing a fixed figure therefore makes little sense, because the same natural writing scores differently depending on length. Treat the percentage as relative context, not an absolute target, and compare terms against each other rather than against an imaginary ideal.

The real risk at the high end is keyword stuffing. Google's spam policies describe it as filling a page with words or phrases to manipulate rankings, often unnaturally or out of context, and warn that such pages may rank lower or be removed from results entirely. You rarely reach that point by writing for a reader. You reach it by writing for a number, repeating a phrase well past what a human would tolerate. If a density check shows one term wildly above everything else, read that section aloud, you will usually hear the problem immediately.

Modern optimization is less about one keyword and more about covering a topic the way a knowledgeable writer would, using related terms, synonyms, and natural variations. That is the principle behind semantic SEO and approaches like TF-IDF, which weigh a term's frequency on a page against how common it is across many pages. You do not need to calculate any of that to benefit from it. Write thoroughly and naturally, then use a density check to confirm you have not leaned too hard on a single phrase or accidentally buried your main subject.

  • Paste the full body text, not a short excerpt, so the percentages reflect the real article rather than a fragment.
  • Sort by the longer phrases, not single words, since two- and three-word combinations show your actual topic more reliably than isolated terms.
  • If one term sits far above everything else, rewrite that passage with synonyms and related words instead of deleting the keyword entirely.
  • Run the check after editing for the reader, not before, so you measure finished prose rather than letting the number shape your first draft.

Read the full guide →

Tool by the Super Simple Digital Tools Team. Reviewed by our editorial team. Free to use, no signup required.

Related tools