Writing a robots.txt File That Helps Crawlers Instead of Hurting Your SEO
By the Super Simple Digital Tools Team · Updated June 2026
The robots.txt file is one of the smallest files on your website and one of the easiest to get wrong. It is the first thing most crawlers fetch before they touch anything else, so a single misplaced slash can block your entire site or, just as bad, leave the door open to areas you meant to fence off. The good news is that the format is simple: a series of groups, each beginning with a User-agent line that names a crawler (or * for all of them), followed by Disallow and Allow rules. Get the structure right and the rest is mostly careful path-matching.
Path matching is where the subtlety lives. Values are matched from the start of the URL path, so Disallow: /blog blocks /blog, /blog/, and /blogging-tips alike, while Disallow: /blog/ blocks only the folder. You can use the wildcard * to match any sequence of characters and $ to anchor the end of a URL, for example Disallow: /*.pdf$ to block PDF files. When an Allow rule and a Disallow rule both match, the more specific (longer) rule wins, which is how you open a single file inside an otherwise blocked directory. Remember paths are case-sensitive even though directive names are not.
The Sitemap directive is the one line almost every site should include. Unlike Allow and Disallow it is not tied to a user-agent and takes a full absolute URL, such as Sitemap: https://example.com/sitemap.xml. You can list several sitemaps or a sitemap index. This is a low-effort way to point every search engine straight at your canonical list of URLs, and it works even if the rest of your file is just an open policy. Adding it costs nothing and helps crawlers discover new pages faster.
The biggest conceptual mistake is treating robots.txt as a privacy or removal tool. It is neither. Because Disallow blocks crawling but not indexing, a blocked URL linked from elsewhere can still surface in search results as a bare link. Worse, if you Disallow a page and also add a noindex tag, the crawler can never read the noindex because it is not allowed to fetch the page, so the page may stay indexed. The correct pattern for removing content is to leave it crawlable and apply noindex, or to require authentication for anything truly sensitive.
Once your file is generated, validate it before you ship it. Paste your real URLs into a tester (Google Search Central offers one) to confirm the right paths are blocked and the important ones are not. After deploying, check Search Console for crawl-blocked warnings over the following weeks, since robots changes can take time to register. Keep the file under control in version control, document why each rule exists, and revisit it whenever you launch new sections or change your URL structure.
- Always include an absolute Sitemap URL line, even if the rest of your file allows everything, so crawlers find your canonical URL list quickly.
- Never use Disallow together with a noindex tag on the same page; the crawler must be allowed to fetch the page to ever see the noindex instruction.
- Block crawl traps like internal search results and faceted filter URLs (for example Disallow: /search and Disallow: /*?sort=) to save crawl budget on large sites.
- Keep sensitive directories out of robots.txt entirely; listing them just advertises them. Protect them with passwords or server-side access rules instead.