Robots.txt

← Back to glossary · Category: Technical

What Robots.txt Means, in Plain Language

A robots.txt file is a small plain-text file that sits at the root of a website, at an address like example.com/robots.txt, and tells search engine crawlers which parts of the site they may request and which parts they should leave alone. It is written in a simple format of groups: a User-agent line naming the crawler the rules apply to, followed by Allow and Disallow lines naming the paths that crawler may or may not fetch. A rule such as Disallow: /admin/ tells any crawler that honours the file not to request anything under the admin directory. A rule such as Disallow: / tells every crawler not to request anything at all. The whole file is usually only a few lines long, and it is one of the most powerful files on a site precisely because it is so small: a single misplaced line can hide an entire site from search engines.

The file's purpose, in Google's own framing, is mainly to avoid overloading a site with requests. A busy crawler can request thousands of pages, and a site with endless filtered URLs, internal search results, calendar pages, or duplicate parameter combinations can generate an effectively infinite supply of low-value pages for a crawler to chase. Robots.txt lets the site owner draw a boundary around the useful part of the site and ask crawlers to spend their attention there, rather than on the machinery behind it. That is a crawl management job, and understanding it as crawl management, rather than as access control or as a privacy tool, prevents most of the serious mistakes people make with the file.

The format is standardised. The Robots Exclusion Protocol, originally defined in the mid-1990s, was formalised as an Internet standard, RFC 9309, which specifies how crawlers should interpret the file. Google's crawlers follow that specification, with a documented set of supported fields and behaviours, and the protocol is voluntary in an important sense: it is a request that well-behaved crawlers honour, not a lock that forces anyone out. A crawler that ignores robots.txt can still fetch a page, which is why the file must never be treated as security. Anything that genuinely must stay private needs a password, not a Disallow line.

What Robots.txt Is Not

Robots.txt is not a mechanism for keeping a page out of Google's index, and this is the single most consequential misunderstanding in technical search work. Google's introduction to robots.txt says this directly: the file tells crawlers which URLs they can access, it is used mainly to avoid overloading a site with requests, and it is not a mechanism for keeping a web page out of Google. Blocking a URL in robots.txt prevents Googlebot from crawling it, but it does not prevent Google from indexing it. If other pages link to the blocked URL, Google can still include it in its index based on those links alone, without ever reading the page content, and the result will typically appear without a description, because Google has no content to describe. To keep a page out of the index, use a noindex tag on the page itself, or put the page behind a login, and make sure the page is not blocked, so the crawler can actually read the instruction.

Robots.txt is not access control, and it is not a privacy tool. The file itself is public: anyone can fetch example.com/robots.txt and read exactly which directories the owner considers sensitive, which is why listing genuinely private paths in the file can perversely advertise them. A staging site, an admin panel, or an internal tool that is protected only by robots.txt is not protected at all. Password protection, network restrictions, or removal from the public web are the correct tools for privacy; robots.txt is a traffic management note to polite crawlers.

Robots.txt is also not a per-page tool. It works on URL prefixes and patterns, not on individual pages with fine-grained conditions. A rule disallows every URL whose path starts with the given prefix, which means a broad rule can accidentally cover pages its author never intended to block. That prefix behaviour is the source of the classic disaster, a rule like Disallow: / that was meant for a staging site and copied to the live site, or a rule blocking a directory that later gains important content. Every rule should be read as a statement about a whole family of URLs, not about one page.

How Robots.txt Works

When a crawler arrives at a site, one of its first requests is for the robots.txt file at the root of that host. The file applies per host, protocol, and port, which means a file at www.example.com does not cover blog.example.com, and a file served over one protocol does not automatically cover the other. The crawler reads the file, finds the group whose User-agent matches it, or the wildcard group that applies to all crawlers, and follows the most specific matching rule for each URL it considers. Where rules conflict, the most specific rule wins: a longer, more precise path beats a shorter, broader one, so an Allow for a specific file can override a Disallow for its directory, under Google's interpretation.

Google supports a small, documented set of fields. User-agent names the crawler or group of crawlers. Disallow names a path prefix the crawler should not request. Allow names a path prefix the crawler may request, useful for carving an exception out of a broader Disallow. Sitemap names the location of the site's sitemap, which is a convenience for crawlers rather than a rule, and it is how many sites point crawlers at their sitemap without any other step. Google supports two wildcards: the asterisk matches any sequence of characters, and the dollar sign marks the end of a URL, so a rule can block all URLs ending in a particular pattern. Fields that some other systems support, such as crawl-delay, are not supported by Google, and adding them to a file aimed at Googlebot has no effect, however reasonable the intention.

Why Robots.txt Matters

It matters first because it guards the site's crawl budget, the limited attention crawlers give to a site in a given period. A small blog with a few dozen pages will rarely notice crawl budget at all, because its whole site can be crawled quickly. A larger site, or a site that generates many near-duplicate URLs through filters, sorting, internal search, or session parameters, can easily spend its crawl attention on pages nobody should find, while important new pages wait. A well-chosen set of Disallow rules steers crawlers away from that machinery and toward the pages that deserve to be found, which is a genuine improvement in how quickly new and updated content is discovered. The guide to crawl budget for a small website explains when this matters and when it does not, and the honest answer for most small sites is that it matters less than content, but still deserves a correct file.

It matters second because a wrong file is one of the few single-point failures in search visibility. Most search problems degrade a site gradually, page by page. A bad robots.txt rule degrades it all at once, because every page the rule covers stops being crawled, and pages that stop being crawled gradually stop being indexed well. The failure is also quiet: the site still loads fine for visitors, the pages still exist, and nothing in the normal browsing experience signals a problem. Only a check of the file itself, or a Search Console report showing blocked pages, reveals it. That combination, total effect plus silent failure, is why the file deserves a deliberate review whenever it changes, and why copying a file from another site or from a staging environment without reading it is a genuinely risky habit.

Common Robots.txt Mistakes

The first common mistake, and the most damaging, is using robots.txt to hide a page from search results. As explained above, a Disallow does not remove a page from the index; it removes the crawler's ability to read the page, which also removes the crawler's ability to read any noindex instruction on that page. The result is a page that remains indexed, often with no description, and cannot be fixed by adding a tag the crawler can never see. The correct sequence for removing a page from search is to leave it crawlable, add the noindex instruction, and let the crawler read it, or to remove the page or password-protect it. The guide to sitemap errors that block indexing shows the same confusion from the sitemap side, where blocked URLs listed as indexable create contradictory signals.

The second mistake is blocking resources the page needs to render. Stylesheets, scripts, and images are often served from directories that look like machinery, and a broad rule aimed at machinery can block them too. When Googlebot cannot fetch a page's rendering resources, it indexes a broken or unstyled version of the page, and the site owner, who sees the page perfectly in a browser, cannot understand why the indexed version looks wrong. The remedy is to allow the resource directories explicitly, and to verify the rendered page in Search Console rather than assuming the rule only affected the intended paths.

The third mistake is the staging file copied to production. A staging or development site is often protected with a blanket Disallow: / rule, which is entirely sensible for staging, and that file is then carried over when the site launches, sometimes by a deployment process that copies everything. The live site then tells every crawler to stay away, and the launch appears successful in every respect except the one that matters for search. A launch checklist that includes opening the live robots.txt file and reading it, line by line, prevents this failure for the cost of thirty seconds.

How to Apply Robots.txt Step by Step

Start by reading the current file. Open your site's robots.txt address in a browser and read every line, asking of each rule what family of URLs it covers and why that family should not be crawled. If you cannot explain a rule, do not keep it out of caution; an unexplained rule is a liability, because it may be blocking something valuable for a reason that no longer applies. Note also whether the file contains a Sitemap line, and whether that line points at a sitemap that still exists. This reading, done once, often finds the problem that months of speculation missed.

Next, decide what genuinely should not be crawled. The usual candidates are internal search result pages, filtered or sorted variants that duplicate existing pages, administrative and staging areas, and any directory whose pages exist for machinery rather than for readers. Do not block pages that should be indexed, pages that carry a noindex instruction you want read, or resource directories that pages need in order to render. Write the list of intentions before writing any rules, so that each rule can be checked against an intention, rather than copied from another site's file whose intentions were different.

After that, test the file before relying on it. Use Search Console's robots.txt tester or an equivalent validator to check a sample of URLs against the new rules: a page that should be blocked, a page that should be allowed, a resource file, and a URL inside a blocked directory that should be excepted. Testing a handful of representative URLs catches prefix mistakes and wildcard errors that reading alone misses. The Broken Link Checker is a useful companion here, because a crawl configuration change is a good moment to confirm that the pages you intend to be found are actually reachable and linked.

Finally, publish, verify, and monitor. Upload the file to the root of the correct host, open the live address and confirm the new content is served, and then watch Search Console's crawl and indexing reports over the following days, remembering that Google caches the file for up to about a day, so the effect is not instant. Record the date of the change and the rules that were changed, so that any later indexing change can be traced back to this file or ruled out. Revisit the file whenever the site's structure changes, a new directory is added, or a staging environment is promoted, because a file that was correct for yesterday's structure is not automatically correct for today's.

How Rankenza Handles Robots.txt

Rankenza's own articles are ordinary, crawlable pages on a single responsive site, with no robots.txt games standing between the reader and the content. Each article starts from research into the top 10 pages for its keyword, filtered by what the searcher actually wants, and the draft is written in original words, reviewed by a human editor, and given internal links from the customer's own sitemap, so that the pages worth crawling are also the pages worth finding. Rankenza does not advise blocking pages that should be indexed, and it does not use robots.txt to hide content that should simply be improved or removed; where a page should not appear in search, the correct tool is the noindex instruction on a crawlable page, as this guide explains.

How This Looks on Rankenza

The comparison below is method against method, without inventing traffic or ranking figures for either side.

Sources: What the Research Says

According to Google Search Central's robots.txt introduction, a robots.txt file tells search engine crawlers which URLs the crawler can access on a site, it is used mainly to avoid overloading the site with requests, and it is not a mechanism for keeping a web page out of Google; to keep a page out of Google, the documentation directs owners to noindex or password protection. Google's robots.txt FAQ adds the documented behaviours this page relies on, including the supported fields and wildcards, the per-host scope of the file, and the treatment of error responses. Google's guide to controlling what is shared with search adds the complementary side: the noindex rule as the tool for keeping content out of results, as distinct from a crawl block. Moz's Beginner's Guide to SEO covers the same file from the practitioner's side, as part of the basic technical setup a site owner should understand before publishing. The Robots Exclusion Protocol itself is formalised as RFC 9309, the standard that defines how crawlers should interpret the file. Where this guide gives no number, that is deliberate: the sources describe rules, scopes, and behaviours, not a measured outcome that would apply to every site.

• Google Search Central: Robots.txt Introduction and Guide
• Google Search Central: Robots.txt FAQ
• Google Search Central: Control the Content You Share on Search
• Moz: The Beginner's Guide to SEO

Frequently Asked Questions

Does blocking a page in robots.txt remove it from Google?

No. Blocking prevents crawling, not indexing. If other pages link to the blocked URL, Google can still index it from those links, typically without a description, because it has never read the page content. To remove a page from search, use a noindex instruction on a page the crawler can reach, or remove the page, or password-protect it.

Where does the robots.txt file live?

At the root of a host, for example example.com/robots.txt. The file is per host, protocol, and port, so a subdomain needs its own file, and a file on one host does not cover another. Crawlers request it early in a visit, and Google caches it for up to about a day, so changes are not instant.

Can I use robots.txt to protect private content?

No, and trying to do so can make things worse, because the file is public and anyone can read which paths you listed. A crawler that ignores the file can fetch the content anyway. Private content needs a password, a network restriction, or removal from the public web; robots.txt is a traffic note to polite crawlers, not a lock.

Should a small site worry about robots.txt at all?

Yes, but briefly. A small site rarely needs elaborate rules, and its crawl budget is rarely a constraint. What it needs is a correct file: no blanket block carried over from staging, no blocked rendering resources, no conflict with its sitemap or its noindex pages, and a current Sitemap line. That check takes minutes and prevents one of the few failures that can hide a whole site at once.