The robots.txt file: how to write it, test it and avoid common mistakes

The Robots Exclusion Protocol is standardised in RFC 9309. The file is plain text in UTF-8, served at /robots.txt on the root of a host. Each combination of protocol, domain and port has its own file: https://example.com/robots.txt does not govern blog.example.com, which needs a file of its own.

The file is built from groups. A group starts with one or more User-agent lines, followed by Allow and Disallow rules. Directive names are case-insensitive, but paths are case-sensitive: /Admin/ and /admin/ are different addresses.

Two limits are worth remembering. First, robots.txt is a request, not protection. Well-behaved crawlers follow it, but the file is public, so anyone can read what you listed. Never use it to hide sensitive URLs; use authentication for that, and see the article on security headers and SEO for related hardening. Second, Google reads only the first 500 KiB of the file and ignores the rest, so a very long list of rules should be shortened with patterns.

The smallest valid file allows everything and points to the sitemap:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

An empty value after Disallow: means nothing is blocked. If there is no file at all and the server answers 404, search engines treat that as no restrictions.

A typical site has a public part that should be crawled and a few utility areas that need no crawling: the cart, the account area, internal search. A file for such a site looks like this:

User-agent: *
Disallow: /cart/
Disallow: /account/
Disallow: /search
Allow: /account/login

Sitemap: https://example.com/sitemap.xml

Line by line: User-agent: * means every crawler that has no group of its own. Disallow: /cart/ blocks every URL that starts with /cart/. Allow: /account/login makes an exception for the login page inside a blocked area. The Sitemap line is not tied to a group and can appear anywhere, but its URL must be absolute. The sitemap.xml guide covers that file's requirements in detail.

The * and $ wildcards

The standard allows two special characters. The asterisk matches any sequence of characters, and the dollar sign anchors the end of the URL:

User-agent: *
Disallow: /*?sessionid=
Disallow: /*.pdf$

The first rule blocks URLs with a sessionid parameter in the query string. The second blocks every URL ending in .pdf. Without the $, the second rule would also catch /file.pdf?download=1.

Separate groups for separate bots

When a crawler finds a group that names it, it follows only that group and ignores the * group. If you add a dedicated group for Googlebot, copy every rule it still needs from the general group.

When several rules match a URL, the crawler picks the one with the longest path. If two matching rules are equally long, Allow wins, which is the less restrictive outcome. That is how Google behaves and how the SEOAudit scanner reads the file.

Search engines can differ in small details, so for critical sections avoid clever combinations. A short file with a few obvious lines is safer than a delicate arrangement nobody remembers.

A word on Crawl-delay: Google ignores it and manages crawl rate on its own. Some other engines honour it, but most sites do not need it.

Also consider how the file itself is served. A 404 on /robots.txt means no restrictions. A 5xx response can make Google pause crawling of the site until the file is reachable again. An unstable robots.txt is therefore a risk to the whole site, not a cosmetic issue. Google describes its handling in the Search Central documentation.

The most common misunderstanding goes like this: "I will block the page in robots.txt and it will disappear from Google." It will not. Disallow stops crawling, not indexing. If other pages link to a blocked URL, a search engine may still list it in results with no description, because it could not read the content.

There is a reverse trap. If you block a page in robots.txt and also put noindex on it, the crawler never sees the tag, since it never opens the page. Google stopped supporting a noindex directive inside robots.txt itself in 2019, so do not rely on one.

To take a page out of the index:

1. Leave the page crawlable.
2. Add a robots meta tag with the value noindex (example below) to the head section, or send an X-Robots-Tag: noindex header (the header is the right choice for PDFs and other non-HTML files).
3. Wait until the crawler has revisited the page, and only then, if you want to, block it in robots.txt.

<meta name="robots" content="noindex">

Google explains the options in its page on blocking indexing. The SEOAudit scanner reports a conflict between robots.txt and a noindex directive, because that combination usually points to a mistake. A similar mix-up arises with wrong canonical URLs: that is a different mechanism and replaces neither robots.txt nor noindex.

Most problems repeat from site to site. These are the ones seen most often.

A staging leftover

Test environments are often blocked completely:

User-agent: *
Disallow: /

If this file travels to production with a release, crawling stops. It is the classic launch mistake, and it is why a robots.txt check belongs in every technical SEO checklist. The SEOAudit scanner flags a file that blocks the whole site.

Blocked styles, scripts and images

Rules such as Disallow: /assets/ or Disallow: /static/ stop Googlebot from rendering the page the way a visitor sees it. The page may look empty or broken to the crawler. Do not block resources needed for rendering. The scanner reports when such files are disallowed.

Blocking a page you want indexed

A broad rule like Disallow: /blog blocks /blog/ and also /blog-archive, because matching works on the start of the path. Check exactly what each pattern catches. The scanner checks whether the page you are analysing is blocked.

Other mistakes

• The file is not at the root, for example /files/robots.txt. It will not be read.
• The Sitemap line is missing or uses a relative URL. The scanner notes when no sitemap is declared in the file.
• Wrong letter case in a path: /Admin/ does not block /admin/.
• Stray characters such as a byte order mark, which not every parser tolerates.
• An HTML page served instead of a file. If the server returns a 404 page with status 200, that is a soft 404. The SEOAudit scanner ignores such HTML answers at /robots.txt and does not treat them as rules.

Companies publish their own crawler names, and you can allow or block them through the same User-agent groups. Examples of tokens are GPTBot, ClaudeBot, PerplexityBot and Google-Extended. The owner of each crawler documents what it does, so check the company's current documentation before adding a rule.

A group that blocks one crawler and leaves the rest alone:

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /cart/

Blocking is a business decision, not a technical one. If you block AI crawlers you may reduce the chance of appearing in answers from those systems; if you allow them, you let them read your text. There is no single right answer. What matters is to decide on purpose and to avoid blocking something by accident with a broad rule.

Here is what the scanner actually does. It reports when AI search bots are disallowed in robots.txt. Bots used for model training are not reported. The scanner reads the file's rules but does not evaluate every AI crawler individually against a full list. For the wider topic, see the page on AI search readiness.

Checking takes a few minutes and needs no special tools.

1. Open the file in a browser. Go to https://your-domain/robots.txt. You should see plain text, not a site page and not an error.
2. Look at the response. In a terminal run curl -i https://example.com/robots.txt. Expect status 200 and a text/plain type. A 5xx status needs fixing at once.
3. Read the Google Search Console report. The robots.txt report shows which file Google fetched, when, and whether it hit parsing problems.
4. Test key URLs. Take the home page, a few categories, an article and one page you deliberately blocked, and confirm the outcome matches your intent.
5. Run the scanner. The free scanner reads the file, applies the rules to the URL you submit and shows conflicts, blocked resources and whether a sitemap is declared.

The scanner is passive: it opens the URL you give it and a few public supporting files, and it does not crawl the whole site. It does not replace a review of every section. For the full picture, which includes robots.txt, see the SEO audit, and for hardening questions the security check.