What a robots.txt rule does
It asks a crawler to stay out of a path. The crawler is asked, not forced, and a polite crawler complies.
A crawler that ignores the file will ignore your rule too. Treat it as a request with a real audience, not as a wall.
A filled policy
Cedar and Slate Interiors, decided with the client on 2026-09-20.
# everything is welcome by default
User-agent: *
Allow: /
# the supplier portal is not for crawlers
User-agent: *
Disallow: /supplier/
# one agent asked to skip the archive of old campaigns
User-agent: ExampleResearchBot
Disallow: /campaign/
Sitemap: https://cedarandslate.example/sitemap.xml
The reasons, in the client’s words. The supplier portal shows prices that are wrong for the public, and the campaign archive is thin.
The mistake that hides pages
Blocking a path in robots.txt stops the crawl, not the indexing. A blocked page can still appear in results, with no description.
Use a noindex tag when you want a page out of the index, and leave robots.txt for crawl control.
Acceptance: the file is right when
- it is served at the root of the host, as plain text, with a 200
- the sitemap line names a file that returns 200
- each disallow rule names a path somebody actually meant to block
- no stylesheet, script or image path appears in a disallow rule
- a blocked page is blocked for the reason written beside the rule
Troubleshooting
| Symptom | Likely cause | What to do |
|---|---|---|
| A blocked page still appears in results | robots.txt blocks crawling, not indexing | Add a noindex tag, then remove the block |
| Nothing is crawling at all | A leftover Disallow: / |
Remove it, and resubmit the sitemap |
| The page renders without styling to a crawler | A stylesheet is disallowed | Remove that rule |
| The sitemap line is ignored | The file is not at the root, or it redirects | Serve it at the root with a 200 |
| A research agent ignores the rule | The rule is a request | Keep it, and rely on the page itself |
Limits
This file does not measure AI visibility, and it cannot prove a crawler obeyed. Visibility is read separately, and every reading is an estimate.
The user agent tokens in this example are illustrative. A real rule needs the token the crawler publishes. That list was not reachable from the environment this page was written in.