A relaunch copies the staging file to production, and Disallow: / sits there blocking the whole site. A plugin update rewrites the file and drops the sitemap line. A server hiccup makes robots.txt return a 500 for a few hours, and Google stops crawling while it lasts. None of this shows up in uptime monitoring, because the homepage is still returning 200 the whole time.
This guide covers what actually goes wrong in production, how Google reacts to each case, and how to monitor the file so a bad rule becomes an alert instead of a traffic graph you explain three weeks later.
What Actually Breaks in Production
| What happens | How it usually happens | What it costs |
|---|---|---|
Disallow: / on the live site | Staging file copied during a launch, migration, or restore from a staging backup | Crawling stops sitewide; rankings decay over weeks |
| robots.txt returns 5xx | Server or PHP error, an overloaded host, a WAF rule | Google temporarily treats the whole site as disallowed |
| robots.txt disappears (404) | File deleted in a deploy, or an SEO plugin disabled | Everything becomes crawlable, including URLs you meant to keep out |
| Sitemap line removed | Plugin rewrite, or a hand-edited file replacing the generated one | Slower discovery of new and updated pages |
| A new Disallow blocks a real section | Someone blocks /search or a parameter and catches more than intended | Whole sections quietly drop out of the index |
| CSS or JS folders blocked | Copy-pasted "security" robots.txt from a forum | Google cannot render pages properly |
| Different file per host | Rules added to www but not the bare domain, or a staging subdomain left crawlable | Half the site is governed by rules you never reviewed |
| File grows past 500 KiB | Generated rules appended by a plugin over time | Google ignores everything past the limit |
The pattern behind most of these: nobody edited robots.txt on purpose. It changed as a side effect of something else.
How Google Reacts to Each Failure
This is what makes robots.txt worth monitoring rather than checking occasionally — the response codes do not fail in the direction you would expect.
| Response | How Google treats it |
|---|---|
| 200 with rules | Rules apply as written |
| 404 or other 4xx | Treated as no robots.txt at all, so everything is crawlable |
| 429, or 5xx | Treated as if the site were fully disallowed; after a while Google falls back to the last cached copy, and if the error persists it starts treating the site as having no restrictions again |
| Redirect to an HTML page | Effectively no valid file; the rules you think exist are not being applied |
| Over 500 KiB | Content past the limit is ignored |
So a broken server gives you the worst outcome of all: crawling stops, and nothing on your site looks wrong.
What to Monitor
Five things, in order of how much damage they prevent.
- Status and reachability. robots.txt should return 200 on every host you care about — bare domain,
www, and any subdomain that serves pages. Alert on 5xx immediately and on 404 the same day. - A diff of the file contents. Any change should produce a notification, even a harmless one. This is the check that catches the staging file, the plugin rewrite, and the well-meaning colleague.
- Verdicts for specific URLs. Pick the paths that matter — a product URL, a category, a top blog post, your main landing page — and confirm each is still allowed for Googlebot. This catches an over-broad rule without you having to read the whole file.
- The sitemap line. Confirm it is present and points at a sitemap that is still valid.
- Asset paths. Confirm your theme's CSS and JS directories are not disallowed.
How Often
After every deploy, and daily in between. Deploys and plugin updates cause most robots.txt incidents, so a check triggered by the release is the one that saves you; the daily check catches changes made by a plugin, a host, or a person outside your release process.
Weekly is not enough. Google can drop a meaningful share of crawling in a week, and the recovery takes longer than the outage did.
Three Ways to Set It Up
1. Search Console's robots.txt report. Shows which robots.txt files Google found for your site, the fetch status, and any parse errors. It reflects Google's own view, which no external tool can, so it is worth checking after any change. It is manual, and it does not test individual URLs or alert you — the legacy robots.txt Tester that did that was retired in late 2023.
2. Test on demand, monitor on a schedule. Use a free robots.txt tester when you are making a change, and let a monitoring tool watch the file continuously between changes. This is the combination most teams end up with: a human checks intent, a machine watches for drift.
3. Roll your own. A scheduled job that fetches robots.txt, compares it to the committed copy in your repository, and fails the build or posts to Slack on any difference. It works well if robots.txt lives in version control, and badly if a CMS or plugin generates it.
However you do it, decide in advance who gets the alert. A robots.txt change is usually an SEO problem caused by a developer or a plugin, so the notification needs to reach someone who can read the rules and someone who can revert them.
Put the Check Where the Risk Is
Every robots.txt incident is the same shape: a change nobody meant to make, on a file nobody was watching, discovered by a drop in impressions weeks later. The fix is not more discipline. It is a check that runs on a schedule and a notification that goes to a human.
If you are already monitoring your site's health after deploys, robots.txt belongs in the same place as your other post-deploy checks — next to page rendering, redirects, and the journeys that make you money.