Allow / disallow rules
Parses every user-agent group and counts the Allow and Disallow rules so you can see what's blocked at a glance.
Free robots.txt Tester
Fetch and parse any site's robots.txt, see its rules and sitemap directives, and test whether a specific path is allowed or blocked for a given crawler.
Test a robots.txt file
Enter a site to fetch its robots.txt. Optionally add a path and user-agent to test whether that path is allowed or blocked.
Free check. No signup, nothing stored.
How NorthDuty crawlability monitoring works
Enter a site to fetch its robots.txt. Optionally add a path and user-agent to test whether that path is allowed or blocked. Recurring checks are configured inside the NorthDuty app.
What this robots.txt tester reports
A single check parses the live robots.txt and tells you exactly how crawlers will treat your URLs.
Per-path testing
Enter a path and user-agent to get an allowed-or-disallowed verdict using longest-match with Allow-wins-on-tie, the behaviour major crawlers use.
Sitemap directives
Lists any Sitemap: lines so you can confirm search engines are being pointed at the right sitemaps.
Missing or empty files
Tells you when no robots.txt exists — in which case everything is crawlable by default.
Why test robots.txt
A single stray Disallow can deindex an entire section of a site, and it's easy to ship one by accident.
- Confirm you aren't accidentally blocking important pages from search engines
- Verify a specific URL is crawlable for Googlebot or another agent
- Check that staging rules didn't ship to production
- Confirm your sitemap is referenced from robots.txt
- Understand how different user-agents are treated
How the robots.txt tester works
No signup — enter a site and optionally a path to test.
1
Enter a site
Provide a domain or URL. NorthDuty fetches robots.txt from that site's root.
2
We parse the rules
The file is parsed into user-agent groups with their Allow and Disallow rules and any sitemap directives.
3
Optionally test a path
Add a path and user-agent to get an allowed-or-blocked verdict, with the matching rule shown.
4
Catch regressions automatically
NorthDuty's health checks cover SEO fundamentals including robots.txt, so an accidental block gets flagged on a schedule.
Go Beyond One-Off Checks
Use the tool preview for a quick answer, then move into recurring monitoring for your most important pages and journeys.
Feature
Website Monitoring
What website monitoring is and how it works: NorthDuty combines default 5-minute health checks, screenshot-based UI diffs, and editable user journey monitoring in one project.
Explore Website MonitoringFeature
Uptime Monitoring
Monitor uptime every 5 minutes by default with HTTP, SSL, DNS, blank-page detection, broken resources, JavaScript errors, and API call tracking.
Explore Uptime MonitoringArticle
Website Monitoring Checklist
Use this website monitoring checklist to decide which pages, signals, journeys, and alerts your team should monitor first.
Read Website Monitoring ChecklistPricing
Pricing
NorthDuty pricing for website monitoring: a free plan through $29/mo Starter, Pro, and Business — covering uptime, visual change detection, and user journeys.
Compare pricing plansFrequently Asked Questions
Answers about this diagnostic preview and when to move into recurring monitoring.
Is this robots.txt tester free?
Yes. It's free and requires no signup. Enter a site to fetch and parse its robots.txt, and optionally test a specific path.
How does the path test decide allowed vs disallowed?
It picks the most specific matching user-agent group, then applies the longest matching rule, with Allow winning ties — the same approach major search-engine crawlers use. Wildcards (*) and end-anchors ($) are supported.
What if a site has no robots.txt?
If no robots.txt is found, the tester reports that — and by the standard, everything on the site is crawlable by default.
Does this change my robots.txt?
No. The tester only reads the live file. To change crawling rules you edit robots.txt on your own server.
Call To Action
Start monitoring your website with NorthDuty today.
A robots.txt mistake can quietly deindex pages for weeks. NorthDuty monitors SEO fundamentals continuously, so an accidental block gets caught fast.
Start on the free plan — add your base URL and monitoring starts in minutes.