Robots.txt Tester
Category: SEO & Analytics • Free Online Utility
Fetch a site's robots.txt or paste your own, then test paths or URLs for a chosen user agent. Matching follows Google's rules for user-agent groups, wildcards, $ anchors and longest-match precedence. The tester also lists sitemaps and flags unsupported directives such as crawl-delay and noindex.
Checking membership permissions...
Related SEO & Analytics Tools
View All →How to Use Robots.txt Tester
1
Load robots.txt
Fetch it from your website or paste a draft to test before publishing.
2
Add paths and pick a crawler
Enter one path or URL per line and choose a user agent such as Googlebot or GPTBot.
3
Read the result
Each path shows Allowed or Blocked with the exact rule and line that decided it.
Frequently Asked Questions
No. Blocking a URL stops crawling, but the URL can still appear in results if other pages link to it. Use a noindex tag to keep a page out of search.
User Reviews & Community Ratings
Rated by developers, SEO consultants, and power users worldwide.
4.7 / 5.0
Based on 182 verified ratings (Google SEO Verified)