Crawler & Indexation Tools
Robots.txt Viewer
Parse, inspect, and validate any website's robots.txt file. Test URL path crawl permissions for Googlebot and Bingbot, and verify sitemap index declarations.
Load Samples:
1
User-Agents
5
Disallow Rules
2
Allow Rules
2
Sitemaps Declared
Live URL Path Permission Tester
BLOCKED (Disallowed): Googlebot cannot crawl
/admin/dashboard.
Matched rule: Disallow: /admin/
🗺️ Declared XML Sitemaps
2 found
🤖 User-Agent Directives & Crawl Permissions
What is Robots.txt?
The Robots Exclusion Protocol (REP) is a standardized text file situated in your website's root directory (/robots.txt) telling search engine web crawlers which URLs they may or may not request.
Allow vs. Disallow Priority
According to Google's official RFC 9309 specification, when both an Allow and a Disallow rule match the same URL, the directive with the longest pattern match takes precedence.
AI Scraper Protection
You can prevent automated AI model scrapers from training on your content without affecting your organic Google search rankings by disallowing bot tokens such as GPTBot, CCBot, and anthropic-ai.
Copied to clipboard!