robots.txt Validator
Fetch any website's robots.txt, validate its syntax, and see the parsed rules grouped by user-agent. Test whether a specific path is allowed or disallowed for a given crawler, and spot common mistakes that block search engines.
robots.txtrobots validatorcrawlerdisallowuser-agentseo
Features
- Fetch robots.txt from any URL
- Parsed rules grouped by user-agent
- Allow/Disallow path tester
- Sitemap directive detection
- Syntax and best-practice warnings