Is the robots.txt validator free?
Yes. Validating robots.txt is completely free, with no signup, and the file is parsed locally in your browser.
Does it upload or fetch my robots.txt from a URL?
No. This version is paste-first or local-file-first and parses everything in your browser, so your file is not uploaded or fetched from a live URL.
What issues does it catch?
It parses user-agent blocks and sitemap lines and flags common problems like duplicate or suspicious directives and basic syntax issues.
Does it simulate every search engine crawler?
No. It focuses on practical parsing and common directive issues rather than simulating the exact behavior of each search engine.
When should I use it?
Use it as a quick QA step during development or just before publishing or handing off a site, to catch robots.txt mistakes early.
Are comments allowed anywhere in the file?
Yes. A hash starts a comment and everything after it on that line is ignored, whether the hash is in column 0 or after a directive. The banner comment at the top of a typical robots.txt is not a problem.
Does robots.txt stop a page from being indexed?
No. Disallow stops crawling, not indexing. A blocked URL can still appear in search results with no snippet if other pages link to it, because Google never fetches the page to see a noindex tag. To keep a page out of the index, allow crawling and serve a noindex meta tag or X-Robots-Tag header.
Should the sitemap line live in robots.txt?
It is a useful backup for crawlers that have no Search Console or Webmaster Tools connection. The line is global, not tied to a user-agent block, and this validator collects every Sitemap line separately and checks that each one is an absolute http or https URL.
Does the file get uploaded anywhere?
No. Parsing happens in your browser with plain JavaScript. Whether you paste the text or pick a local .txt file, nothing is sent to a server, which makes it safe to check a robots.txt for a staging host that is not public yet.
What counts as a user-agent block here?
A block is one or more User-agent lines followed by their directives. If a directive appears before any User-agent line, the validator warns and opens an implicit block for the wildcard agent, which is what most crawlers do in practice but is not something you should ship.
Will it catch a rule that blocks my whole site?
It will show you the parsed blocks and their directives, so a 'Disallow: /' under 'User-agent: *' is visible immediately. It does not simulate path matching, so it will not warn you that this rule blocks everything. Read the block list, that is what it is there for.