Free tool · Search visibility
Robots.txt Validator
The robots.txt validator checks the public file that gives crawlers instructions about which paths they can request. Enter your website address to review its rules before investigating missing search visibility.
Free robots.txt validator
Free to use without an account. View the result here and choose whether to receive it by email.
What the tool covers
Review crawler access instructions
The public robots.txt response, recognised directives, rule structure and sitemap declarations.
The public robots.txt response
The validator requests robots.txt at the website's root and checks the document returned. An error page or inaccessible response needs attention before the rules can be reviewed.
Recognised rule structure
User-agent groups and recognised directives are checked for basic structure. The tool flags lines that need review without claiming to reproduce every crawler's interpretation.
A broad blocking signal
A literal Disallow: / in the wildcard user-agent group is highlighted for investigation. More specific exceptions and crawler behaviour still need a separate review.
Sitemap declarations
The report checks sitemap declarations in the file. A declaration points crawlers to a sitemap; it does not establish that the sitemap or its listed pages are valid.
Understand the limits
Robots.txt controls compliant crawling, not access security or guaranteed indexing. This is a focused validation, not a simulation of every crawler.
Understanding your result
Review crawler rules in the context of your site
A rule can be deliberate on a staging site and a problem on a public shop. Confirm which domain you checked and what should be discoverable before changing the file.
Readable rules are only the starting point
A successful syntax check does not guarantee that every intended page can be crawled. Review the paths and user-agent groups that apply to the particular crawler you want to allow.
Review broad blocks before editing
Check whether a wildcard Disallow: / is intentional and whether more specific rules affect the outcome. For a public WordPress site, review the source of the file and its search visibility settings before removing a restriction.
Validate the declared sitemap separately
Open the declared XML address in the sitemap validator. For missing search results, also review page-level indexing instructions and your search console, because robots.txt alone cannot explain every indexing decision.
Check the file at the site root
Crawlers look for robots.txt at the root of a website, such as example.com/robots.txt. The rules apply to that scheme, host and port. The validator checks the file served publicly, which can differ from a file you edited if a plugin, cache or hosting rule generates the response.
Review broad blocks before changing them
A user-agent group names the crawler a rule applies to. A Disallow rule blocks a path for matching crawlers, while Allow can make a more specific exception. A site-wide block deserves a closer look, but staging and private development sites can intentionally restrict crawlers. Confirm the site's purpose before removing a rule.
Keep crawling and indexing separate
Blocking a URL in robots.txt does not reliably remove it from search results or protect private content. Use authentication to secure private pages. For public pages that should not be indexed, review the appropriate noindex instructions and ensure the crawler can read them.
Questions about this tool
What does a robots.txt validator check?
It reads the public robots.txt response and checks recognised directives, rule structure and sitemap declarations. The findings help you investigate crawler access rules.
Does a missing robots.txt file stop Google indexing my site?
No. A missing file does not by itself block crawling. Search visibility also depends on page access, indexing directives and other search engine decisions.
Can robots.txt keep a private page secret?
No. Its rules are public and depend on crawler cooperation. Use proper authentication and access controls for private content.
Will this validate every bot's interpretation?
No. Crawlers differ in supported extensions and behaviour. Review the relevant crawler's documentation when a rule is intended for that service.
Can I email the robots.txt validation?
Yes. Use the optional email form after starting the check to receive a copy of the result.
Free website widget
Add this tool to your website
Let visitors use the Robots.txt Validator on your own page, including the option to email their results. Copy the code into an Elementor HTML widget or a WordPress Custom HTML block, then preview the published page.
Get the embed code
The widget fits the width of its container. Increase the height value if you want more of the report visible at once; longer results can be scrolled inside the widget.
BugShield hosts and runs the tool. Results and email addresses stay inside the widget and are not shared with the page embedding it. If your editor removes the code or your site blocks external widgets, ask your website administrator to allow frames from bugshield.co.uk.
Need help with the next step?
A BugShield developer can review the findings and help you decide what your WordPress site needs.
Explore WordPress support