What robots.txt controls
The robots.txt file lives at the root of a domain and tells crawlers which paths they may or may not request. It is the first file most search engine bots fetch before crawling a site. A single typo here can quietly block your entire site from Google, so testing it matters.
This tester fetches the live robots.txt, parses each user-agent block, and lists every Allow and Disallow rule plus any Sitemap declarations and Crawl-delay values. It flags whether the file is missing, returns an error, or accidentally blocks all crawlers.
How to read your results
Each user-agent group applies to one or more bots. A line of `Disallow: /` under `User-agent: *` blocks every crawler from the whole site, which is almost never what you want on a live site. An empty `Disallow:` line means nothing is blocked.
Robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in search results without a snippet if other sites link to it. To keep a page out of the index entirely, allow crawling and use a noindex meta tag instead. Always list your XML sitemap with a `Sitemap:` line so crawlers can discover your URLs.
Related guides
Common mistakes to avoid
- Leaving a development Disallow: / rule live in production.
- Expecting robots.txt to remove a page from the index (use noindex instead).
- Placing robots.txt in a subfolder instead of the domain root.
- Blocking CSS or JS that Google needs to render the page.
- Forgetting to declare the XML sitemap.
Frequently asked questions
Does robots.txt stop a page from being indexed?
No. It only controls crawling. A disallowed URL can still be indexed if other pages link to it. Use a noindex meta tag to remove a page from the index.
Where should robots.txt live?
Always at the root of the domain, for example https://example.com/robots.txt. It does not work in a subfolder.
What does Disallow: / mean?
It blocks crawlers from every URL on the site. On a production site this almost always means a mistake.
Should I list my sitemap in robots.txt?
Yes. Adding a Sitemap: line helps search engines discover all of your important URLs faster.