Generate a correct robots.txt file to tell search engine crawlers which parts of your site to crawl or skip, with your sitemap location included.
How it works
- Select a preset policy (Allow All, Disallow All, or Custom).
- Add specific directories to block or allow.
- Provide your XML sitemap URL to output a fully compliant robots.txt text.
Frequently asked questions
Does Disallow hide a page from Google?
No — it asks crawlers not to crawl it, but it can still be indexed if linked. To keep a page out of results, use a noindex meta tag instead.
Where does robots.txt go?
At your domain root, reachable at https://yourdomain.com/robots.txt. It only applies to that exact host.
Should I block staging sites?
Block all crawlers on staging, but never rely on it for security — it is a public file and only a polite request.