Robots.txt generator
Tell search engines and other crawlers which parts of your site they may visit.
Runs entirely in your browser. Nothing you type is sent anywhere.
Crawler names change. Check each company’s own documentation before relying on this list.
How to use the robots.txt generator
robots.txt asks crawlers which paths not to visit. It doesn’t hide pages: a blocked page can still appear in search results if other sites link to it, so use a noindex tag for that.
- List the folders crawlers shouldn’t visit, such as admin or checkout pages.
- Add your sitemap address.
- Tick any AI crawlers you want to opt out of.
- Upload the file to the root of your site, so it opens at yourdomain.com/robots.txt.
Questions
Does robots.txt hide a page from Google?
No. It asks crawlers not to visit, but a blocked page can still appear in results if other sites link to it. Use a noindex tag to keep a page out of search results.
Do all crawlers obey robots.txt?
Reputable ones do. It’s a request, not a lock, so never rely on it to protect private information.
Is anything I type sent to a server?
No. Everything is generated in your browser.