Robots.txt Generator
Build a robots.txt file that tells search engines which parts of your site to crawl, with optional blocks for AI training crawlers.
Loading the tool
robots.txt is a request, not security. Well-behaved crawlers follow it, but it does not hide private pages.
How to use Robots.txt Generator
- Choose the default rule and add paths to disallow.
- Add your sitemap address.
- Copy the file to the root of your website.
Privacy and limits
This tool runs in your browser. What you type is processed on your device and is not sent to our server.
Questions
- Is Robots.txt Generator free to use?
- Yes. You do not need an account and there is nothing to pay.
- Does it work offline?
- Once the page has loaded, the tool does its work on your device. Some browsers need a connection only to load the page itself.