Robots.txt Generator
Generate a robots.txt file for your site.
R
Robots.txt Generator
Control crawler access.
How It Works
Use Cases
Frequently Asked Questions
What is a robots.txt file?
It is a text file that tells search engine crawlers which pages or files the crawler can or cannot request from your site.
Where should I put the robots.txt file?
It must be placed in the root directory of your website (e.g., yoursite.com/robots.txt).