Online Robots.txt Generator
Instantly generate a robots.txt file to guide search engines on how to crawl your website. Protect private directories, manage crawl budgets, and link your XML sitemap effortlessly.
Global Settings
Restricted Directories
Add paths you want to hide from search engines. (Applied to Default Access).
Specific Bot Rules
Override the default settings for specific crawlers.
robots.txt
# Generated robots.txt will appear here...Next Steps: Download this file as robots.txt and upload it to the root directory of your website (e.g., https://yourwebsite.com/robots.txt).
Why do you need a robots.txt file?
A robots.txt file is a simple text file placed in the root directory of your website. It acts as the gatekeeper for search engine crawlers (like Googlebot or Bingbot). It uses the Robots Exclusion Protocol (REP) to instruct these bots on which pages they should crawl, and which they should ignore. This is highly useful for hiding internal search result pages, admin dashboards, staging areas, or stopping aggressive bots from crashing your server by crawling too fast.
How to Use the Robots.txt Generator
- Select the Default Access rule. "Allow all bots" is recommended for 99% of public websites.
- If you have private areas (like
/wp-admin/or/private/), add them under Restricted Directories. - Provide the absolute URL to your XML Sitemap so search engines can discover your content faster.
- If a specific bot is causing trouble, use the Specific Bot Rules to override the default access and block them entirely.
- Review the live-generated code on the right side.
- Click Downloadand upload the file to your web server's root folder (e.g.
https://yoursite.com/robots.txt).
Frequently Asked Questions
Where should the robots.txt file be placed?+
The robots.txt file MUST be placed in the top-level root directory of your website. For example, if your site is www.example.com, the file must be accessible precisely at www.example.com/robots.txt. If you put it in a subdirectory, search engines will not look for it and will assume no rules exist.
Does a robots.txt file hide my page from search results?+
Not always. Using a robots.txt file to 'Disallow' crawling will stop the bot from reading the page content. However, if other sites link to your page, Google might still index the URL itself and display it in search results without a description. To completely block a page from being indexed, you should use a <meta name="robots" content="noindex"> tag in the HTML of the page instead.
What is Crawl-delay?+
The Crawl-delay directive tells crawlers to wait a specific number of seconds between requests. This helps prevent aggressive bots from overloading your server and slowing down your site for real users. Note that Googlebot completely ignores the Crawl-delay directive (you must manage Google's crawl rate in Search Console), but bots like Bingbot and Yandex do support it.
What is User-agent: * ?+
The asterisk (*) is a wildcard character. When you use User-agent: *, you are instructing every single search engine crawler on the internet to follow the rules listed below it.