robots.txt Generator
Control How Search Engines Crawl Your Site

A robots.txt file is a text file that tells search engine crawlers which pages or files they can or can't request from your site. It's part of the Robots Exclusion Protocol (REP) and must be placed at your website's root (e.g., https://example.com/robots.txt).

No! robots.txt is not a security measure. It only requests that well-behaved crawlers don't access certain paths. Malicious bots will ignore it, and the file itself is publicly accessible. Never use robots.txt to hide sensitive information - use proper authentication and access controls instead.

The asterisk (*) is a wildcard that matches all user agents (crawlers). Rules under "User-agent: *" apply to all bots unless overridden by more specific rules. For example, if you have rules for both "*" and "Googlebot", Googlebot will follow the Googlebot-specific rules.

"Disallow" tells crawlers not to access a specific path. "Allow" explicitly permits access to a path, which is useful for allowing specific URLs within a disallowed directory. For example, you might disallow /private/ but allow /private/public-page.html.

It depends on your preferences. Blocking GPTBot, ChatGPT-User, and other AI crawlers prevents your content from being used for AI training. However, some AI features (like ChatGPT browsing) may provide traffic to your site. Consider your content strategy and business needs.

No, Google ignores the Crawl-delay directive. To control Google's crawl rate, use Google Search Console's crawl rate settings instead. However, Bing, Yandex, and some other crawlers do respect Crawl-delay.

Add your sitemap URL at the end of your robots.txt file using the "Sitemap:" directive. You can include multiple sitemaps. Also submit your sitemap directly to Google Search Console and Bing Webmaster Tools for faster indexing.

No. All processing happens entirely in your browser using JavaScript. No data is ever transmitted to any server. Your robots.txt configuration remains completely private. The tool works fully offline once the page is loaded.