robots.txt Generator
Control Search Engines and AI Crawlers

A robots.txt file is a text file that tells search engine crawlers which pages or files they can or can't request from your site. It's part of the Robots Exclusion Protocol (REP) and must be placed at your website's root (e.g., https://example.com/robots.txt).

No! robots.txt is not a security measure. It only requests that well-behaved crawlers don't access certain paths. Malicious bots will ignore it, and the file itself is publicly accessible. Never use robots.txt to hide sensitive information - use proper authentication and access controls instead.

The asterisk (*) is a wildcard that matches all user agents (crawlers). Rules under "User-agent: *" apply to all bots unless overridden by more specific rules. For example, if you have rules for both "*" and "Googlebot", Googlebot will follow the Googlebot-specific rules.

"Disallow" tells crawlers not to access a specific path. "Allow" explicitly permits access to a path, which is useful for allowing specific URLs within a disallowed directory. For example, you might disallow /private/ but allow /private/public-page.html.

It depends - and the answer is not the same for every AI bot. Treat them as two groups. Training crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended, CCBot, Bytespider, Amazonbot, Meta-ExternalAgent, DuckAssistBot, cohere-ai) build the corpora that models are trained on; blocking them opts you out of training but does nothing for live answers. Answer-engine crawlers (ChatGPT-User, Perplexity-User, OAI-SearchBot) fetch your page in real time to answer a user's question and usually cite you, so blocking them forfeits AI referral traffic. Most sites should block training and allow answers - that is the "Block AI Training" preset on this page.

llms.txt is a proposed standard (Jeremy Howard / Answer.AI, September 2024) for an AI-friendly site index. Where robots.txt says "stay out of these paths", llms.txt says "if you're an LLM with a limited context window, here are my most important pages in Markdown, ranked." Both live at your site root, both are voluntary, and they answer different questions. Generate yours with the RJL.io llms.txt Generator.

Claude-Web was Anthropic's earlier user-agent string and is now deprecated. The current names are ClaudeBot (training) and anthropic-ai (legacy token still honored). Other robots.txt generators often still ship the old name, which silently does nothing on modern crawls - we use the current names and keep anthropic-ai as a belt-and-braces second entry.

No, Google ignores the Crawl-delay directive. To control Google's crawl rate, use Google Search Console's crawl rate settings instead. However, Bing, Yandex, and some other crawlers do respect Crawl-delay.

Add your sitemap URL at the end of your robots.txt file using the "Sitemap:" directive. You can include multiple sitemaps. Also submit your sitemap directly to Google Search Console and Bing Webmaster Tools for faster indexing.

No. All processing happens entirely in your browser using JavaScript. No data is ever transmitted to any server. Your robots.txt configuration remains completely private. The tool works fully offline once the page is loaded.