Blocks GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot (training and AI-answer crawlers). Google-Extended only affects Gemini, not Google Search.
Google ignores changefreq and priority; other search engines may use them.
User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
About Robots.txt & Sitemap Generator
Generates a robots.txt file and an XML sitemap for a website, the two standard files search engines use to understand what to crawl and index.
Enter your site's URL and crawling preferences, and the tool generates both files ready to upload to your website's root directory.
Running this once before launching a new site is a five-minute technical SEO step that's easy to forget entirely if it's not part of a standard pre-launch checklist.
Watch: Robots.txt & Sitemap Generator in Action
- When should I generate these files?
- Running this once before launching a new site is a five-minute technical SEO step that's easy to forget entirely if it's not part of a standard pre-launch checklist.
- What does Block the whole site (staging) do?
- It writes a robots.txt with User-agent: * and Disallow: /, which asks all crawlers to stay out of everything, and it ignores your Disallow paths. Remove the block before launch, or real visitors may never find you in search.
- Which crawlers does the training-crawler checkbox cover?
- It adds a Disallow: / block for each of GPTBot, ClaudeBot, CCBot, Google-Extended and PerplexityBot. The list is fixed, and compliance depends on each operator honoring robots.txt.
- Why were some of my pages skipped?
- The note under the sitemap tab lists them. A full address on a different host or scheme than your Domain is skipped, as is text that cannot form a valid address. Duplicates are merged silently.
- Do I have to type the domain as https://example.com?
- No. If you leave out the scheme, https:// is added. A port stays, a path is dropped, and the host must contain a dot or be localhost, otherwise a red message asks for a valid website address.
- Does robots.txt stop a page from being indexed?
- Not reliably. It controls crawling, not indexing, and a blocked URL can still be listed if other pages link to it. To keep a page out of results, use a noindex directive on a crawlable page or password protection.
How to use Robots.txt & Sitemap Generator
Enter your site's address in Domain; a missing https:// is added and only the origin is kept, so any path you type is dropped. List paths to keep crawlers out of under Disallow paths, one per line, and list your URLs under Pages, one per line, either as paths like /about or as full addresses on the same site.
Four settings sit beside the paths. Block the whole site (staging) replaces the rules with a single Disallow of the root. The checkbox that blocks training crawlers adds a separate refusal for GPTBot, ClaudeBot, CCBot, Google-Extended and PerplexityBot. Include lastmod (today) stamps each URL with the current date. Change frequency offers none, daily, weekly, monthly or yearly. The robots.txt and sitemap.xml tabs show the two files, each with Copy and Download.
When to use it
Use it when launching a small or new site that needs a starter robots.txt and a sitemap listing its main pages, when you must keep a staging copy out of search results with a blanket block, or when you want to steer crawlers away from a few utility folders such as /admin/ and need syntax that is correct on the first try.
When not to use it
Do not use robots.txt to protect private content: the file is public and only asks well-behaved crawlers to stay away, and a blocked page can still appear in search if other sites link to it. Use authentication or a noindex directive for that. For a large site with thousands of changing pages, generate the sitemap from your CMS or build step so lastmod reflects real edit dates.
Tips for best results
The Sitemap line in robots.txt always points to your domain's /sitemap.xml, so upload the sitemap under that exact name and location in the site root. A Disallow entry typed without a leading slash gets one added, and wildcards you type, such as *.pdf, pass through as written. If the note under the sitemap tab reports skipped addresses, they belonged to a different host or were malformed.
Limitations
Include lastmod writes the same current date on every URL, so it does not reflect when each page really changed. A priority value is not offered and crawl delay is not exposed. Disallow entries apply to all crawlers, and the whole-site block ignores your path list entirely. A note mentions the 50,000 URL cap per sitemap file, but the tool does not split longer lists into several files.
Privacy
The domain, paths and page list are turned into text by your browser, and downloads are created on your device from that text. The tool never fetches your site or checks that listed pages exist, so your domain is not contacted. The site logs only the tool name and a device identifier for usage counting, never the addresses you entered.
Next: OG / Twitter Card Preview
Check how your pages preview when shared on social media
More from SEO Tools: