SEO Tool
Robots.txt Generator
Generate a clean starter robots.txt with user-agent, allow, disallow, and sitemap rules for common SEO setups.
Tool Area
Everything runs in the browser.
Use this robots.txt generator when you need a readable draft for a launch, migration, staging environment, or client handoff. It keeps the common syntax tidy, but it still expects you to review the rules before uploading them because robots.txt can affect how search engines crawl important sections of a site.
- Do not block CSS, JavaScript, or important content unless you are sure.
- Use noindex meta tags for indexing decisions; robots.txt mainly controls crawling.
- Add the sitemap line when you want crawlers to discover your XML sitemap faster.
How to use
- Choose the user-agent, usually * for a broad rule.
- Add allow, disallow, and sitemap values only where they are needed.
- Copy the generated robots.txt output and test it before deployment.
Common use cases
- Draft a starter robots.txt file for a new site.
- Prepare crawl-control examples for clients, staging sites, or migrations.
- Avoid basic formatting mistakes when adding sitemap or disallow rules.
FAQ
Does it cover advanced robots rules?
No. It is designed for common starter patterns, not every advanced directive.
Can robots.txt remove a page from Google?
Not reliably. It can stop crawling, but noindex is the cleaner signal when you want a page kept out of search results.
Can I leave some fields empty?
Yes. Empty fields are skipped from the output.