Free SEO Tool
Robots.txt Generator
Generate a robots.txt file to control search engine crawling.
Configuration
Set up your robots.txt preferences
Generated robots.txt
Place this file in your website's root directory
User-agent: * Allow: / Disallow: /admin Disallow: /private
Best Practices
📁
Root Directory
Place robots.txt in your website's root directory (e.g., https://example.com/robots.txt)
🎯
Specific Paths
Use specific paths for disallow rules (e.g., /admin/ instead of /admin)
🗺️
Include Sitemap
Include your sitemap URL to help search engines discover your content
🔍
Test Your File
Test your robots.txt file using Google Search Console
⚠️
Not For Security
Remember that robots.txt is publicly accessible and not meant for security
📝
Case Sensitive
Paths in robots.txt are case-sensitive. Be precise with capitalization