Free Online Tool
robots.txt Generator
Create the perfect robots.txt file for your website in seconds. No signup, no cost โ just fill in, copy, and upload.
100% Free
No Signup
Google Friendly
Instant Download
Step 1 โ Choose Your Website Type
Pick a preset to auto-fill the best settings for your site
Step 2a โ Sitemap & Crawl Settings
Tell search engines where your sitemap is
๐ Found in Yoast SEO โ General โ Features โ XML Sitemaps
โ ๏ธ Crawl delay is ignored by Googlebot. Only useful for Bingbot or server-overloaded sites.
Step 2b โ Block Pages from Search Engines
Toggle ON to hide pages from Google, Bing, and other bots
Step 2c โ Custom Allow / Disallow Paths
Manually add specific paths to block or allow
Tip: The special Allow rule for
/wp-admin/admin-ajax.php keeps AJAX features working even when wp-admin is blocked.
Step 2d โ Block Specific Bots
Select bots to completely block from crawling your site
Googlebot, Bingbot, and other search engine bots are never blocked by this section โ only selected bots above.
๐ How to Upload to Your Website
1
Click Download โ save the
robots.txt file to your computer2
WordPress: Use Yoast SEO plugin or upload via FTP to your
/public_html/ folder3
Verify it’s live by visiting
yoursite.com/robots.txt in your browser4
Test with Google Search Console โ Settings โ robots.txt tester
โ Frequently Asked Questions
What is a robots.txt file and why do I need it? +
A robots.txt file tells search engine bots which pages on your website they can or cannot visit. Without it, bots crawl everything โ including admin pages, login pages, and duplicate content โ wasting your crawl budget and potentially hurting your SEO rankings.
Where exactly do I upload the robots.txt file? +
Upload it to the root directory of your website so it is accessible at https://yoursite.com/robots.txt. In WordPress, you can add it via FTP to the public_html folder, or use the Yoast SEO plugin (SEO โ Tools โ File Editor) to edit it directly from your dashboard.
Does robots.txt affect my Google rankings? +
Yes, indirectly. A well-configured robots.txt helps Googlebot focus on your most important pages, avoids duplicate content issues, protects your crawl budget, and prevents sensitive admin pages from appearing in search results. Incorrect settings can accidentally block important pages from indexing.
Can I block just one bot, like AhrefsBot, without blocking Google? +
Yes! robots.txt allows you to write rules for specific bots using their User-agent name. Use the “Block Specific Bots” section in this tool to block bots like AhrefsBot, SemrushBot, and MJ12bot individually โ without affecting Googlebot or Bingbot.
Is robots.txt enough to protect my admin pages? +
No โ robots.txt only asks bots not to crawl those pages, it does not enforce any security. Malicious bots may ignore it entirely. For real security, always use strong passwords, two-factor authentication, and access control on your server. robots.txt is an SEO tool, not a security tool.
