FREE SEO TOOL · WORDPRESS PRESET · AI CRAWLER OPTIONS
Robots.txt Generator – Free Tool for WordPress and Any Website
Choose a preset, add the folders you want to block and your sitemap. Get a clean robots.txt file you can copy or download, with optional rules for AI crawlers like GPTBot.
Start with
Rank Math and Yoast use /sitemap_index.xml. WordPress core uses /wp-sitemap.xml.
WordPress options
AI crawlers
This keeps your content out of AI model training. It does not remove you from Google Search, and AI search tools that fetch pages for users use other bots.
Your robots.txt
Robots.txt Rules Explained
Robots.txt is a small text file at the root of your website. It tells search engine crawlers which parts of the site they should not visit.
| Line | Meaning | Example |
|---|---|---|
User-agent: | Which crawler the rules are for (* means all) | User-agent: * |
Disallow: | A path the crawler should not visit | Disallow: /wp-admin/ |
Allow: | An exception inside a blocked folder | Allow: /wp-admin/admin-ajax.php |
Sitemap: | Full URL of your XML sitemap | Sitemap: https://www.example.com/sitemap_index.xml |
Empty Disallow: | Nothing is blocked | Disallow: |
Common Robots.txt Mistakes
Disallow: / on a live site
This blocks your whole website. It often gets left behind after a site moves from staging to live.
Blocking CSS and JS
Do not block /wp-content/ or /wp-includes/. Google needs your styles and scripts to see the page properly.
Using it to hide pages
A blocked page can still show in Google if other sites link to it. Use a noindex tag instead.
Forgetting the sitemap
The Sitemap line helps every search engine find your pages, not just Google.
Test after uploading
Open yoursite.com/robots.txt in a browser, then check the robots.txt report in Google Search Console.
WordPress SEO plugins
Rank Math and Yoast let you edit robots.txt from the dashboard, so you do not need FTP.
Learn Technical SEO the Practical Way
Robots.txt is one small part of technical SEO. Learn crawling, indexing, site speed and Search Console with live websites in our small-batch classes in Coimbatore.
Frequently Asked Questions
Where do I put the robots.txt file?
In the root folder of your website, so it opens at yourdomain.com/robots.txt. In WordPress you can also edit it from Rank Math or Yoast.
What is the best robots.txt for WordPress?
A simple one: block /wp-admin/ but allow admin-ajax.php, optionally block internal search, and add your sitemap. Do not block CSS, JS or images.
Does robots.txt stop a page from appearing in Google?
No. It stops crawling, not indexing. To keep a page out of search results, use a noindex meta tag and let Google crawl it.
Should I block AI crawlers like GPTBot?
It is your choice. Blocking training crawlers such as GPTBot, ClaudeBot and Google-Extended keeps your content out of AI model training. It does not affect normal Google Search rankings.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay, so this tool leaves it out. Bing and some other crawlers support it.
Is this robots.txt generator free?
Yes. It is free, needs no login, and works in your browser.
More Free Digital Marketing Tools
All free, no sign-up. See all tools
SEOSERP Preview ToolSee how your page title and meta description look on Google, with pixel length and keyword checks.Open tool →AnalyticsUTM BuilderCreate tracking links for social posts, emails and ads, and see which GA4 channel each visit lands in.Open tool →Local SEOGoogle Business Profile ChecklistAudit your Google Business Profile with 26 checks and get a to-do list sorted by impact.Open tool →Google AdsGoogle Ads Budget CalculatorTurn a budget or lead goal into clicks, leads, cost per lead, ROAS and break-even CPC.Open tool →CareerDigital Marketing Salary CalculatorEstimate your monthly pay by role, experience, city and skills, and see how it grows.Open tool →CareerDigital Marketing QuizPractise 50 interview questions on SEO, Ads, social media, content and GA4 with instant answers.Open tool →SEOSchema Markup GeneratorCreate JSON-LD structured data for local business, article, product, course, FAQ and more, ready to paste.Open tool →Social MediaWhatsApp Link GeneratorCreate a wa.me click-to-chat link with a pre-filled message, plus a ready-made website button.Open tool →Social MediaInstagram Hashtag GeneratorGet a mix of broad, niche and local hashtags for your business, topic and city, ready to copy.Open tool →YouTubeYouTube Tag GeneratorTurn your video topic into relevant YouTube tags and stay within the 500-character limit.Open tool →YouTubeYouTube Hashtag GeneratorGet relevant hashtags for videos and Shorts, and preview the three that show above your title.Open tool →Social MediaLinkedIn Text FormatterAdd bold, italic and bullet points to LinkedIn posts, and preview the see-more cut-off.Open tool →YouTubeYouTube Thumbnail Size CheckerCheck your thumbnail against 1280×720, 16:9 and 2 MB, then resize and download it in one click.Open tool →YouTubeYouTube Earnings CalculatorEstimate daily, monthly and yearly YouTube income in rupees from views and RPM.Open tool →Social MediaInstagram Engagement Rate CalculatorWork out engagement rate by followers or reach from likes, comments, saves and shares.Open tool →SEOFAQ Schema GeneratorTurn questions and answers into FAQPage JSON-LD plus matching FAQ HTML, with bulk paste.Open tool →Social MediaWhatsApp QR Code GeneratorMake a QR code that opens a WhatsApp chat with your message, and download it as PNG or SVG.Open tool →SEOURL Slug GeneratorTurn titles into short, lowercase, hyphenated slugs, in bulk, with a keyword check.Open tool →Social MediaInstagram Bio GeneratorCreate clear Instagram bios with emojis and a call to action, all under 150 characters.Open tool →YouTubeYouTube Description GeneratorWrite SEO-friendly video descriptions with chapters, links and hashtags in seconds.Open tool →YouTubeYouTube Title GeneratorGet catchy, keyword-rich video title ideas with a length check for search.Open tool →Social MediaInstagram Caption GeneratorGet captions for product, offer, tip and festival posts with a hook and call to action.Open tool →Social MediaInstagram Username GeneratorGet username ideas that follow Instagram's rules, and check your own idea.Open tool →YouTubeYouTube Channel Name GeneratorGet catchy channel name ideas for your topic, each with a matching @handle idea.Open tool →BrandingSlogan GeneratorGet short, catchy slogan and tagline ideas for your business, shop or brand.Open tool →Social MediaTamil Captions for Instagram60+ short Tamil captions with English meaning for posts, reels and status. Tap to copy.Open tool →SEOMeta Description GeneratorWrite SEO-friendly meta descriptions with your keyword and a call to action, sized for Google.Open tool →Social MediaLinkedIn Headline GeneratorGet keyword-rich LinkedIn headlines for students, job seekers and professionals.Open tool →SEOBlog Title GeneratorGet SEO-friendly blog title ideas for your keyword, with a Google length check.Open tool →SEOKeyword Density CheckerSee your keyword density and top phrases, and spot keyword stuffing in your content.Open tool →EmailEmail Signature GeneratorCreate a professional email signature for Gmail or Outlook, with no watermark.Open tool →ContentReadability CheckerGet your Flesch score and grade level, and find long sentences and passive voice.Open tool →ContentHeadline AnalyzerScore your headline out of 100 and get tips to make it more clickable.Open tool →Social MediaContent Calendar TemplateCreate a monthly content plan with post ideas and formats, ready for Excel or Sheets.Open tool →CareerDigital Marketing Resume BuilderBuild an ATS-friendly digital marketing resume and download it as a PDF.Open tool →Local SEOGoogle Review Link GeneratorCreate a direct Google review link, QR code and WhatsApp message for customers.Open tool →ContentReading Time CalculatorSee reading time and speaking time for blogs, video scripts and speeches.Open tool →SEOMeta Tag GeneratorCreate copy-ready SEO meta tags for title, description, robots and canonical.Open tool →SEOOpen Graph GeneratorCreate Open Graph and Twitter card tags with a live social share preview.Open tool →EmailSpam Words CheckerHighlight spam trigger words in your email and get a risk score with fixes.Open tool →Local SEOLocal Business Schema GeneratorCreate LocalBusiness JSON-LD with address, map location and day-by-day opening hours.Open tool →EmailEmail Subject Line TesterScore your email subject line and preview it in a phone and desktop inbox.Open tool →AdsCPM CalculatorFind CPM, total ad cost or impressions from your budget in seconds.Open tool →BusinessProfit Margin CalculatorSee your real profit, margin and markup after fees, GST and ad cost.Open tool →BusinessLTV CalculatorFind customer lifetime value, lifetime profit and your LTV:CAC ratio.Open tool →AdsCTR CalculatorFind click-through rate, or the clicks and impressions you need.Open tool →AdsCPC CalculatorFind cost per click, total spend or clicks from your budget.Open tool →AnalyticsConversion Rate CalculatorFind your conversion rate, expected leads or the traffic you need.Open tool →BusinessWebsite Cost CalculatorEstimate website build and yearly costs in India for DIY, freelancer or agency.Open tool →AdsCPA CalculatorFind your cost per lead or sale, the budget you need, or expected conversions.Open tool →CareerSEO QuizTest your SEO knowledge with 10 random questions and instant explanations.Open tool →SEOKeyword Clustering ToolGroup a long keyword list by topic and plan one page or ad group per cluster.Open tool →AdsFacebook Ads Cost CalculatorEstimate reach, clicks and leads from your Meta ads budget, or the budget for your goal.Open tool →AdsBreak-even ROAS CalculatorFind the minimum ROAS your ads need, plus target ROAS and break-even CPA.Open tool →AnalyticsMarketing ROI CalculatorFind the return on your marketing spend, on revenue and on real profit.Open tool →CareerFreelance Rate CalculatorFind the hourly, day and project rate you need as a freelancer.Open tool →Social MediaSocial Media Image SizesImage sizes for every major platform, plus a checker for your own image.Open tool →SEO.htaccess Redirect GeneratorCreate 301 redirects, HTTPS and www rules as copy-ready .htaccess code.Open tool →AdsKeyword CombinerMix word lists into every keyword combination, with Google Ads match types.Open tool →AnalyticsA/B Test CalculatorCheck if your A/B test result is significant, with uplift and sample size.Open tool →AI ToolsAI Prompt GeneratorTurn a simple idea into a clear, structured prompt for ChatGPT, Gemini or any AI.Open tool →AI ToolsImage Prompt GeneratorBuild detailed AI image prompts for ChatGPT, Gemini, Midjourney or Stable Diffusion.Open tool →AI ToolsVideo Prompt GeneratorBuild AI video prompts with camera movement, style and sound for Veo, Sora or Kling.Open tool →AI ToolsGemini Prompt GeneratorCreate Gemini photo editing prompts for headshots, festival looks and more.Open tool →AI ToolsFacebook Post Prompt GeneratorBuild ChatGPT or Gemini prompts that write Facebook posts for your business.Open tool →AI ToolsCanva AI Prompt GeneratorWrite prompts for Canva Magic Media, Magic Design and Magic Write.Open tool →AI ToolsProduct Photography Prompt GeneratorCreate AI prompts for white-background listings, lifestyle and ad product photos.Open tool →AI ToolsLogo Prompt GeneratorCreate AI logo prompts for icon, wordmark, initials, badge and mascot ideas.Open tool →AI ToolsYouTube Script Prompt GeneratorBuild ChatGPT or Gemini prompts that write YouTube scripts and Shorts.Open tool →
