Your robots.txt Might Be Blocking AI Visibility
Introduction
This article builds directly on the previous one. If Article 166 explained how to check whether AI crawlers are visiting your site, this article focuses on the most common ways businesses accidentally block them, and the specific fixes for each.
The reason this deserves its own article: in our audits, roughly 30 to 40% of business websites have robots.txt configurations that partially or completely block AI crawlers. Not intentionally. Accidentally. Through rules that were set up years ago for SEO purposes and never updated for the AI era.
This means a third of businesses attempting AI search optimization are building citations, publishing content, and implementing structured data while their website, the single source they control most directly, is invisible to the AI tools they're trying to reach.
The five most common robots.txt mistakes
Mistake 1: The blanket block with Google-only exception.
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
This is surprisingly common. It was set up to prevent scraping while keeping Google happy. The problem: it blocks every AI crawler except Google's. ChatGPT's search mode (via Bing), Perplexity, Claude, and every other AI tool's crawler is denied access.
The fix: add explicit allow rules for AI crawlers you want to access your site:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Bingbot
Allow: /
Mistake 2: Blocking all bots except a whitelist that doesn't include AI crawlers.
Some security-conscious sites use a whitelist approach: block everything, then explicitly allow only known crawlers. If the whitelist was created before 2023, it almost certainly doesn't include AI crawlers because they didn't exist yet.
The fix: Add AI crawler user agents to your whitelist. Review and update your whitelist annually to include new AI crawlers as they emerge.
Mistake 3: Overly broad Disallow rules that catch AI crawlers.
User-agent: *
Disallow: https://yazeo.com/insights/
Disallow: /resources/
These rules were set up to prevent thin content or duplicate pages from being indexed. But they also block AI crawlers from your blog and resource content, which is often the most valuable content for AI recommendations. AI-optimized content that AI can't access is wasted effort.
The fix: Remove or narrow the Disallow rules. If specific pages need blocking (duplicate content, internal tools), block those specific URLs rather than entire directories.
Mistake 4: Security plugin or CDN blocking AI crawlers.
WordPress security plugins (Wordfence, Sucuri, iThemes Security), CDN providers (Cloudflare), and hosting-level firewalls sometimes block crawlers they don't recognize. AI crawlers are relatively new, and some security tools classify them as suspicious by default.
This blocking doesn't appear in your robots.txt file. It happens at the server or network level, which makes it harder to diagnose. Your robots.txt might allow GPTBot, but your firewall blocks it before it reaches the robots.txt file.
The fix: Check your security plugin settings, CDN bot management rules, and hosting firewall logs. Whitelist AI crawler IP ranges and user agents. Most security tools have documentation on how to whitelist specific bots.
Mistake 5: Using noindex meta tags on key pages.
Some websites use noindex meta tags on pages that shouldn't be indexed by Google (thank you pages, internal tools). But these tags also tell AI crawlers not to process those pages. If noindex is applied to your services page, about page, or other critical business information pages, AI crawlers will skip them.
The fix: Audit your website for noindex tags on pages that should be AI-accessible. Remove noindex from any page that contains business information, services, content, or entity data you want AI to process.
How to audit your entire site for AI crawler issues
Here's a step-by-step audit process.
Step 1: Read your robots.txt file.
Go to yourdomain.com/robots.txt. Read every rule. Check whether any Disallow rules affect AI crawler user agents (GPTBot, ClaudeBot, PerplexityBot, Bingbot). Check whether any blanket rules (User-agent: *) block crawlers that aren't explicitly allowed.
Step 2: Check your security tools.
Log into your CDN dashboard (Cloudflare, Sucuri, etc.), your hosting control panel, and any WordPress security plugins. Look for bot blocking rules, challenge rules (CAPTCHAs for bots), or rate limiting that might affect AI crawlers.
Step 3: Check server logs for AI crawler activity.
If AI crawlers are absent from your logs despite no robots.txt block, the block is happening at the server or network level (Mistake 4).
Step 4: Scan for noindex tags.
Use a site crawler tool (Screaming Frog, Sitebulb, or even manual inspection of key pages' source code) to check for noindex meta tags on your most important pages.
Step 5: Test AI access directly.
Ask Perplexity a question your website content should answer. Check whether your site appears in the source citations. If it doesn't, something is preventing Perplexity from accessing your content. Ask ChatGPT (in search mode) a similar question. If your content isn't reflected in the answer despite being relevant, a crawling issue may be the cause.
The cost of accidental blocking
Let's quantify what accidental AI crawler blocking costs.
If your website is the most authoritative, most detailed, most current source of information about your business (as it should be), and AI crawlers can't access it, AI tools are making recommendations about your business based on third-party sources alone.
Those third-party sources may include: directory listings (which might be outdated), review platforms (which reflect customer opinion but not your current services), and old web content (which might describe your business as it was three years ago, not as it is today).
The information gap between "what AI knows from third-party sources" and "what AI would know if it could read your website" directly affects recommendation accuracy, description quality, and recommendation probability. Businesses with AI crawler access give AI the most current, most comprehensive, most entity-specific data. Businesses without it give AI scraps.
Fixing AI crawler access is often the single highest-impact, lowest-effort fix in an AI optimization strategy. It takes 15 minutes to update robots.txt. The impact can be the difference between AI knowing you and AI guessing about you.
Not sure if your site is blocking AI crawlers? Run your free AI visibility audit at yazeo.com for a complete assessment of your AI crawler accessibility alongside your entity signals, structured data, and recommendation status.
Key findings
- 30 to 40% of business websites accidentally block AI crawlers through outdated robots.txt rules, security plugins, or CDN configurations.
- The five most common mistakes are: blanket blocks with Google-only exceptions, outdated whitelists, overly broad Disallow rules, security tool blocking, and misapplied noindex tags.
- Blocked AI crawlers mean AI tools rely exclusively on third-party data for your business, which may be outdated, incomplete, or inaccurate.
- Fixing crawler access takes 15 minutes and is often the single highest-impact technical fix for AI visibility.
- A full audit requires checking robots.txt, security tools, server logs, and meta tags to identify all blocking points.
Frequently asked questions
The 15-minute fix that changes everything
You might have spent months building citations. Weeks creating AI-optimized content. Hours implementing structured data. All of that work produces less impact if your robots.txt file, set up five years ago by a developer who's long gone, is blocking the AI crawlers that need to read your website.
Check. Fix. Test. 15 minutes. Then everything else you've built works harder.
Run your free AI visibility audit at yazeo.com and find out if your website is working for you or against you in AI search. The audit checks what the crawlers see, not just what your team sees. That difference might be the most important thing you learn this quarter.