BRRH AI Automation
•9 min read

How to Audit Your Website's Robots.txt for AI Crawler Access

A step-by-step robots.txt audit for GPTBot, ClaudeBot, and PerplexityBot access, so your Las Vegas business shows up in AI Overviews and ChatGPT answers.

robots-txtai-crawlerstechnical-seoaeo
How to Audit Your Website's Robots.txt for AI Crawler Access

A page can rank on page one of Google and still be invisible to ChatGPT. If your robots.txt file quietly blocks the crawler that feeds an AI answer engine, none of your on-page SEO work reaches that engine at all. This guide walks through exactly how to check, what to fix, and which crawlers to name explicitly so your site shows up where people are increasingly asking questions instead of searching.

Table of Contents

Why Robots.txt Access Decides Whether AI Overviews Can See Your Site

Here's the uncomfortable part: a page can be optimized, fast, and ranking well in traditional search results, and still never get cited in a ChatGPT answer or a Google AI Overview, because the crawler that would have fetched it was turned away at the door.

That door is your robots.txt file, and more sites have an unlocked problem here than most owners realize. seoreport.dev's August 2026 re-crawl found that 13.3% of 45 re-audited robots.txt files restricted at least one AI crawler, often without the site owner knowing it was happening.

For a Las Vegas small business, the stakes are practical, not theoretical. Being excluded from AI-generated answers means losing a share of discovery traffic that never touches a traditional search results page at all; the searcher asks a question, gets an answer with a citation, and clicks through without ever seeing your listing in a ranked list.

The crawler that trains a model and the crawler that fetches a live answer are often two different bots. Blocking one doesn't block the other, and most robots.txt files were never written with that distinction in mind.

It's worth bolding that distinction because it trips up a lot of audits: a training crawler collects content to improve a model over time, while a search/answer crawler fetches a page in real time to generate the specific answer a user just asked for. Rules written for one don't automatically cover the other.

What Robots.txt Actually Controls (and What It Doesn't)

Robots.txt is a plain-text file sitting at the root of your domain (yourdomain.com/robots.txt) that tells well-behaved crawlers which paths they're allowed to request. It is a courtesy instruction, not a security mechanism. Any crawler that chooses to ignore it, can.

That last point matters more than it sounds. A permissive robots.txt means nothing if a firewall, CDN rule, or WAF is silently dropping the same crawler's requests at the network layer instead. richieaubin.com flags this exact gap as a reason to test the file actually being served on every hostname your site responds to, not a cached copy sitting in a CDN edge node from six months ago.

A robots.txt audit is only half the job. The other half is confirming the server doesn't contradict what the file promises.

It also helps to separate two levers that get confused in the same audit. Robots.txt governs crawling, meaning whether a bot is allowed to request a page at all. A noindex tag or canonical tag governs indexing, meaning whether a page that was successfully crawled actually gets stored and served in results. A page can pass one check and fail the other.

The AI Crawlers Worth Naming Explicitly in Your Robots.txt

Generic wildcard rules written years ago for old-school scrapers often net-block newer AI agents by accident, simply because the user-agent string changed and nobody updated the file. The fix is naming each crawler explicitly rather than relying on a single catch-all block.

CrawlerOperatorPurpose
GPTBotOpenAITraining and search
ChatGPT-UserOpenAILive browsing during a chat session
ClaudeBotAnthropicTraining
PerplexityBotPerplexitySearch and answer generation
Google-ExtendedGoogleGemini and AI Overviews training
CCBotCommon CrawlFeeds many downstream models
AmazonbotAmazonAlexa and search

A single User-agent: * block can technically cover all of these, but it also makes it harder to audit who's actually allowed in when something breaks. Giving each crawler its own User-agent group, even if the Allow rules are identical, makes future audits and future exceptions much easier to manage.

How to Audit Your Robots.txt File in 6 Steps

Run through these in order. Each step catches a different failure mode, and skipping one is usually how a blocked crawler goes unnoticed for months.

  1. Fetch the live production robots.txt directly (yourdomain.com/robots.txt), not a staging or local copy, and record the HTTP status code and timestamp.
  2. List every User-agent group in the file and note which AI crawlers are named explicitly versus covered only by a wildcard * rule.
  3. Cross-check the named list against the current major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot), since providers add new agents faster than most teams update their files.
  4. Open the robots.txt report inside Google Search Console to confirm the version Google last fetched matches what you expect. agilitycms.com recommends this specific step for catching caching mismatches between what you think is live and what Google actually sees.
  5. Run a free checker such as the AI Crawler Access Checker from tools.scoreroute.com, which audits robots.txt against 19 named AI bots and shows the exact rule blocking each one, or the AI Readiness Checker from apify.com, which scores robots.txt, llms.txt, and schema.org markup together for whether ChatGPT, Claude, and Perplexity can crawl the site.
  6. Separately verify firewall and CDN bot-management rules aren't blocking a crawler your robots.txt allows. These two systems are configured independently, and a clean robots.txt file means nothing if your WAF has a bot-management rule quietly dropping the same request.

If you're also setting up an llms.txt file alongside this audit, our guide on how to set up llms.txt for a small business site walks through that companion step.

Common Robots.txt Mistakes That Quietly Block AI Search Visibility

Most robots.txt problems aren't intentional. They're leftovers from a deploy, a migration, or a rule nobody revisited after the crawler landscape changed.

  • A staging Disallow: / rule pushed to production during a deploy, blocking every crawler including AI bots until someone notices traffic drop off weeks later.
  • Treating robots.txt as case-insensitive when crawler user-agent matching is actually case-sensitive, so a typo'd agent name silently fails to match either direction.
  • Blocking an entire subdirectory like /blog/ to stop generic scraping, not realizing that directory holds the long-form content AI answer engines pull most often.
  • Assuming one blanket rule covers both training and real-time search crawlers, when providers like OpenAI often use separate agents for each purpose.

Each of these is a quiet failure. Nothing errors out, nothing breaks a build, the site just stops showing up where it used to.

What We Check Before Calling a Site AI-Ready

A clean robots.txt file is necessary but it is not sufficient on its own. We've seen this firsthand on a cleanup pass where robots.txt came back clean, the sitemap had zero errors, and every page returned a 200 status, yet only 39 of 51 submitted pages actually indexed. Word count alone didn't close that gap; internal linking did. Pages that weren't linked from anywhere else on the site sat unindexed no matter how long or well-written they were.

A clean robots.txt is a gate being open. It doesn't guarantee anyone walks through it. Indexation and AI citation both depend on internal linking and crawl budget doing their part too.

Before we call a site AI-ready, here's the checklist we run:

  • Robots.txt live-fetched from production and parsed line by line
  • No unintended wildcard blocks catching AI crawlers by accident
  • Google Search Console fetch matches what's actually live
  • Firewall and CDN bot-management rules tested separately from robots.txt
  • Sitemap submitted with zero errors and every page returning a 200

This kind of technical pass is something we fold into both new builds and ongoing care plans. If you're starting from scratch, our web design Las Vegas page covers what that looks like for a new site, and if you're trying to get repeat technical checks like this off your own plate, business automation Las Vegas covers how we handle recurring audits without someone having to remember to run them manually. For a broader look at how AI Overviews specifically weigh these signals, see how to rank in Google AI Overviews as a local business.

FAQ

Does blocking GPTBot in robots.txt stop my site from appearing in ChatGPT answers? It can, depending on which agent you block. OpenAI uses separate user-agents for training (GPTBot) and live browsing (ChatGPT-User). Blocking GPTBot alone may still let ChatGPT fetch pages in real time; blocking both removes you from training data and live citations.

How often should I audit my robots.txt for AI crawler access? Check it after every major deploy or CMS migration, since staging rules or plugin defaults sometimes get pushed to production by accident. A quarterly review alongside your sitemap and Search Console check catches drift even without a deploy.

Is a robots.txt audit enough to guarantee AI search visibility? No. Robots.txt only controls crawling. A page can be fully crawlable and still miss AI citation if it's not indexed, lacks clear structured content, or sits behind a firewall rule that contradicts the robots.txt file. Treat it as one checkpoint, not the whole audit.

What's the difference between robots.txt and a firewall blocking AI crawlers? Robots.txt is a voluntary instruction that well-behaved crawlers choose to respect. A firewall or CDN bot-management rule actively blocks requests at the network level regardless of what robots.txt says. You need to test both, since they're configured in different systems.

Which AI crawlers should a small business explicitly allow? At minimum, name GPTBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), and Google-Extended (Gemini and AI Overviews) in their own robots.txt groups rather than relying on a wildcard rule, since new agents are added faster than most teams update their files.

If you haven't looked at your robots.txt file since it was first generated, that's the next ten minutes worth spending: fetch the live version, run it through the AI Crawler Access Checker, and confirm Search Console agrees with what you're seeing.

Need help with this?

AI Visibility Audit

Get cited in ChatGPT, Perplexity, and Google AI Overviews — a one-time audit of how AI search sees your brand and how to fix it.

See the AI visibility audit →

GET THE GAME PLAN

Plain-English notes for Las Vegas business owners on local SEO, Google and Meta ads, websites, and AI automation — sent only when something's worth sending.

How to Audit Your Website's Robots.txt for AI Crawler Access