We checked what AI tools find when they visit 95 small-business websites in Los Angeles, in October 2026. No site we tested singled out ChatGPT's or Perplexity's search crawlers, though 5 blocked Claude's. Several sites had little or nothing for a crawler to read once it got in.
- About 1 in 20 home pages showed a crawler no readable text at all (5 of 95), and 8 had fewer than 100 words.
- More than half didn't label what kind of business they are in the structured data search engines read (53 of 95).
- About 1 in 5 home pages didn't show the phone number from the business's own Google listing (20 of the 92 listings that have one).
- About 1 in 7 turned away at least one AI crawler (14 of 95). Most of those blocks hit OpenAI's training crawler, which OpenAI says is used for training only.
How we tested
We started with Google Maps listings within about 9 miles of downtown Los Angeles and kept independent businesses: a claimed listing, at least 5 reviews, their own website and no chains. From those, we picked 10 at random in each of 10 categories: dentists, plumbers, restaurants, hair salons, gyms, law firms, accountants, real estate agents, auto repair shops and coworking spaces. Several picks that turned out not to be independent, such as chains, a dealership and a nonprofit, were replaced with the next random pick. Five sites were down or refused every visitor, which left 95.
Scripts did the fetching and counting. For each site, they read robots.txt, loaded the home page as a normal browser and under the names of four AI crawlers, and compared the page with the business's Google listing. Anything that looked blocked was tested twice more, and we read every block page and every empty page by hand.
Cobalt Bloom, which publishes this guide, plans to build and look after websites that fix these issues. We've reported every result, including the ones that don't help that case.
What this can't show. We checked home pages only, so a phone number, hours or structured data on another page don't count here. We sent each crawler's name, not the crawler itself. A site that checks where a visitor comes from could treat the real one differently. Anthropic doesn't publish the full identification string for Claude's search crawler, so we used its name in the standard format. It's one day, one city and 95 sites, so treat every share as rough, give or take about 10 points.
Can AI crawlers get in?
AI companies send programs called AI crawlers to read websites. Some look up and fetch pages so the tool can answer a question and cite its sources. Others collect text to train future AI models. The first kind is the one that matters for being found and cited. From each company's own documentation:
- ChatGPT: OAI-SearchBot puts pages into ChatGPT's search answers, and ChatGPT-User fetches a page when someone asks about it. GPTBot is for training only. OpenAI says sites that block OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links" (OpenAI).
- Claude: Claude-SearchBot and Claude-User are for search and answers. ClaudeBot is for training (Anthropic).
- Perplexity: PerplexityBot puts pages into Perplexity's results, and Perplexity-User fetches a page when someone asks. Perplexity says neither is used for training (Perplexity).
- Google: AI Overviews and AI Mode have no extra requirements beyond normal Google search. A page has to be indexed and eligible to show in Google with a snippet, and even then Google doesn't guarantee it will appear (Google).
We tested ChatGPT's, Perplexity's and Claude's search crawlers, plus GPTBot.
What we found: 14 of the 95 sites turned away at least one of those names. GPTBot was blocked on 11 (10 every time, 1 some of the time) and Claude's search crawler on 5 (4 every time, 1 some of the time). None of the 14 blocked ChatGPT's or Perplexity's search crawlers.
robots.txt, the file most guides tell you to check, showed the block on only 3 of the 14, and no robots.txt in the test blocked an AI search crawler. The rest sat in the settings in front of the site. On 5 sites it was the AI crawler setting in Cloudflare, a service many sites use to screen traffic, and all 5 blocked GPTBot only. On 5 it was a server or security rule refusing a crawler by name, and on 4 the site answered a crawler with "too many requests" on its first visit.
Some of these blocks may be deliberate, especially the Cloudflare ones on GPTBot. Two more sites turned away all four crawler names but weren't counted: one checked whether a crawler was real, which the real crawlers would likely pass, and one turned away every automated visitor, our normal-browser test included.
Check yours: open yourbusiness.com/robots.txt in a browser and look for OAI-SearchBot, PerplexityBot or Claude-SearchBot followed by Disallow: /. Then ask whoever runs your site to run the test we used, since robots.txt can't show a firewall block:
Could you check whether our site blocks any AI crawlers? From a terminal, run this once with no name, as a baseline (on Windows, use
curl.exeand-o NULinstead of-o /dev/null):
curl -sL -o /dev/null -w "%{http_code} %{size_download}" https://ourdomain.com/Then run it with
-A "NAME"added, once for each of these names: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, GPTBot. Please run each a few times, since some blocks come and go.The first number is the response code and the second is the page size. A different code from the baseline (403 means refused, 406 means not acceptable, 429 means too many requests) means something stopped it. If the code is 200 but the size is very different, run the command again without
-o /dev/nullto see what came back. A block page or a "verify you're human" page counts as blocked. If anything is blocked, please tell me which setting caused it: Cloudflare's bot settings, the host's firewall or a security plugin.
The test uses each crawler's name, not the crawler itself. Treat a block as a strong hint and a pass as a good sign, but neither as proof. The firewall's own logs show what happened to the real crawler.
Is there anything on the page to read?
Some sites are built so that the words only appear after the browser runs code. Google's crawler runs that code. The AI companies don't say whether theirs do, so it's safest to assume some don't. For a crawler that doesn't, a page built this way is empty even though nothing blocked it.
What we found: 8 of the 95 home pages had fewer than 100 words a crawler could read without running code, and 5 had none. Four of the five were built as JavaScript apps that send an empty shell and fill it in later. The fifth was a business whose Google listing linked to a parked domain, a holding page from the domain company, instead of a website. So 4 sites were empty to crawlers that don't run code, against none that blocked ChatGPT's search crawler.
Check yours: open your home page, right-click and choose "View Page Source." (In Safari, first turn on the Develop menu in Settings, then Advanced.) Use Find to search for a short, plain phrase you can see on the page, such as one of your services. If it isn't in the source, a crawler that doesn't run code can't see it either. While you're at it, click the website link on your Google listing and make sure it opens your site.
Does the site say what kind of business it is?
Structured data is code in a page that labels facts for machines: this is a dentist, here is the address, these are the hours. Google describes it as "explicit clues about the meaning of a page" (Google). The labels sit in the page, so any AI tool that reads the page has them too.
What we found: 53 of the 95 home pages had no structured data saying what kind of business they were (a LocalBusiness label or one of its types, such as Dentist or Plumber). Only 27 listed opening hours this way, and 35 used sameAs to link the site to the business's other profiles. Some sites may put this on another page, such as a contact page, which we didn't check.
Check yours: paste your address into Google's free Rich Results Test. If it finds no "Local business" item, or one missing your phone number, address or hours, the structured data is missing or incomplete. Google's guidelines say structured data should describe things that are visible on the page, so the facts belong in the page too.
Are the basics on the home page?
When an AI tool describes a business, it can compare what the website says with other sources, such as the Google listing. If the site leaves the basics out, the tool has less to go on.
What we found: 20 of the 92 businesses with a phone number on Google didn't show it on their home page, counting the visible text and tap-to-call links. We could find opening hours on 45 of the 95 home pages, in the text or the structured data. Street addresses were often missing too, but some businesses that go to their customers, such as plumbers, leave theirs off on purpose.
Check yours: open your Google listing next to your home page. The name, phone number and hours should match, and the home page should show at least the phone number and the area you serve.
What to do with what you find
- Fix anything that stops a search crawler. That might be a line in robots.txt, a checkbox in your website builder, or a rule in your firewall or a security plugin. Each AI company's documentation, linked above, lists the names to allow. OpenAI says ChatGPT's search can take about 24 hours to notice a change to robots.txt.
- Make sure the words are in the page. If your site sends an empty page until code runs, ask whoever built it about serving the text in the page itself, which many website tools can do.
- Label the business. Add structured data with your business type, address, phone, hours and links to your other profiles, and check it with the Rich Results Test.
- Put the basics on the home page, matching your Google listing.
None of this guarantees an AI tool will recommend you, but it gives one that looks at your site the facts it needs.