Website code on a screen, the kind of robots.txt settings that decide whether AI crawlers can read a medical practice site
One small file on your website, robots.txt, decides whether ChatGPT and Perplexity are even allowed to read you. Most owners have never looked at it. Photo via Pexels.

A dermatologist we talked to this summer had done everything right. Clean website, good reviews, real photos, the works. But when a patient asked ChatGPT for a skin doctor in her area, the practice never came up. The competitor two miles away did, every time. She assumed it was some ranking mystery. It was not. Buried in her website settings, a single toggle her old web guy had switched on was telling every AI tool on the internet, in plain language, do not read this site. She was invisible to AI by her own hand, and had no idea.

This is the part of the AI conversation nobody explains to practice owners. You have probably read that publishers are blocking the bots behind ChatGPT to protect their content. That is real. But whether you should do the same thing is a completely different question, and for a local medical practice the honest answer is almost always no. Let us walk through why, and how to make sure your site is not turning patients away without your permission.

First, what an "AI crawler" actually is

An AI crawler is just a program that visits your website and reads it, the same way Google has crawled the web for decades. What changed is that AI companies now run their own crawlers, and here is the key thing most articles skip: they run more than one, and the different ones do very different jobs.

There are really two families that matter to you:

The distinction is the whole ballgame. Blocking a training crawler mostly affects whether your writing helps build tomorrow's model. Blocking an answering crawler decides whether a patient searching inside ChatGPT tonight sees your practice at all. As the analysts at TechnologyChecker put it after studying millions of sites, publishers now block the training bots and allow the answering bots on purpose. Block the wrong one and you do not protect anything. You just delete yourself from the answer.

Why blocking is a publisher problem, not yours

The websites racing to block AI crawlers are the ones whose entire business is content: The New York Times, big magazines, large blogs that sell ads against articles. For them the math is simple. If an AI eats their reporting and hands it to readers for free, they lose the click, the subscription, and the ad. Blocking is self defense.

Your practice is the mirror image of that. You do not sell articles. You sell appointments. You want to be the answer when someone asks an AI who to see. Being read, quoted, and recommended by ChatGPT is not a threat to your business, it is free distribution to the exact person looking for you. So when you copy the newspaper's playbook and block the bots, you are using a shield built for a problem you do not have, and paying for it with the one thing you actually want: to be found.

And a lot of sites are doing exactly that. In a study of the top 2,000 websites, researchers found that about 19% block AI crawlers, and 136 of them were blocking without even knowing it. GPTBot is the single most blocked crawler on the web right now. The trend is real, it is spreading, and small business sites keep getting caught in the current.

~19% of the top 2,000 websites block AI crawlers, and dozens were doing it by accident, unaware they had turned themselves off. Source: analysis of top 2,000 sites, 2026.

The accidental block is the real danger

Here is what actually bites practices. Almost nobody sits down and decides, on purpose, to hide from AI. Instead a setting somewhere flips itself on and no one notices for months. The usual culprits:

The scary part is how silent it is. Nothing breaks. Your site loads fine, Google still works, the phone still rings sometimes. There is no error, no warning, no line in your analytics that says "a patient asked ChatGPT about you and you were not allowed to appear." The loss is completely invisible, which is exactly why it goes unfixed for so long. It is the same invisible leak we wrote about when we looked at what ChatGPT tells patients about your practice, just one layer earlier in the pipe.

Why this matters more every month

You might be tempted to file this under "someday." It is not a someday problem. The 2026 Patient Choice Report from rater8 found that 47% of patients had used AI tools like ChatGPT and Perplexity to research a provider, up from 31% just nine months earlier, and reliance on AI to actually pick a doctor doubled in a year. Patients aged 45 to 60, the ones with steady insurance and regular visits, led the whole trend.

So the audience asking AI for a doctor is large, growing fast, and skews toward your best patients. If your website is blocking the answering crawlers, none of that reaches you. You are not in the running. And it stings extra because AI referred visitors tend to arrive with high intent, they were literally just told you are a good fit, so a blocked site is not losing tire kickers, it is losing people ready to book.

How to check your own site in five minutes

You do not need a developer to find out where you stand. Do this right now:

If you find blocks you never meant to set, the fix is usually a two minute change: flip the Cloudflare or Squarespace toggle back off, or have whoever manages your site remove those lines. If reading a robots.txt file makes your eyes glaze over, that is fair, most owners have never opened one. It is the kind of thing we check for free on a call, along with the rest of your local and AI search footprint.

Our honest take at EtherealMinds

We will say it plainly, because the internet makes this sound more complicated than it is. For the overwhelming majority of medical practices, do not block AI crawlers. Allow them. You are a local business whose goal is to be discovered, and every one of these tools is a new way for a patient to discover you. Hiding from them is like taking your name out of the phone book to protect your privacy, and then wondering why the phone stopped ringing.

If you want to be precise about it, the tidy setup is the one the data says works: allow the answering crawlers without question (OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-Extended), and treat GPTBot, the pure training bot, as an optional judgment call. Blocking GPTBot will not hurt your Google rankings and will not remove you from ChatGPT's live answers, so if the idea of your service pages feeding a model bothers you, block that one alone and lose nothing that matters. But for most practices with ordinary service and location pages, there is little to guard and no reason to overthink it. Allow everything and move on.

The bigger point is that letting AI read you is only step one. Being crawlable does not make you the recommendation. The AI still has to find you accurate, consistent, and clearly the right answer, and that comes from a fast, well structured website built to be read by machines and humans alike, matching business listings, and real reviews. Opening the door is necessary. Being worth walking through is the rest of the job.

Open the door, then be ready when they knock

Think about the full journey for a second. A patient asks ChatGPT for a doctor. For that to end at your practice, three things have to be true, in order. The AI has to be allowed to read your site, which is the crawler question we just covered. It has to decide you are the best answer, which is the search and reputation work. And when it hands the patient your number and they call at 9pm, someone, or something, has to actually answer. Miss any one of the three and the whole chain breaks.

That last mile is where a lot of practices lose the patient they worked so hard to earn, which is why we pair the search side with an AI receptionist that picks up instantly, day or night, answers the common questions, and books the visit while the patient is still holding the phone. It is all one connected patient acquisition system: let the AI in, be the answer it gives, and be there when it sends someone your way. Block the crawlers and none of it ever gets to start.

So before you touch anything else this week, go read your own robots.txt. It is the cheapest audit in marketing, and for more practices than you would guess, it is the difference between being in the conversation and being left out of it entirely.

Not sure if your site is hiding from AI?

Book a free strategy call. We will check your robots.txt live, tell you exactly which AI crawlers you are allowing or blocking, ask ChatGPT and Perplexity about your practice on the spot, and map the fixes that put you back in the answer.

Book a free strategy call →