Cloudflare Changed Its AI Settings Yesterday. Find Out If Your Website Was Included.

Yesterday Cloudflare changed the default settings on a large chunk of the internet. If you've never heard of Cloudflare, that's normal, and it might still be sitting in front of your website right now.

Cloudflare is the company a lot of web developers put between your site and everybody trying to reach it. It makes pages load faster and it blocks junk traffic. They handle something like a fifth of the web. Whoever built your site probably signed you up for the free plan four years ago and never brought it up again.

They announced this back on July 1. It went live September 15.

What actually changed

Cloudflare stopped treating AI bots as one thing. There are three buckets now and each one gets its own rule.

Search bots read your page so they can point somebody at you later. Those are still allowed everywhere. Training bots take your words to feed into a model, and those get blocked by default now. Agent bots are an AI going out to grab your page right this second because a real person just asked a question about you, and those get blocked by default too.

The new defaults only kick in on pages that carry ads, and only for new sites plus free plan customers who never touched their settings. If you pay Cloudflare and you already picked your own preferences, nothing moved on you.

So for most businesses around here, nothing broke yesterday. Your site doesn't run ads. I'm not telling you to panic.

I'm telling you those settings exist and you've never once looked at them.

The bucket that costs you money

That third one is what I actually care about.

OpenAI by itself runs three separate crawlers doing three separate jobs. GPTBot collects training data. OAI-SearchBot builds the index. ChatGPT-User is the one that goes and reads your page live, in the moment, because somebody typed "who does HVAC in Sylvania" and ChatGPT went looking for an answer.

That third one is a customer. Right now. It's also the only one of the three that shows up in your analytics as a referral you can see.

Plenty of people clicked a big button in 2025 that said block AI bots, because at the time that felt like the responsible move. If your web guy did that, he turned off the crawler that brings you business along with the two you were actually worried about. One switch, three very different results, and nobody told you which was which.

I get the instinct. Nobody loves the idea of a billion dollar company hoovering up their words for free. That's fair and I'm not going to argue you out of it. But the customer asking a question about your shop and the training run are not the same visitor anymore, and now you can finally tell them apart.

What to do this week

Find out whether your site is behind Cloudflare at all. Ask whoever built it. Or go to cloudflare.com and see if you've got a login sitting in your password manager. If nobody can answer that in ten minutes, that's its own problem worth solving.

If you are on it, open Security settings and find the AI crawler controls. Then decide what you want each of the three categories to do instead of letting a default decide for you. For a local business that wants to get found, I'd let search through and let agents through. Training is a judgment call and it's yours to make.

Then test it instead of trusting the dashboard. The summary count in there lumps blocked requests together with rate limits and plain old dead links, so it reads a lot scarier than reality. Requesting one real page as each bot and reading the status code back gives you the truth in about a minute.

And while you're in there, open your robots.txt and read it. Half the sites I check have a line somebody added years ago that nobody at the company remembers putting there.

None of this is urgent in the sense that your phone stops ringing today. It's the kind of thing that quietly costs you for a year before anybody notices, which is exactly why it's worth an hour now.