WordPress AI Crawlers: What to Allow and Block (2026)

AI crawlers are automated bots that fetch your WordPress pages for three different reasons: to train a model, to index you for AI search, or to read a page live while somebody is chatting with an assistant. Whether they get in is decided by your robots.txt file, your host and your CDN, and almost never by llms.txt, which controls nothing at all. That distinction matters more this quarter than it did last year, because Cloudflare changes what happens by default on 15 September 2026, and because AI bots have gone from a rounding error in your logs to a measurable share of your traffic. This guide covers what the crawlers actually want, which of the three access layers can enforce anything, how to tell whether your host is already blocking bots without telling you, and how to make the allow-or-block call deliberately.

What AI crawlers want from your WordPress site

Not every AI bot is doing the same job, and lumping them together is how site owners end up making a decision they never intended. Three purposes are worth separating. Training crawlers pull your content into a model's weights: you get no visit and no attribution, and this is the category publishers object to most. Search crawlers index your pages so an assistant can cite you when someone asks a relevant question, which is the closest thing left to the old bargain with Googlebot and the route to being mentioned inside ChatGPT, Perplexity and AI Overviews. Agent fetches happen in real time, when a person asks an assistant to check a price, read a spec or complete a task for them.

The exchange is wildly different across those three. Training crawlers have historically accounted for the large majority of AI crawl volume while sending back almost nothing, which is precisely why Cloudflare and others began classifying bots by purpose rather than by company. It is also why "block AI" and "allow AI" are both bad settings: one of those categories is a customer standing at your door, and the same switch turns all three off.

Does llms.txt control AI crawlers?

No, and this is the most common misunderstanding we run into. llms.txt is a plain text file that describes your site and points at the pages you consider important. It is a suggestion aimed at tools that choose to read it, not a permissions file, and nothing is obliged to fetch it or act on it. Ahrefs examined 137,210 domains and found that 97% of the llms.txt files published on them were never requested at all during May 2026. Of the small number that were fetched, most of the requests came from SEO audit tools rather than from AI systems.

That does not make the file worthless. It costs nothing to publish, it is a tidy summary of your best content, and the convention may still gain traction. We publish one on this site. What it will not do is keep anybody out or let anybody in, so if you added llms.txt and considered AI access handled, that box is still unticked. Getting cited by AI engines is a content and structure problem, covered in our guide to GEO and AEO. Controlling who reaches the content in the first place is an infrastructure problem, which is the rest of this article.

The three layers that actually decide access

Access to a WordPress site is settled in three places, and only one of them can enforce anything.

The practical implication is that your settings page and your reality can disagree for months. A robots.txt file that welcomes GPTBot means nothing if the request is dropped at the edge, and an SEO plugin's "allow AI crawlers" toggle only writes a line in a text file. Enforcement lives one layer below WordPress, which is exactly where most owners never look.

Is your host already blocking AI bots without telling you?

Quite possibly, and you would not necessarily know. Managed WordPress hosts and CDNs ship opinionated defaults, and blocking AI scrapers became a popular one during 2025. When the block happens at the edge, the request never reaches PHP, so nothing appears in your analytics plugin and nothing looks broken. The site simply stops being cited, gradually, and nobody can point at a cause.

Three checks, in this order. Open yoursite.com/robots.txt and read what is actually there, including whatever an SEO plugin generated on your behalf. Then open your CDN or bot-protection dashboard and find the AI crawler section, which on Cloudflare is now organised by crawler purpose rather than by vendor. Then read raw server logs and search for GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot. The logs are the only source of truth in that list: the first two tell you what you configured, and only the third tells you what happened. If your hosting provider will not give you raw access logs, fix that before you tune anything else, because you are otherwise guessing.

What changes on 15 September 2026

Cloudflare is moving from a single AI-bot switch to three categories defined by purpose: search, agent and training. On 15 September 2026, for all new domains onboarding to Cloudflare, the training and agent categories will be blocked by default on pages that display ads, while search remains allowed by default. Existing sites are not migrated automatically, and customers can set their own preferences ahead of the date.

Two consequences worth planning for. If you put a new WordPress site behind Cloudflare after that date and it carries advertising, agent traffic is refused unless you change the setting, which means an assistant sent by a real customer to check your prices comes back empty-handed. And if you have been treating AI access as one on-off decision, that framing is finished: the question is now which purposes you accept, which is a better question anyway.

What to allow, and what to block

There is no universally correct answer, because it depends on whether your content is the product or the advertisement for it. For a business that wants to be found, this is a defensible default.

• Allow search and answer crawlers: being absent from AI answers is a real and quiet cost, and this is the category that still sends people back to you
• Allow agent fetches: if you sell, quote or take bookings, an agent acting for a customer should be able to read the page
• Decide training on its own merits: a publisher whose archive is the product usually blocks it, a service business that wants to be known usually does not care
• Block unidentified scrapers: anything that will not identify itself, ignores robots.txt or hammers your server is a bandwidth problem, not an audience
• Write the decision down and re-check it quarterly: your host or CDN can change its defaults without asking you

Then make sure the content is worth crawling, because access is necessary and not sufficient. Engines cite pages that answer a question directly, in clean structure, with facts they can verify against a named source. That is the work our generative engine optimization service does, and it is the other half of the same problem: our technical SEO team handles the crawl and index side, and the two only pay off together. If you are also weighing what the new AI plumbing in core changes for your stack, our post on AI in WordPress core covers it.

Bottom line: llms.txt does not control AI crawlers, robots.txt only asks nicely, and the one layer that can genuinely refuse a bot sits at your host or CDN, probably running a default you never chose. Decide which crawler purposes you want, verify it in your logs rather than on a settings page, and look again after 15 September 2026.

Frequently asked questions