AI search

Should you block AI crawlers?

Published 26 August 2026 · Updated 26 August 2026 · 6 min read

Blocking AI crawlers removes you from AI answers. For most businesses that is a cost with no matching benefit.

Read with AI

This is one of the few genuinely new decisions AI has created for site owners. It is also one where the default advice given to publishers has been repeated, unhelpfully, to small businesses whose situation is nothing alike.

Two kinds of crawler

Training crawlers gather content used to train future models. Blocking them limits reuse but does not remove you from today’s answers.
Retrieval crawlers fetch pages live to answer a question now. Blocking them removes you from citations in that assistant.
Search crawlers such as Googlebot serve ordinary results and, in Google’s case, also feed AI Overviews. Blocking it removes you from search entirely.

What blocking actually costs

If you sell services locally, being named by an assistant when somebody asks for a recommendation is free visibility. Blocking retrieval removes that possibility while doing nothing to stop competitors being named instead. For a plumber or an accountant, that is a clear loss.

When blocking is reasonable

Your content is the product: a publisher, a data provider, a paid research library.
You are negotiating or hold a licensing arrangement for your archive.
You host proprietary material that should not be summarised anywhere.
Crawler traffic is genuinely straining your infrastructure, in which case rate limiting usually beats blocking.

How to control access properly

A related question is whether to publish an llms.txt file, covered in llms.txt explained. Both decisions sit naturally within technical SEO Bristol, and any competent agency should have a view. Ask them before assuming.

Good to know

Frequently asked questions

Will blocking AI crawlers protect my content from being copied?

Partially at best. It reduces authorised collection, and it does not stop scraping by anyone ignoring the rules. Treat it as a policy signal rather than enforcement.

Does blocking affect my Google rankings?

Not if you only block AI specific crawlers. It affects rankings severely if you block Googlebot, which is why the distinction matters.

What do most businesses do?

Most allow everything, because visibility is worth more to them than control over reuse. Publishers are the main exception.

Can I allow citation but block training?

Broadly yes, by permitting retrieval crawlers and disallowing training ones. The details differ by provider and change, so check current documentation before implementing.

Written by the CompareSEO Bristol editorial team. We are an independent directory and we do not sell SEO services, so nothing here is written to win your business. Read our methodology for how we compile and check what we publish.

Compare agencies

More on ai search and seo

Start here AI search and SEO: what actually changes AI has changed how answers are presented, not how they are found. The pages that get cited are, overwhelmingly, the pages that already ranked. 9 min read Google AI Overviews explained The generated summary at the top of Google results is assembled from pages Google already trusts. Getting into it is a ranking problem before it is anything else. 7 min read How to become the source AI cites Assistants quote passages, not pages. Writing in self contained, verifiable passages is most of the work. 7 min read Generative engine optimisation: real or rebranded? Perhaps a fifth of what is sold as generative engine optimisation is new. The rest is the SEO you should already be buying. 6 min read Getting found in ChatGPT ChatGPT names businesses it can find and verify. Being findable, consistent and well reviewed is most of what you can control. 6 min read Perplexity and citation led search Perplexity shows its working. That makes it the best free diagnostic available for how AI systems see your site. 5 min read

Explore more

Other guide sections

How to choose an SEO agency Most bad SEO experiences start at the shortlist stage, not the delivery stage. Get the selection right and the rest becomes far easier to manage. 10 min read How much does SEO cost in the UK? Most UK small businesses pay between £500 and £2,500 a month for SEO. Where you sit in that range depends on your market, the state of your website and who is actually doing the work. 9 min read What is SEO? A plain English guide SEO is the work of making it easy for search engines to find, understand and trust your website, so that people looking for what you sell arrive on it. 9 min read The local SEO guide for UK businesses For most businesses serving a place, local search is where the enquiries are. It is also the part of SEO that moves fastest and costs least. 10 min read Does your industry change your SEO strategy? The mechanics of search are the same everywhere. What changes by sector is how people search, what convinces them, and what you are allowed to say. 8 min read SEO reports explained A good SEO report answers three questions: what did you do, what happened, and what are you doing next. Most reports answer none of them. 8 min read