For most businesses selling a service, no. Blocking AI crawlers removes you from answers that will be generated with or without you, handing the citation to a competitor who stayed accessible. The calculation changes if your content is the product, such as a publisher, a research firm or a paid database, where uncompensated reuse is a direct commercial loss rather than a marketing opportunity.
Not all crawlers do the same job
This is the part that gets flattened in most discussions. Different crawlers from the same company serve different purposes, and blocking them has different consequences.
| Purpose | What it does | Effect of blocking |
|---|---|---|
| Training | Collects content to train future models | You are less likely to be part of what a model knows. No effect on live answers today |
| Live retrieval | Fetches pages while answering a question | You cannot be cited in answers that use retrieval. This is the costly one |
| Search indexing | Builds an index that also feeds AI features | Can affect ordinary search visibility too. Check carefully before blocking |
Because these overlap, a blanket block in robots.txt frequently costs more than the person writing it intended. Read what each agent you are about to block is used for.
The case for blocking
- Your content is the product. Publishers, research houses, data providers. Reuse without attribution or payment is lost revenue.
- You have licensing arrangements that a free crawl undercuts.
- The material is sensitive and you would rather it were not summarized by anyone.
- You have a considered position on consent and are willing to accept the visibility cost.
The case against
- The answer gets generated anyway. Blocking removes your name from it, not the answer itself.
- Your competitors are visible in it. Someone gets cited in your category and it will not be you.
- Referral traffic from these tools exists and is growing, at least for now.
- Being described accurately matters. If you are absent from the accessible sources, what remains is whatever third parties said about you.
The middle options people miss
- Block selectively by directory. Allow your public marketing content, block a paid archive or a members area.
- Block training, allow retrieval, where the operator separates them. This keeps you citable today without contributing to future models.
- Block specific operators rather than all of them, based on whether they send anything back.
- Review it on a schedule. Whatever you decide this quarter may be wrong next year, and a robots.txt written once is never revisited.
Before you change anything
- Check your server logs for which agents actually visit you, rather than acting on a list from an article
- Confirm what each agent you intend to block is used for, because some feed ordinary search too
- Note the date and the reasoning somewhere, so the next person understands why
- Remember robots.txt is a request. Well-behaved crawlers honor it and not everything is well-behaved
If you sell a service and your website is marketing, blocking is usually the wrong trade. If your content is the thing people pay for, it is a genuine commercial decision and deserves a proper conversation rather than a default.