All resources

Review 10 min read

Should I block AI training on my website? Opt out if you like, but keep AI answers

The new controls finally separate AI training from AI answers. Make the training decision based on how you earn money, and keep the answer bots open.

Should you block AI training on your website? I would block it only if your content itself is what you sell. For most small and medium businesses, the training decision matters less than people fear. The real risk is blocking the bots that fetch your pages when ChatGPT, Copilot, Gemini or Perplexity build an answer. Site owners often block them by accident, along with the training bots. My rule is simple: decide on training based on how you make money, and keep the answer bots open.

The timing matters. On 30 October 2026 Microsoft Bing announced separate, independent preferences for AI training and AI grounding. On the same day Google and OpenAI said they plan to honor one shared robots.txt training signal. For the first time, most site owners can make these two choices separately.

Training and answers are two different jobs

Microsoft's definitions are the clearest I have seen. AI training means using your content to change the learned parameters of a model. AI grounding means using your content as input to help inform an AI-generated answer. Indexing and ranking pages in ordinary search results is neither.

Training is indirect: your page becomes one of many sources a model learns from. Grounding happens at the moment someone asks. A customer asks a question, a bot fetches your page, and the assistant can name you and link to you. If you block grounding, the assistant can't use your page for that answer. It builds the answer from other sources, and those may include your competitors.

Until this autumn, most site owners treated training and grounding as one decision. Since 2023, Microsoft used NoArchive and NoCache to control both. Google's Google-Extended token still covers both. Long "block all AI bots" lists mixed the two types together. OpenAI, Anthropic and Perplexity already had separate bots, but few people read the documentation that closely.

Bot or tokenWhat it doesBucket
GPTBot (OpenAI)Controls whether your content is used for trainingTraining
OAI-SearchBot (OpenAI)Controls whether a site can appear in ChatGPT search resultsAnswers
ChatGPT-User (OpenAI)Opens a page when a user asks ChatGPT toAnswers
ClaudeBot (Anthropic)Collects training dataTraining
Claude-SearchBot, Claude-User (Anthropic)Improve search results and fetch pages for user questionsAnswers
PerplexityBot, Perplexity-UserSurface and link sites in Perplexity answers; fetch pages for user actionsAnswers
Google-Extended (token, not a crawler)Gemini training and, according to reporting, grounding in the Gemini appsBoth
Googlebot, BingbotSearch, plus Google's AI Overviews and AI Mode, and Microsoft's Copilot groundingAnswers

The roles come from reporting on OpenAI's crawlers, Anthropic's help center, Perplexity's crawler guide and Google's list of crawlers. One detail: robots.txt rules may not apply to user-initiated fetchers such as ChatGPT-User and Perplexity-User, because a person asked for the page. A firewall block does stop them, though. I won't repeat the robots.txt syntax here. Our map of llms.txt and other AI-crawler signals in 2026 shows how these signals fit together.

What changed on 30 October 2026, and what didn't

The shared signal is a robots.txt group named User-Agent: __usage_ai-training. It comes from an IETF Internet-Draft by Kevin Kelley of Anthropic. An Internet-Draft is a working proposal, not a finished standard. Google will treat an opt-out through this token as a Google-Extended disallow, unless you name Google-Extended explicitly. OpenAI says it will honor the token in the next few months, and its current robots.txt rules stay in effect until then. Reports say Anthropic has started to implement it, but I have not seen an Anthropic page that confirms this.

What didn't change is just as important. Under the draft, robots.txt files published today express no shared preference, because they don't have this group yet. Your existing GPTBot, ClaudeBot and Google-Extended rules are the ones doing the work. OpenAI, Google and Anthropic say the per-bot controls remain available.

DateWhat happens
15 Dec 2026Temporary grounding settings open in Bing Webmaster Tools, only for sites that use NoArchive/NoCache
1 Feb 2027Bing starts applying the new preferences as it recrawls pages
1 Mar 2027Bing stops reading NoArchive/NoCache for training and grounding
3 May 2027The current Internet-Draft expires
30 Sep 2027Bing's temporary grounding settings expire
2027 / "coming months"Google / OpenAI support for the shared token

Where the split still isn't clean

Google: training and Gemini answers come together

Search Engine Journal and Dataconomy report that Google-Extended also covers grounding in the Gemini apps. They also say Google's documentation describes no way to separate the two. So if you opt out of training with the new token, you may also drop out of Gemini app answers. A separate Google-Extended group overrides the shared token, but it is still one switch for both uses. Google Search, AI Overviews and AI Mode are not affected, because Googlebot controls them, not Google-Extended.

Microsoft: grounding is set page by page

At Bing, you can set training for the whole site in robots.txt and override it for single pages. Grounding is set only per page, through a meta tag or the X-Robots-Tag header. Microsoft says, "There is no site-wide robots.txt grounding preference." Both preferences default to Allow. There is also a middle option, ai-grounding:disallowextended. It keeps your pages available to Microsoft's own experiences, such as Copilot and Microsoft 365, but excludes grounding services Microsoft sells to third parties.

These preferences control how Microsoft uses your content, not whether Bingbot can crawl it. An Allow does not override a robots.txt rule that blocks Bingbot. Microsoft also has to recrawl a page before it sees a new setting.

A wildcard rule is not an opt-out

Under the draft, a User-Agent: * group must not be read as a training preference. If you think your blanket rules already say "no training", they say nothing under the new system.

Should I block AI training on my website? A decision rule

Ask one question: is your content itself the product you sell? If it isn't, training on your service pages costs you little. If it is, protect it.

BusinessAI trainingAI answers
Local service, clinic, law firm, agencyYour call; I lean toward allowKeep open
Online store, SaaS, B2B lead generationYour call; I lean toward allowKeep open
Publisher with a paywalled or licensable archiveDisallow, or licenseOpen on public pages; consider DisallowExtended at Bing
Content is the product: courses, data, recipes, stock mediaDisallowDecide page by page: open for sales pages, closed for the product

For example, imagine a dental clinic. Its prices, opening hours and treatments are not a product anyone can resell. Blocking training protects nothing valuable. Blocking answers means ChatGPT can't check those facts when a patient asks for a clinic nearby.

"But AI steals my traffic": the strongest objections

Publishers with licensable content

This is a fair concern. Opting out of training is a reasonable choice if your archive is worth money. But Bing says its preferences "do not establish licensing or payment arrangements". The draft also lets private agreements, such as licensing deals, override the shared signal. So the token protects your options, but it doesn't negotiate for you. And you don't have to remove your public pages from answers to keep those options.

AI summaries reduce clicks

Yes, they do. Pew Research Center studied the March 2025 browsing data of 900 US adults. Users who saw a Google AI summary clicked a traditional result in 8% of visits, against 15% without one. They clicked a link inside the summary in about 1% of visits. That is a real cost. But blocking answers doesn't bring the old search page back. It only removes you from the summary that now sits on that page.

Blocking doesn't return the clicks

The only evidence I have seen points the other way, and it is thin. A secondary report describes one site tested by SEO Cyrus Shepard. Its regular organic clicks fell while it was out of AI Overviews and AI Mode, and they recovered when it returned. That is a single site, so it proves nothing on its own. Still, I have not seen good evidence for the opposite.

Open access is not a guarantee, so measure it

Microsoft says it plainly: "Neither choice guarantees inclusion, traffic, or payment." If you disallow grounding, Microsoft warns you will see reduced brand visibility in AI experiences. If you allow it, your pages only become eligible. Whether an assistant names you depends on clear, unique facts on those pages and on what other sources say about you.

That is why I think AI visibility should be measured before and after a change, not promised. Our research on Share of Model and measuring AI visibility explains the approach. In our Sereda.ai case study, ChatGPT first said it did not know the HR platform. Sereda.ai published an AI Page on 22 October 2025. Two researchers independently scored the answers to 4 questions on 5 criteria, on a 100-point scale. The scores rose from 24 to 87 in ChatGPT and from 28 to 91 in Gemini. "Before" was September to early October 2025, and "after" was early December 2025. That is one case, not a typical result.

For the technical side, check your server logs. The answer bots should get 200 responses, not 403 errors, redirect loops or firewall challenges. A user-agent string can be faked, so verify important bots by IP or reverse DNS. If you don't want to read raw logs, Chetver Pulse shows which AI bots visit, what they read and where they hit errors.

AI bot requests on a demo site: which ChatGPT, Claude, Gemini and Perplexity bots came, and what they read. After you change your rules, use a view like this to confirm the answer bots still reach your pages.
AI bot requests on a demo site: which ChatGPT, Claude, Gemini and Perplexity bots came, and what they read. After you change your rules, use a view like this to confirm the answer bots still reach your pages. (Chetver dashboard on a demo site)

What I would do in your place

  1. Fetch what your site actually serves. Open your live robots.txt and check the meta robots tags and X-Robots-Tag headers on key pages. CDN and firewall settings can change the file without you noticing. HasData reported a Cloudflare change that removed AI opt-outs from the robots.txt files of 118 sites.
  2. Label every rule "training" or "answers". A rule that blocks OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot, Bingbot or Googlebot should be there only because you chose it on purpose.
  3. If you want out of training, add a separate __usage_ai-training group. Keep your GPTBot, ClaudeBot and Google-Extended rules for now. Decide on purpose whether you can accept losing Gemini app answers.
  4. If you use Bing's NoArchive or NoCache, plan the switch before 1 March 2027. Move training to the new robots.txt group and grounding to page-level tags.
  5. Check the logs again after every change. The answer bots should still get 200 responses.

Start with step 1 today. It is quick, and it shows you which decisions your site has already made without you.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot controls whether your content is used for training. OAI-SearchBot controls whether your site can appear in ChatGPT search results, and you set the two independently. ChatGPT-User handles pages a user asks ChatGPT to open.

What is __usage_ai-training and is it live yet?

It is a shared robots.txt group for expressing an AI training preference, defined in an IETF Internet-Draft. It is not live at the big vendors yet. Microsoft starts applying it from 1 February 2027, Google plans support in 2027, and OpenAI says the coming months. Until then, your existing per-bot rules, such as GPTBot and Google-Extended, are the ones that work.

Will opting out of AI training remove me from Gemini?

It can. Google will read a shared-token opt-out as a Google-Extended disallow. Reporting says Google-Extended also covers grounding in the Gemini apps. Googlebot controls Google Search, AI Overviews and AI Mode, so the opt-out does not affect them.

Does User-Agent: * Disallow count as an AI training opt-out?

Not under the new draft. A wildcard group must not be read as a training preference. A robots.txt file without a __usage_ai-training group expresses no shared preference. If you want to opt out of training, you need the dedicated group.

I use Bing NoArchive or NoCache. What do I need to do?

Plan the switch before 1 March 2027. After that date, Bing no longer reads NoArchive or NoCache for training or grounding. Set training with the __usage_ai-training group in robots.txt and grounding with page-level meta tags or the X-Robots-Tag header.

If I let AI bots read my site, will AI assistants mention me?

Not necessarily. Microsoft says neither choice guarantees inclusion, traffic or payment. Open access makes your pages eligible. Whether an assistant names you depends on clear, unique facts on your pages and on what other sources say about you.

llms.txt and an AI Page for your site, written by us.

Basic

$29/ mo per site

or $290 a year, 2 months free

  • An AI Page and a file for AI bots, refreshed monthly
  • A weekly website check and a list of fixes
  • A summary of AI bot requests

30-day money-back guarantee · cancel anytime