Major AI search engines respect robots.txt directives, but the policy choice for Canadian businesses is non-trivial. The relevant user agents to consider: **GPTBot** — OpenAI's web crawler used to train GPT models and power ChatGPT browse. Blocking GPTBot prevents your content from being used to train OpenAI models, but also prevents your content from appearing in ChatGPT's web responses. **ClaudeBot / anthropic-ai** — Anthropic's web crawlers for Claude. Blocking prevents Claude from citing your content in its responses. **PerplexityBot** — Perplexity's crawler. Blocking removes you from Perplexity's citation pool entirely. **Google-Extended** — Google's separate user agent for Gemini and other AI products. Importantly, Google-Extended is separate from Googlebot — blocking Google-Extended does not affect traditional Google Search indexing. This lets you opt out of AI training while remaining indexed for Search. **CCBot** — Common Crawl's bot, used as a corpus for many AI training pipelines. Blocking CCBot removes your content from many downstream AI training datasets. **The strategic decision for Canadian businesses:** For most Canadian businesses, the value of being cited by AI search engines exceeds the value of withholding training data. AI search drives meaningful brand awareness and qualified traffic, especially for B2B and high-consideration purchases. The withholding case is strongest for paywalled news sites, paid research, and proprietary methodology that needs IP protection. If you want to allow AI search citation but not AI training, the cleanest implementation is: allow GPTBot, ClaudeBot, and PerplexityBot (these are used for inference/citation); block Google-Extended (Google AI training) and CCBot (Common Crawl). This is a defensible middle position. **Sample robots.txt for the middle position:** ``` User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / # All other crawlers (including GPTBot, ClaudeBot, PerplexityBot, and Googlebot) allowed by default ``` Note that some AI products bypass robots.txt for direct user-initiated browsing requests. Robots.txt directives apply to systematic crawling, not to per-query inference fetches in many implementations.