AI assistants

An assistant can know a business from two sources. The first is its training data, which is months old by the time a model is released and rarely holds more than a passing mention of a small company. The second is a live search while the answer is written. The assistant turns the question into several related searches, a technique Google calls query fan-out, reads a handful of results and composes an answer with links to some of them.

The assistants do not share one index. Google's AI features use the Google index, while OpenAI, Anthropic and Perplexity run their own search crawlers, OAI-SearchBot, Claude-SearchBot and PerplexityBot, so a page has to be readable for each of them. Answer engine optimisation (AEO)AEOAnswer engine optimisation: the work of making pages easy for AI assistants to find, read and cite. It builds on SEO, because assistants mostly answer from a live web search.In the glossary is the work of being found and cited on this second route.

When an assistant answers which provider to choose or how a product works, it names a few sources, and those sources are the short list the reader sees. For Google's AI Overviews and AI Mode, Google states that a page needs nothing beyond what search already requires: it must be indexed and eligible to appear with a snippet. No AI text files or extra markup are needed. Work done for SEOSEOSearch engine optimisation: the work of making pages easy for search engines to crawl, index and choose as a result for relevant queries.In the glossary therefore carries over. Promises of special AEO techniques deserve scepticism, also because answers vary from user to user and from session to session.

Access comes first. Providers run separate crawlersCrawlerA program that loads web pages automatically, follows their links and passes the content to a search index or a training data set. Each identifies itself by name, such as Googlebot or OAI-SearchBot.In the glossary for search and for training, and robots.txtrobots.txtA text file at the root of a website that tells crawlers, one by one by name, which paths they may load. It controls crawling, not whether a page appears in search results.In the glossary treats each one on its own. OpenAI states that blocking GPTBot keeps content out of model training while OAI-SearchBot can still show the site in ChatGPT search. Anthropic splits the same way between ClaudeBot, Claude-SearchBot and Claude-User, and says all three honour robots.txt. Bot protection on a hosting or CDN service can block AI crawlers without anyone noticing, so its settings belong on the launch checklist. The content itself should arrive as HTML from the server, because not every crawler runs JavaScript.

Beyond access, the evidence is thinner. In a 2024 study by Aggarwal and colleagues, adding citations, quotations and statistics raised a page's visibility in a simulated generative search engine by up to 40 percent. Pages that put the answer in the first sentences and write sections that make sense on their own are easier to quote. Outside the page, an Ahrefs study of 75,000 brands found that mentions on other websites correlated far more strongly with presence in AI Overviews than backlinks did (0.66 against 0.22). The study shows a correlation and does not prove cause.