Why AI answers change between runs

What can change between two requests

The engine we sample today is OpenAI GPT with web search, reached through an API rather than the consumer app. OpenAI's developer documentation describes the web search tool as a way for its models to access up-to-date information from the internet and provide answers with sourced citations. Which pages a request reads and how it weighs them is not something we can see, so two requests do not have to read the same pages or name the same businesses. We record the engine label with each result so you always know which product produced it.

The free check and the audit sample the API for the same reason we label it. The answer a customer sees in a consumer app can differ from ours on the same day with the same question, and neither is wrong. They are different products reading the web at different moments, and a measurement has to say which one it measured.

Four things can differ from one run to the next, and only the first is under your control: the exact wording of the question; the product and model that answered it; the location the request was made from or asked about; and the moment it ran. Change any of them and you have a different sample, not a repeat. Our methodology page states it plainly: answers can vary between requests, models, locations and products.

Order inside an answer moves too. When the response format allows it, the audit records where your name appears and which other businesses are named. That position is the order inside one sampled answer. It is not a search ranking, and the same question can put you first on one run and leave you out on the next.

Why one check proves very little

A single answer that names a competitor is one observation. It does not show that the competitor is recommended consistently, and it does not show that you are absent consistently. It shows what one request returned at one moment.

That is a measured fact, not a guess. In BrightLocal's Local AI Visibility Study, published in September 2026, which reviewed about 200,000 localized searches across roughly 1,300 business locations on three AI platforms, the average overlap between result sets on repeat searches was only 20 to 33 percent: 23 percent on ChatGPT, 21 percent on Google AI Mode and 33 percent on Google AI Overviews. The same study found that businesses still reappeared more than half the time, and that the platforms disagreed with each other as well. Variation is normal, which is the honest reason to sample many questions instead of trusting one.

The distinction we keep most carefully is between an answer that leaves you out and no answer at all. If an engine is unavailable, a request fails, or no answer comes back, that does not count as an answer that omitted your business. The free check marks each question as Mentioned, Not mentioned or No answer returned, and the audit reports coverage gaps separately from mention rates. A manual check on your phone has no such label, so a failed request can look like a snub.

One check also tempts you to explain it. A missing mention does not tell you why another business appeared, and a later mention does not prove that a change you made caused it. We treat every answer as an observation and keep the explanation separate.

What a wider sample does and does not do

The audit plans 40 to 60 questions for one business in one US city. They are built from your category, your city and state, and the services or customer needs you name, and they cover discovery, specific jobs, buying intent, comparison and questions that name your business directly. You can preview the planned set at checkout before paying.

The questions are grouped by the job they do. Some ask who is best or most recommended, some ask who to hire or who can come out today, some ask about cost, reviews or a comparison, some describe a specific need such as a water heater replacement, and a few name your business directly. Each group is a different way a customer might ask, and an answer that leaves you out of one group says little about the others.

A wider sample does not remove variation. It makes variation visible. When one unusual answer sits among forty, it stops standing in for the whole, and a pattern across many different questions is worth more than a verdict from one. The report keeps two counts apart for the same reason: a discovery rate for questions that do not supply your name, and a branded rate for questions that do. Recognizing a name you supplied is a different result from being named unprompted.

Even then, a rate describes the sampled answers. It is not the share of customers who will see your business, and a single audit cannot establish a trend, a cause or market share. We say this on the methodology page and in the report because the numbers invite stronger claims than they can carry.

How to compare two runs fairly

Record before you compare. Write down the exact wording of the question, the product and model, the date, the location, the full answer, the businesses named and the sources cited. The free checklist calls this comparing like with like, and it is an easy step to skip.

Compare question sets and coverage before you compare percentages. If the second run asked different questions, or fewer of them were answered, the two rates are not the same measurement. If your business was named on the second run and not the first, note it as a change in the sample and resist calling it a result.

Then choose one question for a like-for-like follow-up: the same engine, the same wording, a recorded run time and the missing responses written down. That is the smallest honest experiment you can run, and it is how the final check in the sample report is framed. The audit's question inventory lists what was included and its coverage notes list what was not sampled, so a second audit can be compared set to set before it is compared rate to rate.

Where the free check fits

The free check asks three local buyer questions through one available engine and shows the sampled answers on screen and by email, with the engine named. It needs your business details, an email address and a quick verification, and it runs once per email per day. Three answers are a starting point, not a verdict. Run it, record it, and read it as a sample.