Ask ChatGPT the same question twice and you can get two different sets of brands. That is not a bug in your tracking, it is how generative answers work, and it is the single biggest reason AI visibility dashboards get misread. SQSEO handles this by treating one answer as one sample rather than one measurement: you fan a seed keyword into the prompts your buyers actually use, then run each prompt repeatedly, because a brand that appears in three runs out of ten is telling you something very different from a brand that appears in nine.
Why the same prompt gives different answers
Three separate mechanisms move an answer, and they operate on different timescales.
Sampling is the fastest. Language models generate text probabilistically, so unless temperature is pinned to zero the model can choose a different next token and follow a different path through the same knowledge. Two runs, seconds apart, on an unchanged index, can name different brands in different orders. This is inherent, not a fault.
Retrieval is the second. Modern answer engines are retrieval augmented: they run a search, take the top documents, and generate from those. The retrieval augmented generation approach that underpins this was designed to ground answers in fetched documents rather than in memorised weights, which means the answer inherits every bit of volatility the underlying search has. If the search results move, the answer moves.
Freshness is the third and slowest. Models and indexes are updated, and research on keeping models current with search augmentation found that performance on questions with changing answers depends heavily on how recently the retrieval layer has seen the world. Work on refreshing large language models with search engine augmentation showed that time sensitive questions are exactly where ungrounded models degrade, which is why answer engines lean on live retrieval and why their outputs shift as the web shifts.
What volatility means for your numbers
| What you are measuring | Runs needed for a usable read | What a single run tells you | Main risk | Verdict |
|---|---|---|---|---|
| Do we appear at all | 10 or more per prompt | Almost nothing: absence in one run is not absence | Declaring a problem that does not exist | Never act on one run |
| Share of voice against rivals | 20 or more per prompt, repeated weekly | A snapshot with a wide error bar | Reading noise as a competitor gain | Trend over weeks, never week on week |
| Which sources get cited | 10 or more, collected as URLs | A partial list of the citable pool | Optimising for one citation that was luck | Look for URLs that recur, ignore the rest |
| Effect of a content change | Baseline of 20 before, 20 after, same prompts | Nothing usable | Crediting a change that was drift | Hold the prompt set fixed or the comparison is void |
The practical rule that falls out of this: a number without a run count attached is not a measurement. Any tool or report that says “you appear in 40 percent of answers” without saying how many answers is describing a sample of unknown size.
Citation volatility is worse than mention volatility
Being named is one thing. Being cited with a link is another, and the citation layer moves more.
A study evaluating verifiability in generative search engines examined answers from several commercial systems and found that a substantial share of generated statements were not fully supported by their own cited sources, with citation precision varying widely between engines. The evaluation of verifiability in generative search engines matters here for a practical reason rather than an academic one: if the link between a claim and its citation is loose, then the citation you won this week was partly a retrieval accident, and it can be lost without anything about your page changing.
This is why tracking citations by URL over many runs is more useful than tracking a citation count. A URL that appears in seven of twenty runs is genuinely in the citable pool for that prompt. A URL that appeared once is a coin landing on its edge. Grouping by URL and counting recurrence turns a noisy signal into a stable one, and it is the same discipline behind tracking LLM mentions and citation gaps rather than chasing individual appearances.
The engines differ, and they differ in how much they move
Perplexity, ChatGPT search, Google AI Overviews and Claude do not behave alike. Systems that retrieve aggressively and cite heavily move more between runs, because they inherit more of the search layer’s variance. Systems that lean harder on model knowledge move less between runs but drift more between model versions, which is a slower and larger jump.
Google AI Overviews sit in their own category because they are attached to a ranking system with its own volatility, and Google’s own documentation on AI features in search is explicit that these surfaces are generated per query rather than served from a fixed index of answers. Analysis of click behaviour around AI Overviews, including Ahrefs data on AI Overviews reducing clicks, points at the same conclusion from the traffic side: what appears above the organic results is not stable enough to treat any single observation as a position.
The takeaway for a tracking setup is that engine comparisons need equal run counts per engine. Comparing twenty ChatGPT runs against five Perplexity runs and concluding that Perplexity is more consistent is comparing sample sizes, not engines.
How many runs is enough
The honest answer is that it depends on how large a change you need to detect, and the arithmetic is ordinary sampling arithmetic rather than anything specific to AI.
If your brand truly appears in half of answers for a prompt, ten runs gives you a result somewhere in a wide band around five, and seeing four one week and six the next is pure noise. Twenty runs narrows that band meaningfully. Fifty narrows it further but costs proportionally more, and past a point the underlying reality has changed before you finish measuring.
For most teams the workable settings are ten runs per prompt for a weekly pulse across a broad prompt set, and twenty to thirty runs on a small set of decision critical prompts where you need to detect real movement. Spend the extra runs on the prompts that matter commercially rather than spreading them evenly, which is the same prioritisation logic behind picking prompts in the first place.
Build the prompt set before you buy the tracker
Volatility makes prompt selection more important, not less, because every prompt you track costs runs and runs are the scarce resource. A tracker pointed at forty vague prompts at five runs each produces forty unreliable numbers. The same budget on twelve high intent prompts at twenty runs each produces twelve numbers you can act on.
This is where the research step earns its place. SQSEO fans a seed keyword into the questions people actually ask in your niche, so the prompt set is grounded in real query language rather than in what the team assumed buyers say. Doing that before subscribing to a tracker is the difference between measuring the right twelve things carefully and the wrong forty things badly, and it is the same argument made in best AI visibility tools: the tool measures, the research decides what is worth measuring.
What to do about volatility in practice
Fix the prompt set and leave it fixed. Every prompt you add or reword resets the comparison, so treat the prompt list like a survey instrument rather than a live document. Keep a changelog if you must change it.
Run at a consistent cadence and a consistent time. Answers drift with the web, and the web drifts with the news cycle, so a Monday morning run and a Friday evening run are not directly comparable for anything topical.
Report bands, not points. “Between 30 and 50 percent of runs, twenty runs per week over four weeks” is a statement someone can act on. “42 percent” is a statement that invites a meeting about a two point drop that never happened.
And separate the two questions that get conflated: are we in the citable pool for this prompt, and how often do we surface from it. The first is a content and authority problem. The second is partly luck, and no amount of optimisation removes the sampling variance underneath it.
Model version changes are the jump, not the drift
Run to run variance is noise you can average away. A model version change is not, and the two get confused because both show up as a moved number.
When an engine ships a new model, or changes how it retrieves, the entire behaviour of the answer can shift at once: which brands it considers comparable, how many sources it pulls, whether it names vendors at all for a given question type. That is a step change, and averaging across it destroys the very thing you want to see. The Stanford AI Index report documents how quickly capability and deployment have moved year over year, and the practical consequence for anyone tracking visibility is that a twelve month trendline usually spans several different underlying systems.
The handling is straightforward once you name it. Keep a dated log of engine and model changes alongside your tracking data, and when you see a discontinuity, check the log before checking your content. A number that moves fifteen points in a week and stays moved is a system change. A number that moves five points and comes back is sampling.
Prompt phrasing moves answers more than most content changes
One underrated source of variance is the prompt itself. Asking “what is the best X” and “which X should I use for Y” are different retrievals and frequently return different brand sets, even though a human would call them the same question.
This cuts two ways. It is a risk, because a tracker whose prompts drift in wording is measuring a moving target. And it is an opportunity, because the phrasings your buyers actually use are the ones worth winning, and they are often not the phrasing a marketer would write. Real query language beats invented query language, which is the whole reason the prompt discovery step comes before the tracking step rather than after it.
It also explains a common frustration: a brand that appears reliably for one phrasing and never for a near synonym. That is not a ranking problem to fix, it is two different questions with two different citable pools, and the fix is content that answers the second question directly rather than more optimisation of the page that already wins the first.
What this means for reporting to people who are not in the weeds
The reporting habit that survives contact with a leadership team is to lead with the band and the sample size, then the trend, then the action. “35 to 40 percent of 240 answers per week, flat over four weeks, and the four URLs doing the work are these” is a slide someone can make a decision from.
What does not survive is a single headline percentage with a week on week arrow next to it, because the arrow will be wrong roughly half the time and the credibility cost of chasing phantom drops is high. Setting the expectation early, that these numbers move and the movement is measured rather than eliminated, is easier than explaining it after the first noisy week.
A worked example: reading a real tracking week
Take a prompt set of twelve buying intent questions, twenty runs each, four weeks running, one brand tracked against three rivals.
Week one, your brand appears in 96 of 240 answers, 40 percent. Week two, 84 of 240, 35 percent. That five point drop reads like a problem and is well within the range you would expect from sampling alone at this volume. Nothing has happened.
Week three, 88 of 240. Week four, 91 of 240. The four week picture is 35 to 40 percent, flat, and the week two dip was noise. Anyone who reacted to it in week two spent a fortnight investigating a change that did not exist.
Now the citation layer. Across all 960 runs, thirty of your URLs were cited at least once, but only four appeared in more than fifty runs. Those four are your actual citable assets for this prompt set. The other twenty six are noise, and treating them as wins would send your content team in twenty six directions.
Key takeaways
AI answers vary by design, through sampling, through retrieval, and through freshness, so one answer is one sample rather than one measurement. Attach a run count to every number or the number means nothing. Citations move more than mentions, so group by URL and count recurrence rather than chasing individual appearances. Give equal runs to each engine before comparing them, fix the prompt set and cadence, and report bands rather than points. Above all, spend your run budget on the handful of prompts that decide revenue rather than spreading it thinly across everything.
