Because the tracker and your chat window are not asking the same question under the same conditions, and even when they are, AI answers vary from one run to the next. A tracking tool typically sends a fixed set of prompts through an API or a clean session, with a declared location and no memory, several times, and reports how often your brand appeared. Your own chat window runs one prompt, once, inside an account that knows your location, your language, your history and often your company, and the model has frequently learned from your earlier questions that you care about your own brand.
Both views are real. They answer different questions: the tracker estimates what a typical buyer sees across many conversations, and your window shows what one very untypical buyer, you, sees in one. SQSEO users run into this constantly, which is why our own public tracking reports every share with its sample size and an interval rather than a single verdict.
Reason one: a single answer is one draw from a wide distribution
The biggest source of disagreement is not personalisation. It is ordinary variability. SparkToro and Gumshoe had 600 volunteers run the same 12 prompts through ChatGPT, Claude and Google’s AI, 2,961 runs in total, and found there is less than a 1 in 100 chance that ChatGPT or Google’s AI returns the same list of brands in any two of 100 answers. The same list in the same order was rarer still.
So when a report says you appear in 40% of answers for an intent and you open ChatGPT and do not see yourself, nothing is necessarily wrong. A 40% appearance rate means you will be absent from most individual answers. Checking once and concluding the tool is broken is like flipping a coin, getting tails and concluding the coin has no heads. The reverse also happens: you see yourself in your window and wonder why the report says 15%.
The fix is to compare like with like. If you want to check a tracker against your own experience, run the same prompt ten or twenty times in fresh sessions and count. That gives you a rough frequency to hold against the tool’s figure, and the two will usually be much closer than one screenshot suggested.
Reason two: your account is personalised, the tracker usually is not
Consumer AI assistants increasingly personalise. Memory features carry facts from earlier conversations, custom instructions shape tone and focus, and chat history gives the model context about what you work on. If you have spent weeks asking an assistant about your own product, it knows your product matters to you. That can make your brand more likely to appear in your answers than in a stranger’s.
Search based answers add the ordinary personalisation of search. Google explains that results can differ between people because of time, context such as location, language and device, and personalised results based on information in your Google Account. AI Overviews and AI Mode sit on top of Google Search, so the same context can shape what they retrieve and cite.
A well designed tracker removes as much of this as it can. It runs logged out or through an API, with memory off, and declares its settings explicitly. That is the right choice for measurement, because the goal is to estimate what a typical buyer sees, not what someone with your history sees. It also guarantees that your window and the report will differ whenever your personalisation pushes in a particular direction.
Reason three: location, language and model version
Location changes answers even for identical prompts, especially for anything with local or regional intent: suppliers, services, retailers, pricing, availability. API based trackers usually set a location on purpose. Anthropic’s web search tool, for instance, accepts a user_location parameter with city, region, country and timezone to localise results, and OpenAI’s web search tool takes an approximate user location in the same way. If the tool is set to the United States and you are in the Netherlands, you are effectively looking at two different markets.
Language works the same way. A prompt in Dutch retrieves and cites different sources from the same prompt in English, and a tracker running English prompts tells you little about Dutch buyers.
Then there is the model itself. The chat product may route your question to a different model or mode than the API model the tracker uses, may have search turned on where the tracker has it off or the reverse, and is updated on its own schedule. A report built last week on one model version and your check today on another can disagree for reasons neither of you controls.
| Factor | Your own chat window | Typical tracking tool | Effect on the comparison |
|---|---|---|---|
| Number of runs | One prompt, once | Many prompts, several runs each | Single checks mislead in either direction |
| Memory and history | Often on, knows your interests | Off or clean session | Can raise your brand in your window |
| Location | Your real location | A declared location, often one market | Different markets, different answers |
| Language | Your interface language | The language of the prompt set | Different sources retrieved |
| Model and mode | Whatever the product routes to, search on or off | A fixed API model and configuration | Different model, different answer |
| Prompt wording | How you would ask | How the prompt set was written | Different phrasing, different answer |
| Timing | Right now | Aggregated over days or weeks | Answers drift over time |
Reason four: the prompts are not the same prompts
Even when you type “the same question”, you rarely type the same words. Real people phrase one need in very different ways. In the SparkToro study, the prompts volunteers wrote for one scenario had an average semantic similarity of only 0.081. Small wording differences matter more than intuition suggests: research on prompt sensitivity found differences of up to 76 accuracy points from formatting changes alone in open models. Your phrasing reflects your knowledge of the category, often including the category name you use internally, which a buyer may never use.
Trackers have the opposite problem. Their prompt sets are written by someone, often generated by a model, and can lean toward a style that real buyers do not use. That is a real weakness of some tools, and a good reason to ask any vendor how prompts are produced. Seeding a prompt set from real query language, rather than writing it from scratch, is the method we describe in how to use synthetic prompt generation for AI tracking.
Reason five: time
Answers change as indexes update, as sources are added and removed, and as models are retrained or reconfigured. A tracker that reports a weekly average is showing you the middle of a moving distribution. Your check at 9am on Tuesday is a single point in it. Over a few weeks the two should broadly agree; on any given day they may not. The longer view is in how stable AI answers are over time.
A worked example
Suppose a report says your brand appears in 30% of answers for the intent “project software for a small agency”, from 240 answers over four weeks, with an interval of roughly 25% to 36%. You open your assistant, ask your own version of the question once, and you are not mentioned. That result is entirely consistent with the report: at 30%, you are absent from seven answers in ten.
Now suppose you instead run five of the tool’s actual prompts, ten times each, in a clean session with memory off and the tool’s location, and you are named in 14 of 50 answers, 28%. That is close to the report, and you can trust its trend lines. If you had been named in 2 of 50, or 40 of 50, you would have a concrete discrepancy to take to the vendor, with the prompts, dates and settings to back it up.
How to tell whether the tracker is wrong
Differences are expected, but trackers can also be wrong, and some are. A few checks separate the two.
Ask for the sample size. A share without a count of answers behind it is not a measurement. If the tool reports 30% and cannot tell you whether that is 3 of 10 or 300 of 1,000, treat it as anecdote.
Ask for the configuration. Which model, which mode, search on or off, which location and language, logged in or out, and on which dates? A tool that cannot say cannot be compared with anything, including itself next month.
Look at the raw answers. Good tools let you open the actual responses behind the number. Read a sample. If your brand is mentioned but the tool counted it as absent, or it counted a different company with a similar name, you have found a parsing problem, not a visibility problem.
Reproduce a slice yourself. Pick three prompts from the tool’s set, run each ten times in a clean session with memory off and the same location, and count. If your count lands inside or near the tool’s interval, the tool is measuring what it says. If it is far outside repeatedly, question the method. Our review of whether free AI trackers are accurate goes through these checks in more depth.
Which number should you use?
Use the tracker’s figure for decisions and trend lines, provided its method passes the checks above. It is the closer estimate of what buyers across your market see, and it is repeatable, which your window is not.
Use your own window for qualitative insight. It is excellent for reading how an assistant describes you, which sources it cites, which competitors it groups you with and whether any facts are wrong. Those observations are valuable even though the frequency is not.
And when stakeholders compare the two, explain the gap before it becomes a debate. The simplest framing is that the report is a poll and the chat window is one interview. Both are informative. Only one of them tells you how common something is.
Key takeaways
AI tracker reports differ from your own chat window because they measure different things under different conditions: many prompts and runs versus one, a clean session versus a personalised account, a declared location and language versus yours, a fixed API model versus whatever the product routes to, and an average over time versus a single moment. Individual answers are highly variable, with under a 1 in 100 chance of the same brand list repeating, so a single check cannot confirm or refute a report. Judge a tracker by its sample sizes, its declared configuration, access to raw answers and whether you can reproduce a slice of it. SQSEO’s own tracking follows those rules, and its free research gives you the real buyer phrasing to build a prompt set that reflects your market rather than your own vocabulary.
Frequently asked questions
Why do AI search tracker reports differ from what I see in my own chat window?
Because the tracker runs many prompts several times in clean, non personalised sessions with a declared location and model, while your window runs one prompt once inside an account with memory, history, your location and language. AI answers also vary heavily between runs. Compare a report with a tally of repeated clean runs, not a single screenshot. SQSEO reports every tracked share with its sample size and interval for this reason.
Is my own ChatGPT result more accurate than the tracker?
It is accurate for you, in that moment. It is not representative of other buyers, because your memory, history, location and phrasing are specific to you. For how often buyers in general see your brand, a well configured tracker is the better estimate.
Does ChatGPT memory affect whether my brand appears?
It can. Memory, custom instructions and chat history give the model context about your interests, which can make your own brand more likely to appear in your answers. Trackers usually run with memory off to avoid this.
How can I check whether a tracking tool is reliable?
Ask for sample sizes, the exact model and configuration, location and language, and the dates of the runs. Open the raw answers behind the numbers, and reproduce a few prompts yourself ten times each in clean sessions. If your tallies land near the tool’s figures, its method holds up.
Why does my brand appear in some answers and not others?
Assistants sample their answers and draw on retrieval that changes, so the list of brands varies from run to run. A brand with a 40% appearance rate is absent from most individual answers. Track the frequency over many prompts and runs rather than any single answer.
