Table of Contents
I’ve watched this industry build and discard a dozen tracking methods since 2006, from rank checkers that got IP-banned weekly to the rise of SERP APIs that made life easier almost overnight. So when a new way of measuring visibility shows up, my first question is always the same: does the shortcut actually measure the thing it claims to measure? A new research paper just gave a fairly blunt answer for one of the shortcuts a lot of AI-visibility tools are quietly relying on, and it’s worth sitting with.
The core finding
Researchers from the National University of Singapore and Xi’an Jiaotong University set out to test something a lot of us have assumed without checking: can you approximate what ChatGPT cites by running the same question through Google search results, or through OpenAI’s own API with web search turned on?
The short answer is: only partially, and mostly at the site level, not the page level.
Combining Google’s organic results with OpenAI’s Responses API found a little over a quarter of the exact pages that GPT-5.5 Instant actually cited in its answers. When the researchers looked at sites instead of individual pages, that number roughly doubled, landing just under half. The same pattern held for GPT-5.3 Instant, with slightly higher numbers across the board.
In other words, these stand-in methods are decent at telling you whether a domain shows up in ChatGPT’s answers. They’re much weaker at telling you whether a specific page on that domain is the one being cited.
How they actually tested it
What makes this paper worth paying attention to is the method, not just the headline figures.
The team captured the real data flow from signed-in ChatGPT sessions while the assistant was generating answers, so they could see the underlying search queries, which pages came back, and which of those pages ChatGPT actually picked as sources. They pulled 80 questions from Search Arena, a public dataset of real conversations people have had with search-enabled AI chatbots, split evenly between English and Chinese, and spanning four task types: recommendations, fact lookups, synthesis, and analysis.
Each question ran 20 times apiece on GPT-5.3 Instant and GPT-5.5 Instant between late March and late July 2026, with fewer runs on two earlier models that were retired mid-study.
To build the comparison data, they pulled up to five pages of Google organic results per question through SerpAPI, averaging around 43 results per query, and separately queried OpenAI’s API with web search enabled 20 times per question, all within a few days of the matching ChatGPT runs.
Why pages and sites behave so differently
This is the part that actually matters if you run or buy AI-visibility tracking.
Google’s results alone matched a small share of cited pages, in the single digits, but closer to a quarter of cited sites. The API’s source list did better on both counts, and pages the API cited in its own generated answer were a noticeably closer match to what ChatGPT cited than pages that merely showed up in its broader search results.
Part of the explanation is that ChatGPT doesn’t search the way you’d expect. In the runs where search activity was visible, neither GPT-5.3 Instant nor GPT-5.5 Instant used the exact wording of the original question, even once. An earlier model did, but only in a small fraction of runs. And for Chinese-language questions, both newer models ran the majority of their visible searches in English instead, which lines up with a pattern another research group documented earlier this year in fan-out queries.
There’s also a surprising amount of run-to-run variation. Roughly four in ten of the distinct pages that showed up for a given question across GPT-5.3 Instant’s 20 runs appeared in only one of those runs. GPT-5.5 Instant showed a similar spread. Sites were more stable across runs than individual pages were, but the authors are careful to note that a page showing up more consistently doesn’t necessarily mean it’s more likely to get cited. It just correlates.
What this means for anyone tracking AI visibility
If your agency, or a tool you’re paying for, reports ChatGPT citation data pulled from Google rankings or from the API rather than from the actual ChatGPT interface, treat the page-level numbers with real skepticism. The site-level numbers hold up better, but even those aren’t a perfect stand-in.
The researchers’ own recommendations are straightforward: track at both the page and site level rather than just one, rerun the same prompt multiple times instead of trusting a single pass, and remember that ChatGPT is rewriting your question into its own search queries behind the scenes, not searching your exact phrasing. They also flag something worth asking any vendor directly, whether the data behind a citation report came from the API or from the live ChatGPT app, since this paper shows those two sources don’t behave the same way.
Worth noting too: this is a preprint posted on arXiv, which means it hasn’t gone through peer review yet.
One more wrinkle: the models keep changing
Every model tested in this paper predates GPT-5.6 and GPT-6, which are already rolling out to different tiers of ChatGPT users. That matters because past model swaps have visibly shaken up citation behavior before. Around the GPT-5.5 rollout earlier this year, separate data on German-language citations showed day-to-day swings jump sharply above their normal baseline for a stretch.
The paper’s authors say as much themselves: monitoring approaches need to be re-evaluated whenever the model behind ChatGPT changes. Until someone repeats this comparison on GPT-6, these figures describe a snapshot of models that are already being phased out.
