Summarizing “Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations”: A Hamsterdam Research Post

By Ethan Lazuk

Welcome to another edition of Hamsterdam Research! 🐹

This is where we look at recent AI research papers to learn what they’re talking about and explore their hypothetical implications for the future of search and SEO/GEO strategies.

This time, we’ll look at research called, “Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations.”

It was submitted to arXiv on October 11, 2026, and its authors are Yunxuan Jiang, Shiqi Zhang, Anqi Xing, Xiangyu Chang, and Xiaokui Xiao.

First off, why should SEO/GEO professionals care about this research?

This study helps explain what happens between ChatGPT retrieving a webpage and deciding whether to cite it, highlighting retrievals vs. citations, an important distinction for understanding and measuring AI search visibility.

The authors show that a webpage can be relevant enough to retrieve but never appear as a citation.

They also find that different model settings show different query-rewriting behavior when it comes to query fanout.

The researchers also found substantial variation in retrieved webpages across repeated runs, where the same prompt doesn’t necessarily produce the same sources or citations every time.

Another finding was that Google rankings aren’t a reliable substitute for ChatGPT citation data. The researchers compared ChatGPT’s cited sources with Google Search results and the OpenAI Responses API, but even combining those external sources recovered only approximately 26–28% of the URLs cited by the studied ChatGPT Instant settings.

The study also found greater consistency at the hostname and entity levels than at the exact-URL level — meaning different pages from the same website may appear across repeated searches, which suggests brand and domain visibility may matter more than individual URL tracking suggests.

The ultimate takeaway for SEO/GEO professionals is that GEO strategies and measurement need to account for model differences. ChatGPT’s search behavior differed across the model settings studied, meaning query rewriting, retrieval frequency, and citation-related signals weren’t identical.

Let’s start by reviewing the paper’s abstract.

“Generative engines such as ChatGPT Web retrieve webpages to answer user queries, but only a subset are shown as citations. Understanding this retrieval-to-citation process is important for generative engine optimization (GEO), which aims to improve content visibility in generative-engine responses. However, prior studies mainly analyze final responses, leaving the internal execution process and its variation across model settings less understood. To this end, we conduct an observational study of four ChatGPT Web settings from ChatGPT 5.2 to 5.5, covering Instant and Thinking modes at the URL, hostname, and entity levels. We examine how ChatGPT rewrites queries, retrieves candidate webpages, and selects references. The Instant settings remain mostly single-round, while GPT-5.4 Thinking frequently performs multiple retrieval rounds. Across repeated runs, candidates retrieved more often are also more likely to be cited. We then evaluate Google Search and the OpenAI Responses API as external proxies. Responses API candidates provide broader coverage, while its citations more closely match sources cited by ChatGPT Web. Both proxies match ChatGPT Web better at the hostname and entity levels than at the exact-URL level. Even when combined, they recover only about one quarter of cited URLs. We further test whether these external signals remain predictive when combined. Within the external candidate set, a supervised model combining them reaches AUCs above 0.74 for GPT-5.3 and GPT-5.5 at all three levels. These findings suggest that GEO should distinguish retrieval visibility from citation visibility, account for rewritten queries and run-to-run variation, and consider hostname- and entity-level presence in addition to exact URLs. Differences across settings also suggest that GEO strategies should be re-evaluated when the deployed setting or serving environment changes.”

Next, let’s break down the paper’s key vocabulary terms:

• Query Rewriting: The process of reformulating a user’s original query into one or more search queries that an AI system executes to retrieve information.
• Candidate-to-Citation Funnel: The process through which ChatGPT retrieves candidate webpages and selects a subset of them as references for its generated answer.
• Retrieval Visibility: Whether a webpage appears among the sources retrieved by an AI search system, regardless of whether it is ultimately cited.
• Citation Visibility: Whether a retrieved source is selected as a citation in an AI-generated response, rather than simply appearing in the retrieval results.
• External Proxy: An outside system, such as Google Search or the OpenAI Responses API, used to approximate or predict ChatGPT’s retrieval and citation behavior.

Esta bien. Let’s take a deeper look at the research paper’s contents now.

If you want to follow along, you can grab a PDF or the HTML version on arXiv.

The paper has 9 main sections.

We’ll focus on the most relevant ones below.

Introduction

Below I’ll paste in excerpts from the introduction that provide essential context for understanding the paper:

“In traditional search engines such as Google Search, webpage visibility mainly depends on retrieval and ranking. In contrast, generative engines such as ChatGPT Web introduce an extra citation selection step. They retrieve webpages to generate an answer, but only a subset of the retrieved webpages are shown to users as citations. Understanding how generative engines select webpages for citation, and which signals remain informative across different model settings, can help brands improve their visibility and share of voice in generative-engine responses, a practice known as generative engine optimization (GEO).”

“… we conduct an observational study of four ChatGPT Web model settings from ChatGPT 5.2 to 5.5, spanning Instant and Thinking modes. We examine retrieval and citation behavior at the URL, hostname, and entity levels. We first examine how ChatGPT rewrites user queries, retrieves candidate webpages, and selects references from server-sent event (SSE) traces. We then evaluate whether Google Search and the OpenAI Responses API can approximate this process from externally observable results. We find that the observed model settings differ in how ChatGPT Web rewrites queries and retrieves webpages. Among the Instant settings, GPT-5.3 and GPT-5.5 use fewer executed queries than GPT-5.2 and remain mostly single-round, while the GPT-5.4 Thinking setting frequently performs multiple retrieval rounds and narrows the search target in later rounds. Repeated runs retrieve many different webpages, but candidates that appear more often across runs are also more likely to be cited. Longer retrieval snippets provide a weaker but consistent citation signal. We find that Google Search and the OpenAI Responses API only partially reproduce ChatGPT Web. Among the proxies, Responses API candidates provide broader coverage of sources cited by ChatGPT Web, while Responses API citations more closely match them. Both proxies match ChatGPT Web more closely at the hostname and entity levels than at the URL level.”

“These findings suggest that GEO should distinguish retrieval visibility from citation visibility, account for rewritten queries and run-to-run variation, and consider hostname- and entity-level presence in addition to exact URLs. Since retrieval behavior, proxy effectiveness, and citation signals differ across observed settings, GEO strategies should be re-evaluated when the deployed model setting or serving environment changes.”

Generative Engine Optimization Implications

The authors identify four key takeaways for GEO from their research:

1. Retrieval visibility does not guarantee citation.
2. GEO monitoring should account for rewritten queries and repeated runs.
3. External proxies are more informative at the hostname and entity levels than at the exact-URL level.
4. GEO strategies should be re-evaluated when the model setting or serving environment changes.

Conclusion

“This work presents a multi-setting observational study of how ChatGPT Web rewrites queries, retrieves candidate webpages, and selects references across four settings from ChatGPT 5.2 to 5.5. We find that retrieval behavior differs across settings and that candidates retrieved more often across repeated runs are more likely to be cited. Google Search and the OpenAI Responses API only partially reproduce ChatGPT Web, with stronger results at the hostname and entity levels than at the exact-URL level, while their signals can predict which sources ChatGPT Web cites. These findings suggest that GEO should distinguish retrieval visibility from citation visibility, account for rewritten queries and repeated runs, and consider hostname- and entity-level presence in addition to individual webpages. Future work will extend this study to more generative engines and model settings.”

Knowing what we do now, why should SEO/GEO professionals care about “Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations”?

The biggest takeaway from this research is that SEO/GEO professionals need to think beyond rankings and retrieval to understand how their content becomes part of AI-generated answers.

Here are several more practical takeaways, connecting the study’s findings to actions SEO/GEO professionals can take:

1. Optimize for search intent and query variations, not just exact keywords

Build content around topics and search intents, including relevant subtopics, comparisons, questions, and terminology variations. This is supported by the research finding that ChatGPT frequently rewrites users’ original queries into different search queries and some model settings execute multiple searches and progressively narrow their focus.

2. Measure retrieval visibility and citation visibility separately

Look for situations where content appears in retrieval results but isn’t selected as a citation, then investigate possible reasons.

3. Track domain- and entity-level visibility alongside individual URLs

Evaluate whether multiple pages from a website appear across AI search results for relevant topics and consider content gaps where a website lacks useful resources addressing important customer questions.

4. Don’t treat Google rankings as a direct predictor of ChatGPT citations

According to the research, Google Search and the OpenAI Responses API combined recovered only 26.2–28.3% of the cited URLs in the studied ChatGPT Instant settings. Thus, don’t assume a number-one Google ranking guarantees AI citation visibility.

5. Segment GEO monitoring by model and search experience

The research found that ChatGPT’s retrieval and citation behavior differed across the model settings studied, including Instant and Thinking configurations. As a practical action, record the model, mode, and testing conditions whenever possible, but don’t assume results from one model accurately represent every ChatGPT experience.

Caveat: The researchers did not experimentally validate specific content optimization tactics, so the study is particularly valuable as a framework for more informed GEO measurement and hypothesis-driven experimentation, rather than a new checklist of guaranteed citation improvements.

Outro

This research taught us that SEO/GEO strategies should address three distinct challenges: 1. getting content discovered, 2. getting it selected as a citation, and 3. measuring that visibility reliably.

For practitioners, the clearest immediate opportunities are improving how we research conversational search intent, expanding visibility measurement beyond individual URLs, testing prompts repeatedly, and avoiding overreliance on Google rankings as proxies for AI citations.

I hope you’ve enjoyed this edition of Hamsterdam Research! 🐹

Feel free to comment below or contact me with your feedback.

Stay tuned for another new article, hopefully next week, or check out related research posts below.

Until next time, enjoy the vibes:

Thanks for reading. Happy optimizing!