Evaluating “Characterizing Web Search by Conversational LLM Agents” for SEO/GEO Insights: A Hamsterdam Research Post

By Ethan Lazuk

Welcome to another edition of Hamsterdam Research! 🐹

This is where we look at recent AI research papers to learn what they’re talking about and explore their hypothetical implications for the future of search and SEO/GEO strategies.

This time, we’ll look at research called, “Characterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responses.”

It was submitted to arXiv on September 16, 2026, and its authors are Mahsa Amani, Seungeon Lee, Abhisek Dash, Asmaa El Fraihi, Yunah Jang, Elisabeth Kirsten, Qinyuan Wu, Krishna P. Gummadi, Manish Gupta, Abhilasha Ravichander, Muhammad Bilal Zafar, and Soumi Das.

To get us started, why should SEO/GEO professionals care about this research?

For SEO and GEO professionals, the study matters because it suggests that visibility in AI search is not one ranking event, but a sequence of retrieval, reformulation, selection, synthesis, and attribution decisions.

This research gives us a rare look at what happens between a user’s conversational prompt and the final AI-generated answer. The paper breaks that process into three stages:

1. whether the model decides to search the web
2. how it formulates and reformulates queries
3. how it uses retrieved pages to construct its response

This research shows that AI visibility can depend on several separate decisions happening in sequence. A model may decide not to search at all. If it does search, it may issue several queries, refine them using conversation history and previous search results, and make those queries increasingly specific around things like entities, locations, and dates.

An especially important finding for SEO/GEO professionals is that the different AI platforms are surfacing different slices of the web. The researchers found substantial platform-level differences in which domains appeared in search results. For example, Reddit and YouTube were prominent in the results observed for ChatGPT and Grok, while they were absent from Claude’s results in the study. That means visibility in AI search may depend not only on traditional relevance or authority, but also on the retrieval system sitting behind the particular conversational platform.

And then there is the citation layer. The researchers found that AI systems tended to cite more authoritative and popular domains, but they also found that a meaningful portion of generated claims were supported by retrieved pages that never received a citation in the final response. For SEO/GEO practitioners, this creates a particularly useful distinction, which is that being retrieved, influencing an answer, and being visibly cited aren’t necessarily the same thing.

Let’s start by reviewing the paper’s abstract.

Here’s the abstract, which is worth a read in full:

“Conversational LLM agents increasingly rely on Web search, yet the end-to-end lifecycle of agentic search remains poorly understood. We present the first study of Web search across four major conversational platforms (ChatGPT, Claude, Grok, and DeepSeek), combining real-world user interactions (invivo) with controlled experiments using the same platform’s models by their APIs (invitro). We investigate the quality of agentic decisions to invoke Web search, their strategies to formulate queries, the potential domain preferences in the search results they receive, and the choices they make when transforming search results into grounded responses. We find that Web-search decisions vary substantially across platforms and models, while more frequent Web-search invocation does not necessarily yield better response quality. We further show that conversational agents employ different complex querying strategies and that platform specific search engines return search results from their preferred domains. Finally, although responses are largely grounded in search results, some claims rely on uncited search results, raising concerns about attribution and reliability. Our findings have important implications for the design of future AI agents and Web search tools optimized for conversational retrieval.”

Next, let’s break down the paper’s key vocabulary terms:

• Agentic Web Search: Web search performed autonomously by an LLM agent as part of answering a user’s prompt. The agent decides whether to search, what to search for, and how to use the retrieved information.
• Web-Search Decision: The stage where an LLM determines whether its internal knowledge is sufficient or whether it needs to search the web.
• Query Formulation: The process of transforming a user’s often long conversational prompt into one or more shorter search-engine queries.
• Query Reformulation: The process of modifying subsequent search queries based on earlier search results, conversation history, and other context.
• Fan-Out Queries: Multiple search queries generated and issued in parallel from a single user prompt, allowing an agent to explore several aspects of a question simultaneously.
• Parametric Knowledge: Information encoded within the language model itself through training, as opposed to information retrieved from the live web during a search.
• Domain Preference: The tendency of a platform’s search system to return results disproportionately from certain websites or domains. The study found substantial differences in these preferences across AI platforms.
• Grounding: The extent to which claims in an AI-generated response are supported by evidence from retrieved or cited web sources.
• Citation Rate: The proportion of URLs returned by the search system that ultimately appear as citations in the generated response.
• Harness Instructions: System-level instructions that guide an LLM agent’s behavior, including when and how it should invoke web search tools. The paper finds that both these instructions and the underlying model influence search behavior.

Bueno. Let’s take a deeper look at the research paper’s contents now.

If you want to follow along, you can grab a PDF or the HTML version on arXiv.

The paper has six main sections.

We’ll focus on the most relevant ones below.

Introduction

“The ability of agents powered by large language models (LLMs) to autonomously search the Web is fundamentally altering how we access online information,” they write. “Today, millions of users rely on conversational AI platforms like ChatGPT OpenAI, Claude Anthropic, Grok xAI, and DeepSeek DeepSeek-AI as Web search assistants, trusting them to navigate the Web and generate accurate answers.”

They next talk about web search and agentic web search:

“The life cycle of agentic search (see Figure 2) similarly begins with the model recognizing and deciding that its own parametric (internal) knowledge is not sufficient to answer a user’s prompt. The next stage involves querying a search tool, both in parallel and in sequence, reformulating the queries based on retrieved results. The final stage involves synthesizing the search results from multiple (potentially contradicting) sources with the model’s own parametric knowledge to generate a coherent response.”

“To conduct our study,” they write, “we adopted the data collection strategy described in (Karnam et al., 2026) and collected conversational data donated by real-world users of four chatbot platforms, namely ChatGPT, Claude, Grok, and DeepSeek.”

“To our knowledge,” they continue, “our work is the first to investigate Web search by multiple chatbots invivo, i.e., when users are interacting with them for their normal, daily conversations.”

“To our knowledge,” they write, “our work is the first to systematically study how the choice of model family impacts Web search.”

They also describe the implications of their findings for users and designers of conversational AI agents. This is worth reading in full:

“For users, invoking Web search does not guarantee higher-quality responses. Moreover, Web-grounded responses can reflect platform-specific preferences for particular sources, rather than a comprehensive view of the Web. For designers, the substantial variation across search-calling decisions, querying strategies, and citation practices highlights the need to optimize and evaluate the entire search life cycle. In particular, designers should pay attention to source attribution, as agents may use information from those Web sources without giving appropriate credit to them.”

Analyzing Search Calling Decisions

“Web search is invoked when LLM agents decide that they need additional (external) knowledge, i.e., their parametric (internal) knowledge is insufficient, to respond to a user prompt,” they write. “So we begin by analyzing agents’ decisions to search the Web.”

They found that “The fraction of user prompts that result in Web search varies based on topic, even as they show a generally rising trend over time across all topics … for example, Travel-related prompts trigger Web search more frequently than Health-related prompts.”

They also found differences in the rates of web search in an invitro setting (repeated prompting) aspect of their research: “while GPT calls Web search for only 140 (14% of) prompts, Claude calls Web search for 825 (83% of) prompts.”

“Clearly,” they write, “LLM agents are making very different decisions on which prompts they invoke Web search.”

They next talk about harness instructions as a factor in web search activity:

“The striking differences in Web search behaviors by our LLM agents could potentially be attributed to two main factors: their harness instructions and models’ self-decisions. By harness instructions, we refer to explicit guidelines provided to models, often as part of their system prompt, on when they should invoke specific tools such as Web search. Models interpret and follow these instructions, when making their web calling decisions.”

“Next,” they write, “we investigate whether the observed differences in Web search behavior by LLM agents are justified, i.e., whether they result in improvement in quality of their final responses.”

Ultimately, they found that “While Web searches do tend to improve the response quality in general, they are not guaranteed to improve and the improvements do not fully compensate for the missing parametric knowledge.”

Analyzing Search Queries and Results

First they give some context on how web search works:

“Once an LLM agent decides to invoke Web search, it transforms a verbose user prompt into one or more Web queries. The queries are then answered by platform-specific search engines, which return Web pages (URLs) relevant to the queries, which can be used to both refine the queries and finally, generate the response. Accordingly, we first analyze the agents’ querying strategies and then investigate the results from their local search engines.”

They next talk about fanout queries:

“Unlike traditional Web search, where users typically issue a single query and manually refine it if needed, conversational agents can autonomously orchestrate the querying process by generating multiple Web queries, which can be issued both in parallel as fan-out queries and in sequence across iterations. The agents’ freedom to orchestrate multiple queries is typically specified in their harness instructions.”

“In our invivo dataset,” they write, “we observe that LLM agents frequently issue multiple queries per prompt, with the median number varying between 2 (Claude) and 4 (Grok), with the maximum number going much higher than 10.”

“Interestingly,” they write, “while ChatGPT, Grok, and DeepSeek generate queries in parallel fan-outs at each iteration, Claude only generates one query at each iteration, resulting in much deeper (over many iterations) search for Claude.”

“Next,” they continue, “we investigate how user prompts are reformulated into Web queries, including how iterative queries are reformulated to narrow down the search space, potentially using results from prior queries.”

One of their findings is that “Web queries are more concise than user prompts but longer than traditional queries.”

They also found that “New query terms get derived from the conversational context and prior search results.” They continue: “As search progresses across iterations, queries increasingly incorporate information from previous search results.”

Lastly, they found that “Web queries keep getting more specific over iterations.” They continue: “Web queries consistently become more specific (leading to higher Likert scores) by introducing constraints like exact dates, locations, and named entities.” Finally, “Rather than repeatedly issuing similar searches, conversational agents progressively reduce ambiguity and narrow the search space, enabling better search results.”

Next, they “investigate the domain preferences of the search engines employed by ChatGPT, Claude, Grok, and DeepSeek.”

They found that “Search engines built for LLMs return few results per web query and vary from one platform to another.” They continue: “On average, search engines on Claude and Grok return around 10 results per query, while DeepSeek returns far fewer (around 5) and ChatGPT returns far more (around 15).”

They also found that “Search engines show preference towards domains that vary from one platform to another.”

Their summary on this section also raises an interesting question about user awareness of LLM behavior:

“Compared to traditional search engines, search engines built for LLM agents return only a small number of results (typically, less than 30) and these results are frequently concentrated in a few domains and these preferred domains vary from one search engine to another. It is very unclear if users of LLM agents today are aware of such preferences and if they potentially constitute unwanted/harmful biases on the platforms.”

Analyzing Search Responses

“We now analyze how conversational agents synthesize the final response for user prompts using the returned search results from different sources together as well as their own parametric knowledge,” they write.

They found that “across platforms and our invivo and invitro setups, responses by LLM agents include, on average, between 1 and 8 citations.”

They also found that “Agents primarily cite URLs returned by Web search” while “only a small fraction (less than 1% to 5%) of the cited URLs are generated by the model’s parametric knowledge.”

Another finding was that “Agents preferentially cite more authoritative domains.”

“Having examined how agents select citations from retrieved search results,” they write, “we next investigate the extent to which claims are grounded, i.e., supported by evidentiary content in URLs cited and/or returned as search results.”

They found that “Most grounded claims are supported by cited sources.” However, “a substantial fraction of claims are instead supported by other cited URLs within the same response, suggesting that claims are not always strictly attributed to citations listed immediately following them.”

They also found that “A substantial fraction of claims rely on uncited search results.” They continue: “Across ChatGPT, Claude, and Grok platforms, these uncited, yet searched URLs, consistently account for 14% to 53% of all claims across both invivo and invitro settings.”

Lastly, they found that “Ungrounded claims are less factual than grounded claims.” They explain: “After our attempts to ground claims in URLs that are cited and returned as search results, we are left with claims that remain ungrounded – these are unexplained claims that are (potentially) generated from models’ parametric knowledge.” They continue: “upwards of 15% to 20% of all claims remain ungrounded.”

The implications of this are very interesting for SEO/GEO professionals:

“We find that claims supported by associated citations and other cited sources achieve the highest factuality scores (3.47, 3.35 in ChatGPT), followed by claims supported by search results that are not cited (3.33). In contrast, ungrounded claims (potentially generated by parametric knowledge) consistently receive the lowest factuality scores (2.84) across platforms.”

Conclusion

“Conversational LLMs are increasingly becoming Web-search agents that must decide when to search, how to search, and how to incorporate retrieved information into their responses. By combining real-world conversational traces with controlled experiments across four major platforms, we characterize this end-to-end search lifecycle. We find substantial differences across agents at each stage: Web-search invocation varies considerably and does not uniformly improve response quality; agents engage in distinct, increasingly specific querying strategies; retrieved results exhibit strong platform-specific domain preferences; and, while responses are largely grounded in retrieved evidence, a non-trivial fraction of claims rely on uncited search results or remain unsupported by the observable search trace. These findings show that conversational Web search should be understood and evaluated as a sequence of interconnected decisions rather than as a single retrieval step. Building more reliable search agents will therefore require improving not only retrieval itself, but also when agents search, how they seek evidence, and how faithfully they attribute that evidence in their responses.”

Knowing what we do now, why should SEO/GEO professionals care about “Characterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responses”?

SEO/GEO professionals should care about this research because it shows that visibility in AI search is not determined by a single ranking event.

The paper breaks conversational search into a chain of decisions where the model first decides whether it needs the web at all, then formulates one or more search queries, then it may refine those queries over several iterations, review the returned results, and finally decide which information to use and which sources to cite.

That chain of decisions has several important implications for SEO/GEO:

• AI platforms search differently. The study found large differences in when ChatGPT, Claude, Grok, and DeepSeek chose to invoke web search, meaning the same topic may depend heavily on live retrieval on one platform but mostly on parametric knowledge on another.
• The user’s prompt is not necessarily the search query. Agents compress conversational prompts into shorter queries and can issue several searches, including iterative and fan-out queries. Those searches become more specific over time by adding dates, locations, and named entities.
• Different AI platforms surface different parts of the web. The researchers found strong platform-specific domain preferences. For example, Reddit and YouTube appeared prominently in search results for ChatGPT and Grok while being absent from Claude’s results in the study.
• Retrieval and citation are not the same thing. A page can influence an AI-generated answer without ever receiving a visible citation. The researchers found that a meaningful share of claims could be traced to retrieved-but-uncited URLs.
• Authority appears to matter after retrieval, too. When choosing what to cite, the agents tended to favor more authoritative or popular domains over the broader set of URLs they retrieved.

Perhaps the biggest takeaway from the paper is that AI search visibility is a multi-stage process involving retrieval, query reformulation, source selection, synthesis, and attribution, not simply a new version of the traditional SERP.

Outro

I hope you’ve enjoyed this edition of Hamsterdam Research! 🐹

Feel free to comment below or contact me with your feedback.

Stay tuned for another new article, hopefully next week, or check out related research posts below.

Until next time, enjoy the vibes:

Thanks for reading. Happy optimizing!