Summarizing “Can AI Chat Replace Traditional Search? Evidence from a RAG-based Live Events Experiment”: A Hamsterdam Research Post

By Ethan Lazuk

Welcome to another edition of Hamsterdam Research! 🐹

This is where we look at recent AI research papers to learn what they’re talking about and explore their hypothetical implications for the future of search and SEO/GEO strategies.

This time, we’ll look at research called, “Can AI Chat Replace Traditional Search? Evidence from a RAG-based Live Events Experiment.”

It was submitted to arXiv on August 20, 2026, and its authors are Rakesh Allu and Evgeny Kagan.

First off, why should SEO/GEO professionals care about this research?

This paper challenges the assumption that often underlies discussions about the future of SEO/GEO that conversational AI represents an inevitable improvement over traditional search interfaces.

The first takeaway is that AI search adoption isn’t the same as AI search superiority. In an experiment involving 362 participants, the researchers found that a standalone AI chat interface produced less satisfying search experiences than traditional filters or a hybrid interface. This suggests the future of search may be more hybrid than purely conversational.

Another takeaway is that search intent may determine when AI search is most useful. The authors found a distinction between exploratory search and targeted retrieval. Surprisingly, when users knew what they wanted, conversational AI performed comparably to traditional search, but when people were browsing without a particular event in mind, chat-only search performed significantly worse.

A third takeaway is that conversational AI could narrow product and brand discovery. The authors identify a phenomenon they call the “echo chamber effect.” Without visible categories or menus, users relied on what they already believed was available when constructing prompts. As a result, they explored narrower portions of the event catalog and sometimes searched for things that weren’t available.

A fourth finding is that AI search could change how consumers evaluate results. The study identifies “prompting fatigue,” where users expend effort composing prompts and interacting with AI, potentially leaving less attention for evaluating the results. “As a result,” they write in the abstract, “users often pick one of the first displayed suggestions rather than examining the list more carefully.”

The final finding is that more capable AI models don’t necessarily produce better search experiences. The authors found no statistically significant differences in the measured search outcomes between GPT-4.1 mini, GPT-5.2, and GPT-5.4.

In short, the researchers didn’t find AI was categorically inferior, but rather that the way AI is integrated into a search experience (and the intent behind the user’s search) matters considerably.

Let’s start by reviewing the paper’s abstract.

“Conversational AI tools are being increasingly integrated into online search and browsing interfaces. To understand how such tools affect user behavior and search outcomes, we run a pre-registered, incentivized online experiment in which participants search a catalog of several thousand events scraped from a major live-events platform. We compare three interfaces: a classic, attribute-based filter interface (Filter), a GenAI chat interface (Chat), and a combined interface offering both (Hybrid). In both AI conditions, AI output is restricted via retrieval-augmented generation (RAG). Our main result is that, relative to the other two conditions, the Chat condition performs the worst. The effect is robust across AI models (gpt-4.1-mini, gpt-5.2 and gpt-5.4) and is driven mainly by users who perform open-ended exploration (as opposed to users searching for a specific artist, venue or sports team). Furthermore, there are no significant differences on any metrics between Filter and Hybrid. We also document two mechanisms. First, Chat creates an echo chamber: without a visible menu, users construct queries based on what they think is contained in the catalog. As a result, they frequently search for event types that are not available in the catalog, and they search too narrowly. Second, Chat leads to prompting fatigue: it shifts effort towards prompting, and away from inspecting and evaluating the output. As a result, users often pick one of the first displayed suggestions rather than examining the list more carefully. Taken together, our results suggest that AI is best used as a complementary tool to existing filter-based search rather than as a replacement.”

Next, let’s break down the paper’s key vocabulary terms:

• Retrieval-Augmented Generation (RAG): An AI approach that retrieves information from an external knowledge source and uses it to generate responses grounded in that information, rather than relying solely on a model’s training data.
• Echo Chamber Effect: A phenomenon where conversational search reinforces users’ existing assumptions by returning results based on what they already believe or know, potentially limiting their exposure to unfamiliar options.
• Prompting Fatigue: The mental effort associated with repeatedly constructing and refining AI prompts, which can reduce the attention users devote to evaluating search results and making decisions.
• Open-Ended Exploration: A type of search behavior where users browse available options without a specific target in mind, allowing them to discover products, services, or experiences they hadn’t previously considered.
• Hybrid Search Interface: A search experience that combines traditional navigation tools, such as filters and keyword search, with conversational AI, allowing users to explore information through multiple interaction methods.

Bravo. Let’s take a deeper look at the research paper’s contents now.

If you want to follow along, you can grab a PDF or the HTML version on arXiv.

The paper has six main sections.

We’ll focus on the most relevant ones below.

Introduction

Below I’ll paste in excerpts from the introduction that provide essential context for understanding the paper:

“Digital platforms have traditionally helped consumers navigate large inventories of products and services through attribute-based filters and text-based keyword search. A typical example is the search page of Ticketmaster, a leading live-events platform (Figure 1). The page is organized around category tabs, a location filter, a date filter, and a keyword box. To find an event users must translate their preferences into some combination of these predefined fields. More recently, platforms have started experimenting with a different design, in which users type their queries in natural language. These conversational (AI-based) interfaces are built on generative language models that have become sufficiently capable for routine commercial use. The promise of conversational (AI-based) search is that it can accommodate open-ended queries and preferences that are difficult to express through pre-defined filters.”

“Despite widespread enthusiasm among industry leaders, it is an open question whether AI adds value to consumer search and exploration, and how this value differs for specific use cases and implementations. On the one hand, the classic argument from the search literature (Stigler 1961, Bakos 1997, Brynjolfsson et al. 2003) is that a less-constrained search should produce greater value for the consumer. This suggests that AI should improve search outcomes as it relaxes the hard constraints of attribute-based filters and keyword-based search and lets users express moods and occasions that cannot be easily codified. … On the other hand, AI may increase search frictions: forming and typing natural-language queries and iterating with the AI may demand more time and effort than using filters to select a subset of the available options.”

“While it is easy to think of AI adoption as a binary decision, platforms must also decide how AI should be integrated with existing search tools. Should AI be treated as a substitute for or a complement to conventional search? Having access to both modalities may be especially valuable when consumers can specify some preferences precisely but can express others only in broad or subjective terms.”

They next talk about their experimental setup:

“Within the experiential goods category, we use a live-event ticketing platform (ticketmaster.com) as our empirical setting. … Participants’ task is to search the inventory of events with the goal of selecting the event that they find most appealing. …

We conduct three between-subjects treatments. In the first treatment, which we will refer to as Filter, participants use a classic, attribute-based interface with drop-downs for event date and type (Sports, Arts & Theater, Music) and a search box. The search box performs simple text matching: it returns events whose text contains the exact string of characters entered by the user. …

In the second treatment, which we will refer to as Chat, participants use a chat interface backed by a large language model (LLM) that has been given access to the same inventory of events as the filter-based search through Retrieval Augmented Generation (RAG), and that returns events with the closest match (“closest match” will be defined later). Finally, in the third treatment, which we will refer to as Hybrid, participants see the same interface as the Filter treatment with the search box now powered by an LLM. That is, Hybrid includes design elements of both Filter and Chat treatments.”

Next they preview their results:

“Our main result is that the value of conversational AI search depends on its implementation. … we find that the treatments are ranked as follows in terms of search performance: $\text{Hybrid}\approx \text{Filter}>\text{Chat}$. In particular, Chat is significantly outperformed by both Hybrid and Filter on three of the four main outcome measures. … we find that search intent is a moderator of the treatment effects. In particular, users performing open-ended exploration value their choices significantly less in the Chat arm relative to the other two arms, whereas users performing retrieval (looking for a specific artist, venue, or team) do just as well under Chat as they do in the other conditions. Finally, our results do not depend on which model powers the chat: whether the responses are generated by our legacy model (gpt-4.1-mini), or by newer models (gpt-5.2 and gpt-5.4) does not significantly predict any of the outcome variables.

Why does the Chat condition perform poorly, while the Hybrid condition performs on par with the traditional filter-based search? We trace this back to two mechanisms: the echo chamber effect and the prompting fatigue effect. The echo chamber effect occurs because Chat users, unlike those in Filter and Hybrid, do not see a menu of available event types. They must instead compose queries from what they believe is available in the catalog, and the model reflects those beliefs back in the form of search results. Search results are therefore limited to the part of the catalog the user already had in mind, or the user finds that the searched events are not in the catalog at all. Searching too narrowly and searching out-of-catalog are both associated with lower valuations. The prompting fatigue effect occurs because the Chat interface redirects the user’s effort and attention away from carefully examining the retrieved events and towards constructing effective queries and waiting for the system to return a list of events. This causes users to pick results near the top of the list, which in turn is associated with lower valuations. Together, the mechanisms suggest that structured interfaces provide consumers with a map of the feasible choice space, whereas a blank conversational interface makes users construct that choice space themselves, which can lead to poor search outcomes.”

If nothing else, reread the final few sentences from that last paragraph, particularly about how the echo chamber effect and prompt fatigue together “suggest that structured interfaced provide consumers with a map of the feasible choice space, whereas a blank conversational interface makes users construct that choice space themselves, which can lead to poor search outcomes.”

Results

I’ll summarize the results section below:

1. Traditional and hybrid search outperformed AI-only search

AI-only search performed the worst overall, producing lower user satisfaction and longer search times.

2. AI chat made searching slower and less satisfying

Chat users spent an average of 3.57 minutes searching, compared with 2.75 minutes using traditional filters and 2.91 minutes using Hybrid.

3. AI chat was particularly ineffective for exploratory searches

Among exploratory searchers, Chat produced event valuations $15.89 lower than Filter and $21.14 lower than Hybrid, based on the regression estimates. This suggests conversational AI may be better suited to targeted searches than open-ended discovery.

4. More advanced AI models didn’t improve results

Upgrading to a more capable language model didn’t translate into measurably better search outcomes in this experiment.

5. Hybrid search offered no measurable advantage over traditional filters

Researchers initially expected Hybrid to outperform both alternatives by giving users more flexibility, however, Hybrid performed similarly to traditional filters across all four primary measures.

Practical Implications and Outlook

Below are excerpts from the final section:

“Our experiment compared three versions of an event-discovery platform: a traditional interface with filters and keyword search, a conversational AI interface, and a hybrid offering both. …

Our comparison of the three interfaces was motivated by the rapid deployment of conversational search across digital platforms, and, just as in our experiment, the value of these tools in practice depends on how they are implemented and in what settings they are used. …

We believe that the divide between platforms that have successfully deployed AI tools and those that have not is not coincidental. Rather, it is consistent with the moderating effects of search intent identified in our experiment. Travel booking platforms that have moved quickly tend to serve users who have a specific target, such as booking a hotel in a specific location or finding a flight for a specific route. These use cases are defined by a well-defined, pre-conceived target – precisely the scenarios where we find conversational AI to do little harm. In contrast, the platforms that have been hesitant to deploy AI serve users who are more interested in open-ended exploration. Both Netflix and Airbnb sell experiences, and experiences are found by browsing, which is exactly the type of search where chat appears to perform worst. …

Layering a conversational interface on top of a structured filter, as in our Hybrid condition, has no detectable cost. In contrast, replacing the filter with chat lengthens search, narrows the portion of the catalog users see, and lowers the value of what they choose. Importantly, platforms need not invest in state-of-the-art LLMs to offer this functionality. Our results show that valuations are invariant to the model powering the chat (gpt-4.1-mini, gpt-5.2, gpt-5.4). In other words, users are not sensitive to the specific language used by the AI to describe its search results. This suggests that retrieval quality (as opposed to response generation) is what drives success or failure in the deployment of these AI systems, and even a lightweight AI model, such as gpt-4.1-mini, can perform RAG-based retrieval effectively.”

Knowing what we do now, why should SEO/GEO professionals care about “Can AI Chat Replace Traditional Search? Evidence from a RAG-based Live Events Experiment”?

Since we’ve already discussed the results of the study at length, here are practical takeaways extrapolated from the research findings:

1. Treat unsuccessful searches as a source of content and audience research.

For businesses with on-site search or AI assistants, unsuccessful queries could reveal unmet customer needs, confusing product descriptions, or missing information. This is based on the researcher’s finding that 49% of Chat participants made at least one out-of-catalog query, compared with 29% of Filter participants and 19% of Hybrid participants.

2. Create content that helps consumers discover their options, not just answer specific questions.

This research suggests a useful role for content: helping people understand what options exist before they know what to search for. Consider category guides, product-selection guides, comparison tables, alternative-product pages, and curated collections.

3. Make product and service catalogs easier to understand and navigate.

The researchers found that having a visible representation of available categories helped users explore the catalog. This supports a practical reason to maintain descriptive category pages, logical site hierarchies, contextual internal links, and clear navigation.

4. Expand GEO measurement beyond brand mentions and citations.

Consider tracking additional metrics where possible: how many relevant alternatives consumers examine, which products they compare, how frequently they refine searches, and whether AI-assisted visits generate qualified conversions.

Outro

The overall lesson from this research for SEO/GEO practitioners is that optimizing for discovery should involve understanding not only what information a search system retrieves, but also what opportunities consumers have to explore, evaluate, and choose from that information.

My most interesting takeaway personally was how prompting fatigue can lead people to choose from the first available results rather than doing further research.

I hope you’ve enjoyed this edition of Hamsterdam Research! 🐹

Feel free to comment below or contact me with your feedback.

Stay tuned for another new article, hopefully next week, or check out related research posts below.

Until next time, enjoy the vibes:

Thanks for reading. Happy optimizing!