Exploring “Collaborative Search Meets Large Language Models: An Empirical Study” for SEO/GEO Insights: A Hamsterdam Research Post

By Ethan Lazuk

Welcome to another edition of Hamsterdam Research! 🐹

This is where we look at recent AI research papers to learn what they’re talking about and explore their hypothetical implications for the future of search and SEO/GEO strategies.

This time, we’ll look at research called, “Collaborative Search Meets Large Language Models: An Empirical Study.”

It was submitted to arXiv on November 3, 2026, and its authors are Peiyuan Gong, Jiamian Li, Erhan Zhang, Fengming Liang, Kexin Liu, Yue Chen, and Jiaxin Mao.

First off, why should SEO/GEO professionals care about this research?

This paper examines how AI changes people’s search behavior.

Rather than our usual focuses of rankings, citations, and visibility, this research raises a more fundamental question, which is what happens to the relationship between searchers, sources, and information when AI becomes an active participant in the search process?

From this controlled study of 80 students completing collaborative research tasks, there are at least five takeaways we can make:

1. AI is changing how people interact with search results.

Participants using an AI agent that synthesized web information clicked fewer search results than participants using a conventional in-chat search tool. This illustrates how AI-generated answers can reduce the need to inspect individual sources, with potential implications for organic traffic and how we measure visibility.

2. Search is becoming more conversational and context-dependent.

The study’s CoSearchAgent interpreted conversations involving multiple people, reformulated queries, asked clarifying questions, and retrieved information based on the evolving discussion. For SEO/GEO, this highlights why understanding broader information needs matters alongside optimizing for individual keywords.

3. Convenience doesn’t necessarily mean better information.

Although LLM-based assistance reduced perceived effort and improved collaboration, conventional search tools produced richer final reports. This suggests that AI summaries can make information easier to consume while sacrificing detail, a consideration for content creators who want their expertise represented accurately.

4. Source credibility and verification still matter.

Participants appreciated answers supported by inspectable sources, but they also reported questionable references, irrelevant links, and incomplete information. This raises questions about the role of trustworthy, comprehensive source material in AI-mediated discovery.

5. AI search may increasingly support collaborative decision-making.

The study found that LLM-based agents generally improved participants’ ability to share information and coordinate their research. For marketers, this opens an interesting perspective on how AI could influence group research, whether people are comparing products, evaluating vendors, or exploring complex purchasing decisions.

Let’s start by reviewing the paper’s abstract.

“Collaborative search enables users to address complex information needs jointly. Advances in large language models (LLMs) enable autonomous agents to support collaborative information seeking within shared communication spaces. However, little is known about how these agents shape search experiences, behaviors, and outcomes compared with conventional search tools. We conduct a user study with 80 students working in 40 pairs. Each pair completes four collaborative search tasks under four scenarios: QAAgent, which answers queries using an LLM’s parametric knowledge; CoSearchAgent, which uses multi-user conversational context to reformulate queries, retrieve web information, and generate source-grounded answers, while asking clarifying questions when needed; an in-chat search plugin that returns result lists; and manual Google Search. After each task, participants complete several questionnaires assessing their search experience and independently write a report; after all tasks, they participate in individual interviews. Results show that LLM-based agents reduce perceived workload, support collaboration, and reshape communication and evidence-use practices. They also encourage more independent paraphrasing, albeit at the cost of report richness. To the best of our knowledge, this is the first systematic empirical comparison of LLM-based agents and conventional tools for collaborative search across search experience, user behavior, and collaborative outcomes.”

Next, let’s break down the paper’s key vocabulary terms:

• Collaborative Search: The process of multiple people working together to find, evaluate, share, and synthesize information to address a common information need.
• Conversational Query Understanding: The ability of a search system to interpret a user’s information need using the context of an ongoing conversation, including previous messages, references, and unstated details.
• Parametric Knowledge: Information encoded in a language model’s parameters during training, which the model can use to generate answers without retrieving external sources.
• Web Evidence Extraction: The process of identifying and extracting relevant passages or information from retrieved webpages to support an answer.
• Source-Grounded Answer Generation: The process of generating an AI response using information retrieved from external sources, with supporting citations that allow users to inspect the underlying evidence.

Bravo. Let’s take a deeper look at the research paper’s contents now.

If you want to follow along, you can grab a PDF or the HTML version on arXiv.

The paper has seven main sections.

We’ll focus on the most relevant ones below.

Introduction

Below I’ll paste in excerpts from the introduction that provide essential context for understanding the paper:

“… little research has systematically compared LLM agents with traditional search tools in collaborative search. It remains unclear how their richer, adaptive assistance affects search experience, shapes user behavior, and influences collaborative outcomes. …

To address this gap, we conduct a user study with 80 students organized into 40 pairs, systematically comparing two forms of LLM-based assistance with two conventional search tools. Specifically, QAAgent answers users’ questions directly using the LLM’s parametric knowledge, whereas CoSearchAgent (Gong et al., 2024) interprets the multi-user conversational context, rewrites queries, proactively asks clarifying questions, retrieves information from the web, and generates source-grounded answers. We compare these agents with two conventional search tools: SearchAgent, a lightweight plugin that returns ranked result lists within the group conversation, and manual Google search, through which collaborators search independently and share their findings in the conversation. Each pair completes four open-ended collaborative search tasks, one under each scenario. After each task, each participant completes several questionnaires assessing their search experience and then writes a brief search report. After completing all four tasks, they participate in individual interviews.

Our results reveal that LLM-based assistance reshapes collaborative search by reducing perceived task demands and strengthening teamwork. QAAgent delivers the broadest collaborative gains, improving mutual awareness, information sharing, and exchange efficiency. CoSearchAgent provides the best agent experience and reduces evidence inspection through synthesized, web-grounded answers. However, these experiential benefits do not necessarily improve report quality. In the plug-in setting with a fixed word limit, reports assembled from copied webpage content are of higher quality than those based on copied agent responses, as webpages provide richer information for selection and synthesis. This helps explain why Google Search and SearchAgent produce richer reports, whereas LLM-based agents encourage independent wording. Overall, automated retrieval and synthesis reduce effort, but latency and clarification can offset time savings, while direct answers raise concerns about completeness and trust.”

Figure 2 from the paper shows the four different search scenarios participants used:

Results

The results section includes some interesting findings, which I’ll summarize below:

• AI reduced workload: Both AI agents made searching feel easier and less demanding than Google Search.
• QAAgent improved collaboration the most: It received the highest ratings for communication, information sharing, and teamwork.
• CoSearchAgent provided the best overall experience: Participants preferred its combination of AI-generated answers and web sources, while conventional search results received the highest reliability rating.
• AI reduced engagement with search results: CoSearchAgent users clicked 43% fewer results than SearchAgent users (13.18 vs. 23.20 per task).
• AI changed information sharing: Participants relied more on information already available within conversations rather than manually sharing webpage links.
• AI didn’t necessarily save time: Google Search had the shortest average interaction duration (16.05 minutes), while CoSearchAgent had the longest (20.48 minutes).
• Traditional search produced richer reports: Google Search scored highest for factual richness (4.56/7), compared with 4.21–4.24 for the AI agents.
• AI encouraged more independent wording: Reports created with AI assistance showed significantly less textual overlap than those produced using traditional search.
• There was ultimately a trade-off: AI simplified research and synthesis but sometimes sacrificed the depth available from original webpages.

In summary, AI-assisted search made collaborative research feel easier, but it also reduced direct engagement with sources and didn’t necessarily improve research quality.

For SEO/GEO professionals, the findings highlight a potential shift from users discovering and exploring webpages to consuming information synthesized by AI, with implications for website traffic, content visibility, and the value of original sources.

Discussion

Below is a summarization of key discussion points from the paper:

• AI reduced effort but didn’t always save time: AI agents simplified searching and information synthesis, but slow responses and unnecessary clarifying questions sometimes prolonged research.
• Source credibility remained a concern: Users appreciated AI-generated summaries with supporting links but encountered incomplete answers, questionable sources, and irrelevant information.
• AI improved collaboration but could interrupt workflows: Shared AI-generated answers helped participants coordinate, while excessive explanations and poorly timed questions sometimes disrupted conversations.

Conclusion

Below is the full conclusion:

“In this work, we compare LLM-based and conventional support for lightweight collaborative search through a study of 80 participants in 40 pairs. Results indicate that LLM assistance generally reduces perceived task demands and improves collaboration, with effects varying by assistance type. QAAgent supports the most active, responsive conversations and receives the highest collaboration ratings, while CoSearchAgent offers the strongest overall experience and reduces result inspection through source-grounded synthesis. However, users remain concerned about latency, incomplete answers, source credibility, and excessive or poorly timed interventions. Conventional tools produce richer reports than the LLM-based agents, partly because webpages provide more detail than length-limited agent responses. LLM-based reports show less wording overlap across reports. Among scenarios with browsable Web evidence, CoSearchAgent combines the highest report independence with factual richness close to that of the other scenarios. Report quality is also similarly consistent within pairs across scenarios. These findings suggest that collaborative search agents should help teams make sense of shared evidence while leaving users in control of the assistance they receive.”

Knowing what we do now, why should SEO/GEO professionals care about “Collaborative Search Meets Large Language Models: An Empirical Study”?

One of the biggest opportunities for SEO/GEO professionals, as implied by this research paper, is to rethink content optimization around how people research, collaborate, and make decisions with AI assistance, rather than focusing exclusively on rankings and citations.

In terms of practical takeaways from the research, here are five:

1. Optimize for evolving information needs, not just individual queries.

AI agents can interpret conversations and reformulate search queries as users refine their needs. Therefore, we should build content that anticipates follow-up questions, changing requirements, and increasingly specific comparisons, rather than targeting isolated keywords.

2. Create content that supports group decision-making.

This is a niche suggestion for group research, but participants wanted AI agents to organize information, connect evidence, and help divide research tasks. Therefore, we could develop comparison matrices, evaluation criteria, decision checklists, and stakeholder-specific resources that teams can reference when evaluating options.

3. Make source verification easier.

Participants valued inspectable references but encountered questionable sources and irrelevant supporting links. To solve for this, we can provide clear attribution, links to primary research, publication dates, and transparent methodologies so users can independently verify important claims.

4. Use layered content to balance efficiency and depth.

AI-generated summaries simplified research but sometimes lacked the factual richness of original webpages. Structure pages with concise answers followed by detailed explanations, supporting evidence, examples, and caveats can solve for this, accommodating both quick information retrieval and deeper investigation.

5. Test AI visibility across multi-turn conversations.

The research demonstrates how conversational context influences query interpretation and information retrieval. Thus, we should go beyond testing single prompts and simulate realistic research journeys involving follow-up questions, comparisons, changing criteria, and multiple stakeholders, then evaluate which sources AI retrieves, what information it preserves, and what it omits.

Outro

The biggest lesson for SEO/GEO professionals from this paper is that content optimization should account for the entire AI-assisted research process, not just the moment a source gets retrieved or cited. In other words, create content that is useful for discovering information, verifying claims, comparing alternatives, and supporting collaborative decisions.

I hope you’ve enjoyed this edition of Hamsterdam Research! 🐹

Feel free to comment below or contact me with your feedback.

Stay tuned for another new article, hopefully next week, or check out related research posts below.

Until next time, enjoy the vibes:

Thanks for reading. Happy optimizing!