Summarizing “Web vs. LLMs: An Empirical Study of Learning Behaviors of CS2 Students” from an SEO’s perspective (Hamsterdam Research)

The emergence of generative AI chatbots like DeepSeek, ChatGPT, and Gemini has caused SEO professionals to reevaluate their strategies for content visibility.

The question is how will users engage with chatbots versus how they’ve used the web traditionally, and what are the implications of those differences in behavior for SEO strategies?

In this Hamsterdam Research post, we’ll take a look at a study that analyzes how students engage with LLMs versus “traditional online methods” to learn programming concepts.

While not related to SEO directly, this paper can help us glean insights into user behavior more broadly that we can use for our own strategies.

Introducing the paper

The paper is called “Web vs. LLMs: An Empirical Study of Learning Behaviors of CS2
Students.”

It was published recently on January 22nd, 2025 by Aayush Kumar, Daniel Prol, Amin Alipour, and Sruti Srinivasa Ragavan.

Let’s start with the paper’s abstract (my highlights throughout), then we’ll get into the details:

“LLMs such as ChatGPT have been widely adopted by students in higher education as tools for learning programming and related concepts. However, it remains unclear how effective students are and what strategies students use while learning with LLMs. Since the majority of students’ experiences in online self-learning have come through using search engines such as Google, evaluating AI tools in this context can help us address these gaps. In this mixed-methods research, we conducted an exploratory within-subjects study to understand how CS2 students learn programming concepts using both LLMs as well as traditional online methods such as educational websites and videos to examine how students approach learning within and across both scenarios. We discovered that students found it easier to learn a more difficult concept using traditional methods than using ChatGPT. We also found that students ask fewer follow-ups and use more keyword-based queries for search engines while their prompts to LLMs tend to explicitly ask for information.”

– Abstract

So far, we see that traditional methods (related to web search) were more effective for students to learn than using ChatGPT. We also see there were differences in the behavior of using web search and LLMs regarding more keyword-based queries for search engines with fewer follow-ups.

With that context, let’s take a look at the paper’s contents …

Introduction

“Large Language Models (LLMs) such as ChatGPT have gained popularity for their ability to assist learners through interactive and personalized guidance,” the authors write. “These systems can provide high-quality explanations, generate examples, and answer questions in real-time.”

However, LLMs have drawbacks, both in terms of inaccuracies as well as impacting how users learn, such as fostering critical thinking, which is why “the research on whether or not to adopt LLMs in learning contexts is divided.”

“There is also a paucity of empirical studies comparing learning with LLMs against traditional web-based learning (e.g., using video tutorials, documentation, and websites such as Stack Overflow) – both in terms of quantitative outcomes and qualitative behaviors,” write the authors. “As a result, making informed decisions about learning choices becomes hard for both students and educators alike.”

That’s the focus of their paper, “to address this gap by comparing the effectiveness of learning programming concepts with AI vs. traditional web (NoAI) resources.”

Their study involves 32 participants at a large engineering university in India “learning new topics in programming concepts with LLMs and traditional web resources.”

Their results “suggest that the participants performed better on the post-study evaluations in the NoAI condition,” meaning traditional web resources. They “also observed a statistically significant difference in between the types of prompts in AI and queries under NoAI conditions.”

So far as SEOs, this should give us promise that users might stick with traditional web resources for informational purposes, despite the convenience of generative AI chatbots. We should also extrapolate that user behavior between LLMs and web search engines will be different, which might impact the strategies for getting content to appear in either place.

Related work

The researchers next discuss related work, largely in the context of learning programming. This section isn’t as relevant to SEOs, but let’s pull out a few notable quotes.

“Early experiments with AI-based feedback mechanisms have shown promise in delivering helpful and high-quality feedback, although challenges remain to balance automated assistance with learning objectives,” they write.

“ChatGPT’s conversational interface has profoundly impacted programming education,” they continue. The interface “enables personalized learning experiences and promotes student agency,” while ChatGPT is also effective “in debugging and concept clarification.”

However, “AI tools accelerate learning for some students while potentially hindering others,” the researchers note.

As for traditional web search, “Research on information seeking in the web has identified fundamental patterns in how developers learn programming online.” For example, “developers adapt online examples to their needs.”

Research questions

The researchers have two primary research questions, including whether learning outcomes and information seeking strategies differ in AI and NoAI (traditional web search) treatments.

Methodology

The researchers spend a large amount of time discussing their methodology. Again, this isn’t necessarily relevant to us as SEOs, so we’ll summarize some of the key insights here.

To answer their research questions, the authors “conducted a user study with 32 participants, comparing their learning of programming concepts with and without AI.”

“Each participant learned two different programming concepts, namely currying and immediately invoked function expressions (IIFE) in Javascript. They were instructed to learn one of these tasks using only ChatGPT (GPT 3.5 Turbo) and the other using traditional online resources (e.g web search, videos) starting with a Google search page, without the use of any Generative AI tools. We call these two conditions AI and NoAI respectively.”

In terms of data collection, the authors “gathered participants’ solutions to the two assessments (i.e., quiz and debugging task) and scored them for further analysis.”

Results

“Participants learning with AI performed significantly worse than those learning through web search in the theoretical quiz for currying,” they write.

“Participants preferred learning by example across both treatments,” they continue. “They did not spend much time coding while learning, often using the IDE only to verify external code. Some participants watched videos while learning as a more dynamic alternative to websites.”

This part is interesting, as we’ve seen a rise in the importance of video content for SEO strategies in the generative AI age, both in terms of videos as sources in AI Overviews as well as the fact that multimodal AI models can use video to inform answers just as easily as text.

The researchers also discuss query analysis, which might be an interesting section for SEOs.

“Participants wrote a total of 148 search queries (Google + YouTube) in the NoAI treatment and a total of 237 prompts to ChatGPT,” suggesting fewer follow-up questions were asked during web search.

To that end, “Most (18 out of 32) participants used keywords based on the learning objectives as the starting point for their web information foraging.”

“As participants progressed in their learning, querying differences surfaced again. AI prompts were often follow-ups to previous prompts, in contrast to those in web searches. At the same time, a higher percentage of web queries were copied directly, from sources such as prior websites they had visited. These differences in information seeking behaviors could be considered adaptations to the environment. For example, ChatGPT offered byte-sized answers to specific questions and afforded easy follow-up questions in a more chatty manner, whereas web searches offered a longer list of diverse, potentially useful sources that participants had to sift through and choose to learn from.”

In short, “Participants used more follow-up queries when using LLMs, and phrased their queries differently across learning conditions, showing a tendency to use keyword-based queries for search engines and to explicitly ask for information in their queries to LLMs.”

We’ll skip the section on threats to validity and jump to concluding remarks.

Conclusions

The researchers conclusions cover several areas, summarized below:

Perform fast and learn slow

“ChatGPT and other LLMs can improve students’ productivity in performing programming tasks, but their effectiveness in helping students learn and understand new concepts is unclear. In our study, while AI helped participants slightly more than web search in successfully debugging programs, a notable number of participants in the AI condition lacked understanding of their proposed fixes. Similarly, participants in the AI treatment performed poorly in the quiz for currying, the more difficult task, as compared to participants in the NoAI treatment, suggesting that the nature of productivity may be at odds with learning. While productivity emphasizes on the speed in completion of tasks, successful learning can be slow and tedious, and ChatGPT might not be the best tool for learning, especially in the absence of sufficient metacognitive abilities.”

In other words, ChatGPT was an efficient method for completing tasks, but students may not have learned or understood the concepts as clearly as they did using web search.

Importance of example-based learning

“Learning theories have emphasized the importance of examples in learning. Examples can provide learning opportunities for learners to imitate experts, discern analogies and contrast among different problems. In our study, participants across treatments showed a tendency to learn through examples – participants in the AI treatment spent 51% of their time learning by example while those in the NoAI treatment spent 40% of the time learning by example and an additional 10% learning through videos. This emphasizes the importance of incorporating high-quality examples in pedagogy.”

Though a niche topic, we can see how providing examples is helpful for information seekers. As SEOs, we can understand how this plays into demonstrating experience in content by showing examples of the work.

Need for holistic LLM responses when learning

“Participants in our study tended to write keyword-based queries when using search engines, which led them to websites containing human-curated content about programming concepts. While using LLMs, participants wrote more questions and explicitly asked for information in their prompts. Since LLMs reply directly to these prompts, their responses might only cover the portion of information about the concept that is asked for in the prompt. Our results suggest that students are more likely to ask follow-ups and thus dive deeper into areas they have already explored previously while using LLMs. This might lead learners down a rabbit hole, preventing them from gaining a more complete understanding of the concept, thus explaining participants’ poorer performance in AI condition for the harder task. Future research can explore how LLMs can understand student intent and provide more holistic responses rather than generating direct responses to prompts for exploratory learning. One approach would be to draw inspiration directly from web searches that afford learning from multiple perspectives by utilizing multiple LLM responses. In our study, while using web search, participants often visited more than one website for one search query (211 websites + videos, from 148 search queries).”

In other words, web searches provide multiple perspectives and deeper context than LLM responses, which aids in the learning of information. One fear among SEOs is the willingness of users to click through to view web results from AI search. These findings suggest that might become a best practice, at least for information seeking behavior.

Text is NOT enough!

“Modern LLMs generate high-quality textual responses to user prompts. They can provide examples as well as high-quality explanations of these examples. However, participants in the NoAI treatment also expressed interest in non-textual learning resources, notably videos. Some participants chose to watch videos as they are more dynamic than text-based responses, and allow them to better understand the process behind using the concept. Others cited them to be more human. These preferences are consistent with prior work by Jackman et al. on the benefits of YouTube as a tool to demystify complex concepts. Thus, when choosing generative AI tools as learning resources in educational settings, it is imperative to choose ones that allow for heterogenous content types for different learning preferences and styles. For example, can augment its generated textual content with links to video resources. However, there are opportunities for research and development in multimodal language models that can help in creating more effective learning content (e.g., videos, interactive visualizations) and human-like experiences for diverse subjects and learners.”

As SEOs, we can take away from this the value of having multiple content formats, including images and videos.

Our takeaways as SEOs

While this study doesn’t speak to SEO concerns directly, it does highlight how web search might continue to be an essential component of information seeking, despite the availability of AI chatbots.

Of course, this research was done on ChatGPT 3.5, rather than ChatGPT Search, so we can assume the outcomes would be different in that case.

Still, the fact that web search provided a more diverse range of viewpoints that enabled better learning of, in this case, programming gives me hope that, despite the convenience of LLMs, visiting original sources on the web will continue to be a best practice for those looking to truly learn a subject matter.

Until next time, enjoy the vibes:

Thanks for reading. Happy optimizing! 🤗

Related posts