Exploring “Does the Citation Matter? Quality, Grounding, and Physician-Rubric Accuracy of Google’s AI Overviews for Medical Information” for SEO/GEO Insights: A Hamsterdam Research Post

By Ethan Lazuk

Welcome to another edition of Hamsterdam Research! 🐹

This is where we look at recent AI research papers to learn what they’re talking about and explore their hypothetical implications for the future of search and SEO/GEO strategies.

This time, we’ll look at research called, “Does the Citation Matter? Quality, Grounding, and Physician-Rubric Accuracy of Google’s AI Overviews for Medical Information.”

It was submitted to arXiv on October 29, 2026, and its authors are Sam Agnoli, Brigid Reilly, Lily Barna, and Emőke-Ágnes Horvát.

First off, why should SEO/GEO professionals care about this research?

This research seeks to answer whether being cited by an AI search engine actually means that your content is authoritative, accurately represented, or even used to generate the answer.

In an analysis of 3,611 healthcare questions, researchers from Northwestern University found that:

AI Overviews are highly prominent for healthcare queries.

AI Overviews appeared for 96.4% of the researcher’s questions.

Traditional rankings and AI citations diverge.

Only 44.4% of Google’s top five traditional search results were also cited in the corresponding AI Overview. Ranking highly doesn’t necessarily guarantee AI visibility.

Citation selection doesn’t always favor the most credible sources.

AI Overviews disproportionately cited commercial health publishers over academic and government sources, and cited pages scored worse on four of six credibility measures, although the differences were relatively small.

A citation doesn’t guarantee that a claim is supported.

After adjusting for classifier error, researchers estimated that only 56.7% of assessable cited statements were supported by their sources.

Being cited isn’t the same as contributing to an accurate answer.

The researchers found no meaningful relationship between citation grounding and the medical accuracy or completeness of AI-generated responses.

For SEO/GEO professionals, an implication of this research is that AI citation visibility, source authority, factual grounding, and answer quality may all be distinct. A brand could earn citations without its content being faithfully represented, while a credible source might be overlooked despite performing well in traditional search.

This research also raises questions about how we measure SEO/GEO success. If an AI system cites a webpage without accurately grounding its claims in that page, simply counting citations may provide a misleading picture of a brand’s actual influence on AI-generated answers.

The broader takeaway is that as SEO/GEO evolves, professionals may need to think beyond rankings and citation frequency toward how accurately their content is represented, whether their expertise is reflected in AI-generated answers, and whether those answers genuinely serve users.

Caveat: this research examined healthcare queries over three days in September 2026. Its findings are particularly relevant to high-stakes, YMYL content, but shouldn’t automatically be generalized to every industry or AI search platform.

Let’s start by reviewing the paper’s abstract.

“Google’s AI Overviews (AIOs) are the first result users encounter when searching the web, shifting the work of selecting and evaluating sources from the user to Gemini. This may be especially consequential for health information, given that Google’s traditional algorithm was designed to prioritize quality sources, and inaccurate information carries health risks. We evaluated 3,611 benchmark health questions, comparing each AIO against that question’s top five traditional results on source selection and quality. We also evaluated the AIOs’ statement grounding in their cited sources and how accurate AIOs were against physician-written rubrics. Overviews appeared for 96.4% of questions but drew on substantially different evidence than traditional search: only 44.4% of the top five results were cited by the corresponding AIO, which instead over-represented commercial health publishers (e.g., WebMD, Healthline) at the expense of academic and government sources (e.g., Cleveland Clinic, National Health Service). AIO-cited pages scored lower than traditional results on four of six well-established credibility criteria. The grounding of AIO statements to their sources was poor: after correcting for classifier error, an estimated 56.7% of cited statements were supported by their cited page, and a further 20.8% of all statements were factual claims with no citation at all. When AIOs were graded against physicians’ rubrics, they earned 38.2% of all available points (e.g., accuracy 54.2%, completeness 21.6%), and among the subset of AIOs whose physician rubric anticipated a specific inaccurate or potentially harmful claim, 22.2% of AIOs contained the harmful content. Notably, the accuracy of AIOs was unrelated to the grounding of their statements. We recommend AIO literacy interventions for users and holding AIOs at least to Google’s own health-ranking standards.”

Next, let’s break down the paper’s key vocabulary terms:

• Retrieval-Augmented Generation (RAG): A technique that combines large language models with externally retrieved information to generate responses, often with citations to supporting sources.
• Statement Grounding: The degree to which an AI-generated claim is supported by the source it cites. A statement can include a citation without actually being supported by that source.
• HONcode (Health on the Net Code): A framework for evaluating the trustworthiness of online health information. The study uses criteria related to source authority and whether content supports rather than replaces medical professionals.
• JAMA Criteria: A set of four standards for evaluating the credibility of medical information: authorship, attribution, currency, and disclosure.
• Physician-Written Rubrics: Evaluation criteria developed by medical professionals to assess the accuracy, completeness, and safety of AI-generated healthcare responses, including identifying potentially harmful claims.

Bravo. Let’s take a deeper look at the research paper’s contents now.

If you want to follow along, you can grab a PDF or the HTML version on arXiv.

The paper has six main sections.

We’ll focus on the most relevant ones below.

Introduction

Below I’ll paste in excerpts from the introduction that provide essential context for understanding the paper (I included most of the introduction because the information was pertinent, I felt, to SEO/GEO):

“Google’s AI Overviews (AIOs) are now the first result its 2.5 billion monthly users encounter when searching for information online (Pichai, 2026). In practice, AIOs employ the large language model (LLM) Gemini with retrieval-augmented generation (RAG) to provide tailored answers with citations to users’ questions. AIOs supersede the traditional search algorithm on an opt-out model (Bianchi, 2025), functionally shifting the work of selecting and evaluating sources from the user to Gemini. Rather than decomposing a complex question into multiple searches, users can now search for a single question with many details and caveats, and are provided with a synthesized answer from several sources (Google, 2024). When users receive an AIO, they are less likely to click links from the traditional search and rarely check the AIO against its sources (Chapekis and Lieb, 2025). This suggests that AIOs’ accuracy and fidelity to their citations are increasingly the primary safeguards of online information for users. However, AIOs were deployed with little understanding of how they might impact the information ecosystem.

Consider that Google’s traditional search algorithm has been carefully developed over the past 30 years to provide high-quality information in a variety of contexts (Google, 2019a; Google, 2019b). Google has particularly strict criteria for health information, ranking credible sources like academic and government domains more highly than commercial domains (Google, 2019b; Strzelecki, 2020). Yet emerging evidence suggests that AIOs may draw upon lower quality sources than traditional searches. One study of AIOs in non-healthcare domains showed that AIOs rely on more user-generated content (e.g., anonymous social media posts) than traditional search results (Hagar and Diakopoulos, 2026). These lower quality sources could be associated with harmful or misleading claims. For example, an audit of AIOs for baby care and pregnancy questions observed that basic medical safeguards were omitted in the vast majority of AIOs (Hu et al., 2026). A related concern is that AIOs may make claims that are not grounded in their sources, with one study of non-medical queries finding that 11% of claims were unsupported by their citations (Xu et al., 2026). It is unclear whether AIOs inherit Google’s previous quality standards in health domains, which were developed to protect users from unnecessarily complex and potentially harmful information (Kitchens et al., 2014; Google, 2019a; Google, 2019b).

Studying AIOs in the context of healthcare is particularly important given that health-related queries are frequently high-stakes, and inaccurate or misleading information can lead to worse outcomes, including increased mortality (Swire-Thompson and Lazer, 2020). Related work on health content from state-of-the-art LLMs found that nearly half of common health-related answers are misleading and/or incomplete (Tiller et al., 2026), which is especially concerning given that people trust LLMs as much as doctors (Shekar et al., 2025). A recent systematic review suggested that, in addition to increasing the volume of health misinformation, AI also increases its perceived credibility as compared to human-written content (Saeidnia et al., 2026). Additionally, AIOs (Sun et al., 2024) and LLMs (Xu et al., 2023) are generally preferred to traditional searches for the rapidity and clarity with which users receive an answer. Taken together, AIOs may represent an outsized threat to the healthcare informational ecosystem that has yet to be characterized.

The current study audited AIOs in the high-stakes domain of health to test whether their citations represent credible sources, grounded statements, and accurate answers. We queried 3,611 health questions to Google from HealthBench (Arora et al., 2025), OpenAI’s benchmark for realistic health conversations, each paired with physician-written grading rubrics, and evaluated each AIO in four complementary analyses. First, we examined the degree of overlap between AIOs’ sources and the top 5 traditional sources. Second, we compared the quality of sources across both groups using two widely used healthcare website evaluation instruments: Journal of the American Medical Association (JAMA) criteria and Health on the Net Code (HONcode). Third, we examined the degree to which statements within each AIO were grounded by their citations using HallOumi, an open-source claim verification classifier that has previously been used to evaluate AIOs in a similar context (Mickle et al., 2026). Finally, we graded each AIO on physician-written criteria using GPT-4.1 to assess the accuracy and completeness of AIO answers (Arora et al., 2025), and tested whether the quality and grounding of AIOs’ citations predicted their accuracy and completeness. …

Empirical work demonstrated that AIOs are generated more often for long queries as compared to short queries (Grossman et al., 2026), suggesting that they may increase as a function of a query’s complexity. …

Though AIOs cite more sources than a user may encounter with a traditional search, this diversity could let in lower quality sources. For example, AIOs frequently index anonymous, uncredentialed user-generated comments on internet forums as evidence for their summaries (Stox, 2025; Hagar and Diakopoulos, 2026). …

Contemporary LLMs do a decent job of correctly answering general health questions, but performance decreases as queries become more complex (Narula et al., 2023; Fernández-Pichel et al., 2025). Unfortunately, LLMs also tend to be overconfident in their claims (Chen et al., 2023; Mielke et al., 2022), and LLM citations signal credibility (Ding et al., 2025) that could misinform people about the quality of advice (Steyvers and Peters, 2026).”

Results

The results section includes some interesting findings regarding AIO behavior, for example:

“The analytic sample included 3,481 questions with retrieved AIOs. Of these, 3,474 had usable AIO text (at least 200 characters), and 3,103 also had at least one readable webpage in both the AIO and traditional groups, forming the sample for all paired source comparisons (see Table A1; for descriptives, see Table 2). AIOs averaged 283.2 words (SD = 121.7). For AIOs that cited at least one source (n = 3,154), they cited on average 8.50 sources (SD = 4.07). However, some AIOs cited no sources (n = 327). In contrast, we capped traditional searches to five sources and usually retrieved all 5 sources (n = 3,322; four or less sources n = 159).”

I’ll summarize the rest of the section below:

1. AI Overviews frequently appeared but sometimes provided no citations.

AI Overviews appeared for 96.4% of the 3,610 retrieved healthcare searches. AIOs that included citations referenced an average of 8.5 sources. Questions with high complexity were more likely to produce uncited AI Overviews (10.6%) than low-complexity questions (4.2%). High-risk questions were also more likely to receive answers without citations (17.6%) compared with low-risk questions (6.4%). Finally, longer questions were associated with fewer citations, suggesting that AI Overviews may become less transparent as queries grow more complicated.

2. AI Overviews cited different sources than traditional Google results.

Only 44.4% of Google’s top five traditional search results were also cited in the corresponding AI Overview. The remaining 55.6% appeared in traditional results but were not cited in the AI Overview. Traditional search favored academic medical institutions and government health organizations, while AI Overviews cited commercial health publishers more frequently, suggesting that traditional rankings and AI citation selection follow different patterns.

3. Traditional search results generally had higher source-quality scores.

Researchers evaluated sources using six JAMA and HONcode credibility measures. Traditional results outperformed AI Overview citations on four measures: authority, disclosure, currency, and attribution (authorship and complementarity showed no statistically significant differences).

4. Many AI-generated statements were not supported by their citations.

Researchers examined 24,060 statements across 3,474 AI Overviews. Only 50.6% of statements could be evaluated directly against their citations. Of those evaluable statements, 47.1% were classified as supported by the cited sources. After correcting for classifier errors, the estimated support rate increased to 56.7%. Another 20.8% of all statements were factual claims without citations. Citation support declined as question complexity increased, falling from 52.2% for low-complexity questions to 46.3% for high-complexity questions.

5. AI Overviews struggled with medical completeness and sometimes included harmful information.

AI Overviews earned just 38.2% of available points across physician-written evaluation criteria. They scored 54.2% for accuracy but only 21.6% for completeness. Most lost points reflected missing information (89%) rather than explicitly incorrect claims (11%).

6. Better citation grounding did not translate into more accurate answers.

Researchers found virtually no correlation between how well statements were supported by citations and how accurately or completely AI Overviews answered medical questions (ρ = 0.01). AI Overviews without citations scored similarly to those with citations on accuracy and completeness (40.4% versus 39.2%).

7. Results were largely consistent for shorter questions.

Researchers repeated their analyses using 1,714 shorter, single-turn questions to better reflect typical search behavior. The main conclusions remained largely unchanged.

Discussion

Below is a summarization of key discussion points from the paper:

AI Overviews and traditional search prioritize different sources:

Only 44.4% of the top five traditional search results appeared as citations in AI Overviews. AI Overviews favored commercial health publishers, while traditional search more frequently surfaced academic institutions and government health organizations.

AI Overviews may apply weaker source-quality standards:

AIO-cited sources scored lower on four of six credibility measures, particularly authority and disclosure. That said, the differences were relatively small, and previous research on non-medical queries produced different findings.

Citations frequently fail to support AI-generated claims:

Only 47.1% of evaluable cited statements were classified as supported by their sources, increasing to an estimated 56.7% after accounting for classifier errors. Support declined as question complexity increased.

Missing information was a bigger problem than outright inaccuracies:

AI Overviews scored only 21.6% on medical completeness. Their relatively short responses frequently omitted information physicians considered important. Additionally, 22.2% of AIOs contained specific inaccurate or potentially harmful information when their evaluation rubrics identified such errors to avoid.

Citation grounding did not predict answer accuracy:

Researchers found no meaningful relationship between how well citations supported statements and how accurately AI Overviews answered medical questions. Answers without citations performed similarly to those with citations.

AI-generated answers may not depend heavily on retrieved sources:

The researchers proposed several possible explanations, including that Gemini relies substantially on knowledge acquired during training, that retrieved sources contain outdated or inconsistent information, or that citations are selected after an answer is generated. (These remain hypotheses rather than confirmed explanations.)

Citations can create a misleading sense of credibility:

Previous research suggests that users trust AI-generated answers more when citations are present. The study raises concerns that this trust may be misplaced when citations don’t actually support the information presented.

Google should strengthen safeguards for medical AI Overviews:

The authors recommend applying traditional search’s healthcare quality standards to AI Overviews, prioritizing authoritative medical sources, and withholding AI-generated advice for questions that are too complex or risky.

Conclusion

Below is the full conclusion:

“Google’s AIOs show considerable promise for increasing the efficiency and accessibility of health information. However, our project raises healthcare concerns regarding the quality of AIOs’ sources, the fidelity of statements to those sources, and the overall accuracy of AIOs. In doing so, we hope that people seeking health advice will use AIOs with caution (if at all), and that Google and other web directories will expediently address these problems. Billions of people rely on search engines to make informed decisions in their lives. If AI is to be the default tool for these searches, it should supply accurate, grounded information. When that is not possible, AIOs should not mislead us.”

Knowing what we do now, why should SEO/GEO professionals care about “Does the Citation Matter? Quality, Grounding, and Physician-Rubric Accuracy of Google’s AI Overviews for Medical Information”?

AIO insights aside, this paper suggests several practical takeaways for SEO/GEO professionals:

1. Audit AI answers for missing information:

First identify important details, caveats, or exceptions that AI Overviews leave out then use those gaps to develop content that addresses users’ needs more thoroughly.

2. Test complex, multi-part prompts:

The study found that citation grounding declined as question complexity increased. Test prompts involving multiple conditions or nuanced scenarios to uncover weaknesses and content opportunities.

3. Develop expert-defined answer rubrics:

Create checklists of what accurate, complete answers should include for your industry’s important queries and use these rubrics to evaluate AI responses and guide content improvements.

4. Measure how accurately AI represents your content:

Go beyond counting citations and examine whether AI-generated answers preserve your content’s meaning, evidence, and qualifications. Track any misrepresentations or omissions alongside citation visibility.

Outro

The broader lesson from this research is that SEO/GEO success shouldn’t be measured by citations alone.

Understanding the quality and completeness of AI-generated answers can reveal opportunities that traditional rankings and visibility metrics miss.

I hope you’ve enjoyed this edition of Hamsterdam Research! 🐹

Feel free to comment below or contact me with your feedback.

Stay tuned for another new article, hopefully next week, or check out related research posts below.

Until next time, enjoy the vibes:

Thanks for reading. Happy optimizing!