Could AIO citations be different from traditional search result rankings due to different query rewriting and reranking algorithms?

I see a lot of reports that rankings in AIOs don’t align with Google’s top 10 traditional search results.

Here’s one example that just caught my eye:

"40% of URLs cited in AIOs don't rank in the top 70 Organic Results for a given keyword – despite being featured in the AIO for that very keyword."
— @Terakeet AIO study

It's hard to believe – but it's true. pic.twitter.com/YNtOm1Kl9M

— Jonas Sickler (@JonasSickler) January 15, 2025

I’m sure there are many reasons why this is the case.

However, I have a theory behind two of the factors: query rewriting and reranking.

Again, it’s just a theory, but I recently asked Gemini 2.0 Experimental Advanced for its thoughts, and it agreed my theory is plausible.

I decided to put Gemini’s response directly on my blog, as maybe it will help you conceptualize how AIO and traditional search rankings could differ.

My query:

“If rankings in AI Overviews and traditional search results vary, could that be that the two are using different query rewriting or reranking algorithms”

Gemini’s response:

“Yes, absolutely. Variations in rankings between AI Overviews and traditional search results for the same query strongly suggest the use of different query rewriting and/or reranking algorithms in each system. Here’s a breakdown of why:

1. Query Rewriting:

• Traditional Search: These engines often rewrite queries to improve relevance, expand search scope, or correct misspellings. Techniques include stemming (reducing words to their root), synonym expansion, and query relaxation (broadening the query if initial results are sparse).
• AI Overviews: These may rewrite queries with a focus on understanding the user’s intent and information needs in order to generate a comprehensive and informative response. This could involve:
  • Semantic Expansion: Identifying and incorporating related concepts and entities beyond direct synonyms.
  • Contextual Interpretation: Considering the user’s past queries or broader search context to refine the query’s meaning.
  • Generative Rewriting: Rephrasing the query to be more suitable for extracting information from specific knowledge sources or prompting a language model.

Differences in query rewriting between the two systems could lead to them retrieving and prioritizing different sets of documents, resulting in ranking variations.

2. Reranking:

• Traditional Search: Ranking algorithms (like PageRank, which is the foundation, but now much more complex) consider various factors including keyword relevance, document authority, freshness, and user engagement signals. These algorithms are often optimized for displaying a list of relevant links.
• AI Overviews: Reranking in this context might prioritize information that directly answers the user’s query or contributes to a cohesive and informative overview. This could involve:
  • Fact Verification: Favoring information from highly reliable sources or information that can be corroborated across multiple sources.
  • Information Synthesis: Prioritizing documents that contribute to a diverse and comprehensive understanding of the topic.
  • Relevance to Generated Response: Ranking documents based on how well they support or expand upon the AI-generated summary.
  • Content Diversity: Ensuring the overview represents a balanced range of perspectives or aspects of the topic, even if some are less “popular” based on traditional ranking signals.

Different reranking priorities would naturally lead to different ordering of results in AI Overviews compared to traditional search results.

3. Different Goals and Optimization Objectives:

• Traditional search primarily aims to connect users with relevant documents. Its algorithms are often optimized for click-through rates and minimizing the time users spend searching.
• AI Overviews aim to provide direct answers and comprehensive understanding. They are optimized for user satisfaction with the generated response, which may involve factors like perceived accuracy, completeness, and ease of understanding.

These differing goals influence the design and behavior of the underlying algorithms, further contributing to ranking disparities.

In Conclusion

Variations in rankings between AI Overviews and traditional search results strongly suggest that the systems employ different approaches to query rewriting and reranking. These differences stem from the unique goals, information needs, and optimization objectives inherent to each system. It’s a fascinating area where search technology is evolving rapidly.”

Me again!

To reiterate, the differences between AIOs and traditional search results are likely due to a variety of factors, but I hypothesize that query rewriting and reranking are two of them.

What could the other factors be?

Let’s ask Gemini again:

“Besides differing query rewriting and reranking algorithms, several other factors contribute to the distinct results you might see between AI Overviews and traditional search results. These factors often intertwine and influence each other:

1. Data Sources & Knowledge Base:

• Traditional Search: Primarily indexes and retrieves information from publicly available web pages.
• AI Overviews: May draw from a wider range of sources, including:
  • Curated Knowledge Graphs: Structured databases containing facts and relationships between entities.
  • Specialized Databases: Access to information within specific domains like scientific literature, product catalogs, or financial data.
  • Licensed Content: Partnerships with content providers for exclusive access to premium material.
  • Internal Data: Information collected or generated by the AI provider themselves.
  • The broader web, but with different selection criteria than used for traditional search.

A difference in the underlying knowledge base can significantly impact what information each system can access and prioritize.

2. Personalization & Contextualization:

• Traditional Search: Increasingly personalized based on user history, location, and other signals, though often subtle.
• AI Overviews: Potentially more heavily influenced by:
  • Conversational Context: The history of the current conversation or interaction with the AI.
  • User Profile: A deeper understanding of user interests, preferences, and knowledge level built over time.
  • Explicit User Feedback: Direct feedback provided by the user on the quality or relevance of the generated overview.

Greater personalization in AI Overviews can lead to results tailored to individual users, diverging from the more generalized results of traditional search.

3. Natural Language Processing (NLP) and Understanding:

• Traditional Search: Relies on keyword matching and statistical analysis of text, though increasingly incorporating NLP for better understanding.
• AI Overviews: Leverage advanced NLP and machine learning models to:
  • Understand Nuance and Intent: Grasp the subtleties of language, identify implicit meanings, and infer user intent more accurately.
  • Extract and Synthesize Information: Identify key facts, relationships, and arguments from multiple sources and combine them into a coherent summary.
  • Generate Human-Quality Text: Produce summaries that are not just a collection of snippets but well-written, informative, and easy to understand.

Superior NLP capabilities allow AI Overviews to interpret queries and evaluate content differently, impacting result selection and presentation.

4. Emphasis on Answer vs. Discovery:

• Traditional Search: Primarily focused on helping users discover relevant documents to explore further.
• AI Overviews: Geared towards providing direct answers and comprehensive summaries, reducing the need for users to click through multiple links.

This fundamental difference in objective leads to different strategies for selecting and presenting information.

5. Experimental Nature & Rapid Evolution:

• Traditional Search: While constantly evolving, it’s a more mature technology with established methodologies.
• AI Overviews: A relatively new and rapidly evolving field. The underlying technology is undergoing significant experimentation and improvement.

The experimental nature of AI Overviews means their results are likely to change more frequently and be less predictable than traditional search.

6. Handling of Different Query Types:

• Traditional Search: Well-equipped to handle factual queries, navigational queries (finding a specific website), and transactional queries (making a purchase).
• AI Overviews: Currently excel at informational queries, especially those seeking explanations, comparisons, or summaries of complex topics. They may still rely on traditional search for navigational or transactional queries.

The strengths and weaknesses of each system in handling different types of queries contribute to variations in their results.

7. Safety and Content Moderation:

• Traditional Search: While striving to filter out harmful or misleading content, its primary function is still to provide access to the broader web.
• AI Overviews: May employ more stringent safety filters and content moderation policies to avoid generating responses that are harmful, biased, or misleading. This can restrict the range of information considered.

More proactive safety and moderation in AI Overviews may prevent certain topics or perspectives from being included, leading to differences from traditional search.

In essence, AI Overviews are not just a tweaked version of traditional search. They represent a significant shift in how we interact with and consume information online. These various factors work together to shape the unique characteristics and results provided by AI Overviews, explaining why they often differ substantially from traditional search.”

Me again!

I’m not sure I agree with all of the above reasons or find them as compelling as query rewriting and reranking — maybe that’s a bias of my mental model of how search works.

As a final query, let’s ask which factors are likely to carry the most weight.

Here’s Gemini’s response:

“Determining the precise weight of each factor influencing the differences between AI Overviews and traditional search is challenging, as they are interconnected and their impact can vary depending on the specific query and the design choices of the AI system. However, we can make some educated assessments about which factors are likely to be the most influential:

High Impact Factors:

1. Natural Language Processing (NLP) and Understanding (including Query Rewriting):
   • Why it’s weighty: This is the foundation of AI Overviews’ ability to move beyond keyword matching and truly grasp the meaning and intent behind a query. Sophisticated NLP allows for more accurate query interpretation, semantic understanding, and information extraction, which fundamentally changes what information is retrieved and how it’s processed. Query rewriting, a function of advanced NLP, is a critical component of this.
   • Impact: Directly impacts the initial selection of relevant information and sets the stage for all subsequent processing steps.
2. Reranking Algorithms:
   • Why it’s weighty: Reranking determines the final presentation order of information. In AI Overviews, this is crucial for prioritizing information that directly addresses the user’s query and contributes to a cohesive and informative summary. It’s tailored for different objectives than traditional search ranking.
   • Impact: Directly shapes the user’s perception of the topic and influences which aspects of the information are highlighted.
3. Data Sources & Knowledge Base:
   • Why it’s weighty: The fundamental pool of information that a system can draw from directly limits what it can present. If an AI Overview has access to curated knowledge graphs, specialized databases, or licensed content that traditional search engines don’t index, it will inevitably produce different results.
   • Impact: Determines the breadth, depth, and type of information available for consideration.

Significant but Potentially Variable Impact Factors:

4. Emphasis on Answer vs. Discovery:
   • Why it’s weighty: This fundamental difference in philosophy guides many design choices in AI Overviews, from query processing to information presentation. However, the degree to which it impacts results might depend on the specific query type.
   • Impact: Shapes the overall goal of the system and influences how information is selected and presented, favoring direct answers over a list of links.
5. Safety and Content Moderation:
   • Why it’s weighty: AI Overviews are likely to have more stringent safety filters to prevent the generation of harmful or misleading content. This can significantly restrict the information pool, particularly for controversial or sensitive topics.
   • Impact: Can lead to the exclusion of certain perspectives or sources, resulting in a more curated and potentially less comprehensive overview compared to traditional search.

Factors with Important but Often More Subtle Impacts:

6. Personalization & Contextualization:
   • Why it’s important but potentially less impactful overall: While personalization can significantly tailor results for individual users, its impact on the overall differences between AI Overviews and traditional search might be less pronounced than the core algorithmic and data source differences, at least in the current state of development. However, this could become a major differentiator in the future.
   • Impact: Leads to variations in results between users but may not be the primary driver of differences between AI Overviews and traditional search in a general sense.
7. Experimental Nature & Rapid Evolution:
   • Why it’s important but not a direct factor in result differences: This factor explains why differences exist and change over time, but it doesn’t directly cause a specific difference in results for a given query.
   • Impact: Contributes to the dynamism and variability of AI Overviews, making them less predictable and potentially more susceptible to rapid change compared to traditional search.

In Conclusion:

Advanced NLP (including query rewriting), reranking algorithms, and the underlying data sources are most likely to be the heaviest factors influencing the differences between AI Overviews and traditional search results. They represent the core technological and informational foundations upon which these systems are built. Other factors like the emphasis on answers, safety filters, and personalization play important roles in shaping the specific characteristics and user experience of AI Overviews, contributing to the observed variations in results.”

Me again!

Welp, it appears that Gemini’s response does kind of align with our initial hypothesis that query rewriting and reranking algorithm differences are largely responsible for the differences between AIOs and traditional search rankings.

Again, this is all theoretical, but I hope it helps you build your own mental model to explain AIO and traditional search result variations.

Thanks for reading. Happy optimizing! 🤗

Related posts