Exploring “Scalable In-context Ranking with Generative Models,” a Google Research Paper, & Why SEO/GEO Professionals Should Care (A Hamsterdam Research Post)

By Ethan Lazuk

Welcome to a new edition of Hamsterdam Research! 🐹

If you’re new here, this is where we look at recent AI research papers to learn just what the heck they’re talking about and explore their hypothetical implications for the future of search and SEO/GEO strategies.

This time, we’ll look at a research paper from Google called, “Scalable In-context Ranking with Generative Models.”

It was published on October 8, 2025, and its contributors are Nilesh Gupta, Chong You, Srinadh Bhojanapalli, Sanjiv Kumar, Inderjit Dhillon, and Felix Yu.

But before we dive in, why should SEO/GEO professionals care about this research paper?

In short, the paper shows how LLMs can evaluate and rerank many candidate documents after an initial retrieval stage, reinforcing that content must first be retrievable and then clearly relevant to the specific query. The paper also suggests that self-contained, well-structured passages may be easier for AI systems to evaluate efficiently, giving a hint toward actionable takeaways for SEO/GEO strategies.

To start, we’ll review the paper’s abstract.

In case you’re on a mobile device and can’t read that, here’s the abstract in HTML (with my highlights):

In-context Ranking (ICR) is an emerging paradigm for Information Retrieval (IR), which leverages contextual understanding of LLMs by directly incorporating the task description, candidate documents, and the query into the model’s input prompt and tasking the LLM to identify relevant document(s). While it is effective, efficiency is a significant challenge in this paradigm, especially as the candidate list grows due to quadratic / super-linear scaling of attention operation with context length. To this end, this paper first identifies inherent and exploitable structures in the attention of LLMs finetuned for ICR: (1) inter-document block sparsity – attention is dense within each document block but sparse across different documents in the context; and (2) query-document block relevance – the attention scores from certain query tokens to a document block in middle layers strongly correlate with that document’s actual relevance. Motivated by these observations, we introduce BlockRank (Blockwise In-context Ranking), a novel method that adapts the attention operation in an LLM by (a) architecturally enforcing the observed inter-document block sparsity, reducing attention complexity from quadratic to linear without loss in performance, and (b) optimizing query-document block relevance for true relevant documents during fine-tuning using an auxiliary contrastive training objective, improving retrieval in attention. Experiments on BEIR, MSMarco and NQ with Mistral-7B demonstrate that BlockRank Mistral matches or outperforms existing SOTA listwise rankers and controlled fine-tuned baseline while being significantly more efficient at inference ($4.7\times$ for $100$ MSMarco documents in context) and scaling gracefully to long-context shortlists – around $500$ documents in-context ($\sim 100K$ context length) within a second, presenting a scalable and effective solution for ICR. We will make our code available here.

Next, let’s break down the key vocabulary terms from the research paper.

• Information retrieval (IR) is the process of finding relevant documents for a query. It is the foundation of both traditional search and AI retrieval systems.
• In-context ranking (ICR) involves giving an LLM a query and multiple candidate documents, then asking it to identify or rank the most relevant ones.
• First-stage retriever is the initial system that searches the larger document collection and produces a shortlist for more detailed evaluation.
• Candidate documents are the shortlisted pages or passages that may be relevant to the query and are evaluated during reranking.
• Reranking is the reordering of retrieved documents using a more sophisticated model that can make deeper relevance judgments.
• Query-document relevance is the degree to which a particular document satisfies the user’s query or information need.
• Attention mechanism is the process an LLM uses to determine which tokens, passages, or documents deserve the most focus.
• Inter-document block sparsity is the paper’s finding that the model pays relatively little attention between separate documents and focuses mainly within each document.
• Structured sparse attention is BlockRank’s method of limiting attention to the connections most useful for ranking, reducing computational costs.
• BlockRank is the paper’s proposed ranking model, designed to evaluate large candidate lists efficiently while maintaining or improving retrieval performance.

That covers the abstract and its basic concepts.

Cool. Let’s now delve into the research paper for some deep learning.

If you wish to follow along, you can grab a PDF or the HTML version on arXiv.

The full paper has six sections, including: 1) Introduction, 2) Problem Setup and Related Work, 3) Emergence of Structured Attention in In-context Ranking, 4) BlockRank: Blockwise In-context Ranking, 5) Experimental Results, and 6) Conclusion.

We’ll summarize each of them below.

1. Introduction

The first paragraph of the paper has a typo, writing “neutral” instead of “neural.” I guess we’re all human!

But it’s worth exploring because of the historical details that help us understand information retrieval today:

Information retrieval (IR) is the problem of finding relevant content from a large document corpora. While sparse retrieval methods based on word-level matching have existed for decades (Robertson et al., 2009; Formal et al., 2021), modern IR systems increasingly leverage deep neutral network based representations, which achieve their success through a superior ability to capture deep semantic relationships (Karpukhin et al., 2020). Recently, generative large language models (LLMs) (Team et al., 2023; Achiam et al., 2023) have emerged as a revolutionary paradigm that transforms many sub-fields of machine learning, including IR. Through pre-training on the web, LLMs absorb an enormous amount of world knowledge and demonstrate remarkable capabilities in dialogue, question answering, reasoning, and beyond (Wei et al., 2022).

The highlighted section above, in particular, matters for SEO/GEO because it explains the shift of IR systems from matching exact words to understanding meaning. While older sparse retrieval systems mainly looked for overlap between the query and the words on a page, neural retrieval systems can recognize that two passages are relevant even when they use different language, because they capture semantic relationships.

The practical lesson for SEO/GEO strategies is that content should not rely only on repeating target keywords (like inserting fanout queries into text). Rather, content should clearly explain the topic, related entities, relationships, user intent, and supporting context so retrieval systems can understand what the page is about. Keywords still matter, especially in hybrid retrieval systems, but semantic completeness is increasingly important.

Next, the paper gets into a core topic of in-context ranking:

The powerful capabilities of LLM open up novel approaches for IR as well. One emerging paradigm is the In-context Ranking (ICR) (Lee et al., 2024; Ma et al., 2023), which directly leverages an LLM’s contextual understanding capabilities. In this setup, a query and a list of candidate documents are formatted together within the LLM’s input prompt (see Figure 3), tasking the model to identify the most relevant document(s), often through the generative decoding process. ICR holds the promise of considering the query and all candidates simultaneously while performing relevance judgements.

As SEO/GEO professionals, we should care about ICR because it represents a possible selection stage after retrieval. In other words, an AI system may place several candidate documents into context and compare them directly to decide which are most relevant to the query. That means SEO/GEO is not only about getting content retrieved but it’s also about making the content satisfy the specific information need better than similar competing documents.

In terms of practical implications, the content we create should have:

• Direct answers to specific questions
• Clear entities and relationships
• Distinctive facts, evidence, or expertise
• Self-contained passages that remain understandable when extracted
• Strong alignment with the user’s actual intent, not just broad topical relevance

One note: while the paper explains an emerging retrieval and ranking method, it doesn’t prove that ChatGPT, Google AI Overviews, or other AI assistants currently use this exact ICR setup.

However, the authors of the research paper note a limitation of LLM-based ICR:

Despite this promise, LLM-based ICR introduces significant efficiency challenges. As the number of candidate documents increases, the input context length grows rapidly, making inference computationally expensive and memory-intensive, due to quadratic/super-linear complexity of the attention mechanism.

In their research, the authors consequently propose BlockRank:

Building up on these insights, we propose BlockRank (Blockwise In-context Ranking), an efficient in-context ranking method. BlockRank introduces two modifications (visualized in Figure 2) to standard LLM architecture and fine-tuning: (1) architecturally, it imposes a structured sparse attention in which document tokens attend only causally to their own content and shared instruction tokens, reducing attention complexity from quadratic to linear; and (2) it incorporates a contrastive learning objective to explicitly optimize internal attention from signal-carrying query tokens toward relevant documents, which helps the BlockRank model in two fronts: (a) attend strongly to the relevant document in context, improving retrieval quality (see Table 3); (b) the ability to reliably infer relevance based on attention concentration during the prefill stage, leading to further speedups in inference compared to iterative decoding (see Section 4.3).

If the above sounds a little technical, here’s a clearer explanation: BlockRank makes LLM-based document ranking faster and more accurate in two ways:

First, BlockRank limits how much each document interacts with the others, so the model can process large groups of documents more efficiently.

Second, BlockRank trains the model to focus more on the documents that best match the query, allowing the model to identify relevance from its attention patterns without generating a full answer step by step.

For SEO/GEO professionals, the main idea is that BlockRank helps an LLM compare more candidate documents, identify the most relevant ones, and do it with less computing time.

Let’s look at Figure 2 from the paper and discuss what it’s showing us:

The figure above shows how BlockRank turns a long ranking prompt into a more efficient document-selection process.

From left to right:

1. The model receives one long prompt containing instructions, several candidate documents, a query, and a request to identify the best document.
2. The prompt is split into separate blocks, including the instructions, each individual document, and the query.
3. Ordinarily, a standard fine-tuned LLM would process everything with broad attention and generate the answer token by token.
4. But BlockRank changes this process so each document mostly focuses on its own content and the shared instructions, rather than interacting heavily with every other document.
5. BlockRank also trains certain query tokens to focus more on the relevant document. Those attention patterns become relevance scores, which the model can use to rank the documents directly.

For SEO/GEO professionals the main takeaway is that BlockRank helps an LLM compare a large set of candidate documents, identify which ones best match the query, and do so faster than a standard LLM-ranking approach.

That concludes the introduction. Now let’s delve into some of the details of the paper.

2. Problem Setup and Related Work

This section is a little technical.

In short, it “formally defines the ICR task addressed in this paper.” The authors “also review relevant prior work that uses LLM for IR, and position our BlockRank method in the context of the literature.”

For our sake, I’ll ask ChatGPT to summarize the key takeaways relevant to SEO/GEO professionals, which I’ll reword:

AI ranking can be a two-stage process.

A fast first-stage retriever selects a smaller group of candidate documents, then an LLM compares those candidates more carefully. The SEO/GEO implication is that content must first be retrievable before it can be selected, cited, or recommended.

Your content competes against a shortlist, not necessarily the entire web at once.

In the paper, the LLM reranks around 100 documents that have already passed the initial retrieval stage. The practical goal is therefore both candidate-set inclusion and strong performance during reranking.

Broad topical relevance alone may not be enough. Content also needs to align closely with the specific query and information need.

The researchers found that placing the query before the documents improved performance because the model could evaluate each document according to the exact information need. In other words, SEO/GEO content should answer specific questions and intents, rather than merely covering a general topic.

AI systems may compare competing documents together.

Unlike models that score each page independently, ICR allows an LLM to consider the full candidate list in one context. A page may be judged relative to similar alternatives, which makes distinctive facts, original evidence, clear explanations, and direct answers especially valuable.

The hardest competition comes from semantically similar content.

The paper focuses on “hard candidates,” which are documents that appear relevant and may discuss nearly the same subject. When it comes to content, it’s not just about being relevant; it’s about making it easy for the model to determine why your content is the best match.

LLM ranking can consider more than keyword similarity.

Because the LLM processes the query and candidates contextually, it can potentially follow complex instructions, such as finding documents that support, contradict, compare, or explain something. This reinforces the importance of clearly expressing claims, relationships, viewpoints, and evidence.

Efficiency affects how many sources an AI system can evaluate.

Standard LLM attention becomes expensive as more documents are added. BlockRank is intended to make larger candidate-set comparisons practical, which could eventually allow AI systems to evaluate more sources before selecting an answer.

Document structure matters inside the ranking system.

The experiment separates the instructions, individual documents, and query into identifiable blocks. This does not prove that a particular HTML structure improves SEO/GEO, but it supports the broader value of content that can be cleanly segmented and understood as self-contained passages.

3. Emergence of Structured Attention in In-context Ranking

This section examines how a standard fine-tuned LLM internally processes a query and a list of candidate documents during ICR. The researchers use Mistral-7B, fine-tuned on MSMarco, and analyze how attention moves among the instructions, query, and individual documents.

They identify two main patterns:

1. Documents are processed mostly independently

Document tokens concentrate most of their attention on other tokens within the same document, rather than interacting heavily with every other candidate. The authors call this inter-document block sparsity.

Said plainly, the LLM appears to understand each document largely as its own block. Because extensive cross-document attention is often unnecessary, the researchers argue that some of this computation can be removed without harming ranking performance.

2. Certain query tokens point toward the relevant document

Some tokens near the end of the query prompt, including punctuation and formatting tokens, direct much more attention toward the correct document than toward irrelevant candidates. The researchers describe these as signal-carrier tokens because their attention patterns contain a document-relevance signal.

This signal is relatively weak in the model’s earliest layers, becomes much stronger in the middle layers, and then remains or becomes slightly less concentrated in the final layers.

In terms of SEO/GEO takeaways from this section, we learn that an LLM may evaluate a retrieved passage primarily on its own contents while using the query to determine which candidate best satisfies the information need. That supports the value of clear, self-contained passages. (Just note that the study doesn’t directly test webpage optimization or prove that commercial AI assistants use these exact attention patterns.)

4. BlockRank: Blockwise In-context Ranking

This section is the heart of the authors’ contribution, but again, it’s a little technical. So I’ll have ChatGPT summarize it, which I’ll reword:

In short, this section explains how BlockRank makes in-context ranking faster by splitting the prompt into separate blocks for the instructions, each document, and the query. Each document primarily attends to its own content and the shared instructions, while the query can compare all documents. This reduces unnecessary computation and helps the AI model handle larger candidate lists more efficiently.

BlockRank also trains certain query tokens to focus more strongly on the relevant document, turning those attention patterns into relevance scores. Instead of generating a ranking token by token, the AI model can use those scores directly to identify the best documents, which further speeds up inference.

5. Experimental Results

The authors conducted two sets of experiments: “first, an evaluation on the BEIR benchmark to assess zero-shot generalization against state-of-the-art re-rankers, and second, a controlled in-domain evaluation to analyze effectiveness, efficiency, and scalability.”

In case you’re interested, here are the first results:

The table compares BlockRank against other reranking systems across 11 BEIR retrieval datasets. Every model reranks the same top 100 documents initially retrieved by Contriever, and nDCG@10 measures how well each method places relevant documents near the top of the first 10 results.

BlockRank has the highest average score. It does not win every individual dataset, but it performs best overall and leads on several datasets. The main takeaway is that BlockRank can compete with or outperform strong rerankers across varied retrieval tasks, not just the dataset on which it was trained.

And here are the second results:

This table compares BlockRank with traditional retrieval models, encoder-based systems, zero-shot LLMs, and a standard fine-tuned Mistral model on Natural Questions (NQ) and MSMarco. Not only does BlockRank achieve the best overall results, but the table shows that fine-tuning is important, since the zero-shot LLMs perform substantially worse.

6. Conclusion

The authors’ conclusion summarizes their takeaways:

This work addresses the efficiency challenge in In-Context Retrieval (ICR) by analyzing LLM attention, identifying structured sparsity and query-token retrieval signals. We introduced BlockRank, a method that enforces this task-specific sparsity for linear complexity and uses a contrastive auxiliary loss to directly optimize these internal attention signals for relevance. Experiments on MSMarco and NQ show BlockRank (Mistral-7B) matches or surpasses standard fine-tuning effectiveness while being significantly more efficient at inference and training. This offers a scalable and effective approach for LLM-based ICR. However, we acknowledge our current findings are primarily demonstrated on a specific model architecture, and the robustness of the learned attention signals for direct inference across highly diverse tasks needs more investigation.

But why should SEO/GEO professionals care about “Scalable In-context Ranking with Generative Models“?

We can extract at least five main takeaways for SEO/GEO strategies from the research paper.

Retrieval and ranking are separate challenges.

A page first has to enter the candidate set, then outperform other relevant documents during reranking. SEO/GEO strategies should therefore focus on both retrievability and strong query alignment.

AI systems may compare many competing documents at once.

ICR lets an LLM evaluate a shortlist together, so content may be judged relative to other similar pages, not in isolation.

Specific relevance matters more than broad topical coverage.

The paper shows that query-conditioned processing helps the model focus on the facts most relevant to the exact information need. Content should answer specific questions directly rather than just mention the topic.

Clear, self-contained passages may be easier to evaluate.

The model’s attention was strongest within individual document blocks, suggesting that important sections should make sense without requiring extensive context from elsewhere on the page.

More efficient ranking could increase competition.

BlockRank allows an LLM to evaluate larger candidate lists with lower computational cost. As these methods improve, AI assistants may compare more sources before deciding what to cite, recommend, or use in an answer.

Outro

I hope you’ve enjoyed this edition of Hamsterdam Research! 🐹

Feel free to comment below or contact me with your feedback.

Stay tuned for another new article, hopefully next week, or check out related research posts below.

Until next time, enjoy the vibes:

Thanks for reading. Happy optimizing!