What are zero-shot pointwise LLM rankers and intermediate relevance labels, and why should SEOs care?

Welcome to a new Hamsterdam Research post. 🐹

This time, we’ll take a look at a paper from Google Research titled, “Beyond Yes and No: Improving Zero-Shot Pointwise LLM Rankers via Scoring Fine-Grained Relevance Labels.”

The paper’s authors include Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky.

We’ll start with the paper’s abstract and then dig into more of its details.

But first, why should SEOs and marketers care about this research paper?

This paper explores how to make LLMs better at understanding the concept of relevance. We’ll get into the details, but the authors incorporate fine-grained relevance labels into LLM prompts to enable them to differentiate degrees of relevance more accurately.

In short, this research has implications for improved search results by improving the assessment of relevance in LLMs with better understanding of search intent, as fine-grained labels enable LLMs to capture subtle differences in the relevance of documents.

If the standing advice for SEOs is to create helpful content, this paper helps elucidate how search engines might assess that helpfulness in terms of document relevance.

Let’s delve into the abstract.

“Zero-shot text rankers powered by recent LLMs achieve remarkable ranking performance by simply prompting. Existing prompts for pointwise LLM rankers mostly ask the model to choose from binary relevance labels like “Yes” and “No”. However, the lack of intermediate relevance label options may cause the LLM to provide noisy or biased answers for documents that are partially relevant to the query. We propose to incorporate fine-grained relevance labels into the prompt for LLM rankers, enabling them to better differentiate among documents with different levels of relevance to the query and thus derive a more accurate ranking. We study two variants of the prompt template, coupled with different numbers of relevance levels. Our experiments on 8 BEIR data sets show that adding fine-grained relevance labels significantly improves the performance of LLM rankers.”

As we see, existing prompts for pointwise LLM rankers ask the model to choose binary Yes or No labels, but the researchers are focused on intermediate relevance labels “for documents that are partially relevant to the query.”

A pointwise LLM ranker is an LLM that scores one query and one document at a time and ranks documents based on those scores.

The two main categories of pointwise LLM rankers include relevance generation — the LLM is prompted to answer whether a document is relevant to the query — and query generation — the LLM is prompted to generate a query based on the document, then the documents are ranked based on the likelihood of the LLM generating the query.

Pointwise LLM rankers are typically used for zero-shot ranking, or tasks where the LLM has not been specifically trained on the dataset or task before.

Let’s talk about intermediate labels vs. Yes/No labels.

The authors propose adding intermediate relevance labels as options in the prompts, in addition to Yes and No labels. They argue this allows the LLM to make more nuanced distinctions between documents.

What I find interesting here is Google’s AI Overviews, where sources can be included in answers that weren’t necessarily present in the normal search results. Just a thought.

The authors’ reasoning is that documents can have varying degrees of relevance to a query, including documents that may not directly answer the query but still contain helpful information. Including these can perhaps result in more relevant search results.

In terms of their findings, the authors noted that using prompts with intermediate relevance labels improved the performance of LLM rankers across different datasets, consistently. This suggests that using such labels allows the LLMs to better distinguish between documents with varying degrees of relevance.

Again, why should SEOs and marketers care?

If our goal is to produce helpful content for an audience, such advancements with regard to relevance on Google’s end can present new opportunities for content creation. If we’re less dependent on being directly relevant to a query and more on being tangentially helpful to it, we can focus less on targeting keywords and more on creating for users.

Thanks for reading. Happy marketing! 🤗

Related posts