Summarizing “Twenty Days Later: A Pre-Registered Test of History-Based Forecasts of Brand Mentions in Gemini Answers”: A Hamsterdam Research Post

By Ethan Lazuk

Welcome to another edition of Hamsterdam Research! 🐹

This is where we look at recent AI research papers to learn what they’re talking about and explore their hypothetical implications for the future of search and SEO/GEO strategies.

This time, we’ll look at research called, “Twenty Days Later: A Pre-Registered Test of History-Based Forecasts of Brand Mentions in Gemini Answers.”

It was submitted to arXiv on October 14, 2026, and its author is Dmitrij Żatuchin.

First off, why should SEO/GEO professionals care about this research?

This research asks whether brand mentions in AI-generated answers are stable enough to provide meaningful marketing data, a fundamental question for SEO/GEO professionals.

The first point raised is over how the unpredictability of AI responses creates a measurement challenge. Because AI-generated answers can vary between requests, SEO/GEO professionals need to distinguish normal variations in responses from meaningful changes in a brand’s visibility.

A principal finding from the study was that there was considerable stability over approximately 20 days in brand mentions. The researcher tested 12 software brands across 54 questions using Gemini 3.1 Pro Preview. Of the 202 brand-and-question combinations that consistently produced either a mention or no mention in the previous measurement wave, 194 remained unchanged roughly 20 days later, and none completely reversed.

These findings could help us as marketers to determine how often to measure AI visibility. The researcher found that three answers per question from one session predicted results approximately 20 days later nearly as accurately as nine answers collected across three sessions. This finding raises interesting questions about whether repeated daily measurements are necessary for certain SEO/GEO tracking purposes.

The finding also reinforces the importance of measuring visibility at the individual query level. Historical mentions for a specific brand-question combination predicted future mentions much better than the brand’s overall mention rate. For SEO/GEO professionals, this suggests that tracking the questions and consumer needs associated with a brand may be more informative than relying on a single aggregate visibility score.

Caveat: This study involved one model (Gemini), one follow-up interval, 12 established software brands, and API-generated answers that did not use web search. It also counted mentions regardless of whether they were positive or negative.

Let’s start by reviewing the paper’s abstract.

“Counts of which brands a large language model names are useful only if they hold long enough to act on. We test this with a pre-registered replication. The same 162 requests (54 questions in English, Estonian and Finnish about three software categories, four of twelve candidate brands per question, three repeats) were sent to gemini-3.1-pro-preview in three waves within about ten hours on 22 and 23 September 2026, and again on 13 October 2026, 20.3 days after the third wave, with identical request bodies. The plan and its decision rule were frozen, hashed and registered on OSF before the first call. The outcome is a lexical mention label per answer and brand, 648 per wave. The primary forecast, each brand-and-question cell’s mention rate over the nine earlier answers, scored Brier 0.0281 and log loss 0.126, against Brier 0.0259 for the six-answer forecast of the 8-hour wave and a registered ceiling of 0.0559. It beat brand history (0.104), the overall rate (0.234) and a constant 0.5 (0.250) on both scores and in all nine question families (exact sign-flip $p=0.0039$), so durability holds under the registered rule. A forecast frozen from the first wave alone scored 0.0599, 0.0587 and 0.0604 at horizons of 1.5 hours, 9.6 hours and 20.7 days. Of the 202 cells unanimous in the 8-hour wave, 194 were unchanged at D1 and none reversed. The latest wave alone was nearly as accurate as the pooled history (Brier 0.0292; pooled lower in 5 of 9 families, $p=0.28$). Every answer reported the requested model version, and none issued a search query although search was enabled. A blind human check of 96 labels agreed with the rule on 94; both disagreements were adjudicated as rater omissions, so after adjudication the rule matched on 96 of 96. The test covers one model, one interval and twelve known brands, and estimates no decay rate.”

Next, let’s break down the paper’s key vocabulary terms:

• Brand Mention Rate: The proportion of AI-generated answers to a specific question that mention a particular brand, used to measure how consistently that brand appears.
• History-Based Forecasting: Using historical AI-generated answers to predict whether a particular brand will appear in future responses to the same question.
• Brier Score: A statistical measure of how accurately predicted probabilities match actual outcomes. Lower scores indicate better predictions, with 0 representing perfect accuracy.
• Pre-Registration: Documenting a study’s methods, hypotheses, and evaluation criteria before conducting the experiment to reduce the risk of selectively interpreting results.
• Forecast Durability: The extent to which predictions based on historical observations remain accurate over time, indicating whether underlying brand mention patterns are stable.

Bravo. Let’s take a deeper look at the research paper’s contents now.

If you want to follow along, you can grab a PDF or the HTML version on arXiv.

The paper has 8 main sections.

We’ll focus on the most relevant ones below.

Introduction

Below I’ll paste in excerpts from the introduction that provide essential context for understanding the paper:

“People ask AI assistants which products to consider, and companies, agencies and researchers measure which brands the answers name, usually by sending the same question several times and counting. Such a count is worth something only if it still describes the answers when someone acts on it, days or weeks later. Single answers vary from run to run [9, 11]; what matters for planning is whether the rate at which a question names a brand stays put.

We treat this as a forecasting problem. How well do the answers collected to a fixed question predict which brands the same question names later? If a forecast built on past answers keeps its accuracy at a long horizon, the underlying rates held; if its accuracy decays, they moved. Proper scoring rules measure this directly [2, 4], and simple baselines separate stability at the level of a question from the general popularity of a brand.”

“In September 2026 we collected three waves of answers from gemini-3.1-pro-preview to the same 54 questions about twelve software brands. Forecasts frozen before the second and third waves passed their gates at horizons of about 1.5 and 8 hours. Horizons of hours say nothing about weeks. This note reports D1, the same 162 requests sent 20.3 days after the third wave, under a plan that was frozen, hashed and registered before the first call, with the decision rule fixed in advance [8].

Under the registered rule, durability holds. A forecast frozen from the first wave alone kept its accuracy from 1.5 hours to 20.7 days (Brier between 0.0587 and 0.0604), and the most recent wave alone forecast D1 nearly as well as the pooled history. No D1 answer searched the web, so what held is the model’s stored association of these brands with these needs, under an unchanged version string.”

Results

I’ll summarize the results section below:

The main finding is that historical brand mentions in Gemini remained highly predictable after approximately 20 days. The researcher found little evidence that brand mention patterns deteriorated over that period, suggesting AI visibility measurements may be more stable than individual AI-generated answers imply.

All 162 requests completed successfully using the same Gemini model version, allowing a consistent comparison over time. This means the researcher successfully recreated the original testing conditions approximately 20 days later, without any search-grounded responses introducing additional variability.

The researchers’ primary forecasting method, which used nine previous answers to predict whether a brand would appear in future answers to the same question, performed very well. The takeaway being that brand mention patterns remained highly predictable over 20 days, especially when predictions were based on specific questions rather than a brand’s overall popularity.

Furthermore, the findings weren’t limited to one type of software or customer need, but rather question-specific historical mentions consistently provided more accurate forecasts than general brand-level data.

Interestingly, using just three answers from the latest measurement session produced almost the same forecasting accuracy as using nine answers collected across three sessions. However, under these conditions, collecting additional answers within a short period provided limited measurable benefits. This means a relatively small sample may be sufficient to estimate brand mention probabilities.

In summary, historical observations provided useful estimates of future brand mentions, sometimes outperforming predictions based on a smaller sample of much more recent answers. That said, forecasting accuracy varied across software categories, languages, and markets, implying that AI brand mention stability may differ by product category, market, and language. Although, these subgroup comparisons were descriptive, so the study cannot establish that the observed differences would generalize.

In terms of implications for SEO/GEO professionals, we have three main ones:

1. AI brand visibility may be more stable than we assume.
2. Question-level tracking appears more informative than overall brand mention rates.
3. More frequent measurement isn’t necessarily better measurement.

The broader implication is that SEO/GEO measurement may benefit more from selecting representative questions and consistently monitoring them rather than repeatedly collecting large numbers of answers over short periods.

Discussion

Below are excerpts from the discussion section:

“For these questions and this model, the rate at which a question names a brand held for 20 days. Most cells are all-or-nothing and stay that way: across twelve answers spread over three weeks, 182 of 216 cells were named every time or never. …

The latest wave alone was nearly as accurate as the pooled history. Three answers per question from one session predicted the answers of 20 days later almost as well as nine answers from three sessions. For a measurement design this means that a small sample per question from one session carries most of the information, and that more sessions within a few hours add little. The three history waves lie within 10.2 hours of one another, so this study cannot say whether waves spread over weeks would add what waves spread over hours do not. …

D1 shows that sampling at temperature 1.0 and day-to-day serving conditions did not move the rates over three weeks; it gives no evidence about a model update, answers that do search, or other providers. A practical reading: for a fixed model version, a monthly measurement of brand mentions is still current when it is used, and a change in the reported model version is the event that calls for a new one.”

Knowing what we do now, why should SEO/GEO professionals care about “Twenty Days Later: A Pre-Registered Test of History-Based Forecasts of Brand Mentions in Gemini Answers”?

The biggest takeaway from this study is that is that AI-generated answers can vary while the underlying patterns of brand visibility remain relatively stable. For SEO/GEO professionals, this has important implications for how we track AI visibility, evaluate performance, allocate resources, and develop strategies.

Here are more practical takeaways from the research:

1. Track brand visibility at the individual query level

The study found that historical mention rates for specific brand-question combinations were much better predictors of future mentions than a brand’s overall mention frequency.

2. Prioritize consistent measurement over excessive measurement

One of the study’s most interesting findings was that three answers per question from a single session predicted brand mentions approximately 20 days later nearly as accurately as nine answers collected across three sessions.

3. Consider whether monthly AI visibility reporting is sufficient

The researchers found little deterioration in forecasting accuracy over approximately 20 days. In fact, 194 of 202 previously unanimous brand-question combinations remained unchanged after that period. However, the research doesn’t establish that monthly tracking is sufficient for ChatGPT Search, Google AI Overviews, or other search-grounded systems.

4. Distinguish normal AI variability from meaningful visibility changes

We know that AI responses are nondeterministic, meaning identical prompts can produce different answers, however, the study suggests that variability in individual responses doesn’t necessarily mean the underlying brand mention probabilities are changing.

5. Keep measurement conditions consistent and monitor model updates

An important feature of the study was that the researchers used identical requests and confirmed that the model version remained unchanged. Keep in mind that a change in brand mentions might reflect a model update rather than the impact of your SEO/GEO activities.

Caveat: Although web search was enabled during the experiment, none of the responses actually performed a search. The findings therefore concern relatively persistent brand associations in a particular Gemini model, rather than information retrieved from live search results.

Outro

The findings of this research suggest three important principles for SEO/GEO strategies:

• Measurement design matters.
• AI visibility isn’t necessarily as volatile as individual answers make it seem.
• Stable brand associations may require sustained strategic effort to change.

Ultimately, the study offers encouraging evidence that AI brand visibility can be measured in a repeatable, potentially actionable way, at least within the controlled environment tested.

For SEO/GEO professionals, the opportunity is to build measurement frameworks that distinguish genuine changes in brand visibility from normal fluctuations in AI-generated responses, while remaining careful about what those measurements can (and can’t) tell us.

I hope you’ve enjoyed this edition of Hamsterdam Research! 🐹

Feel free to comment below or contact me with your feedback.

Stay tuned for another new article, hopefully next week, or check out related research posts below.

Until next time, enjoy the vibes:

Thanks for reading. Happy optimizing!