Excerpts from Google’s Anti-Trust Trial Debrief: As an SEO, Here’s What Stood Out to Me from USA v. Google LLC (Document #833)

By Ethan Lazuk

We’ve learned all sorts of interesting information from Google’s anti-trust trial that’s pertinent to SEO, especially from Pandu Nayak’s testimony.

[Aside: If that testimony interests you, this post about rankings system questions touches on it, or you can read insightful articles from AJ Kohn and Danny Goodwin or try this GPT Marie Haynes built that references the testimony’s transcript.]

Based on the value of past court documents, I was interested to come across this SER story on the morning of March 1st about a post-trial debrief:

Google in its post trail debrief says its search quality has improved, not degraded https://t.co/aEmqGAMdQY hat tip @gsterling pic.twitter.com/5vuKXD5Fiu

— Barry Schwartz (@rustybrick) March 1, 2024

Coffee in hand, I followed the breadcrumbs from there to a copy of the brief shared by Greg Sterling:

Google boldly claims in its "default search" antitrust post-trial brief that search quality has improved, not degraded. https://t.co/cR807jZsFj

— Greg Sterling 🇺🇦 (@gsterling) February 29, 2024

It was filed and uploaded to CourtListener on February 23rd and is known as “Document #833”:

Having that coffee was fortunate, as it turned out to be 123 pages of litigious writing.

But it was pretty easy to follow, and there were some hidden gems SEOs may find interesting, at least I thought so personally.

If you haven’t read it yet, the good news is that I highlighted everything that I found interesting.

Mostly, this included excerpts related to user behavior, how search works, and other interesting data or statistics.

[Aside: After publishing, I checked the SERPs and found a couple of articles that discussed this document, including an analysis by Nicola Agius, “Google: Don’t punish us for our success in Search” (SEL), and a short news article from the Verge …. I also found the government’s response, which I haven’t read yet, but will add in the future!]

The bad news is that I didn’t plan ahead and I was reading it sideways on my iPhone (it’s not a mobile-friendly document) and taking screenshots on a whim, so the photos here may be annoying to view on mobile.

That said, I added text excerpts to make the content mobile-friendly (and crawlable) and also topical headings. I kept all of the screenshots in chronological order but noted their page numbers.

For the sake of convenience, I also put some of my favorites below first in their own section.

Top excerpts (with thoughts)

Tight on time? Before getting into all 123 pages worth of excerpts, here are the key ones I saw in document #833.

“Consumers do not rely on general search engines for all of their information needs—rather, as the evidence at trial demonstrated, consumers search using the provider that best meets their needs, whether it be a general search engine, specialized search engine (often known as a specialized vertical provider), or social media channel. Users seamlessly switch between these different providers, depending on the type of information they seek.”

– Page 12 of 123

Thoughts: What I find notable here is the triad of general search engines, specialized vertical providers (SVPs), and social media channels. In this document, Google often argues that it’s really competing with these other experiences more than other general search engines like Bing or DuckDuckGo. One interesting thought is the advent of AI answer engines, like Perplexity AI. Though Perplexity isn’t mentioned in the brief (but Neeva is later), some of these AI search experiences could meet users needs for specific types of information or search intents, particularly for younger users. The search landscape definitely could become more fragmented.

“user search behavior reveals that users do not conduct “one-stop-shop” searching, that is, search across a variety of topics in one sitting using a single search provider. Instead, when searching online, users typically search for information in single-topic visits. Where a search visit includes more than one query, the queries are usually topically similar or otherwise within the same vertical segment.”

– Page 17 of 123

Thoughts: So much of “SEO content” is informational. Often, it’s difficult to tie the ROI of that content to a direct conversion, but this supports one potential reason why — “single-topic visits.” That said, the notion of “topically similar” vertical segments supports the idea of topical clusters and itself could include conversions. Also, Dr. Mark Israel, a “Google Expert” and contributor of information in this brief, is someone whose work I plan to look into more.

“new general search engines can avoid many of the costs historically borne by an incumbent like Google. As Neeva recognized, advances in the ability of computers to understand language “could be used as a short circuit to make ranking better,” and “when it comes to figuring out concepts for a query, related queries, or correcting misspellings that people often have when they type in or speak queries, [Neeva was] able to very, very successfully do that.””

– Page 42 of 123

Thoughts: This gets at one of the most interesting themes from this document, which is that user interaction data has become less important for search quality given the capabilities of natural language processing and machine learning. Navboost got a lot of attention as a result of this trial (more on that soon), but so too were other systems mentioned, like RankBrain.

“The fact that prior increases in scale have not translated into share shifts is consistent with both the law of diminishing returns and the record evidence of all the factors that impact search quality that have nothing to do with user interaction data. FOF ¶¶ 253-332. To be sure, some user interaction data is useful to search engines, but beyond a certain point, the utility diminishes such that there is no additional meaningful utility in the data.”

– Page 75 of 123

Thoughts: This is a pretty interesting claim that user interaction data has little impact on search quality. It ties into the theme above on machine learning.

“Professor Edward Fox’s experiment puts to rest any contention that Microsoft’s failure to outcompete Google is a function of its smaller query stream, as opposed to its failure to invest and innovate in search as Google has.4 The results of his experiment demonstrate that both Google and Bing are already at the point of diminishing returns, and thus the consistent quality gap between the two search engines cannot be attributed to the fact that Google receives more queries than Bing. A company as efficient as Google could have search quality similar to Google even at Microsoft’s scale, and conversely a company as efficient as Google but with Microsoft’s scale would not meaningfully benefit from an increase in user interaction data.”

– Page 75 of 123

Thoughts: This references an interesting study saying Google’s quality advantage over Bing isn’t based on user interaction data, to the earlier points. Also interesting, as we’ll see, is this study was likely done using Google’s production ranking systems.

“the overwhelming majority of the quality gap between Google and Bing cannot be explained by the different volumes of user interaction data available. FOF ¶¶ 349, 351-88. And the testimony of Google engineers confirmed that Google’s use of user interaction data has always been but one of many inputs into Google’s systems, has decreased over time, and that the reliance on that data is in the midst of a dramatic change given the rise of large language models that employ fundamentally different techniques to solve the same task, i.e., predicting the usefulness of documents given the query. FOF ¶¶ 253-332. For all of these reasons, there is no evidence that any share shift would generate any meaningful quality improvement.”

– Page 76 of 123

Thoughts: This is a very interesting concept that user interaction data has decreased as an input for Google Search, instead being replaced by LLMs that predict “the usefulness of documents given the query.”

“As Professor Fox explained, his DRE is a form of “scalability” study, a “tried and true” design widely employed in academia and industry that he uses “all the time.” FOF ¶¶ 391-96. Moreover, his experiment drew on processes that Google has found sufficiently reliable to use in the ordinary course. See FOF ¶¶ 397-99. He further ensured that his experiment would retrain those components of Google’s search engine (i.e., NavBoost, Term Weighting, QBST, RankBrain, DeepRank, and RankEmbedBERT) whose functionality is most influenced by user interaction data. See FOF ¶¶ 352-53, 361-63. He likewise meticulously cataloged every signal within Google’s search engine that had an impact of at least 0.01% on Google’s search result rankings and explained how they fit into his study.”

– Page 121 of 123

Thoughts: We have a list of “components of Google’s search engine … whose functionality is most influenced by user interaction data,” and they include NavBoost, Term Weighting, QBST, RankBrain, DeepRank, and RankEmbedBERT. This is a nice follow up to the reporting done on Pandu Nayak’s testimony. Also, Professor Fox catalogued “every signal” with at least a 0.01% impact on “Google’s search result rankings,” and explained its purpose, which implies that number must be reasonable. Makes you wonder about the number of signals that have a smaller than 0.01% impact, and how it all fits together.

All excerpts from USA v. Google post-trial debrief

If you’re curious what other excerpts SEOs might want to know about Google’s anti-trust trial debrief — known as, “UNITED STATES OF AMERICA v. GOOGLE LLC: Redacted Document — Document #833 — feel free to dig in!

Consumer insights and user behavior

“Consumers do not rely on general search engines for all of their information needs—rather, as the evidence at trial demonstrated, consumers search using the provider that best meets their needs, whether it be a general search engine, specialized search engine (often known as a specialized vertical provider), or social media channel. Users seamlessly switch between these different providers, depending on the type of information they seek.”

– Page 12 of 123

“Further, many users today, and younger users in particular, use social media sites like Facebook, Instagram, TikTok, and X (formerly Twitter) to search for information and online content of interest to them. These social media sites are incredibly popular and successful.”

– Page 14 of 123

“Google’s business model (and that of other general search engines) is to strive to answer every user query, whatever the topic. FOF ¶¶ 77, 636-51, 968. Countless other companies follow a different model, whereby they focus on providing users with information in response to queries about a particular topic, for example, hotels or food delivery. FOF ¶¶ 652-66. Social media channels such as Instagram and TikTok follow yet another model that provides users a variety of ways to access information”

– Page 16 of 123

“user search behavior reveals that users do not conduct “one-stop-shop” searching, that is, search across a variety of topics in one sitting using a single search provider. Instead, when searching online, users typically search for information in single-topic visits. Where a search visit includes more than one query, the queries are usually topically similar or otherwise within the same vertical segment.”

– Page 17 of 123

“As explained by Dr. Mark Israel, “People decide, as they need to search, who can fulfill that query[,]” and “each query made by a user is a meaningfully distinct choice about where to go get information.” Tr. 8392:2-8393:19 (Israel); FOF ¶ 925. This observation resonates with lived experience: think of Zappos for finding shoes, or Expedia for flights. It is also confirmed by Dr. Israel’s Google panels data analysis, which shows that users make different search decisions across the different verticals: reliance on general search engines, as opposed to SVPs, differs significantly by vertical.”

– Page 18 of 123

“That users may start on a general search engine does not render SVPs uncompetitive for those queries—SVPs exert competitive pressure on Google to provide a more compelling or useful experience”

– Page 18 of 123

“his analysis revealed that Google faces greater competition for users with shopping queries from Amazon than it does from Microsoft’s Bing, and greater competition for users with local queries from Yelp than it does from Bing.”

– Page 19 of 123

“The increasing usage of social media channels, including for the sort of queries that had not historically been thought of as a target of social media channels, was the focus of a 2021 presentation that Vice Presidents of Search Elizabeth Reid and Pandu Nayak gave to the Alphabet Board. FOF ¶¶ 976-77; Tr. 8203:23-8210:18 (Reid). Among the findings presented to the Board was Google’s user research that showed “63% of daily TikTok users age 18 to 24 stated that they use[d] TikTok as a search engine in the last week.””

– Page 22 of 123

“Evidence presented at trial showed that for online shopping searches, users are twice as likely to begin their search on Amazon than they are Google.”

– Page 22 of 123

“As Dr. Israel explained, empirical data shows that consumers frequently visit a Meta platform (Facebook or Instagram), Amazon, or a series of other websites during the same web session in which they search on Google. It is far more unusual, however, for a consumer to visit Google Search and another general search engine (e.g., Bing or DuckDuckGo) within that same session.”

– Page 26 of 123

“consumers no longer make decisions in the linear fashion envisioned by the funnel, with search ads allegedly driving only final purchasing decisions. Consumers are driven to purchase a product by a variety of ad types across a number of settings. FOF ¶¶ 1065-78. This fact is not lost on advertisers, who use multiple types of digital ads to achieve the same objectives.”

– Page 26 of 123

“Professor Whinston offers in support of the user-side market definition suffers significant methodological flaws. He relies on Comscore search panel data for the proposition that 77% of search sessions begin on general search engines; he contends this demonstrates that users “one stop shop” by using general search engines as a “gateway to the internet.” Tr. 4614:9-4615:16; FOF ¶¶ 937-39. The Comscore data sample, however, was taken from Windows PCs, and thus is not indicative of user search behavior more broadly. It fails to account for mobile queries on general search engines (which outnumber desktop queries), and the usage of mobile apps, including SVP apps that serve as a starting point for user queries in many commercial verticals.”

– Page 29 of 123

“Professor Jonathan Baker offers confirms Dr. Israel’s finding that users search in discrete visits focused on specific topics. Professor Baker’s analysis of user search behavior on Google showed that nearly 80% of user visits—with a visit defined as a series of user actions followed by at least five minutes of user inactivity—involve a single vertical category. Another 15% involve just two vertical categories. It is only by looking at all user searches over a 24-hour period that Professor Baker finds that users often search in more than one vertical, and even then, more than 70% of users search in three or fewer verticals.”

– Page 30 of 123

“The number of general search queries has more than doubled in the ten years between 2011 and 2021, from about 30 billion in 2011 to above 70 billion by 2021.”

– Page 33 of 123

Search engine innovations

“The record is replete with evidence of Google’s search innovations, which range from paradigm-shifting breakthroughs in artificial intelligence to an endless stream of improvements to specific categories of queries based on insights from its teams of search quality engineers and human raters.”

– Page 35 of 123

“Google is relentless in its commitment to the user experience and to improving its ability to offer useful search results, even going so far as to expand the modes by which users are able to search (e.g., Google Lens) and inventing artificial intelligence models that completely transformed search.”

– Page 36 of 123

“The fact that Google has not adopted all of the proposals it has considered is not an exercise of monopoly power, but rather an inherent part of competitive product development. That is especially true of the privacy-related features identified by Plaintiffs because of the indisputable tradeoffs they entail, including to search quality and the ability to detect fraud or other malicious conduct. FOF ¶¶ 1113-23. Google’s decision to preserve features that have a positive impact on search quality while offering customizable settings to suit users’ privacy preferences is a reasoned business judgment consistent with a competitive marketplace, not a degradation of product quality that evidences monopoly power.”

– Page 37 of 123

“Neeva also deployed modern machine learning techniques to develop its own systems for ranking search results without access to the volumes of user interaction data available to established search engines.”

– Page 41 of 123

“new general search engines can avoid many of the costs historically borne by an incumbent like Google. As Neeva recognized, advances in the ability of computers to understand language “could be used as a short circuit to make ranking better,” and “when it comes to figuring out concepts for a query, related queries, or correcting misspellings that people often have when they type in or speak queries, [Neeva was] able to very, very successfully do that.””

– Page 42 of 123

Default search engine status

“Search engine usage data from Apple devices reflected that nearly of all search queries on iOS devices came from a search access point other than the Safari default. FOF ¶ 1413. This fact, alone, confirms that the Safari agreement is not exclusive. Moreover, when Mozilla changed the default search engine on Firefox from Google to Yahoo in 2014, the large majority of search queries on Firefox returned to Google—and other Firefox users “switched back” by accessing Google through the Chrome browser.”

– Page 46 of 123

“As evidenced by Google’s success where it is not the default (as depicted in Figure 7, below), users can and do switch to alternative general search engines both on the same browser and on the same device.”

– Page 47 of 123

“notwithstanding Microsoft’s agreements and whatever “foreclosure” one could argue resulted, Google competed for all of those users, and in fact won a substantial majority of user queries conducted on Windows PCs”

– Page 50 of 123

“Under Professor Kevin Murphy’s calculation, using a choice screen as a conservative upper bound for foreclosure based on Plaintiffs’ theory that a choice screen is a proxy for some notion of “parity,” the Browser Agreements account for a share shift of just 0.9% of search queries”

– Page 52 of 123

“reality that a majority of consumers will use Google even if another search engine is “exclusively” pre-installed or set as the default.”

– Page 53 of 123

“the evidence shows that a majority of users find a way to use their preferred search engine when it is not set as the default.”

– Page 53 of 123

“Apple and Mozilla—whose products depend upon a search engine for both browser performance and consumer enjoyment—chose to have a default search engine in their browsers because they believe it is best for users, and have chosen Google to be in that default position because Google provides the best services to users.”

– Page 56 of 123

“the agreements also incentivize browser developers to improve the browser’s speed, stability, ease of use, and other attributes that encourage users to conduct more searches (and thereby generate more revenue to be shared with the browser developer)”

– Page 63 of 123

“Apple’s selection of Google is not universal across all markets: in those countries where Apple determined that another search engine provided superior search quality to Google, Apple chose a different search engine and carved those countries out of its agreement with Google.”

– Page 65 of 123

Search result quality

“Since 2009, Microsoft has approached Apple a number of times to propose that Bing become Safari’s default search engine. Each time—in 2009, 2013, 2015 to 2016, 2018, and 2020—Apple declined to do a default deal with Bing out of concerns regarding Bing’s product quality. FOF ¶¶ 1304-39, 1391. Apple arrived at this conclusion again and again after taking into account information from Microsoft regarding its plans for improvement and investment, as well as Apple’s own internal assessments of Google’s search quality compared to Bing’s.”

– Page 65 of 123

“Apple conducted a head-to-head evaluation, which confirmed that Google continued to have a strong lead in search relevance.”

– Page 66 of 123

“As Mozilla CEO Mitchell Baker recounted, “When we entered this agreement, there was nothing in the world like Google. Prior to Google, there had been Excite and Infoseek and the Yahoo! directory. I am not sure if Microsoft had started their search at this time, but Google was way ahead. Like search in those days was miraculous and so there was nothing like Google. I mean, it’s hard to remember how earth-shattering search was when Google changed the game with their page rank from what, you know, Yahoo! had been doing or the other things that we called search.””

– Page 67 of 123

“During the years Yahoo was the default search engine, Mozilla observed a decline in both (1) usage of the default search functionality in Firefox and (2) the number of users of the Firefox browser.”

– Page 67 of 123

“When determining which search provider to replace Yahoo with, Mozilla evaluated various options and chose Google on the basis that it was “what [their] users want.””

– Page 67 of 123

“Mozilla advised the Department of Justice, in a letter sent weeks before this lawsuit was filed, that it would be significantly harmed in its ability to compete were it not permitted to enter into a default search agreement with Google, because Google provides the best user experience and is preferred by Firefox users, whereas Bing has poor retention, lower search volume, and lower monetization rates.”

– Page 68 of 123

“the quality of the search engine has a significant impact on the browser experience. (The reverse is equally true: as noted above, higher-quality browsers expand search output, and thus search providers have strong incentives to improve browsing.)”

– Page 70 of 123

Quality and scale (user interaction data)

“Prior increases in scale have not translated to larger share shifts. Microsoft’s own data reveals that when it more than doubled its scale overnight by syndicating results to Yahoo, it saw no material increase in quality.”

– Page 74 of 123

“this occurred when Windows PCs, on which Microsoft had virtually all preinstallation, made up the vast majority of search usage, and (iii) this occurred well before today’s modern large language models emerged with their considerably lessened need for user data.”

– Page 74 of 123

“The fact that prior increases in scale have not translated into share shifts is consistent with both the law of diminishing returns and the record evidence of all the factors that impact search quality that have nothing to do with user interaction data. FOF ¶¶ 253-332. To be sure, some user interaction data is useful to search engines, but beyond a certain point, the utility diminishes such that there is no additional meaningful utility in the data.”

– Page 75 of 123

“Plaintiffs presented no evidence that quantities of user interaction data beyond what Microsoft already has would have any meaningful impact on a search engine’s ability to serve high-quality results to its users. Indeed, Dr. Ramaswamy testified that Neeva—which built a modern search engine that relied on machine learning techniques—could compete successfully with Google with approximately 2.5% of general search users.”

– Page 75 of 123

“Professor Edward Fox’s experiment puts to rest any contention that Microsoft’s failure to outcompete Google is a function of its smaller query stream, as opposed to its failure to invest and innovate in search as Google has.4 The results of his experiment demonstrate that both Google and Bing are already at the point of diminishing returns, and thus the consistent quality gap between the two search engines cannot be attributed to the fact that Google receives more queries than Bing. A company as efficient as Google could have search quality similar to Google even at Microsoft’s scale, and conversely a company as efficient as Google but with Microsoft’s scale would not meaningfully benefit from an increase in user interaction data.”

– Page 75 of 123

“Plaintiffs nonetheless at trial tried to make much of Google’s own use of the user data it receives. In particular, Plaintiffs seize on internal references to the value of that user interaction data. But again, Professor Fox’s experiment disproved the contention that this data gives Google an insurmountable advantage”

– Page 75 of 123

“the overwhelming majority of the quality gap between Google and Bing cannot be explained by the different volumes of user interaction data available. FOF ¶¶ 349, 351-88. And the testimony of Google engineers confirmed that Google’s use of user interaction data has always been but one of many inputs into Google’s systems, has decreased over time, and that the reliance on that data is in the midst of a dramatic change given the rise of large language models that employ fundamentally different techniques to solve the same task, i.e., predicting the usefulness of documents given the query. FOF ¶¶ 253-332. For all of these reasons, there is no evidence that any share shift would generate any meaningful quality improvement.”

– Page 76 of 123

Android devices

“less than 1% of U.S. search queries would shift to rivals were a choice screen to be implemented on Android devices.”

– Page 80 of 123

“The hotseat is one of the most significant places for the placement of browsers on an Android device. Browsers have accounted for 36-37% of search revenue on Android devices in recent years, and most of that is attributable to a browser preloaded in the hotseat.”

– Page 82 of 123

“The evidence at trial demonstrated that the Google Search Widget is popular with users and placement of the Google Search Widget on devices increases search usage. FOF ¶ 1686. For instance, a Google study found that “devices without [the Widget] have up to . . . 10% fewer total search [daily active users] and 12% fewer total queries.”

– Page 95 of 123

“Similarly, the evidence at trial showed that access to a high-quality browser like Chrome, the most popular browser in the U.S., increases web usage and search usage. FOF ¶¶ 907, 912- 13, 916, 1687. As explained by Google CEO Sundar Pichai, Google developed Chrome in part because “[w]e realized just improving the state of browsers would overall help users use the web more, will increase online activity and increase search usage, including Google’s usage. And the correlation was pretty clear to see, and we had seen that.””

– Page 95 of 123

“During the term of Google’s Android Agreements, search usage on mobile devices has increased significantly. From 2012 to 2022, Google search queries on mobile increased more than 500%. DXD-37.091. And on Android devices in particular, this growth has recently largely come from the increased intensity of search. Since early 2018, the sale of Android devices in the U.S. has remained flat, while Google search queries on Android devices have continued to grow significantly.”

– Page 96 of 123

“(“[E]very iPhone looks exactly the same, has exactly the same services on it[.]”). To help address this issue, the preload and placement provisions in Google’s MADAs and RSAs help ensure a high-quality baseline for a consistent user experience across enrolled devices, improving Android’s ability to compete with Apple. FOF ¶¶ 1704-05. In addition, this consistent user experience facilitates switching among Android devices, encouraging consumers to stay within the Android ecosystem when purchasing a new device.”

– Page 98 of 123

“Android devices are typically offered at a lower price point than iOS devices. FOF ¶ 1710. In fact, 40% of Android devices sold are priced below $200. Tr. 9851:2-9852:18 (Murphy); DXD-37.095; FOF ¶ 1710. This availability of lower-priced Android devices has increased competition with Apple and even led Apple to offer lower-priced iPhones to compete against Android. FOF ¶ 1711; Tr. 9419:10-22 (Rosenberg (Google)). Lower-priced devices lead to more mobile users, and thus more mobile search.”

– Page 99 of 123

Professor Edward A. Fox’s “data reduction” experiment (DRE)

“To hear Plaintiffs tell it, Google’s lead in search quality is little more than a function of its access to user interaction data. While Plaintiffs did nothing to test their hypothesis, Google did. Google retained Professor Edward A. Fox—a renowned computer scientist with decades of experience—to design a “data reduction” experiment (“DRE”). Drawing on standard tools of information retrieval, Professor Fox’s DRE involved a comparative quality evaluation of the search results generated by Google’s systems when trained on a smaller quantity of user interaction data. The DRE’s results were unequivocal: even when reduced to Bing’s scale, Google’s measured quality dropped by an amount equal to just 2.9% of Google’s famously sizable quality lead over Bing.”

– Page 119 of 123

“Professor Fox explained, “[his] assignment was to determine the extent to which the amount of . . . user interaction data would affect the quality of a search engine, in particular Google, relative to the difference in quality between it and Bing.” Tr. 7818:22-7819:2; see also FOF ¶ And what he found was unequivocal: “[t]he vast majority of the Google-Microsoft Search quality gap must be explained by factors other than the volume of user interaction data.””

– Page 120 of 123

“As Professor Fox explained, his DRE is a form of “scalability” study, a “tried and true” design widely employed in academia and industry that he uses “all the time.” FOF ¶¶ 391-96. Moreover, his experiment drew on processes that Google has found sufficiently reliable to use in the ordinary course. See FOF ¶¶ 397-99. He further ensured that his experiment would retrain those components of Google’s search engine (i.e., NavBoost, Term Weighting, QBST, RankBrain, DeepRank, and RankEmbedBERT) whose functionality is most influenced by user interaction data. See FOF ¶¶ 352-53, 361-63. He likewise meticulously cataloged every signal within Google’s search engine that had an impact of at least 0.01% on Google’s search result rankings and explained how they fit into his study.”

– Page 121 of 123

“Professor Fox’s DRE used multiple querysets (i.e., Training, Launch, and Covert), multiple quality metrics (e.g., IS4@5 and NDCG), and multiple samples (i.e., both a Low and High Mobile sample compared to a frozen 100% baseline). See, e.g., FOF ¶¶ 354-63, 371-84. To provide further context, Professor Fox also presented results not just for queries in the aggregate but also for tail queries specifically and for each of his six retrained components separately. See, e.g., FOF ¶¶ 374-76”

– Page 122 of 123

“what he found was unequivocal in its results: When reducing the amount of user interaction data from Google’s quantity (i.e., the 100% sample) to Bing’s (i.e., the Low Mobile sample), the resulting quality drop was not statistically different and amounted to a mere 2.9% of the day-to-day Google-Bing quality gap.”

– Page 122 of 123

Fin. 🙂

“Robots are my next of kin”

I hope you found this summary of excerpts interesting, or they help contribute to your own research or articles.

I’ll likely revisit this post to add related resources and other materials in the near future, as well.

Until next time, enjoy the vibes:

Thanks for reading. Happy optimizing! 🙂