Exploring language model embeddings for Bayesian optimization & why marketers should care.

Welcome to a new week of 🐹 Hamsterdam Research, where we’re looking at a Google DeepMind paper, Predicting from Strings: Language Model Embeddings for Bayesian Optimization.

The paper was submitted to ArXiv on October 14th, 2024, and includes several authors, including Tung Nguyen, Qiuyi Zhang, Bangding Yang, Chansoo Lee, Jorg Bornschein, Yingjie Miao, Sagi Perel, Yutian Chen, and Xingyou Song.

In short, this paper introduces a new method called “Embed-then-Regress” for applying Bayesian Optimization to complex search spaces. These can include permutations and combinations. The method uses language models to embed string representations of candidate solutions into fixed-length vectors, which are then used for in-context regression with a Transformer model.

In technical terms, this approach allows for more flexible and efficient optimization across different domains, achieving comparable results to traditional Gaussian Process-based methods. We’ll dive into more of what that means below.

But what about in practical terms for SEOs and marketers?

This paper could have implications for analyzing search queries, predicting user engagement, personalizing search results (this is the one I’m most interested in), A/B testing marketing campaigns, predicting campaign performance, or even identifying trends and patterns from large datasets of marketing data (like web traffic or social media engagement).

We’ll go through those more below, as well.

But first, here’s the paper’s abstract with my bolding:

“Bayesian Optimization is ubiquitous in the field of experimental design and blackbox optimization for improving search efficiency, but has been traditionally restricted to regression models which are only applicable to fixed search spaces and tabular input features. We propose Embed-then-Regress, a paradigm for applying in-context regression over string inputs, through the use of string embedding capabilities of pretrained language models. By expressing all inputs as strings, we are able to perform general-purpose regression for Bayesian Optimization over various domains including synthetic, combinatorial, and hyperparameter optimization, obtaining comparable results to state-of-the-art Gaussian Process-based algorithms.”

– Abstract (my bolding)

If that sounds a little daunting, we’ll review some of the high-level takeaways.

In the introduction, the authors write that “Recent progress in large language models (LLMs) have demonstrated the flexibility and versatility of representation of information as strings, which allow for a wider range of data formats to be encoded for subsequent processing.”

“Strings” in this context refers to textual representations of data used as input for the language model. In short, the authors propose converting everything to text first, from code snippets to molecular structures and beyond.

They continue explaining how “The potential of LLMs for universal learning-based regression is considerable, allowing for regressors that can be generalized across multiple tasks, thereby mitigating the task-specific limitations of current methods.”

This basically means having a single regression model for multiple tasks rather than separate models for specific tasks. Regression models predict continuous numerical values based on a given input data, essentially finding the relationship between the input features and the output variable. An example might be a model to predict house prices based on their square footage.

They explain how their work focuses “on improving the flexibility of regressor-guided search methods through the use of LLM-based embeddings, which map arbitrary strings to fixed-length vectors to be used in downstream tensor-based regressor models, such as an in-context learning (ICL) based Transformer.”

Figure 1 shows the Embed-then-Regress process for Bayesian Optimization:

Let’s break down the figure’s contents.

The Search Space represents the possible solutions for a problem being optimized. The stars in the blue space are different candidate solutions.

The Embedding Space is where each candidate solution from the Search Space is converted into a string representation. A language model (Transformer) then embeds these strings into fixed-length vectors in the embedding space.

In-Context Regression is where the embedded vectors are then fed into a regression model, likely a Transformer. This model learns to predict the performance (represented by the curve) of a candidate solution based on its embedding. The colored dots represent the predicted performance of each embedded solution.

What we haven’t yet mentioned is “Bayesian optimization” and what that refers to. The transformed embeddings used for performance prediction are the foundation of Bayesian optimization, which uses the prediction.

As an iterative process, Bayesian optimization would repeatedly select promising candidates (based on the predicted performance), evaluate and measure the actual performance of the chosen candidate, then update the model with the new data, improving its predictive accuracy over time.

In conclusion, the authors note how “Our method, Embed-then-Regress, demonstrates the versatility of using string-based in-context regression for Bayesian Optimization over a variety of problems.” They demonstrated how it can “obtain comparable results against industry-standard GP baselines and allow flexibility in more esoteric spaces such as permutations and combinations.”

Getting back to our takeaways as SEOs and marketers, let’s review the implications of this research in more detail, with insights pulled from Gemini 1.5 Pro:

• Improved understanding of user intent: By analyzing search queries as strings, SEOs can better grasp the nuances of language and optimize content for user intent.
• Enhanced content engagement: Predicting the performance of content variations can help us as marketers create more engaging content.
• Personalized experiences: The research introduces a method that could enable more personalized content recommendations and search results based on individual user preferences.
• Efficient A/B testing: Bayesian Optimization could accelerate A/B testing, helping us as marketers quickly identify top-performing content variations.
• Data-driven campaign predictions: We can use these methods to better predict campaign performance based on historical marketing data to improve resource allocation and ROI.
• Trend identification: Analyzing large datasets of marketing data could help us reveal trends and patterns to inform future strategies.

Of particular interest to me is the idea of personalized experiences. Since Embed-then-Regress can handle various data types as input, it can potentially analyze diverse user data, like search history, demographics, interests, and social media activity.

Regarding in-context learning, the Transformer-based regression model can learn from previous examples (context) to make predictions, such as learning from a user’s past interactions with content to predict their preferences and interests.

In short, the method could be used to refine search results based on individual user profiles, allowing search engines to deliver more relevant results.

Thanks for checking out this rendition of Hamsterdam Research. 🐹

Until next time, enjoy the vibes:

Thanks for reading. Happy marketing! 🤗