The Embedding Language Model (ELM) & Why SEOs Should Care (Maybe)

By Ethan Lazuk

Welcome to a new installment of Hamsterdam Research!

If you’re new here, this is where we look at recent AI research papers to learn what they’re talking about and explore their implications (hypothetically) for the future of search and SEO.

This week, we’ll be looking at a paper from Google Research called, “Demystifying Embedding Spaces using Large Language Models.”

The paper introduces the Embedding Language Model (ELM), which uses an LLM (large language model) to interpret domain-specific embedding spaces.

The paper was submitted on October 6th, 2023 and last updated March 13th, 2024 (per ArXiv.org). The top of the paper says it was “Published as a conference paper at ICLR 2024,” referring to the International Conference on Learning Representations (May 7th to May 11th, 2024) in Vienna, Austria.

Sorry, it’s obligatory to quote that movie line whenever I see Austria mentioned. 😉

Contributors to the research paper include:

• Guy Tennenholtz
• Yinlam Chow
• Chih-Wei Hsu
• Jihwan Jeong
• Lior Shani
• Azamat Tulepbergenov
• Deepak Ramachandran
• Martin Mladenov
• Craig Boutilier

You can learn more about their backgrounds or research on the Google Research website.

The paper’s experiment also uses a dataset called MovieLens 25M, which includes movie ratings and is available on Kaggle.

What’s this research paper about, and why should SEOs care (maybe)?

Ah, glad you asked!

The Embedding Language Model (ELM) is a novel framework that helps bridge a gap between the rich information of embeddings, which can be hard for humans to interpret, and the natural language capabilities of LLMs.

In short, ELM enables users to query and explore embedding spaces using natural language.

As for why SEOs should care about ELM, while the paper doesn’t touch on Google Search and only has one fleeting reference to information retrieval, it does come from Google Research, and if we look at how its capabilities could apply to a search engine (hypothetically), it opens avenues for exploring topics like vector embeddings and semantic search, personalization, and maybe even decoding the deep learning systems used in ranking search results.

Exploring the research paper and ELM

Let’s start with the paper’s abstract:

“Embeddings have become a pivotal means to represent complex, multi-faceted information about entities, concepts, and relationships in a condensed and useful format. Nevertheless, they often preclude direct interpretation. While downstream tasks make use of these compressed representations, meaningful interpretation usually requires visualization using dimensionality reduction or specialized machine learning interpretability methods. This paper addresses the challenge of making such embeddings more interpretable and broadly useful, by employing large language models (LLMs) to directly interact with embeddings — transforming abstract vectors into understandable narratives. By injecting embeddings into LLMs, we enable querying and exploration of complex embedding data. We demonstrate our approach on a variety of diverse tasks, including: enhancing concept activation vectors (CAVs), communicating novel embedded entities, and decoding user preferences in recommender systems. Our work couples the immense information potential of embeddings with the interpretative power of LLMs.” [Highlights added.]

The abstract speaks about “downstream tasks” make use of embeddings, however, “meaningful interpretation” requires more work.

As the paper says relatedly in its introduction:

“The success of deep learning has brought forth a paradigm shift in knowledge representation through the concept of embeddings—dense vector representations that capture high-dimensional information about entities, concepts, or relationships in a compact and useful format. … These embeddings are invaluable, capturing nuanced relationships and semantic structure in data which traditional machine learning (ML) approaches often miss. Nevertheless, understanding these abstract representations remains challenging.”

In other words, embeddings can capture complex information using dense vector representations, like nuanced relationships of meaning or semantic structures, but these can also be difficult or abstract for humans to understand.

That’s where LLMs can help explain dense vector embeddings in natural human language.

Showing how this works will be the focus of the rest of the paper.

But first, a quick detour:

What are the shortcomings of traditional machine learning?

If you’re also curious why the introduction says traditional ML might miss the nuanced relationships in embeddings, I asked Gemini Advanced for clarification and here’s how it explains it:

“Traditional machine learning models often rely on features that are manually designed by humans. … This limits the model’s ability to understand the complex relationships between words and their broader meaning within a sentence.

Natural language is inherently complex. Traditional techniques often turn words into sparse vectors (mostly zeros with one or two positions indicating the word). This blows up the number of dimensions a model has to handle, and those vectors don’t reflect similarity or relationships between words well. …

Deep learning models, like neural networks, can automatically learn dense representations of words, concepts, or any other kind of data. These representations are called embeddings. The model isn’t told what to focus on; it finds meaningful patterns within the data itself. …

Embeddings pack a lot of information into a relatively small vector space. This makes it computationally much easier to find similarities and relationships.”

– Gemini Advanced response

Related readings: If you’re interested in this topic of embeddings, we explored it a few times in Hamsterdam History, including Alan Kent’s mention of neural networks in an ecommerce context or Jeff Dean’s discussion of neural networks, generally. We’ve also spoken more about the foundations of transformer-based architectures in an overview of machine learning and a history of how Google treats synonyms.

Downstream tasks for embeddings and ML interpretability

Returning to the paper on ELM, the researchers go into specifics regarding downstream tasks for embeddings and previous work on ML interpretability:

“By design, the structure and underlying information carried by an embedding is heavily adapted to the idiosyncrasies of the downstream task, posing a substantial challenge to its interpretation and manipulation. Previous work on ML interpretability offers various task-independent means to interpret embeddings, including dimensionality reduction techniques (e.g., t-SNE (Van der Maaten & Hinton, 2008), UMAP (McInnes et al., 2018)) or concept activation vectors (CAVs) (Kim et al., 2018). While very useful, these techniques are fairly narrow in scope.”

It mentions three methods of ML interpretability, including dimensionality reduction techniques, UMAP, and concept activation vectors (CAVs).

Let’s explore each of these a little more, with assistance from Gemini.

1. Dimensionality reduction techniques

Dimensionality reduction techniques compress large datasets (say hundreds of variables) that are tough to visualize or interpret and puts them into lower-dimensional spaces, like 2D or 3D. Here’s an analogy from Gemini:

“Imagine a complex, high-dimensional landscape with valleys, ridges, and other features. Dimensionality reduction techniques project this landscape onto a lower-dimensional plane (like a flat map) while trying to capture the most significant structures and relationships between those features.”

– Gemini Advanced response

There are different types of dimensionality reduction techniques, such as PCA or t-SNE. The best choice depends on the type of data and goals of the analyses.

2. UMAP

Relatedly, we have UMAP, which stands for Uniform Manifold Approximation and Projection. It preserves aspects of PCA (preserving the global structure of data) and t-SNE (local relationships between points) and can be useful for curved or non-linear manifold in a higher dimensional space.

What does that last part mean? Gemini tells us to imagine a dataset:

“Where the data actually follows a pattern that’s not easily captured by flat lines. Think of a sheet of paper crumpled into a ball; it still has structure, but it’s not flat. … Real-world datasets often have complex patterns within them, not just simple linear relationships. UMAP’s ability to handle these complexities gives it the potential to create more informative and accurate 2D or 3D visualizations, helping us see and understand the data better.”

– Gemini Advanced response

You can see an example of a UMAP created by Anthropic for features in Claude, as covered in our article on feature neighborhoods in LLMs.

3. CAVs (concept activation vectors)

CAVs help us understand which concepts in a machine learning model are “activating” for a particular input. CAVs are vectors for concepts in a visual model (as opposed to vectors in semantic search).

CAVs are summaries of the numbers that make up a vector based on training data. So, let’s say we fed a model a dataset of cat images labeled as such. Each would be turned into a data representation that eventually becomes the vector.

As for the “activating” part, here’s an analogy from Gemini:

“Let’s say you provide an image of a fluffy dog. Here’s what might happen inside:

• Dog CAV Projection: The internal state gets projected onto the “dog” CAV, resulting in a large value (strong projection).
• Cat CAV Projection: That same internal state gets projected onto the “cat” CAV, resulting in a much smaller value (weak projection).

This result would tell you that the “dog” concept is strongly activated by the input image, compared to the “cat” concept.”

– Gemini Advanced response

Limitations of these means of interpretability

Now that we understand these three ML interpretability methods, let’s look at why they’re “fairly narrow in scope,” as the ELM paper says.

Gemini explains these techniques have limited understanding of meaning and lack of direction manipulation.

Regarding t-SNE (a dimensionality reduction technique) and UMAP, Gemini explains that they “excel at revealing how data points cluster or relate in relation to each other. However, they don’t inherently explain the meaning of those clusters or what the dimensions they’re displayed in might represent.”

As for CAVs, these “give us a glimpse into which concepts a model activates,” however “they’re still specific to that particular model and the dataset it was trained on” and so “They might not generalize well to other models or domains.”

As for direction manipulation, Gemini explains, “These techniques primarily focus on visualizing or analyzing existing embeddings.”

ELM as a solution

Let’s now look at what the paper suggests, instead of the previously mentioned interpretability techniques:

“As an alternative to such interpretability methods, suppose one could engage with embeddings using natural language to query information not directly expressed by the name or description of the underlying entity/concept. Perhaps one could even extract information from the embedding representations of non-existent or hypothetical entities.”

In this Figure 1, the researchers show us an image where ELM explores hypothetical versions of real movies or plot lines from two movies merged together:

How does ELM work?

Let’s explore what the paper says about how ELM works:

“This paper introduces a novel framework to interpret domain embeddings by leveraging the power of LLMs (Devlin et al., 2018; Liu et al., 2021; Google et al., 2023). Our method seamlessly introduces embeddings into LLMs by training adapter layers to map domain embedding vectors into the token-level embedding space of an LLM, which in turn allows one to treat these vectors as token-level encodings of the entities or concepts they represent.”

Here’s how Gemini helps explain how this works:

“The core idea is to introduce embeddings into the LLM using special adapter layers. These layers are like small translator modules trained to convert the embedding vectors (which are numerical) into a format that the LLM can understand,” i.e., the “token-level embedding space.”

– Gemini Advanced response

The Embedding Language Model (ELM) is, therefore, the “novel language model framework” that uses the special adapter layers (“trained adapters”) to “accept domain embedding vectors as parts of its textual input sequence to allow interpretation of continuous domain embeddings using natural language.”

Lastly, the model is fine-tuned on domain-specific embeddings using the MovieLens 25M dataset mentioned earlier.

Types of embeddings

Two types of embeddings were used, including “behavioral embeddings, trained based solely on user ratings of movies,” and “semantic embeddings … generated using textual descriptions of movies.”

1. Behavioral embeddings

Behavioral embeddings were trained using “matrix factorization (MF).” According to Gemini, this “is a technique used to decompose a large matrix into two smaller matrices that capture the essence of the original data.” The example it gave was taking all of a user’s movie ratings from 1 to 5 and representing “latent factors” that “capture the underlying aspects,” like if that user prefers action movies.

The algorithm used in the matrix factorization was “Weighted Alternating Least Squares (WALS).” This is an iterative algorithm that can help fill in missing values in the original matrix and update latent factors in the two matrices in an alternating fashion to minimize cost function.

Here’s the full explanation from Gemini, as I think it helps us grasp the training aspect:

“Weighted Alternating Least Squares (WALS) is a commonly used algorithm for matrix factorization, especially in recommender systems. Here’s how it breaks down in this context:

Imagine the Scenario:

• You have a large dataset represented as a matrix. This matrix could represent user-movie ratings, where rows are users, columns are movies, and each cell contains a rating (e.g., 1 to 5 stars).
• This matrix is often sparse, meaning many cells might be empty because not every user has rated every movie.

The Goal of Matrix Factorization:

• You want to decompose this large matrix into two smaller matrices that capture the underlying structure of the data.
• These two smaller matrices are called the “latent factor” matrices.

What are Latent Factors?

• Latent factors represent hidden variables or characteristics that contribute to the ratings in the original matrix.
• For example, in the movie rating case:
  • The user latent factor matrix might capture user preferences like “action movie lover,” “comedy enthusiast,” etc.
  • The movie latent factor matrix might capture movie characteristics like “thrilling,” “hilarious,” etc.

How WALS Works:

WALS is an iterative algorithm that works by filling in the missing values in the original matrix and updating the latent factors in an alternating fashion. Here’s the basic process:

1. Start with an Initial Guess: Begin with some initial guess for the values in both latent factor matrices. These initial values can be random.
2. Fix One, Update the Other:
   • Fix the user latent factors (one matrix).
   • Update the movie latent factors (the other matrix) by minimizing a cost function (often the squared difference between predicted and actual ratings). This update considers the fixed user factors and the existing ratings in the original matrix.
   • Then, fix the movie latent factors and update the user latent factors in the same way.
3. Repeat and Refine: Repeat steps 2a and 2b iteratively. With each iteration, the latent factors get refined, and the predicted ratings in the original matrix get closer to the actual ratings.
4. Stopping Condition: The process continues until a stopping criterion is met, such as a maximum number of iterations or when the change in the cost function becomes very small.

Benefits of WALS:

• Scalability: WALS can handle large and sparse matrices efficiently.
• Accuracy: It can achieve good accuracy in predicting missing ratings.
• Interpretability: The latent factors can provide insights into user preferences and movie characteristics.

Limitations of WALS:

• Cold Start Problem: It may struggle with new users or movies for which there isn’t enough data to learn meaningful latent factors.
• Local Minima: The algorithm might get stuck in a local minimum of the cost function, leading to suboptimal results.

Overall, WALS is a powerful and widely used algorithm for matrix factorization, particularly in recommender systems. It helps us understand the hidden structure within user-item interaction data and make predictions about unseen ratings.“

– Gemini Advanced response

2. Semantic embeddings

The semantic embeddings were trained on a pretrained dual-encoder language model (DLM) and generalizable T5-based dense retrievers.

“More specifically, we concatenate plot descriptions and reviews for each movie, and input these to the DLM. We then average the resulting output vectors to generate the semantic embeddings.”

A DLM is a neural network used to compare two pieces of text, such as the textual description of a movie and a query or prompt to determine their semantic similarity.

Meanwhile, T5 refers to specific DLM architecture based on a T5 model, and dense retrievers “are a type of retrieval system that relies on dense vector representations.”

Conclusion of the paper

In their conclusion, the researchers explain how:

“Taken together, our results suggest that ELM offers a powerful, flexible mechanism for understanding, navigating and manipulating complex embedding representations.”

Here is the full paper, “Demystifying Embedding Spaces Using Large Language Models,” if you wish to read or download it yourself:

It’s also available directly via Google Research or Google Scholar.

Now let’s have some fun and speculate how ELM could apply to SEO.

What might this paper suggest for SEO’s future?

This is the hypothetical and thought exercise part of the post. These are hypotheses, not instructions or claims. 🙂

Let’s start with parts of the paper that actually pertain to search or digital marketing.

I saw two that were loosely relevant.

We all know how big Google Shopping has become in many facets of ecommerce SEO.

The first was an example of how ELM could be used in the realm of ecommerce:

“Consider, for instance, an embedding space representing items from an online commerce site, trained using a large corpus of user ratings or purchases, reviews, and other multi-faceted data sources. The embedding representation of an item may implicitly embody intricate details about its quality, usability, design, customer satisfaction, etc. Moreover, suppose one wanted to understand the properties of a hypothetical item at a specific point in the embedding space, say, if we predicted a potential market for such an item. Since no such item (or description) exists, asking a conventional language model to describe this embedding point would be futile. However, a large language model (LLM) trained to interpret the embedding representation itself could handle such a task.”

As SEOs, we’re attuned to look at products in terms of keywords or general concepts (entities). Embeddings can offer even richer ways to think about and describe products.

Beyond a product’s brand or physical characteristics, like style, color, size, or materials, we could understand details based on user ratings and reviews (customer sentiment), purchase histories (accompanying products), and other details.

ELM could also help us translate that product information into a textual form to enrich product descriptions, going beyond basic keyword descriptors or manufacturer descriptions to richer, more unique, and audience-specific content. (Related to this topic, we’ve also covered USER-LLM, which creates embeddings from user interaction data for LLM personalization.)

The other relevant excerpt from the paper was simply a mention of embedding queries and documents in the same latent space. This speaks to vector-based search, or enabling search engines to match which documents are most relevant to a user’s query (search intent) based on semantic similarity (rather than lexical or keyword matching).

Combined with user data as embeddings, this could lead to more personalization in search.

Implied uses of ELM by search engines

For this portion, we’ll use Gemini 1.5 Pro in Google AI Studio.

I’ve uploaded the research paper PDF to the chat prompt and simply asked, “how the concepts in this paper could relate to Google Search,” along with outlining the hypothetical nature of the prompt and more context.

This is just a thought exercise. I believe Google has a joke search engine called Fred. We’ll just call ours “our hypothetical search engine.”

Gemini gives us four potential applications of ELM in our hypothetical search engine.

• Understanding and Interpreting Search Queries: such as reformulating ambiguous or complex queries or classifying queries into user intents for more targeted or personalized results.
• Enhancing Search Result Relevance: such as matching documents and queries based on semantic rather than lexical relationships or personalizing results with user-specific information (e.g., search history or browser behavior)
• Explaining and Interpreting Search Results: such as automatically generating snippets or summaries of search results to provide users with a concise overview
• Exploring and Navigating Information: such as generating related searches or recommended additional content based on a current query and the underlying embedding space or even creating interactive visualizations of search results.

Again, these are only hypotheticals. 😉

The connection of ELM to ranking results

What I did next was ask Gemini to look at Danny Goodwin’s article on how Search works from December 2023 for Search Engine Land, “How Google Search and ranking works, according to Google’s Pandu Nayak,” to draw out connections to ELM.

Here’s what Gemini came up with:

• Interpreting Ranking Signals: ELM could be used to analyze “hundreds of algorithms and machine learning models” and associated signals and “provide human-understandable explanations of why a particular page ranks highly for a query.” (Wouldn’t that be nice!)
• Understanding Deep Learning Systems: Deep learning systems like RankBrain, DeepRank, and RankEmbed BERT are considered “black boxes,” yet ELM could analyze the inner workings of such systems to provide insights into how they interpret queries and documents and determine page rankings.
• Analyzing User Feedback and Click Data: ELM could analyze click data in conjunction with other signals to identify patterns or relationships to improve the search experience for users.
• Personalizing Search Results: ELM could create personalized embeddings for users to tailor search results.
• Evaluating Search Quality: ELM could augment human raters and Information Satisfaction (IS) metrics, helping identify biases in ranking algorithms.

Again, these are all just hypotheticals generated by Gemini 1.5 Pro, but it’s meant to help us grasp how vast and complex a modern search engine could be, and the value looking at AI research papers could have for giving us context.

See you next time!

I hope you’ve enjoyed this installment of Hamsterdam Research!

This was the first article in the series, so I’ve since adjusted the process a bit and updated this paper to reflect some of that.

You can find more related research articles below, and expect a new post about every week.

Until next time, enjoy the vibes:

Thanks for reading. Happy optimizing! 🙂