Woolr

Introduction

This week Google released EmbeddingGemma 2, their latest open embedding model, and so I wanted to give it a poke. Embedding models sit underneath most of the search and RAG systems out there, so a new one is always worth a look.

Fair warning: this is a gentle explore, not a benchmark. I pick a few questions, run small experiments, and follow whatever looks interesting. As such, it's can be a bit rabbit-holey and has spawned a handful of follow-up posts that sit as stubbed out drafts for now. Anyway.

The model in brief

I won't say too much here because you can read it all in the original blog post and the model card. What I will say is this is a proper good model, fair play Googs. Embedding space is 768 dims, the model is only 740M params, so runs easily on a standard laptop. 8K context is solid, frankly we're always pushing up against just how much useful context about an eight thousand token long passage of text can you encode in a representation with just only 768 dimensions? (We can run experiments on that and might do so in a follow-up post.)

Something that has become very popular in new embedding model releases is the inclusion of task-specific prefixes. Although task prefixes trace back to models like T5 (2019) where every task had a dedicated text prefix, they've really proliferated and arguably become the norm since 2022. Embedding models like E5 with its query: and passage: prefixes and nomic-embed's search_query: and search_document: are some of the first we used in production that saw a significant bump in performance when including the prefixes.

Back to EmbeddingGemma 2. Tasks are split into two categories: asymmetric tasks and symmetric tasks. Symmetric have the same prefix on both (or all) pieces of text being compared, asymmetric do not. In the asymmetric cases the document always gets the prefix title: {title} | text: {content}. In the below table I show the task-specific prefixes.

First pokes

Reading through the model card raised a few questions so let's have a gander at the following: do the prefixes earn their keep, where's the similarity floor, and how much of the vector can you throw away while retaining usefulness?

Prefixes: Do they matter?

The TL;DR - yes they matter, but for search at least, not as much as you might think. Prefixes make sense, but how impactful are they? and what happens if your system uses them inconsistently? These are questions I ask because I've worked on systems that had to adapt and work with both query and document prefixes.

Here I decided to explore the prefixes wiht some a simple example query: "What causes the northern lights?". Let's define a positive example, one that answers the question directly, a hard negative example, one that is the same topic but does not answer the query and an easy negative, a completely unrelated sentence. To demonstrate an often overlooked aspect of these models I'll also include a "garbled" sentence of nonsense tokens and a version of the positive shuffled.

Let's look at the cosine similarity of the query against each of these passages. And, let's also look across various prefix set-ups. What we care about most is the margin column, which is the positive minus the hard negative - the bigger this is the better the model is able to discern the correct answer, roughly speaking.

What surprised me here is that although the intended set-up (Query + Document) wins, it's not by much. The gain is genuinely there, but not a big as I would have expected. Another interesting point, only having a prefix on the query side is basically the same as having no prefixes as measured by margin here. Using the STS prompt on both sides inflates every score, negatives included - not surprising, but it confirms that a wrong prefix hurts more than no prefixes.

I should say, I'm taking all these results with a grain of salt. You would want to repeat over many such samples and show the conclusions hold, which I might do in another blog post but for now I'm just 'avin a poke.

My favourite result here is the garbled row scoring better than the easy negative in every scenario. In other words, a random sentence about bread is further away from the query than nonsense is. Should nonsense really be more similar than a coherent but unrelated sentence? The semantics of nonsense are empty by definition, but they still have to land at some point in the space. High dimensional spaces are strange, and cosine similarity as a distance measure is a perhaps a rougher proxy for semantic similarity than it's ubiquitous use (and undeniable practicality) suggests.

Another fun result - the shuffled positive passage, despite being essentially word salad still beats the hard negative. This one I did dig into a bit more, I ran 20 shuffles each of the positive and the hard negative and looked at the mean.

Shuffling the positive only loses us 0.07 on average, and all of the 20 shuffles beats the hard negative. The embeddings care more about which words are there and less about the order they are in. A probable artifact of mean pooling, and it's history in bag of words. Which brings me to the key takeaways from this section, prefixes perturb the margin by hundredths of a point, while shuffling the words cuts it from 0.11 to about 0.04, so the content itself dominates. And if the word order doesn't really matter, small edits that change the semantics but keep the same words ought to be hard for the model to spot.

Where is the similarity floor for this model?

The garbled row got me wondering where "nothing in common" actually sits for this model. So I took the 12 passages and 4 queries from a little search test I ran, embedded them with the STS prompt, and looked at every pairwise cosine.

That heatmap uses the query and document prompts, the way you'd run a real search. For the numbers below I re-embedded all 16 texts with the symmetric STS prompt, so every text could be compared with every other, queries with queries included.

The least similar pair in the whole set still scores 0.69. Mortgage rates vs a question about how sentence embedding models work, and the model reckons they're 0.69 alike. This is the usual anisotropy story: everything gets squeezed into a narrow cone, so cosines start well above zero and only the gaps between them carry any meaning. It also explains the garbled string, or at least my hunch about it. Nonsense has no meaning to push it in any particular direction, so it sits inside the cone, closer to its axis than almost any real sentence, while meaningful text lines the rim. More on that in a bit!

Subtract the mean vector from everything, though, and the floor drops to around zero. That's handy if you're visualising or clustering a fixed corpus. Just don't expect a magic threshold that says "relevant" for raw scores. The floor also moves with the prefix: the same easy negative from earlier scores 0.72 under STS but 0.55 under the intended query + document prefixes.

Where does nonsense sit?

So I went and tested it. I took 5,000 real sentences from Wikipedia and worked out their centroid, the average direction of real text, which is the axis of the cone. Then I threw five flavours of nonsense at it: keyboard mash, word salad (ten random real words, no grammar), strings of symbols, minimal inputs like "", "." and "the", and real sentences with their words scrambled. The question is simple: how close does each one sit to that axis?

Every flavour of nonsense sits closer to the axis than a typical real sentence. Keyboard mash and the minimal inputs are the extreme: the median mash string is more central than essentially all 5,000 real sentences, and even the least central one beats 99% of them. And there's a nice ordering to it. The less meaning, the more central: mash and the minimal inputs closest to the axis, symbols just behind, word salad (real words, no syntax) a bit further out, and scrambled sentences (real content, wrong order) further still. Meaning is what pushes text out towards the rim. That's the garbled string from earlier explained. Sitting near the axis means being moderately close to everything, so nonsense ends up nearer my query than a sentence about bread, which has committed to a direction of its own.

Looking straight down the axis makes the picture clearer. Real sentences form a thin ring: their median sits 41° from the axis, and 98% of them fall between 35° and 46°. Meaningful text really does line the rim. Keyboard mash and the minimal inputs sit well inside that ring at around 31°, symbols at 35°, and the empty document is closest of all at 28.5°. Word salad (37°) and scrambled sentences (39°) land in between, pulled outwards by their real words. One surprise: the nonsense isn't spread evenly around the axis. Mash and symbols are bunched off to one side, and the minimal inputs and the empty document lean another way. So "the middle" is more a handful of no-content regions near the axis than a single point. I'd also wondered whether nonsense might just form its own "garbage cluster" somewhere off to the side. It does cluster, and tightly: keyboard mash strings are 0.88 similar to each other, against 0.57 for real sentences. But those clusters sit near the axis. Pile things up near the middle and of course they end up close together.

My last suspect was the prompt. Every document gets title: none | text: stuck on the front, and mean pooling includes those tokens. So when the text is empty of meaning, maybe the prompt is all that's left. It's a reasonable guess: an empty document, the prompt and nothing else, sits at 0.88 to the centroid. But dropping the prompt entirely barely changes anything, and nonsense is just as central without it. So it's the model itself that puts meaningless input in the middle, not the prefix.

Matryoshka: how much can we chop off?

EmbeddingGemma 2 is trained with Matryoshka representation learning (MRL), so in theory you can chop the 768-d vector down to 512, 256 or 128 dims and keep most of what matters. To check, I took every pairwise similarity in my little corpus at full size and asked how well the ranking of those similarities survives truncation (Spearman's ρ against the 768-d version). As a control, I did the same with randomly chosen dimensions instead of the leading ones. If MRL is doing its job, the leading dims should win.

And they do, from 128 up: 0.83 vs 0.63 at 128, and 0.98 vs 0.93 at 512. Below 128 it flips, and the leading dims do worse than a random pick. Which makes sense once you remember the model was only trained down to 128. Nobody ever asked the first 32 or 64 dims to stand on their own. Retrieval held up at every size, with the right passage still on top, but four queries is about as easy a test as you can set, so I wouldn't read much into that.

I half expected the leading dims to hog the variance, but they only do so mildly. The first 128 dims carry about a quarter of it, against a sixth if it were spread evenly. So the MRL gains come more from what those first dims encode than from how much they vary.

One gotcha worth knowing: truncate_dim=128 slices the vector but doesn't re-normalise it. My 128-d query came back with a norm of 0.57, not 1. model.similarity hides this because it computes a proper cosine, but if you push these vectors into a store that uses dot product, your scores will quietly be off. Pass normalize_embeddings=True as well, or normalise them yourself.

Exploring the space

Something I always like to do for embedding models is to see how the space is actually laid out. So in this section I take a brief look at how topics cluster, and then probe the multilingual aspects somewhat - does the same sentence in different languages ends up in the same place.

Clusters

It's always fun to plot topic clusters, of course the projection process here also plays a big role in how the clusters look - but it's still interesting to see where clsoe clusters and far clusters land and where specific samples within those land w.r.t others. The numbers further down are all computed on the full 768-d vectors, though, so they don't depend on the projection.

The setup: 50 short sentences, ten each on astronomy, physics, cooking, nutrition and software. I paired a couple of topics on purpose, astronomy with physics and cooking with nutrition, to see whether the model keeps near neighbours apart. Then I encoded the same 50 sentences under each prompt and scored how cleanly the topics separate with a silhouette score (higher is better).

Remember "for search at least" from earlier? This is why it was there. For clustering, the prefix makes a big difference. The Clustering prompt scores 0.26, nearly double no prompt at all (0.15), and every other prompt sits in between or below. The same pattern as the search test shows up too: a wrong prompt, like query, document or STS, does slightly worse than none. The model card says to use the Clustering prompt for clustering, and on this evidence it's right.

Zooming out to topic level, a few things jump out. First, the floor again: even the least related pair of topics, astronomy and software, averages 0.74. Second, astronomy and physics are the closest pair (0.82), as you'd hope. Cooking and nutrition are less clear-cut. Nutrition is about as close to physics (0.80) and astronomy (0.79) as it is to cooking (0.80). Which, looking at my sentences, is fair enough: "Vitamin D is produced in the skin when exposed to sunlight" is a science fact first and a food fact second. And software is the loner: the least similar to every other topic, but also the loosest topic internally (0.80 within, against 0.85–0.88 for the rest).

Multi-Lingual

The model card claims 100+ languages, so I gave it a quick sanity check. I took my northern lights sentence in eight languages (English, French, German, Spanish, Japanese, Chinese, Hindi and Swahili) and added a control: an unrelated sentence about buying groceries, in English and in French. If the model is doing its job, the translations should all score high with each other, well above the controls, and the two controls should match each other.

It passes comfortably. The translations score 0.84–0.98 with each other, while the grocery controls sit at 0.69–0.73 against all of them, so even the weakest translation pair beats every control by 0.11. The two controls match each other at 0.94, as they should. A couple of things I liked. French and Spanish are the closest pair (0.98), which is about what you'd expect from two Romance languages saying the same thing almost word for word. And Japanese and Chinese score 0.94 against English despite sharing no characters at all. After the word-salad results earlier, it's nice to see that whatever the model is doing here, it can't be matching on shared tokens. That doesn't contradict the word-salad result, though: within one language the model can lean on which words are there, but across scripts it has nothing to lean on except meaning. Swahili is the weak link, at 0.84–0.89 against everything else. That's still clearly a match, but it fits the card's warning that quality varies by language, and Swahili is likely the least-resourced of the eight. Usual caveat: this is one sentence, not a benchmark.

Next steps

That's enough for one post, below are a couple of the avenues that I have explored that I want to write follow up posts for.

First, hard negatives. The word-order result has been nagging at me: if the model mostly cares which words are present, then edits that flip the meaning but keep the words should be hard for it to spot. Spoiler: they are. A negated answer scores about as close to the query as a genuine paraphrase does. And that matters in real search systems: a query like "I want to watch a movie that isn't scary" lands right next to "I want to watch a movie that is scary", so the user can get exactly the opposite of what they asked for. I've run the same probes across nine open embedding models, and it isn't just EmbeddingGemma 2. That's the next post.

Second, multimodality. Everything here has been text, but the whole pitch of this model is images, audio and video in one space, and I'm excited to dig into that next.