Woolr

Introduction

I ended the last post on EmbeddingGemma 2 on a teaser: if the model mostly cares which words are present and not what order they're in, then edits that flip the meaning but keep the words should be hard for it to spot. Spoiler: they are. But "embedding models are bad at negation" is old news and not very interesting on its own. What I think is more interesting is the pattern of what the model notices and what it doesn't.

One of the main use cases of these models is search. Training models for search often requires mining hard negatives: passages that almost answer the query, but don't. Here's the one we'll use a bunch in this post (adapted from the one used in the EmbeddingGemma 2 model card):

EmbeddingGemma 2 scores the moon version slightly above the real answer (0.870 vs 0.867). Yeah, it's a tiny margin, too small to mean much either way, but the point is it's no worse. And it's not just Gemma: all eight other models I tried rate the moon nearly as close to the query as the sun. It's a nice illustration of a model reading the topic rather than the answer.

Think about what this means for a search system: an agent running a semantic search with this query against a vector DB will happily take the moon passage as the answer. Or take the example from the end of the last post: to EmbeddingGemma 2, "I want to watch a movie that isn't scary" and "I want to watch a movie that is scary" are near-identical (cosine 0.92 with the STS prompt). That's two sentences compared with each other rather than a query against a passage, but the same thing happens in search: as we'll see below, a passage that differs from the right one only by a "not" scores about as well as the right one.

Hard negatives are the expensive, valuable part of a fine-tuning set, so knowing which ones a model actually finds hard tells you where to spend. That's what this exploration ended up being about: what makes a wrong answer a hard negative, and what does "hard" really mean?

A word on scale before diving in: everything here grows out of seven hand-written query/answer pairs (I'll call them anchors), with short, single-sentence answers. From those come 674 one-word edits, a few thousand random-walk steps and nine models, so I'm fairly confident in patterns that hold across every sentence and every model. Where a result rests on one or two examples, I'll say so.

Defining "hard"

Raw cosine is a pretty bad measure. For EmbeddingGemma 2 a random unrelated sentence already scores around 0.53 against a query. Just like NV-Retriever's positive-aware mining, which judges a candidate negative by its score relative to the positive's, I measure hardness relative to the true answer, and to a background of random sentences, so the scale means the same thing from query to query. With 5,000 random sentences from Natural Questions Wikipedia passages as a background, for a negative n:

h = 1 is as close to the query as the real answer, h = 0 is as close as a typical random Wikipedia sentence, above 1 beats the answer.

For discrete labels I borrowed the contrastive-loss view, following Wang & Liu's point that the loss is hardness-aware, with the temperature controlling how much close negatives count. In InfoNCE (MultipleNegativesRanking, if you live in sentence-transformers) each negative's weight relative to the positive is exp((s_neg − s_pos)/τ), so with the common τ = 0.05:

For EmbeddingGemma 2 that puts the hard line at h ≈ 0.88–0.91 and the medium line at h ≈ 0.61–0.69, depending on the anchor. τ = 0.05 is my assumption about the loss, not a property of the model. That's fine for comparing one model's negatives with each other, but the bands don't carry over cleanly between models, which comes back when I compare nine of them.

I also flagged whether each negative would make the top 5 of the 5,000 background sentences. Nearly everything on-topic did, so on a generic corpus "would it surface in search?" says little about hardness; the gap to the positive does the discriminating.

Probing EmbeddingGemma 2

Nine ways to be wrong

Seven query/answer pairs: northern lights, who wrote Pride and Prejudice, soft-boiled eggs, ibuprofen and alcohol, boiling point, who bought Instagram, and who won the 2018 World Cup. For each answer I hand-wrote nine variants. Here they are for the northern lights:

As an independent meaning judge I used an NLI cross-encoder (nli-deberta-v3-base): does the edit contradict the original, follow from it, or neither?

I'll call the five edits that break the answer but keep the topic (negation, entity swap, number swap, role reversal, predicate swap) the logical edits. Together they average h ≈ 0.92, with negation, number swaps and role reversals highest at about 0.95. NLI calls every negation and every number swap a contradiction (though it oddly calls most role reversals entailment), yet six of seven negations stay hard, and a sentence and its own negation have cosine 0.93–0.98.

The paraphrase control averages 0.92, so you could say negation scores as close as or closer than a real paraphrase. I wouldn't lean on that. Five to seven points per row, wide overlapping ranges (paraphrases run 0.78–1.04), and a confound: my negations keep every word of the original, while my paraphrases share only 25–75% of them ("Auroras happen when solar particles collide with gases…"). Part of what this chart shows is lexical overlap, not just negation-blindness. The headline survives either way: flip the meaning, keep the words, and the model barely moves.

This is a replication, not a discovery. NevIR showed bi-encoders do about as well as random at ranking a passage above its negation, CANNOT showed the same insensitivity in sentence-embedding-based metrics, and a 2025 paper calls it "negation blindness" in general-purpose text embeddings. Good to see it hold in a 2026 model; the more interesting bits come next.

Hardness falls off a cliff when the topic changes

How does a hard negative become an easy one? I built chains of edits, one or two small changes per step, from the true answer to off-topic. Hover the points to read each sentence.

Northern lights: sun → moon and h goes up (inverted); particles → dust, still inverted; then northern → city and h drops from 1.0 to 0.57 in one step, straight from inverted to easy. Instagram falls just as sharply: Facebook → Twitter keeps it hard (0.97), then Instagram → Vine on its own drops it to 0.63, right on the edge of easy. Changing the price as well only takes it to 0.59. NLI flags the very first edit in both chains as a contradiction (p ≈ 1.00), so the meaning was gone long before the score noticed.

Both cliffs come when a word from the question changes, while the words holding the actual answer (sun, Facebook) change freely. Two hand-picked chains is an anecdote, so the next two sections try to check it more systematically.

Which words hold it together?

Word by word: for every word in each answer, ModernBERT proposed its top 8 in-context replacements, giving fluent one-word edits, and I averaged how much each position knocks h down.

aurora: “What causes the northern lights?”

Thenorthernlightsarecausedbychargedparticlesfromthesunhittingtheatmosphere.

austen: “Who wrote Pride and Prejudice?”

PrideandPrejudicewaswrittenbyJaneAustenandpublishedin1813.

egg: “How long should you boil an egg for a soft yolk?”

Forasoft,runnyyolk,boiltheeggforaboutsixminutes.

ibuprofen: “Is it safe to take ibuprofen with alcohol?”

Takingibuprofenwithalcoholincreasestheriskofstomachbleeding,sodoctorsadviseagainstit.

boiling: “What is the boiling point of water at sea level?”

Atsealevel,waterboilsat100degreesCelsius.

instagram: “Which company acquired Instagram?”

FacebookacquiredInstagramin2012foraboutonebilliondollars.

worldcup: “Who won the 2018 FIFA World Cup?”

Francewonthe2018FIFAWorldCup,beatingCroatia4–2inthefinal.

barely moves → drop of 0.4+

The colour sits mostly on query-overlap words, ones that also appear in the query (by a crude test: a content word sharing its first four letters with a query word): northern lights, ibuprofen, Instagram, 2018 FIFA World, Pride and Prejudice. The answer-bearing words barely register (sun −0.02, Facebook −0.01, Croatia 0.00, 100 −0.03, Jane Austen +0.02 and +0.06), and for several, replacing them nudges h up. Per position, the correlation between how much the embedding moves and how often NLI says the meaning flipped is weak (Spearman ρ ≈ 0.26).

That the answer words barely register is less damning than it looks, at least for fact swaps like sun → moon. The query is embedded before the model sees any passage, so the only way it can prefer sun over moon is if the query vector has already learned that auroras are solar. To check, I kept each answer sentence fixed and swapped in ten candidate answers (sun, moon, Mars…; Jane Austen, Charles Dickens…). The true answer does get an edge, but a tiny one: across nine models and six anchors it beats the average wrong candidate by 0.02 h or less, while replacing a query word costs 0.13–0.21. On their own the names fare better (every model ranks a bare "Jane Austen" first for "Who wrote Pride and Prejudice?"), so the knowledge is in there; inside a sentence full of the query's words it just gets drowned out. Negation and role reversal have no such excuse: spotting that "are not caused by" doesn't answer "What causes" takes no world knowledge, and the models miss that too.

How far to trust the world-knowledge check

It's small. Six of the seven anchors, with ten hand-picked candidates each, swapped into the true answer's sentence with everything else left alone. Ibuprofen is left out because its answer is a judgement, not a fact you can swap. Across the 54 model–anchor pairs the true answer comes first 10 times (chance would be about 5), with a mean rank of 4.0 out of 10 (chance 5.5). The two number anchors show no knowledge at all, and "six minutes" for a soft yolk is a fuzzy answer anyway, since five or seven would do. It's one run with short candidate lists, so read it as a faint lean, not a measurement.

Deleting words, rather than replacing them, tells the same story: the query's words hold the score up. Delete one word at a time from each negative:

positive

Thenorthernlightsarecausedbychargedparticlesfromthesunhittingtheatmosphere.

negation

Thenorthernlightsarenotcausedbychargedparticlesfromthesunhittingtheatmosphere.

entity swap

Thenorthernlightsarecausedbychargedparticlesfromthemoonhittingtheatmosphere.

role reversal

Thesunhittingtheatmosphereiscausedbychargedparticlesfromthenorthernlights.

predicate swap

Thenorthernlightsareblockedbychargedparticlesfromthesunhittingtheatmosphere.

topical, no answer

ThenorthernlightsarebestseenfromNorway,IcelandandFinlandinwinter.

deleting it lowers h (the word held the sentence close) · deleting it raises h (the word pushed it away)

Deleting northern sinks every row; deleting the word that made the sentence wrong (moon, blocked, not) mostly does nothing or nudges it up. Across all seven anchors a single deletion demoted 30 of 47 negatives, and 5 of the 21 hard-or-inverted ones went straight to easy. In all 30, the most damaging single deletion was a query-overlap word. Deleting not from a negated sentence moved h by +0.04 on average, at most +0.16.

So, as far as seven short sentences can tell you anything, the knob is which word you edit: change the answer and you get a very hard negative, change the topic and you get an easy one. This replicates, at the word level, what Fayyaz et al. (ACL 2025) showed far more carefully: dense retrievers favour literal and entity matches with the query and largely ignore whether the document contains the answer.

One alternative reading I can't rule out: the words a query repeats are also the sentence's central entities (northern lights, Instagram, Pride and Prejudice). So "the model cares about the query's words" and "the model cares about the main named thing in the sentence" are confounded in my seven. Separating them needs answers where the topic entity isn't in the query, which I don't have.

Another candidate is position: Fayyaz et al. also found dense retrievers favour what comes early, and in my answers the query's words often come first. That one the data can partly check. Non-query words that open their sentences barely register (Facebook −0.01, Taking 0.01), so being early isn't enough on its own; France (0.07) is the one that moves a little. Among the query-overlap words, though, earlier ones do matter more in every model, so position may amplify topic words even if it doesn't explain them. Only five non-query content words sit in the first third of a sentence, so read this as a hint.

Drunken walks: fluent edits vs word salad

Automating the hand-built chains from earlier: start at the answer and replace one random word per step, 15 steps, 15 walks per anchor, with two walkers. The fluent walker lets ModernBERT pick a replacement that fits the context, so the sentence stays grammatical while its meaning drifts. The random walker swaps in a random vocabulary word, so it's word salad within a few steps (a favourite, from step 8: "France won date cruel mitotic storey Cup, beating Croatia sale friendly chins final.").

Fluent edits stay hard longer (reader, behave): after 3 edits, 58% of fluent walks were still hard or inverted against 12% of random ones, and 30 of the 105 fluent walks never reached easy within 15 edits, against none of the random ones. Fluent walks lasted longer on all seven anchors in this run. (An earlier run of the notebook, with different random walks, had two anchors the other way round, so I'd trust the pooled gap more than the per-anchor ordering.) My guess is that junk tokens dilute the mean-pooled vector while fluent replacements stay on-topic, but I haven't tested that.

The walks back up the cliff too, if less dramatically than the hand-built chains. In a typical walk that reaches easy, the single biggest step accounts for about half the total fall (median 53%), and across all 105 fluent walks the biggest step replaced a query-overlap word in 90% of them. So the cliff and the topic words are really one finding: the score drops when the question's words go.

Is it even a negative?

This is the bit that matters most if you're generating training data. All 674 fluent one-word edits, each scored twice: by the embedding (h) and by NLI against the original answer.

Of the 674 edits, 82% stayed hard or inverted, so one fluent swap is usually enough to make a "hard negative". The catch: across all 674 edits, NLI says about 47% preserve the meaning, and among the ones that stayed hard it's 54% (a rough and probably high estimate, for reasons below). The embedding scores these meaning-preserving edits about as hard as the real contradictions (median h 0.99 vs 0.95; interquartile ranges 0.97–1.00 vs 0.85–1.00).

A few of the highest-scoring edits from each group. Every one of them scores above the real answer:

That last row is my main caveat on the 47%. NLI asks whether the edit follows from the original, and "It acquired Instagram in 2012" does, but it no longer answers "Which company acquired Instagram?". Entailment isn't "still a valid answer", and nearly half of the entailed edits just replace a function word (by → as). NLI can also err the other way, and a further 17% of edits were judged neutral. So treat 47% as a rough estimate, probably on the high side, not a measured false-negative rate. The right judge asks "does this still answer the query?", which I didn't run.

The lesson itself is well established: hard negatives are riddled with false negatives and the embedding can't reliably police them, which is why RocketQA denoised with a cross-encoder, NV-Retriever thresholds against the positive, and Thakur et al. (2025) relabel with LLM judges. What this adds is a rough size for the problem when negatives are made by one-word perturbation. The closest published relative I found is UNA, which makes negatives by swapping words chosen by TF-IDF.

The intent word does the heavy lifting

So far every edit has been to the answer. Now flip it round and delete words from the query. The logical edits barely move (mostly within ±0.1), since they contain all the query's content words anyway. The on-topic non-answers move a lot more, especially when the deleted word is the intent word, the one that says what's being asked (wrote, won, causes).

Drop wrote from "Who wrote Pride and Prejudice?" and the passage about Elizabeth Bennet jumps from h 0.83 to 1.17, overtaking the true answer. Drop won from the World Cup query and the topical passage goes from 0.72 to 0.99. The northern lights are milder: dropping causes lifts the topical passage from 0.66 to 0.76. Deleting who, what or the does basically nothing. In a larger run on five anchors (the one behind the model comparison below), removing the intent word lifted EmbeddingGemma 2's topical passages from 0.68 to 0.91 on average. So in these examples the intent word does most of the work of separating "about the topic" from "answers the question". Five anchors, so a strong hint rather than a rule.

Prompts move the goalposts

A callback to the prefixes section of the last post: swapping the query/document prompts for the symmetric STS prompt made negation noticeably less hard (mean h 0.95 → 0.81; hard-or-inverted negations went from 6 to 3). STS was the only setup that helped: with no prompts at all, negation scored 0.93, about the same as with the retrieval prompts. So measure hardness with the prompts you'll use in production.

Is it just Gemma?

Same probes, exactly the same texts, across nine open embedding models that are licensed for commercial use and small enough for a laptop (≤ ~1.2B parameters): EmbeddingGemma 2, Qwen3-Embedding 0.6B, harrier-oss-v1 (0.6B and 270M), Granite Embedding R2 311M, pplx-embed-v1 0.6B, Nemotron-3-Embed 1B, BGE-M3 and Arctic-Embed L v2.0, each with its own recommended prompts. Others were left out because of their licences (non-commercial, non-standard or gated), custom code that wouldn't load on current transformers, or size.

For context, here's where each model's floor sits, which is why everything below is in h rather than raw cosine.

Logic-blindness is everywhere

Start with the simplest test: how similar is each model's embedding of a sentence to its embedding of the negated version?

Every model treats them as near-duplicates, from 0.87 (BGE-M3) to 0.99 (Granite R2, which barely registers not), with EmbeddingGemma 2 mid-pack at 0.955.

The same goes for all the logical edits, not just negation, and it holds against the query too:

Every model puts the logical edits (negation, swaps, role reversal) at a mean h of 0.89–0.94, at least as close to the query as the paraphrase control. The bottom rows are the sanity check: topical non-answers sit at h 0.57–0.76, lexical-overlap decoys lower at 0.47–0.60, and off-topic sentences around 0 in every model. Arctic, pplx-embed and BGE-M3 sit at the low end for logical edits (h 0.89–0.90), and BGE-M3 also has the lowest similarity to its own negation, so it might read not slightly better than the rest. It's a small gap though, and nobody escapes it. Consistent with NevIR, across a 2026 line-up.

Nobody can tell a flip from a paraphrase

Back to the one-word edits, keeping only those NLI was confident about (229 contradictions, 304 entailments): can each model's hardness tell them apart? ROC AUC, 0.5 being a coin flip.

All nine land between 0.61 and 0.69, and every bootstrap interval overlaps every other, though the two ends (harrier 270M at 0.61, Granite R2 and Nemotron at 0.69) only just. EmbeddingGemma 2 is at 0.68, tied with most of the field. Those intervals are optimistic, since all 533 edits come from seven sentences, and NLI's "same meaning" label is shaky, as the Facebook → It example showed.

Topic words rule everywhere

Replacing a word that overlaps with the query costs 0.13–0.21 in h in every model. Replacing any other content word (function words excluded, so this isn't just the and by dragging the average down) costs between −0.003 and 0.011. I won't quote a ratio, because the denominators are essentially zero, but in every model the score is driven by query-overlap words and barely touched by the rest, answer words included. The one soft spot is the crude prefix test for "query-overlap".

Same caveat as everywhere else: these are still the same seven anchors, so nine models agreeing tells you about the models, not about how far seven sentences generalise.

Where they differ: how squashed the scale is

The biggest spread is in how long edits stay "hard": Granite R2, EmbeddingGemma 2 and the harrier models keep 52–70% of fluent walks hard after 3 edits, while pplx-embed, Nemotron and Arctic typically drop to easy after 4.

That's mostly my ruler, not robustness. h is normalised per model, but the hard/medium/easy bands use the raw gap with the same τ = 0.05 for everyone, so models with a narrow span between floor and positive (Granite R2 0.24, EmbeddingGemma 2 0.33, harrier 0.35–0.37) turn a given drop in h into a small raw gap and stay "hard" more or less by construction. A narrow span also makes h noisier, since a small cosine wobble becomes a bigger swing in h. In h itself the models barely differ: mean h after three fluent edits sits between 0.85 and 0.91 for all nine, and Qwen3, which decays slowest in h, is mid-table on "still hard" because its span is wide (0.64). So the models differ in how squashed their scale is, which matters for thresholds and loss temperature, not in how well they read meaning.

And for fun

No visible relationship between MTEB score and meaning sensitivity (EmbeddingGemma 2's MTEB score is self-reported). Six points prove nothing either way, but a better leaderboard number doesn't obviously buy a model that notices when the answer changes.

What I'd do with this when building a dataset

1. Hard negatives: edit an answer word, keep the topic words. Swap the entity, number or polarity and leave the query's words alone; one edit is usually enough to make it hard. Changing a topic word does the opposite, usually dropping straight to easy, which in-batch negatives already give you for free.
2. Judge every perturbation-made negative. Around half of fluent one-word edits keep the meaning, so check each with a judge that asks "does this still answer the query?", not just "does this contradict the original?". Skip function words: nearly half of the meaning-preserving edits came from swapping one.
3. Measure hardness against the positive and the model's own floor, with your production prompts.
4. Be careful with keyword-style queries. Without the intent word ("wrote", "won"), on-topic passages that don't answer the question score like the answer, and with no intent word you could argue they are answers. Keep the intent word in synthetic queries if you want the model to learn the difference.

Replications, and what might be new

Most of this replicates known results on a 2026 line-up, as flagged along the way: negation blindness (NevIR, CANNOT, the 2025 semantic-adapter paper), topic and literal matching winning over answer containment (Fayyaz et al.), and false negatives hiding among hard negatives (RocketQA, NV-Retriever, Thakur et al.).

The parts I haven't seen reported in this form (from a small search, so I may well have missed something):

• The word-level substitution and deletion view across nine current models: replacing a query word moves h by 0.13–0.21, replacing any other content word by about 0.01 or less.
• Hardness-decay walks, fluent vs random.
• A rough size for how often fluent one-word edits keep the meaning, which matters for word-replacement pipelines like UNA's.
• The intent-word deletion.

All of it is pilot-scale. To firm it up I'd want many more anchors from real QA data, longer passages, LLM-written perturbations with an "answers the query?" judge, each model's own temperature in the status bands, and the larger models.

Wrapping up

So does the moon cause the northern lights? EmbeddingGemma 2 is about as happy with that as with the real answer, and all eight other models put it nearly as close (h 0.95–1.00). Whether a wrong answer looks right comes down mostly to whether it repeats the question's words, not whether it gets the answer right. Worth remembering next time you're building hard negatives, or wondering why a search came back with a passage that says the opposite of what you asked for.

References

• Weller, Lawrie & Van Durme. NevIR: Negation in Neural Information Retrieval. EACL 2024.
• Anschütz, Miguel Lozano & Groh. This is not correct! Negation-aware Evaluation of Language Generation Systems (the CANNOT dataset). INLG 2023.
• Semantic Adapter for Universal Text Embeddings: Diagnosing and Mitigating Negation Blindness to Enhance Universality. arXiv 2025.
• Fayyaz, Modarressi, Schütze & Peng. Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence. ACL 2025.
• Qu et al. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. NAACL 2021.
• Moreira et al. NV-Retriever: Improving text embedding models with effective hard-negative mining. arXiv 2024.
• Thakur et al. Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMs. Findings of EMNLP 2025.
• Wang & Liu. Understanding the Behaviour of Contrastive Loss. CVPR 2021.
• Shu & Lampos. Unsupervised hard Negative Augmentation for contrastive learning (UNA). arXiv 2024.