Woolr

Introduction

In September TypeSafe AI released Jev, a model that doesn't quite generate text like regular LLMs. You give the model a fixed set of typed choices, and it hands back one with an associated probability, which is exactly what you want for things like classification and routing. Shortly after, Nish Tahir wrote a lovely post, Build your own decision model (code in build-your-own-jev), showing how to get something similar out of an open LLM.

I started by replicating that post, mostly to pad out my own understanding, and then went down the rabbit hole of asking the practical question: the next time I need to build an intent router, how does a homemade decision model stack up against the other ways of building one?

Intent routing is choosing which handler should get a user's request, e.g. sending "turn the lights off" to the smart-home handler. It's often the first step of an agentic system, and it's a decision with a fixed set of answers, so it's a natural fit for a decision model.

I built four routers and ran them on exactly the same test requests. Because they all see the same requests, every comparison below is paired, and the numbers in square brackets are a 95% paired bootstrap interval on the difference. That means resampling the test requests with replacement 10,000 times, recomputing the accuracy difference each time, and taking the middle 95% of the results. If the interval excludes zero the difference is clear; if it straddles zero, the two can't be told apart. Fair warning before we start: it's a handful of models on two datasets, so none of this has been tested beyond them - just an exploration.

The four contenders

For the three approaches that don't use labels, the big design choice turns out to be the route text, which is to say what you call the routes and how you describe them. I tested each one with the datasets' original route names (e.g. qa, iot) and with routes that have a clearer name and a one-line description (e.g. factual_questions: factual questions, word definitions, maths, stock prices and currency exchange rates). I'll call the second set renamed and described throughout. I had a frontier LLM write it, once per dataset, from each dataset's intent taxonomy. Spoiler: it matters a lot.

The data

• CLINC150 (Larson et al., 2019), grouped into its 10 domains plus an out-of-scope route for requests for which no handler exists to handle. The test set is a sample of 100 requests per domain plus 400 out-of-scope, so 29% are out-of-scope. That share was set by the sampling and overall accuracy depends on it, so I also report in-scope accuracy and out-of-scope recall.
• MASSIVE (FitzGerald et al., 2022), English, at the scenario level: 18 routes, full test set of 2,974 requests.

The trained classifiers used each dataset's training split: all 15,250 labelled requests for CLINC150 (only 250 of them out-of-scope) and a 4,000-request sample for MASSIVE. Local models ran on an M3 Max.

Results

I'll just cut to the chase here are report the results, if you're interested there's plenty of discussion further down the post.

CLINC150

With renamed and described routes, Haiku 5.5 was the most accurate router on CLINC150, and logistic regression on EmbeddingGemma 2 vectors came second. The best local decision model, Qwen3-4B, came in behind both.

The embedding model's prompt, logistic regression's regularisation and the number of neighbours were chosen on a 400-request validation sample.

The paired differences between the leaders:

• Haiku 5.5 was 6.0 points [4.1, 8.1] more accurate than logistic regression, and 16.1 points [13.7, 18.4] more accurate than Qwen3-4B.
• Logistic regression was 10.1 points [7.5, 12.6] more accurate than Qwen3-4B.
• Zero-shot EmbeddingGemma 2 was indistinguishable from Qwen3-1.7B, −0.1 points [−3.0, 2.6], and 5.9 points [3.2, 8.6] behind Qwen3-4B.

Which of Haiku and logistic regression is "better" depends on the request. On in-scope requests, logistic regression was 2.4 points [0.8, 4.0] more accurate than Haiku. On out-of-scope requests, Haiku's recall was 27.0 points [21.8, 32.2] higher. Not hugely surprising when you look at the training split: 250 out-of-scope examples against 1,500 for each domain. The classifier is great at what it has seen lots of and poor at "none of the above".

MASSIVE

On MASSIVE, Haiku 5.5 with renamed and described routes reached 90.4%, which was 10.8 points [9.5, 12.2] above Qwen3-1.7B with the same route text (79.6%). With the original names, Haiku got 76.4% and Qwen3-1.7B 58.7%. A linear layer trained on Qwen3-1.7B's internal state with 4,000 labelled requests reached 89.4% (more on that later), though I didn't compare it with Haiku on paired requests.

I've left EmbeddingGemma 2's MASSIVE results out of the comparison. It hit 89.0% zero-shot with MASSIVE's bare route names, which is suspiciously good, and a check found signs it may have been trained on MASSIVE (see the appendix). Its MASSIVE numbers may be measuring memory of the dataset, so only CLINC150 is used to compare the embedding approaches.

What each approach needs and gives

Accuracy isn't the only thing that matters when you pick a router. Here's what I measured, I didn't really measure speed given I was actively using my laptop for other stuff while running these experiments. Of course, latency and cost are practical considerations that once ought to take into account.

Can you trust the confidence?

This is the bit I think is easiest to overlook. A decision model hands you a probability for free, and it's very tempting to take it at face-value, but it isn't calibrated. The "probability" is a softmax over a handful of route scores, and it's badly overconfident. On MASSIVE, Qwen3-1.7B's raw expected calibration error (ECE, the average gap between confidence and accuracy) was 0.28 to 0.64 across the scoring methods I tried. The fitted temperatures were 2.6 to 9.7, so the scores needed dividing by up to about 10 before the confidences meant anything.

The fix is simple-ish: fit one temperature on held-out labelled data (Guo et al., 2017). That brought ECE down to 0.02 to 0.06 on MASSIVE, and to 0.05 to 0.07 on CLINC150 for the three local LLMs (JSON, renamed and described routes). Once scaled, the best decision model was as well calibrated as the trained classifier: on CLINC150, ECE was 0.046 for both Qwen3-4B and logistic regression on EmbeddingGemma 2, and 0.089 for zero-shot EmbeddingGemma 2. Haiku 5.5 gives you no confidence, so there's nothing to calibrate.

Two catches:

• "Zero-shot" stops being zero-label. The temperature was fitted on 1,000 labelled validation requests for MASSIVE and 400 for CLINC150. If you're going to act on the confidence, budget for a few hundred labelled requests even with a router that needs none to make decisions.
• Calibrated on validation isn't calibrated in production. Temperature scaling makes the confidence match accuracy on the data you fitted it on. If live traffic doesn't look like that data, the confidence drifts with it, which is exactly what happened to the thresholds below.

Route text matters for every approach

Rewriting the route names and adding descriptions raised accuracy for every model that reads route text, with one exception: EmbeddingGemma 2 on MASSIVE (see the appendix, it was already so good that - sus). The trained classifiers don't read route text, so they're not in this comparison (see the aside at the end of this section).

With the original names, all three label-free approaches were below 63% on CLINC150. Even a frontier-ish API model is only as good as the route names you give it. For the local LLMs, a control showed the gain depends on each description actually matching its route (below), and route text written by a model recovered part of the gain (below).

Could the route text help the classifier?

I've shown that route names and descriptions improve every other approach that I tested that uses them, so it stands to reason that giving them to the classifier could help too. There are a few ways it could be done:

• Pairwise scoring: score each query together with each route's text
• Prior weights: start the weights from the route-text embeddings
• Blended scores: mix the zero-shot and trained scores
• Extra features: add the zero-shot similarities as inputs

These are not things I've tried!

How many labels does a classifier need?

On CLINC150 I trained logistic regression on EmbeddingGemma 2 vectors with 1 to 128 labelled requests per route, 5 random draws at each size. The mean first passed Qwen3-4B with renamed and described routes (72.2%) at 32 per route, 352 labelled requests in total. At 16 per route, none of the 5 draws was distinguishable from Qwen3-4B. Every size stayed below Haiku 5.5.

The prompt and regularisation here are the ones chosen on validation with all the training data, since a handful of labelled requests can't choose them. The prompt is the short prefix EmbeddingGemma 2 puts in front of every request (task: classification | query: ), which tells it what the vectors are for. The regularisation is logistic regression's strength setting C, which was 100 here. That is the weakest of the four values tried (0.1, 1, 10 and 100), so the fit is only lightly held back from the training requests.

Confidence thresholds mostly missed their target

One of the selling points of a router with a confidence score is selective routing (Geifman & El-Yaniv, 2017): route the confident requests and send the uncertain ones to a fallback, using a threshold picked on labelled validation data. In most conditions where I tried it, the threshold chosen for a 90% accuracy target gave less than 90% on test.

• Local LLMs on MASSIVE gave 85.7% to 90.4%, and 1 of 12 conditions met the target.
• Local LLMs on CLINC150 with renamed and described routes gave 84.0% to 89.5%.
• Logistic regression on CLINC150 routed 99.5% of test requests at 82.5% accuracy. Its out-of-scope recall was 71% on the validation sample and 45% on the test sample, so overall accuracy was 91.8% on validation and 82.3% on test. Part of that is a real difference between CLINC150's validation and test splits (the in-scope accuracy of two of the LLMs was also 3 to 7 points higher on validation), and part is the usual winner's curse from picking a threshold on 400 requests. I could not separate the two and wasn't arsed to dig deeper.

I only tried models up to 4B parameters that return token probabilities, so I can't say whether larger models behave differently. The trained classifier missed too, so the shortfall isn't specific to small LLMs.

The ranking of confidence was still useful even where the threshold didn't transfer. Logistic regression's area under the accuracy-vs-coverage curve was 0.952 on CLINC150, against 0.897 for Qwen3-4B with renamed and described routes. Important to note; Haiku (like any API-based model that doesn't return token probabilities) return no confidence, so they can't be used this way at all.

Building your own decision model

So those are the results, the rest of the post is the deep dive into the local LLM approach, for anyone who wants to build one. I started from the replication of Nish's post, which builds a decision model on CommonsenseQA, and then tested each setup choice on intent routing.

Jev's internals aren't public, so I can't say how it does this. A homemade decision model built on an open LLM, like the one in Nish's post, works in four steps:

1. The prompt lists the allowed answers. In most of my experiments it ends with the first part of the model's reply, a prefill, e.g. The correct domain is: ** or {"domain": ".
2. Score each allowed answer at the position after the prompt. For a one-token answer like a letter, that's the next-token probability. For a route name that spans several tokens, either add up the log-probabilities of its tokens or apply the same rule constrained decoding uses.
3. Normalise over the allowed answers only. The top one is the decision and its probability is the confidence. (That's my definition. Public descriptions of Jev say it returns a separate confidence value for how concentrated the distribution is, which isn't the same number.)
4. Divide the scores by a temperature, fitted to minimise log loss on held-out labelled data (Guo et al., 2017). This changes the confidence but never the decision. Jev is described as being trained for calibrated probabilities. This is the after-the-fact version, and it needs labelled data.

A couple of terms that you'll need as you read through the below:

• Constrained decoding: generation where, at each step, only tokens that can still lead to an allowed answer are permitted. Libraries like outlines (Willard & Louf, 2023) use it to make output match a JSON schema.
• JSON-schema enum: a schema field whose value must be one of a fixed list of strings, e.g. a Pydantic Literal of route names.
• ECE: expected calibration error, the average gap between confidence and accuracy across confidence bins. Lower is better, 0 is perfect.
• Coverage: in selective routing, the share of requests that get routed rather than sent to the fallback.

Local scoring is deterministic, so each local condition is one run. For every experiment from the route-text ones onwards I wrote the hypotheses down before running the models, and the CLINC150 run used a dataset and two models that played no part in designing those hypotheses. There are a lot of intervals in this post, so expect a few borderline "clear" results by chance.

Where you read the answer

Nish's post makes the decision at the first token of the model's reply. The prompt is the question, the options and Answer:, sent as one user message through Qwen3's chat template with thinking disabled, and nothing after it. The script takes the logits at the last prompt position, keeps only the five for the letters A to E, and softmaxes over those five. Top letter wins.

I got the same numbers: 60.0% accuracy against 59.4% in the post, and 62.6% against 62.4% after the post's head fine-tuning. So far so good. But I was curious what the model actually wanted to say at that position, before the softmax over the five letters. Here's the prompt for one question:

<|im_start|>user
Where would you most likely find a bat?
A. Cave
B. Baseball game
C. Attic
D. Zoo
E. Sporting goods store
Answer:<|im_end|>
<|im_start|>assistant
<think>

</think>

Qwen3-1.7B's most likely first token was "The" for 1,220 of 1,221 questions. The five letters together had a median probability of about 1.3 in 10 million. The method is picking the "least unlikely" letter from a distribution where every letter is basically noise, and it still gets 60%! So much signal in the long tail.

The prompt never says how to format the answer, so the model wants to open with a sentence. Letting it generate (greedy, up to 24 tokens), 941 replies began with The correct answer is: and 24 with The best answer is: . 173 replies (14%) didn't state a letter within 24 tokens at all, and 166 of those began by restating the question.

So the obvious fix is to give it that opening sentence. The prefill read adds The correct answer is: ** to the end of the prompt and reads the five letters at the next position. After the prefill, the five letters had a median probability above 0.99.

On all 1,221 questions the prefill read was 6.2 points [4.0, 8.4] more accurate than the first-token read, and it agreed with the generated letter 98% of the time. This replicates what Wang et al. (2024) found across instruction-tuned models: first-token probabilities often don't match the answer the model would actually write, largely because of conversational preambles. The prefill wording was the model's own most common opening on these same questions, and I didn't try any others; one model, one prompt.

The opening sentence depends on the prompt, too. Replacing Answer: with Answer with only the letter (A, B, C, D, or E). gave the letters a median probability of 1.0 at the first reply token, and the first-token read then got 65.5%. Adding a one-line system message (You answer multiple-choice questions with a single letter and nothing else.) gave 67.7%. I didn't run generation with these prompts.

On intent routing the problem is much bigger. On MASSIVE with 18 routes, the first-token read got 17.1%, where always answering the most common route (calendar) gets 13.5%. With the prefill The correct domain is: it got 50.0%, a difference of 32.8 points [30.7, 35.1]**.

Answering with a letter vs the route name

On MASSIVE with the original route names, scoring the route name beat scoring a letter for Qwen3-1.7B. Same instruction, same prefill, each name scored by its first token: 50.0% with letters and 60.8% with names, a difference of 10.8 points [9.4, 12.1]. Names were also far more stable: the share of requests that got the same route under 4 different option orders was 30% with letters and 58% with names.

How big the gap is depends on the route text. Scoring each name by the sum of its token log-probabilities, names minus letters was:

• +10.7 points [9.3, 12.0] with the original names
• +4.1 [2.8, 5.4] with new names only
• −1.1 [−2.3, 0.1] with descriptions added, which is indistinguishable from zero
• +1.5 [0.5, 2.6] with descriptions and new names

On CLINC150 with renamed and described routes, letters vs the JSON format came out differently for each model:

So there's no universal winner on format. Letters do have a well-known failure mode though. On CommonsenseQA, Qwen3-1.7B answering by letter picked B for 29% of questions, where each letter is the right answer about 20% of the time, and only 34% of questions got the same answer under all 5 rotations of the option order. That's the selection bias towards particular option IDs that Zheng et al. (2024) describe across 20 LLMs, showing up again here on a small Qwen.

JSON-schema enums and a structured-output library

If you're using a structured-output library you might wonder whether reading log-probs yourself is the same thing. On a 300-request sample of MASSIVE, reading the log-probabilities at the enum value gave the same decision as outlines' greedy enum decoding for 300 of 300 requests, with the identical prompt. MASSIVE's original route names all start with different tokens, so constrained decoding makes its decision at the first token of the value. On the full test set, scoring each name by the sum of all its token log-probabilities instead changed 6.8% of decisions, but the change in accuracy, +0.4 points [−0.4, 1.2], was indistinguishable from zero.

Two library settings did change results on those 300 requests, and both are easy to trip over:

• Double templating. Passing an already-templated prompt to outlines wrapped it in the chat template twice. That changed 3.7% of routes and cost 2.3 points [0.7, 4.3].
• The idiomatic way, minus one flag. Passing the plain user message and letting outlines apply the chat template, as its docs show, changed 3.3% of routes and also cost 2.3 points [0.7, 4.3]. Here the prompt lacks the empty <think> block, because outlines doesn't pass enable_thinking=False to the template.

And with Qwen3's shipped sampling defaults, two identical runs disagreed on 3.7% of requests [2.1%, 6.4%]. If you want a decision model, turn sampling off.

Route names and descriptions

This is the big one. On MASSIVE with Qwen3-1.7B, changing only the route text took accuracy from 58.7% to 79.6% in the JSON format. Renaming means replacing terse names with longer ones (qa became factual_questions, iot became smart_home_devices); describing means adding a one-line description of each route.

To see why, look at what happens to qa with the original names: in the route-name format, 87% of factual questions went to general. The model has no idea what qa means as a route.

On CLINC150, renamed and described routes beat the original names for all three models, and all 15 tests written down before the CLINC150 run were supported.

The obvious objection is that descriptions just make the prompt longer, and maybe any extra context helps. Hence the shuffled control: each route keeps its name but gets another route's description. Accuracy with shuffled descriptions was lower than with no descriptions at all in every model and format, by 3.8 to 33.1 points on CLINC150 and 6.3 to 37.4 points on MASSIVE. So the gain depends on each description matching its route; added prompt length alone doesn't produce it. (A small length effect can't be ruled out, because shuffled descriptions are actively misleading rather than neutral.)

A few more details from the CLINC150 runs:

• Descriptions alone, keeping the original names, added 16.5 to 27.1 points in JSON.
• Renaming on top of descriptions added a further 4.1 to 6.4 points overall, mostly through out-of-scope recall, which rose 13.0 to 18.5 points. On in-scope requests the change was +1.6 (indistinguishable), −1.6 and +3.8 points for the three models.
• On in-scope requests alone, renamed and described routes were 23.6 to 25.4 points more accurate than the original names.
• With the original names, Phi-4-mini sent 0 of 400 out-of-scope requests to the route called oos, and Qwen3-4B sent 3% of them in JSON. With renamed and described routes (out_of_scope: requests that none of the other domains can handle) the figures were 17% and 50%.
• With the original names, Qwen3-4B was 1.7 points less accurate than Qwen3-1.7B in JSON (41.0% against 42.7%), but 10.2 points more accurate with letters. Bad route text can make a bigger model look worse.

The shuffled condition also tells you which part of the route text each answer format pays attention to. On MASSIVE, letter answers picked the route whose description fit the request 57% of the time and the one whose name fit 13% of the time. Answers given as route names picked the matching name 54% of the time and the matching description 22%; in JSON it was 52% and 26%. On CLINC150, letter answers again followed the description in all three models (46% to 64%, against 4% to 12% for the name). In JSON it varied by model: Qwen3-4B followed the description 42% of the time and the name 26%, while Phi-4-mini followed the name 33% and the description 24%. Which makes sense: when you answer with a letter, the name is just another bit of text on the line; when you answer with the name, the name carries the weight.

Letting a model write the route text

A frontier LLM with the full intent list is one way to get good route text, but can a small local model do it from a few examples? I had Qwen3-4B write a name and description for each MASSIVE route, given all the route names and 8 example requests per route, with three different random draws of examples. Every generated set was clearly better than the original names, by 7.6 to 20.7 points, and clearly worse than the frontier LLM's set, by 3.4 to 13.2 points.

The catch-all route has 189 true requests in the test set; counts are from the JSON format.

The three generated sets were 10 points apart in JSON, and the catch-all looks like the culprit. The set that predicted its catch-all least often had the highest accuracy, and set 3 sent 72% of factual questions to its catch-all. A route called general is a magnet. With only three sets that link is an observation, not a tested result. And to be fair to Qwen3-4B, the frontier LLM worked from MASSIVE's full intent taxonomy, while Qwen3-4B only saw 8 examples per route.

Calibration and thresholds

Nish's post found the decision model badly overconfident, and that replicated closely. On CommonsenseQA, the questions in the top confidence bin (0.9 to 1.0) averaged 98.6% confidence but were only 70.4% accurate; the post reports 98.6% and 70.1% for the same bin. Overall, mean confidence was 88.1% against 60.0% accuracy, an ECE of 0.28. Overconfidence after post-training is well documented: the GPT-4 technical report showed the pre-trained model was well calibrated and post-training made it less so. Qwen3-1.7B is a post-trained chat model, though I didn't test its base model to check that's the cause here.

The post fits its temperature on the validation set, which is also the set it reports results on, so the calibration it reports is a little optimistic. I refit it on a separate 1,000-question calibration split carved out of the training data: T = 3.87 against 3.82, and ECE 0.044 against 0.039 (interval 0.030 to 0.074). So the leak didn't matter much here. The fine-tuned head was overconfident too, less so: ECE 0.18 before scaling and 0.04 after.

On MASSIVE, confidence exceeded accuracy for every scoring method (letter, route name and JSON enum), with ECE of 0.28 to 0.64. A single temperature fitted on 1,000 validation requests brought that down to 0.02 to 0.06, without changing a single decision.

Renamed and described routes also made the confidence more useful, raising the area under the accuracy-vs-coverage curve by 6.2 to 17.6 points on MASSIVE across three formats and by 23.1 to 32.1 points on CLINC150 across three models in JSON.

As covered above, thresholds picked on validation data mostly undershot on test. On MASSIVE, with 500 validation requests and a 90% target, test accuracy was 85.7% to 90.4%; 1 of 12 conditions met the target and 2 of 12 had intervals reaching 90%. On CLINC150 with renamed and described routes it was 84.0% to 89.5%. Part of the CLINC150 shortfall is the samples themselves: overall accuracy was 2.7 to 6.8 points higher on the validation sample than on the test sample for Qwen3-1.7B and Phi-4-mini, so it can't all be pinned on how the threshold was chosen.

Training on labelled data

If you do have labels, you can train a linear layer on the LLM's frozen final hidden state, which is what Nish's post does. On MASSIVE, with 4,000 labelled requests, it reached 82.8% to 89.6% depending on the prompt format and how the layer was initialised. In JSON it reached 89.4%, against 79.6% for the best zero-shot route text (point estimates, not a paired test).

Initialisation matters at the extremes. With 1 labelled request per route, starting the layer from the model's own output weights gave 68.2% against 54.9% from zeros, averaged over 5 random draws. From 4 per route upward, starting from zeros won on all 5 draws at every size, by 3.5 to 4.7 points on average.

On CommonsenseQA the trained layer added 2.5 points [0.2, 4.8]. Simply changing the prompt to ask for only the letter, plus that one-line system message, added 7.7 points [5.2, 10.2], though it also made log loss worse (2.1 to 5.0). I didn't compare the two gains with each other directly, but it's a nice reminder to fix the prompt before reaching for training.

Conclusion and practical recs

Overall this has been a fun exploration, I am trying to be better about time-boxing work like this. You turn over a rock only to find more rocks you want to flip, it's rocks all the way down. Practically, this work reaffirms a few things I think are mostly self-evident, but it's always good to verify. I would say they are:

• Name and describe your routes properly, and check each description fits its route. Renamed and described routes beat the original names by 22.9 to 31.2 points on CLINC150 for the three local LLMs, and by 14.0 points for Haiku.
• Fix the prompt and where you read the answer before you reach for training. Reading the answer after a prefill was 6.2 points more accurate than the first-token read on CommonsenseQA, and a one-line prompt change beat training a linear layer there (7.7 points against 2.5).
• Calibrate the confidence, but leave a margin on any threshold. One temperature, fitted on a few hundred labelled requests, brought ECE down to ~0.04 without changing a single decision. Thresholds picked the same way still mostly landed under the 90% target on test (84.0% to 89.5% on CLINC150), so aim a bit above what you need.

Limits and what's next

Some things I might want to do next if I carry on down this path:

• Jev itself wasn't tested. TypeSafe lists its price as $0.042 per million input tokens with free output, and end-to-end latency of 70 to 500 ms. Those are the vendor's figures.
• Few models. The local LLMs are 1.7 to 4 billion parameters, and there's one embedding model and one API model. A fine-tuned encoder (e.g. ModernBERT) wasn't tested.
• Few routes. The datasets have 11 to 18 routes. More routes, and route names that share their first tokens (e.g. calendar_set and calendar_query), are untested.
• One prompt for Haiku. Haiku 5.5 got a prompt written for the local models, and I only tried that wording. MASSIVE and CLINC150 are public, and whether Haiku saw them in training is unknown.
• Speed. Only measured for EmbeddingGemma 2. For the local LLMs on CLINC150, time per request with renamed and described routes was 3.2 to 4.8 times that with the original names in batched fp32 runs, measured as a side effect of the main runs. Longer prompts aren't free.
• The renamed and described route text was written once per dataset by a frontier LLM, using each dataset's intent taxonomy, and no other wording was tried.

Appendix: did EmbeddingGemma 2 see MASSIVE in training?

EmbeddingGemma 2 behaved very differently on the two datasets. On MASSIVE it reached 89.0% zero-shot with the bare route names, and descriptions added only 0.3 points [−0.6, 1.2]. One labelled request per route gave 86.5% with logistic regression (mean of 5 draws), and the learning curve was flat from 4 per route upward. On CLINC150 the bare names gave 34.4%, and descriptions added 32.0 points.

MASSIVE's intent and scenario classification tasks are part of the MTEB benchmark, and the model card doesn't list the fine-tuning data. If that data paired MASSIVE requests with their label names, the model would match requests to those names from memory. (I had a poke at the model more generally in Exploring EmbeddingGemma 2.)

So I held the request vectors fixed and changed only the route names, with no descriptions. A model relying only on what the names mean should do at least as well with clearer names as with MASSIVE's terse ones.

The clearer names are the ones used on MASSIVE in the route-text experiments, e.g. factual_questions, local_recommendations, smart_home_devices and device_volume.

MASSIVE's own names were 2.4 points [1.3, 3.5] more accurate than the clearer names. With the bare name qa, EmbeddingGemma 2 recalled 93.8% of factual questions, while Qwen3-1.7B recalled 2.4% with the same name. It isn't uniform, though: MASSIVE's audio gave 54.8% recall against 90.3% for device_volume. The check is consistent with training on MASSIVE but doesn't prove it. Until the training data is known, I wouldn't use EmbeddingGemma 2's MASSIVE results to compare it with other routers.

References

• Tahir. Build your own decision model (code). 2026.
• InfoQ. TypeSafe AI releases Jev. October 2026.
• Larson et al. An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (CLINC150). EMNLP 2019.
• FitzGerald et al. MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages. 2022.
• Wang et al. "My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models. Findings of ACL 2024.
• Zheng et al. Large Language Models Are Not Robust Multiple Choice Selectors. ICLR 2024.
• Guo et al. On Calibration of Modern Neural Networks. ICML 2017.
• OpenAI. GPT-4 Technical Report. 2023.
• Geifman & El-Yaniv. Selective Classification for Deep Neural Networks. NeurIPS 2017.
• Willard & Louf. Efficient Guided Generation for Large Language Models. 2023.
• Muennighoff et al. MTEB: Massive Text Embedding Benchmark. EACL 2023.