Woolr

Introduction

I wanted to see how far a small model could get on one narrow job if I fine-tuned it properly. The job is to take a question like "tarantino's worst, please" and turn it into a JSON filter:

{"director": "Quentin Tarantino", "sort": {"by": "rating", "order": "asc"}}

The filter compiles to SQL and runs against a DuckDB database of 95,254 IMDb titles. The model is Qwen3.5-0.8B, and everything from building the data to training and evaluation runs locally on an M3 Max with MLX.

I picked this task for two reasons. First, it's the same kind of problem as text-to-SQL, tool calling and API routing, where a model turns natural language into a constrained query and a program then runs the query. Second, I can score it without an LLM judge. The main metric is execution accuracy, which checks whether the predicted filter selects the same rows as the correct one. Two filters can be written differently and both be right, and execution accuracy counts them both as correct.

The short version:

• Fine-tuning takes execution accuracy from 24.4% to 92.6% on a generated test set (n=500), and from 13.6% to 70.5% on 44 questions I wrote by hand. The baseline is the same model prompted with three examples.
• Qwen3.5 is a hybrid model, and only 6 of its 24 blocks use full attention, so it isn't obvious where the LoRA adapter should go. On the test set, the placement made no measurable difference across a 9.5x range in adapter size.
• On the handwritten questions, the variant that also adapts the GatedDeltaNet layers scored worst of the four.
• When I read the 13 handwritten failures, I found that 5 were mistakes in my labels and my SQL compiler, not the model. One of the model's own errors gets the sort right but inserts a completely different director ("Pier Paolo Pasolini" for Tarantino).

Results

Fine-tuning compared with prompting

Every condition gets the same system prompt, which lists the full schema and the 27 allowed genres. I compare against the few-shot prompt, which adds three worked examples, because it's the fairest baseline.

The fine-tuned model gets 3.8x the few-shot score on the test set and 5.2x on the handwritten set. Against zero-shot, the ratios would be 11.9x and 10.4x. The adapter has 6.39M trainable parameters, which is 0.85% of the model's 752M. I trained it for 300 iterations on a training set of 2,619 examples, which took 33 minutes.

The way the base model fails surprised me. I expected it to invent genre names, e.g., "Science Fiction" for "Sci-Fi", but with no examples in the prompt it produces only 5 invented genres and 4 unknown field names across the 500 test questions. Zero-shot, 92.4% of its outputs parse as JSON, but only 55.8% pass the schema. Of the 456 schema errors, 429 are the same mistake, a range field (year, runtime or rating) given a plain value or a list where the schema wants an object, e.g., "runtime": 120 instead of {"lte": 120}. The model also fills in fields the question never mentions, e.g., "year": 2024, so it treats every field as required when the schema makes them all optional.

Adding three examples to the prompt fixes the format, and schema validity goes up to 97.6%. Execution accuracy only reaches 24.4%, though, because the filters are well-formed but select the wrong titles. So fine-tuning has to fix the meaning of the filters, not their format.

I checked these numbers for leakage and for selection bias. No training question appears in either evaluation set, and no training filter appears in the test set. One training filter does appear in the handwritten set, with a different question.

The 70.5% has some selection bias, because it's the best of four adapter variants (see the next section). Validation accuracy tied three ways, so I picked the variant partly on the handwritten set. I had also looked at the handwritten scores during the learning-rate sweep, before I settled on choosing by validation. Across the four variants, the handwritten score ranges from 54.5% to 70.5%, so the fairest summary is that fine-tuning scores somewhere in that range on questions in my own words, against 13.6% for few-shot.

Where to put the LoRA adapter

Qwen3.5 is a hybrid model. Of its 24 blocks, 6 use full attention (blocks 3, 7, 11, 15, 19 and 23), and the other 18 use GatedDeltaNet, a form of linear attention. The usual LoRA target list is the attention projections (q, k, v and o) plus the MLP projections (gate, up and down), so on Qwen3.5 it adapts attention in only 6 blocks and leaves the GatedDeltaNet layers alone.

Borobia et al. compared LoRA placements on Qwen3.5-0.8B-Base, the pretrained checkpoint of the model I use (I fine-tuned the post-trained Qwen3.5-0.8B). On GSM8K, attention-only LoRA (1.08M parameters) scored 0.398, against 0.375 for adapting every layer and 0.297 untrained. Adapting only the GatedDeltaNet layers dropped the score to 0.148, below the untrained model. They recommend attention-only LoRA as the default. I wanted to see whether their result holds on the post-trained model and a task with structured output, so I trained four variants and kept everything else fixed (learning rate 1e-4, scale 2.0, rank 16, 300 iterations, one seed each):

On the test set, none of the differences are significant. Exact McNemar tests, paired on the same 500 questions, give p ≥ 0.405 for every pair of variants. So a 1.08M-parameter adapter on the 6 attention blocks does as well as a 10.22M-parameter adapter on every layer type. The result agrees with the paper's recommendation on a different task, since the small attention-only adapter shows no measurable loss against the larger ones.

On the handwritten set, the variant that also adapts the GatedDeltaNet layers scored worst. Attention + MLP gets 7 questions right that the all-layers variant gets wrong, and the reverse never happens (p = 0.016). The difference is the only significant one, but it doesn't survive a Bonferroni correction across the six comparisons (0.05/6 = 0.0083). I also trained each variant only once, so I don't know how much a score changes from one training run to the next. The result agrees with the paper's advice to leave the recurrent layers alone, but it's only suggestive evidence.

Attention + MLP is the target list most tutorials use. I included it expecting it to lose to the all-layers variant, but it scored best on the handwritten set, so I used it for every other number in this post.

If you try this with mlx-lm, note that its LoRA keys match the dotted path inside a block (self_attn.q_proj), not the bare name (q_proj). A list of bare names matches nothing, and the only sign of the problem is Trainable parameters: 0.000% in the log. A list that matches only some of the layers would be worse, because training would run without any error. I worked out the expected parameter count for each variant by hand, and all four match what mlx-lm reports to the digit.

What the model still gets wrong

The fine-tuned model misses 13 of the 44 handwritten questions, and 7 of the misses are errors by the model:

The Tarantino question is my favourite, because the model got the hard part right and the easy part wrong. It worked out that "worst" means sorting by rating in ascending order, but it wrote the wrong director. Nothing in the setup stops the model from writing a plausible director who isn't Tarantino. The Spielberg question shows a related problem, since the model put the name in the wrong field. A fix for both is to look names up in the database instead of having the model generate them.

Five of the remaining six misses were mistakes in my evaluation:

• Three questions fail only because the model added "type": "movie" and my label had no type. In this schema, a filter with no type matches any title type, TV included, and English has no natural way to ask for every title type. For "war film, three hours plus" and "meryl streep films rated above 7", the model's reading is arguably better than mine.
• For "anything Kubrick directed, newest first", the model wrote "Kubrick", and my SQL compiler only matches full names.
• The question "top 5 Scorsese pictures by score" needs both fixes, because the model wrote "Scorsese" and added a type. The model also added a Crime genre, but the top five results are the same.

With both fixes and no retraining, the handwritten score is 36/44 = 81.8%. The sixth miss, "twenty family films from the 2000s", is arguable. My label has a limit with no sort, so any 20 matching rows count as correct, and the model added a sort by year. My data pipeline already treats a limit without a sort as unnatural, so my label breaks my own rule.

The same type ambiguity also caused problems in the data pipeline, as the next section describes.

Methodology

Generating the training data from the filters

The obvious way to get training pairs is to have a large model write both the question and the filter. The problem is that language models are least reliable at writing structured output that exactly fits a schema, which is the filter half of each pair. Their mistakes are also systematic, so the same mistakes would appear in both the training and test sets, and the metrics would not show them.

So I generated the pairs in the other direction. Just like back-translation in machine translation (Sennrich et al., 2016), where a model writes a synthetic source sentence for a real target sentence, a program samples the filter first and the language model writes the question. The filter in each training pair always comes from the program, so it always fits the schema. The pipeline works as follows:

1. Sample a filter from the real data, using directors with at least 4 credits, actors with at least 6 credits and genres with at least 200 titles.
2. Run the filter, and drop it if it returns no rows or more than 20,000.
3. A teacher model (Qwen3.5-4B, 4-bit) writes a question for the filter.
4. A lexical check tests whether every constraint leaves some trace in the question. It matches genres through a synonym table, people by surname, and numbers as digits or phrases ("90s", "nineties").
5. The teacher translates its own question back into a filter, without seeing the original.
6. Keep the pair only if both filters select the same rows.

Step 6 is the check on meaning. Deciding whether a question matches a filter would normally need a human or a judge model, but running both filters turns the decision into a comparison of two sets of rows, which the database answers exactly. The idea is close to execution-guided decoding in text-to-SQL (Wang et al., 2018), which runs partial queries to throw out bad candidates during generation. Here, the check runs when building the dataset instead of at inference. Moreover, a pair is kept under the same rule as the metric I report, so the data filter and the evaluation agree on what "correct" means.

The lexical check is cheap and runs first to save time, but it misses some errors. Here are some teacher outputs from early test runs that the pipeline rejected:

The question "listed from best to worst" contains the word "best", so it passes the lexical check, and only the round trip (steps 5 and 6) finds that the sort order is reversed.

The first run turned 7,000 filters into 2,421 pairs, a yield of 35%. Directors and actors were under-represented in the kept pairs, with 12% of pairs having a director or actor against 18% of the handwritten set. When I measured the yield at each stage, filters with a person passed the teacher stages more often than the rest, so the teacher wasn't the cause. The loss came earlier, in step 2. A specific director combined with a genre, a decade and a minimum rating often matches no titles, so step 2 dropped those filters before the teacher saw them. A second run of 2,200 filters, weighted toward directors and actors, added 998 pairs (a yield of 45%). The two runs gave 3,419 pairs in total, which I split into 2,619 for training, 300 for validation and 500 for test. Generation took about 3.2 hours on the M3 Max.

The most useful lesson from the data pipeline was that a high rejection rate pointed to bugs in my generator, not to a filter that was too strict. Twice I assumed the round trip was too harsh, and both times the rejected pairs showed that the round trip was right:

• Filters without a type match both films and series, but English has no natural way to say "any title type". The teacher wrote "movies", and the round trip correctly rejected the pair. The same ambiguity cost three of the handwritten questions.
• The sampler produced filters with a limit but no sort, e.g., "give me 10 action films from the 80s", and any natural way of asking for 10 films implies an order.

A style rule of my own also cost a lot of data. To stop the teacher writing flat questions that just read out the JSON, I rejected any question starting with "What are the" or "Show me". In a pilot of 128 generations, the rule alone threw away 28% of them, including good pairs. The rule was a style preference, but I had applied it like a correctness check, so now the pipeline accepts questions with formulaic openers until they make up 15% of the kept set.

The main limitation of the approach is that round-trip filtering only keeps questions the teacher can already solve, so test accuracy partly measures how well the student learned the teacher's way of phrasing things. The handwritten set exists to measure the gap, and the difference between 92.6% and 70.5% is a rough measure of the cost.

Training

I trained LoRA at rank 16, applied to all 24 blocks, with scale 2.0 and dropout 0.05. The learning rate was 1e-4, with 30 warmup steps and cosine decay to 1e-5, and training ran for 300 iterations at batch size 4, which is just under half an epoch. The loss covers only the assistant's answer. The system prompt makes up about 75% of every example and never changes, so training on it would mostly teach the model to predict text it always receives. Validation loss goes from 0.674 to 0.037 in the first 100 iterations and ends at 0.012.

If you use mlx-lm, note that the learning rate and the scale work together. 1e-4 is a normal LoRA learning rate in peft, where the multiplier alpha/r is usually around 2. mlx-lm applies its scale directly and defaults to 20, so the same learning rate takes steps about ten times larger. At 1e-4 and scale 20, validation loss fell from 0.719 to 0.350 by iteration 60 and then jumped to 2.883 at iteration 120, just after warmup ended. It was still at 0.551 after 600 iterations, while at scale 2.0 training was stable. I then compared three learning-rate and scale pairs at 150 iterations and picked 1e-4 at scale 2.0 on validation accuracy (88.3%, against 86.7% and 87.3%).

The sweep also shows why I chose by accuracy rather than loss. At 1e-5 and scale 20, the run ended with the lowest validation loss of the three (0.016) and also the lowest validation accuracy (86.7%).

Memory was the second problem. A 0.8B model ran out of memory on a 96 GB machine, because of its 248,320-token vocabulary. The logits tensor has size batch × sequence length × vocabulary, which comes to 4 GB in bf16 at batch 8 and length 1,024, before the backward pass. My examples are 295 to 349 tokens long, so I cut the batch size from 8 to 4 and the maximum length to 384. I also turned on gradient checkpointing. Gradient checkpointing usually slows training down, but at batch 4 the speed was 210.9 tokens per second against 212.6 without it, and peak memory fell from 69.9 GB to 13.1 GB.

Evaluation

Each prediction gets four scores, from loosest to strictest about form:

• parse: is it valid JSON?
• valid: does it pass the schema?
• exact: is it the same filter as the label, after normalising?
• execution: does it select the same rows (in the same order, when there's a sort)?

Execution accuracy is the main metric, because different filters can select the same rows, and a user only cares about which rows come back.

There are two evaluation sets. The first is 500 generated questions held out from the pipeline above. The second is 44 questions I wrote myself in phrasings the teacher doesn't use, e.g., "tarantino's worst, please" and "I've got ninety minutes and I want a thriller". Every handwritten label is schema-valid and returns at least one row. With only 44 questions, the 95% confidence interval is about ±14 points, so I use paired tests to compare variants.

I chose configurations by execution accuracy, not validation loss. sciences44/mlx-lora-finetune fine-tuned the same 0.8B model for text-to-SQL with MLX and found that validation loss was misleading when comparing models. Their 4B model had the lowest validation loss but ranked third on accuracy. Their 0.8B model reached 47% semantic accuracy on general SQL, judged by hand. General SQL is a much broader task than my filter language, which is a large part of why my numbers are higher, so the two results aren't directly comparable.

Practical considerations

In a production system, the model wouldn't run on its own, and checks around it would catch some of the errors in this post. The post measures the model by itself, so it's fair to ask which results still matter once the checks are in place.

Constrained decoding would catch the format errors. It forces the model's output to follow the JSON schema, so the model can't produce invalid JSON, an unknown field or a genre like "Long Epic". A name check would catch most of the name errors, by comparing each name in the filter with the question and looking it up in the database. For example, the check would flag "Pier Paolo Pasolini", because the question says "tarantino", and a fuzzy lookup would turn "Kubrick" into Stanley Kubrick. A similar rule could flag the title search for "spielberg", since Spielberg is a known director.

Checks on the output wouldn't catch errors in meaning. Reading "I've got ninety minutes" as at least 90 minutes is wrong, although it gives a valid filter with real values that returns rows, so nothing in the output looks wrong. The same is true for "kids stuff that isn't animated" turned into Animation, the dropped Documentary genre and the extra title search for "anime". The "Long Epic" filter would also still fail under constrained decoding, because the model set the minimum runtime to 120 minutes instead of 200. Of the 7 model errors on the handwritten set, the name check would catch 2, and the other 5 are errors in meaning.

The main result holds with the checks in place. Constrained decoding fixes the format, but the format isn't what holds the baseline back. The few-shot baseline already passes the schema 97.6% of the time and still reaches only 24.4% execution accuracy. The fine-tuned model passes the schema 99.6% of the time on the test set, so constrained decoding would add little to it. The gain from fine-tuning comes from choosing the right filter, and neither a schema check nor a name check can improve the model's choice.

Some of the other problems in the post only come up while building the model. The learning-rate setting, the out-of-memory error and the bugs in the data pipeline affect training, not a deployed system. The type ambiguity is a design decision about the schema. A production system could also ask the user a follow-up question when a filter returns no rows or when the question is ambiguous.

What I'd do next

Here are the next steps, in order of value for the time they take:

1. Fix the two evaluation defects. Make type required, with an explicit "any" value, and apply the change to the sampler, the teacher prompt and the labels. Also let the compiler match surnames. Together, the two fixes take the handwritten score to 81.8% with no retraining, and they should also raise the pipeline's yield.
2. Test whether the system prompt is still needed. The fine-tuned model has probably learned the schema. If accuracy holds with a one-line prompt, I can cut most of the input tokens on every request, which would be a strong practical argument for fine-tuning over prompting.
3. Strengthen the ablation. Train each variant with three seeds, and write about 100 more handwritten questions to narrow the interval to roughly ±8 points.
4. Add the name check between the model and the database. As described in the practical considerations above, it would catch errors like the Tarantino one.

I left more training off the list. I have two all-layers runs with the same learning rate and scale, one trained for 150 iterations in the sweep and one for 300 in the ablation. The longer run raised the test score from 88.0% to 93.6%, but the handwritten score fell from 59.1% to 54.5%. The drop is only 2 questions, but it suggests extra training was already fitting the teacher's phrasing more than the task. The remaining errors come from schema ambiguity and names, not from the model's capacity.

References

• Borobia, Seguí-Mas & Tormo-Carbó. Where Should LoRA Go? Component-Type Placement in Hybrid Language Models. arXiv 2026.
• Sennrich, Haddow & Birch. Improving Neural Machine Translation Models with Monolingual Data. ACL 2016.
• Wang et al. Robust Text-to-SQL Generation with Execution-Guided Decoding. arXiv 2018.
• sciences44/mlx-lora-finetune. GitHub.
• Code for this post: woolr/llm-finetuning.