We Benchmarked Five Small Local Models on 250 Real Vulnerability Descriptions. The Fastest One Lost.

Everything intelligent in the platform, from choosing which tests a change can reach to proposing a repair for a broken selector, runs on small open-source models inside our own servers. No customer code goes to a third-party API. That is a privacy promise, and it comes with an engineering obligation: if we change a model, accuracy must not move without us knowing. So we built a gate that measures it, and then we used the gate to answer a question we had been arguing about: is the fastest small model good enough?

The task and the cases

The task is closed-set classification: given the text of a vulnerability description, name the weakness class it belongs to, from a fixed list of 33. It is the kind of work our security engine does when it turns a scanner observation into a labelled finding. The 250 cases come from a public vulnerability-enrichment corpus with human labels. We strip the weakness identifier out of the description text before the model sees it, because several descriptions quote it, and a benchmark a model can read the answer from measures nothing.

The case set is pinned by a content hash. If anyone regenerates it and gets a different hash, the comparison is invalid by definition. That rule cost us nothing and has already prevented one accidental re-baseline.

How we ran it

All five models ran through the same runner on the same server: eight vCPUs, no GPU, four threads per model, seed 42, temperature zero. Every case gets one retry. An infrastructure failure, for example the container hitting its memory limit, is recorded as null rather than as a wrong answer. We added that rule after the first attempt died at roughly case 180 with an out-of-memory kill; without it, an infrastructure crash would have looked like a model that suddenly got everything wrong.

Because every model answered the same 250 cases, we can compare them per case rather than only by totals. That matters more than it sounds, and the next section is why.

Results

Look at the middle two. Their intervals overlap almost entirely, and if we had only these totals we could not say whether qwen3:1.7b is better than granite3.3:2b or just luckier on this sample. Totals with intervals tell you how uncertain each number is; they do not tell you whether two models differ.

• qwen3.5:4b: 62.8% (Wilson 95% interval 56.7 to 68.6), 15.0 tokens per second decode.
• qwen3:1.7b: 51.6% (45.4 to 57.7), 39.2 tokens per second.
• granite3.3:2b: 50.4% (44.2 to 56.5), 29.7 tokens per second.
• llama3.2:3b: 46.0%.
• llama3.2:1b: 18.4%.

What the paired test says

McNemar's test asks a sharper question: on the cases where the two models disagree, which one is right more often? Between qwen3.5:4b and granite3.3:2b there were 43 cases only the 4b model got right and 12 only granite got right, chi-squared 16.36, p below 0.001. That is not luck.

Against llama3.2:3b, qwen3.5:4b wins with chi-squared 28.45 (p below 0.001) and qwen3:1.7b wins with 4.45 (p below 0.05). Granite versus llama3.2:3b is 2.69, which is a tie. So the 3b Llama model beats nothing on this workload, and it is not the fastest either.

We also reran qwen3:1.7b from scratch. Its running accuracy matched the first run checkpoint for checkpoint, with zero flipped cases. At temperature zero and a fixed seed, that is what should happen; it is still worth checking, because "should" is not a measurement.

What the numbers do not say

• Closed-set classification of a short description is one task. It says nothing about structured output, long context, or code generation. A model that loses here could win on those, and we do not promote or demote a model on a task the gate does not measure.
• Two hundred and fifty cases put roughly plus or minus six percentage points on each accuracy. That is enough to separate 62.8 from 50.4 and not enough to separate 51.6 from 50.4 without the paired test.
• Decode speed on a shared CPU box is not a constant. In a separate warm measurement under load, qwen3.5:4b decoded at 7.8 tokens per second, about a quarter of its peers. The table number and the loaded number are both real; they were taken under different conditions, and we say which is which.

The decisions this forced

granite3.3:2b was a candidate for promotion on the argument that it is twice as fast as the incumbent. The gate said: true on speed, false on accuracy, by 12.4 points, with p below 0.001. That is precisely the trade our rule forbids making quietly, so the promotion was rejected and the reasoning is written next to the numbers.

The Llama models stay installed and licensed, as a last-resort standby only. The licence question had been settled separately; the measurement settled the engineering one. There is no corner of this workload where they win.

The slowest model sits on our interactive tier. That is a capacity risk we recorded, and did not fix on this evidence, because the interactive tier does structured output and this gate does not measure structured output. Changing it on the wrong evidence would be a different mistake.

If you run local models in a product

• Pin the case set by hash, and refuse to compare runs whose hashes differ.
• Strip the answer out of the prompt. Then check that you actually did.
• Record infrastructure failures as null. A crash is not a wrong answer.
• Compare pairwise on identical cases. Totals with overlapping intervals will mislead you in exactly the cases that matter.
• Write the decision rule before you look at the results. Ours is one sentence: the model may change, the accuracy may not.