A compliance finding is a strange thing to grade. It is the most sensitive document a regulated company produces, an itemised account of where the organisation falls short of the law. The moment you send that document to an external API for scoring, you may have created exactly the kind of data event it describes.
We build multi-agent systems that draft these findings. The question that would not go away was who checks the output, and where that checker is allowed to live. For a bank, a hospital or a defence supplier, the honest answer is: on hardware they own, inside their jurisdiction, behaving the same way on the day of an audit as it did on the day of deployment. A frontier model behind an API fails that test twice. The data leaves the building, and the judge can change without notice.
Our answer shipped this week. HivemindEval v2.1 is an open-weight model that scores a compliance finding on six quality dimensions across six UK and EU frameworks: PSD2 SCA-RTS, UK GDPR with the NHS DSPT, MOD JSP 440, Cyber Essentials Plus, DORA, and the EU AI Act. JSON in, JSON out, fixed schema. It runs on one 24 GB GPU at roughly 40 tokens per second.
This article is about the part of a model release that usually gets left out: where it loses, the version of it that failed completely, and why every number below carries a provisional label on purpose.
What we released
Four things, all public. The merged weights, Apache-2.0, on Hugging Face. A frozen benchmark subset of 68 compliance findings with per-item gold labels. The raw, unedited predictions of every model we compared, ours included. And the scoring code, which recomputes our published leaderboard bit for bit on a laptop, confidence intervals included, no GPU required.
That last artifact is the one we care about most. A leaderboard you cannot recompute is a press release with columns.
The full 230-item benchmark stays private as our validation instrument for future versions, but it is committed by hash, so if it is ever disclosed or audited, you can check that nothing was swapped. There is also a 20-page technical report with the complete methodology, the formulas, and the failure analysis. Everything in this article is the short version of something in that report.
What the fine-tuning actually buys
Here is the cleanest experiment in the release. Take the base model, Qwen3-8B, give it the exact prompt our model uses, and run it on all 212 scorable benchmark items. It produces prose. Not one output parses into the schema. Zero out of 212.
The fine-tuned model, on the same items and the same short prompt, produces valid schema output 211 times out of 212.
To make the base model competitive at ranking, you need to hand it a scoring rubric of around ten and a half thousand tokens on every single call. Ours needs a paragraph. That is the trade in one sentence: the fine-tuning buys schema reliability and roughly ten times cheaper prompts.
Notice what I have not claimed. Producing perfect JSON is a reliability property, and it says nothing at all about whether the scores inside that JSON mean anything. We learned to keep those two ideas separate the hard way, which is section five.
The tie, read as a tie
On ranking quality, the headline result is this. Against a rubric-prompted Qwen2.5-72B, a model nine times larger that needs a 46 GB-class device just to load, HivemindEval reaches a cross-tier rank concordance of 0.984 against the 72B’s 0.985. The 95 per cent confidence intervals overlap almost entirely. On Kendall’s tau, the standard rank correlation, our model leads, 0.817 to 0.801.
We describe this as a statistical tie, because that is what overlapping intervals mean, and we would rather undersell it than have someone rerun the numbers and catch us rounding up. The honest footnote is that capable models cluster between 0.96 and 0.99 on this task, which tells you the ordering part of our benchmark is close to saturated. The next benchmark version needs harder items. That is a flaw in our instrument, and it is in the report as one.
What the tie actually establishes is narrower and, for our customers, more useful: for ranking compliance findings by quality, you no longer need the big card. One workstation GPU, inside your own network, gets you there.
Where we lose
Now the table nobody publishes. Our benchmark injects four classes of deliberate defect into otherwise plausible findings: a citation from the wrong jurisdiction, a fabricated control, a missing required control, and items built to sit right at a quality boundary. A good evaluator should score defective items below clean ones. On that measure, HivemindEval is the weakest model we tested.
Grouped bar chart. Probability that a defective item scores below a clean good one, four defect classes by four models, with the chance level at 0.5. wrong_jurisdiction: Qwen3-8B rubric 1.00, Qwen2.5-72B 0.89, Llama-3.3-70B 0.85, HivemindEval v2.1 0.90. hallucinated: Qwen3-8B rubric 0.59, Qwen2.5-72B 0.59, Llama-3.3-70B 0.57, HivemindEval v2.1 0.67. missing_requirement: Qwen3-8B rubric 0.65, Qwen2.5-72B 0.46, Llama-3.3-70B 0.51, HivemindEval v2.1 0.42. band_edge: Qwen3-8B rubric 0.60, Qwen2.5-72B 0.46, Llama-3.3-70B 0.53, HivemindEval v2.1 0.36.
Read the worst cell out loud: on boundary items, our model separates defect from clean at 0.36, which is worse than chance. And on wrong-jurisdiction citations, a core capability for this exact domain, the plain base model with a rubric scores a perfect 1.00 while ours scores 0.90. We were beaten at our own specialty by our own base model with a long prompt.
So the model card says, in plain text, that if your primary need is defect screening rather than quality ranking, a rubric-prompted model is currently the better tool. It felt strange to write that sentence on our own product page. It would have felt worse to hide the table. An evaluator asks to be trusted, and a benchmark that only shows wins is measuring marketing.
The model that scored everything 75
The reason this release is shaped the way it is has a name, and the name is v2.0.
In May, an earlier version of this model was briefly visible on the public Hub. In our internal validation it did something remarkable. It returned the identical overall score, 75, with an identical breakdown across all six dimensions, for every single input. Excellent findings, terrible findings, findings citing the wrong country’s law: 75. Standard deviation zero. And every one of those outputs was flawless, schema-perfect JSON.
The cause was in the training labels. A data fault propagated one constant score vector through the label set, so the model learned exactly what it was taught: a constant. Every check that looks at output form sailed past it. Loss curves looked healthy. JSON validity was perfect. The model was syntactically valid and semantically inert.
Two things came out of that failure. The first is a set of release gates that measure what format checks cannot: score variance across graded inputs, spread between the best and worst tiers, strict ordering. A collapsed model now fails at the door. The second is the design principle running through this whole release, format validity is not judgment quality, which is why the 211/212 figure and the ranking figures live in separate sections and are never blended into one score.
The uncomfortable part is that v2.0 was live while broken, and at least one third-party catalogue captured a copy that outlived our withdrawal. If you obtained v2.0 weights during that window, do not use them. We have requested the stale copy’s removal, and the full account is in the report, because a validation programme that has caught its own worst failure in public is more credible than one that reports only successes.
Why there is no v2.2
The obvious next step after the defect table was a targeted retrain. We prepared the supplemental training data for the two weakest defect classes, checked it for contamination against the frozen benchmark, and then ran one question before spending any compute: can the AI grader that labels our training data actually discriminate the defect we want to fix?
It cannot. On the target class its own separation measured 0.532. Barely better than the coin flip we were trying to escape. Its written reasoning identifies the omissions clearly, and then its numeric scores fail to penalise them, the same decoupling of words from numbers we had already documented at the top of the scale.
A student distilled from a teacher inherits the teacher’s blind spots, at eight billion parameters or eighty. So we cancelled the retrain at a cost of zero training hours, and the roadmap item turned into something else entirely: a human study.
Provisional, on purpose
Here is the caveat that governs everything above. Our band-level gold labels come from controlled construction plus blind adjudication, and our numeric gold comes from human score ranges plus an AI reference grader that we validated only at the low and mid bands. Agreement between this model and human experts is not merely unreported. It is unmeasured.
So the technical report pre-registers the study that settles it. At least three independent UK and EU compliance experts, scoring benchmark items blind, never seeing any model’s output, roughly six to seven hours of work each. Their consensus becomes the first real numeric gold this benchmark has carried, the inter-rater statistics set the ceiling no model should be expected to beat, and the results get published whatever they show. Including the version of events where a prompted baseline beats us on expert agreement. That commitment is in writing, in the report, in advance.
Expert validation programme
We are recruiting UK/EU compliance practitioners for a three-plus reviewer blind validation panel. Around 6–7 hours, sessioned, fully credited or anonymous at your choice. Your judgment becomes the benchmark’s human ceiling.
Join the panelIf you assess PSD2, DORA, UK GDPR, NHS DSPT, JSP 440 or Cyber Essentials Plus findings for a living, that panel is the single most useful thing you could do with seven hours and our benchmark. Everything we have published is, in the end, a carefully constructed claim waiting for your profession’s verdict.
