Solving the rubrics cube
Testing software has always been reassuringly, quite boring. Put two plus two into a function, expect four back, and if you get five, something's broken and you go and fix it. That's largely the same whether it's a unit test or an integration test. Given this input, I expect exactly this output. Pass or fail.
Today, though, more of our software is being built to be autonomous, on top of various AI systems, not least LLMs. Ask it a question and you might get something brilliant, ask again and get a different but equally good answer, then ask a third time and get something that sounds just as confident and is completely wrong. Which is much harder to write an assertEquals for.
How do we then decide what's right and what's wrong? This is the world of evals, the AI equivalent of a test suite, and one of the most useful tools in the evals toolbox is the rubric.
A rubric is a scoring guide made up of a set of criteria, each given a score, that rolls up into a final number. The criteria are often fuzzy, a bit like reviewing a film or a game. You ask whether it was gripping and whether the ending landed, and you give each a mark. That fuzziness is what makes rubrics such a good fit for non-deterministic systems, and LLMs in particular.
The way it works is that we draft criteria for what a good answer looks like across different categories, then weight them by importance, because a factually wrong answer delivered in a lovely tone is still a wrong answer.
We pipe a set of test questions through the model and hand each answer to a separate model acting as the judge, which scores it against each criterion independently. Multiply each score by its weight, add them up, and that's our final rubric score. For example, here's a rubric for a customer support assistant that answers questions from our help docs, with each criterion scored from 1 to 5.
| Criterion | Weight | What a 5 looks like | What a 1 looks like |
|---|---|---|---|
| Accuracy | 35% | Every fact is correct | Contains a clear factual error |
| Relevance | 20% | Directly answers the question asked | Answers a different question entirely |
| Groundedness | 15% | Every claim is backed by the help docs | Invents policies or features |
| Completeness | 15% | Covers everything the user needs to act | Leaves out critical steps |
| Clarity | 10% | Easy to follow, well structured | Rambling or confusing |
| Tone | 5% | Friendly, professional, on brand | Curt, robotic or inappropriate |
If the judge gave an answer a 4 for accuracy, a 5 for relevance and groundedness, a 3 for completeness and a 4 for clarity and tone, it would come out at 4.2 out of 5. Run that across a few hundred questions and you get a single number for how good the system is, while the per-criterion averages show you where it's falling down.
Of course, we're now using an AI to mark an AI, so it's worth checking the judge before trusting it. Scoring 30 to 50 answers by hand, compare them with the judge's scores, and if you mostly agree you can let it mark the rest. Judges also have some known habits. They tend to prefer longer answers, favour whichever answer they see first, and rate outputs from their own model family more kindly. Swapping the order of comparisons and using a judge from a different provider take care of the last two.
It also pays to make the criteria as concrete as you can. "Rate completeness from 1 to 5" leaves a lot of room for interpretation, while "does it mention the refund window?" leaves almost none, so yes/no checklists tend to give far more consistent scores than sliding scales. Even then there's noise.
Importantly, the system is non-deterministic and so is the judge, so a jump from 4.1 to 4.3 might be a real improvement or just a different roll of the dice. Running each question a few times and looking at the spread as well as the average is important to establish confidence.
Ultimately, what this allows us to do is evaluate our non-deterministic system in a repeatable way. We can change the model, alter the prompt, change the data we send, or any other variable - adjusting each one independently, and measuring the outcome against the same rubric.
Not to mention, quality doesn't have to be the only measure either. A 4.5 that costs ten times more than a 4.3, simply because we changed from using cheap flash model to a frontier one at Max thinking, may not be the winner, so cost and latency are worth weighing alongside it. Once you have a score you trust, you can even treat it like any other test and fail the build when it drops, which brings things right back round to traditional CI.
Which following on from my previous blog posts where I've covered this, it lets us create pressure around agentic workflows even for non-deterministic software. Agents thrive when something pushes back on them, like a failing test or a type error, and up to now that pressure has mostly come from deterministic checks. A rubric gives us the same lever for everything else. We can harness an agent in a loop where it changes something, runs the eval, reads the score and the judge's reasoning, and tries again, nudging the score upwards each time. The catch is that an agent will optimise for exactly what you measure, so if your rubric has gaps, the agent will find them.
I've been using rubrics more and more in my own projects, to great effect. Deterministic tests still cover everything they always have, and rubrics pick up the parts where the right answer is a matter of judgement - which is increasingly the measure for autonomous systems.