Solving the rubrics cube
Testing software has traditionally and reassuringly been, quite boring. Put two plus two into a function, expect four back, and if you get five, something's broken and you go and fix it. That's largely the same whether it's a unit test or an integration test. Given this input, I expect exactly this output. Pass or fail, you get the idea.
Today though, our software is increasingly built to be non-deterministic and autonomous, on top of various AI systems, not least LLMs. Ask it a question and you might get something brilliant, ask again and get a different but equally good answer, then ask a third time and get something that sounds just as confident and yet completely wrong. Which is far harder to verify with an assertEquals statement.
How then do we decide what's right and what's wrong in these outputs? Enter the eval, the AI equivalent of a test suite, and one of the most useful tools in the evals toolbox is the rubric.
A rubric is essentially score card made up of a set of criteria, that rolls up into a final number. The criteria are intentionally fuzzy, a bit like reviewing a film or a game. That fuzziness though is what makes rubrics such a good fit for non-deterministic systems, and LLMs in particular.
The way it works is that we draft criteria for what a good answer looks like across different categories, then weight them by importance, because a factually wrong answer delivered in with a convincing tone is still a wrong answer.
We then pipe a set of test questions through the model and hand each answer to a separate model acting as the judge (ideally a model different from the one we're testing for more independent judgement) - this then scores it against each criterion independently. Then you multiply each score by its weight, add them up, and that's our final rubric score!
For example, here's a really simple rubric for a customer support assistant that answers questions from our help docs, with each criterion scored from 1 to 5.
| Criterion | Weight | 5 = | 1 = |
|---|---|---|---|
| Correct | 45% | Every claim is accurate and backed by the help docs | Gets a fact wrong or invents a policy/feature |
| Useful | 35% | Answers the actual question with every step needed to act | Misses the question or leaves out a critical step |
| Clear | 20% | Easy to follow, friendly and on brand | Confusing, rambling, curt or robotic |
If the judge scored an answer 5 for correct, 3 for useful and 4 for clear, it would come out at 4.1 out of 5. Run that across a few hundred questions and you get a single number for how effective the system is. As well, the per-criterion averages show you where it’s falling down - which is importantly what informs us what we need to adjust.
Of course, we're now using an AI to mark an AI, so it's worth checking the judge before trusting it. Scoring a few answers by hand, compare them with the judge's scores, and if you mostly agree you can let it mark the rest. This ensures you and the judge are aligned on the expected output.
Keeping in mind, the judge itself isn't static and just like the outputs being assessed, you might need to tweak this to better align with whatever a "good" output looks like.
Further consideration is that even how we evaluate a particular rubric, doesn't happen in a vacuum, a score of 4.5 that costs ten times more than a 4.3, simply because we changed from using cheap flash model to a frontier one at Max thinking, may not be the winner, so cost and latency are worth weighing alongside it.
Once you have a score you trust - this becomes your baseline, you can even treat it like any other test and fail the build when it drops, which falls naturally into a traditional CI.
Following on from my previous post where I covered some of this, it lets us create pressure around agentic workflows even for non-deterministic software. Agents thrive when something pushes back on them, like a failing test or a type error, and up to now that pressure has mostly come from deterministic checks. A rubric gives us the same lever even when working with tools like LLMs.
Going a step further than just the CI though, we can even harness an agent into a loop to improve a rubric - change something, run the eval, read the score and the judge's reasoning - then try it all over again, nudging the score upwards.
Much like solving a Rubix's Cube, solving the rubric is a game of constantly adjusting and evaluating, whether that's the prompt, the tools, the context. You may never fully solve it - that's fine, if you wanted a fully deterministic answer, you're probably reaching for the wrong technology anyway.