Your Eval CI Green Is Not a Valid Judge

A passing eval test suite tells you your prompt did not regress on a case you already thought to test. It does not tell you your LLM-as-judge is right.

A test suite is a regression gate; the judge is the actual evaluator. They live on different layers and need different validation. The validation has three layers.

Layer 1 is ground truth. 30 to 200 human-labeled production traces (real conversations from your live system), scored by a single domain expert on the same criterion the judge is supposed to measure (helpfulness, factual correctness, safety, whatever your task is). No committee dilution. Without this you have a regression suite, not an eval.

Layer 2 is diagnosis. Does the judge agree with humans beyond chance? Cohen's kappa is the standard answer (roughly: how often the judge and the expert agree, minus what you would get from guessing). It is still useful, but in 2026 it is one signal in a panel. Add true-positive and true-negative rates per failure class, balanced accuracy (which fixes a known kappa weakness: scores shift with how often each class appears), and bias audits: swap A/B answer order to measure position bias, pad responses to measure verbosity bias.

Layer 3 is remediation. Alignment optimizers take your judge prompt plus disagreement data and produce a corrected judge prompt. SIMBA is the default starting point. MemAlign is faster when your humans wrote rationales: it builds a dual memory of guidelines and edge cases, then injects both into the next judge call. GEPA handles complex rubrics. Neither these optimizers nor your eval framework hands you a kappa score for your own judge: MLflow and Databricks use kappa to benchmark their built-in judges, but for yours you compute it on the gold set yourself, before and after.

Eugene Yan and Hamel Husain converge on the operational version: small gold set, binary or near-binary scoring (1-5 rubrics are unactionable), the panel of metrics above, iterate until the judge matches expert judgment at a level you would trust from a junior teammate, then distill to a small classifier if latency matters.

An engineer reported recently that their LLM-judge had kappa of 0.47 while their CI test suite passed green, with a four-figure cost spike from wrong refund decisions. The story is unverified. The failure modes are documented across the LLM-judge literature: position bias when A/B order is swapped, verbosity bias when responses are padded.

What I do when I audit a team's LLM-judge pipeline now: count the human-labeled traces. If the count is zero, the team has not built an eval pipeline yet. The Monday morning version is 50 traces, one expert, two hours of focused labeling.

Two hours buys you the difference between thinking you have an eval and having one.