The Golden Dataset Lifecycle

Most developers building GenAI systems don't have a golden dataset.

No dataset = no reliable evaluation. No evaluation = no way to know if your RAG or Agent is improving... or regressing.

In classic ML, nobody would deploy a model without a test set. Yet in GenAI, it's the norm.

We treat evaluation as a nice-to-have. When it's actually the core of any GenAI system in production.

A golden dataset is the equivalent of TDD for AI. It even has a name: Evaluation-Driven Development (EDD). → You define what "correct" means BEFORE you ship.

And it doesn't need to be perfect from the start. 10-20 samples is enough to begin.

Then production gives you the rest: - Real traces from your AI system (trace first, annotate later) - User corrections - Expert judgments - LLM-as-Judge to scale

Multiple feedback sources, from richest to simplest: - Expert annotations (highest quality, lowest scale) - User corrections from production - LLM-as-Judge (scalable, good for qualitative — align it on expert annotations to keep it reliable) - Code-based checks (fast, deterministic) - Thumbs up/down (easy to collect, low signal)

Each has its trade-off between quality and scalability. The key is combining them.

To evaluate, you'll typically use a mix of: LLM Judge — flexible, covers qualitative aspects Code-based — fast, deterministic, great for structured checks (And for agents: trace-level evaluation to verify reasoning steps)

With all that, your dataset grows organically with every production interaction. Industry benchmarks suggest ~100 samples for solid general coverage, 250+ for statistical confidence.

Whether you're building a RAG or an Agent, the principle is the same.