The 0.6B Student Out-Scored Its 8B Teacher

Metadata for RAG 🎓 A 0.6B model just beat the 8B teacher that trained it (100% vs 95.9% on classification), 🏎️ 8 times faster. Twelve minutes of training on my Mac, and the data never left my laptop 🔒

Every document that enters a RAG pipeline needs metadata (labels, summary) before indexing. My 8B model does it well, at 10 seconds per document. Fine at 100 documents, a four-month queue at a million.

Can a small model replace an 8B on one task?

Not out of the box. I gave the 0.6B model the same instructions as the 8B, on my corpus of financial filings. The task asks for a summary plus a few labels per document, like section type or how numeric the content is. The small model's JSON parses 96% of the time. Then you check the labels, and only 36% are correct, where random guessing gives 33%. Parsing correctly and labeling correctly are not the same skill.

Why not just add examples to the prompt (few-shot)?

That was my next move, and it backfired. Valid JSON fell from 96% to 43%. The summaries also started describing the worked example instead of the document. The model summarized a bank's risk factors as a specialty chemicals company, because that was the company in the prompt example. At this scale, in-context examples leak into the output.

So how do you teach it?

You move the examples out of the prompt and into the model's own weights. The 8B teacher writes the answer for 173 training documents, once, at temperature 0 (no randomness, its most confident answer). Then the student absorbs those answers through QLoRA, a light form of fine-tuning that trains a small added layer while the base model stays frozen. On a Mac the run takes about 12 minutes and peaks at 4.1 GB of memory. Two details carry the result:

▸ grade the model only on the answer it writes, never on the document it read ▸ split train and test by company, not by document, or your score is quietly inflated

The student came out at 100% valid JSON and 100% correct section labels, on 49 held-out documents from companies it never saw, 8 times faster than its teacher.

What did it lose?

It lost faithfulness. The student inherited its teacher's rich, confident writing style, and confident sentences have more ways to be wrong. An LLM judge scored its summaries 0.73, where the untrained base model's short, careful sentences scored 0.83. So I trained a second student from a more careful teacher of the same size. On its own held-out set, faithfulness jumped to 0.98. The labels collapsed to 68% on that same section check, because that teacher barely varied its own labels. The student copies the teacher's habits, not its intent.

Before you spend a training run, audit the teacher's own answers on your labels. Your student will copy them.