Distill Your Docs: an 8B Teacher, a 0.6B Student, and What Actually Transfers

Every document that enters a RAG platform (retrieval-augmented generation: search finds the relevant documents, and an LLM answers from them) needs the same treatment before it reaches the index. The pipeline must produce a faithful 3-sentence summary plus five categorical facets: what kind of section this is, how specific to the company it is, how numeric, how forward-looking, and for which audience. Everything comes out as one JSON object, so retrieval can filter and rank on these fields. It is the most repetitive LLM task in the whole pipeline. It is also one of the easiest to get subtly wrong: when a summary drifts away from its document, every search that retrieves it returns wrong information. To study this end to end, I built the enrichment stage on a public corpus of corporate filings: 250 section excerpts from real SEC 10-K annual reports. This is the kind of dense, formal document that financial and legal RAG products process every day.

An 8B model, meaning 8 billion parameters, does this job well. I used deepseek-r1:8b, a reasoning model: it writes out intermediate thinking steps, called the reasoning trace, before giving its final answer, and it runs locally through ollama. Its summaries pass a strict faithfulness check 98% of the time. Faithful has a precise meaning here: a separate judge model reads the document and the summary, and confirms that every claim in the summary is supported by the document (the protocol section shows how I tested the judge itself). The summaries also capture each section's central point every time, and the model identifies the section type correctly on 98.4% of the 250 documents. The price is latency: 10.1 seconds per document at the median on my Mac, and over 12 seconds at the 90th percentile when the reasoning trace runs long. At thousands of documents, that is the bottleneck of the whole ingestion pipeline. A hosted API would be faster, but it bills per document, forever. And the documents leave your machine, which many corpora cannot accept.

The obvious reaction is the one you probably just had: this is ONE narrow, repetitive task, so why not run a much smaller model? A 0.6B model, 13 times smaller, answers in well under a second on the same hardware. If it can do this single job, the latency problem disappears.

So I tried exactly that, and the first result is a genuine surprise. I gave the base 0.6B model (Qwen3-0.6B, compressed to 4-bit precision so it fits in very little memory) the same instructions as the teacher, on 49 held-out documents: documents set aside from the start and kept out of all training, so no model has an unfair advantage on them. It produced valid JSON 95.9% of the time, and its summaries passed the faithfulness check at 0.83, meaning 83% of them passed. Judge scores appear as fractions of 1 from here on. Filing prose is formal and predictable, so even a small model can copy its shape. If the task were "summarize these documents into JSON", you could almost stop here.

Then you look at the part retrieval actually filters on, and you hit the wall. Here is a simple test. When I built the corpus, I took each excerpt from a known part of the report: the business section, the risk factors, or the financial analysis. So for every excerpt, the right answer to "which section is this?" is known in advance. No teacher, no judge, just a direct check. The base model gets it right on 36.2% of documents. Random guessing among three choices gives 33%. The other facets have no ground truth to check against, so I score them differently: I compare each label to the one the 8B teacher gives on the same document. That measures imitation rather than truth, a distinction that will matter later in this article. By this measure the picture barely improves. The numeric density label, a three-way choice between low, medium and high, matches the teacher's on 36.2% of documents (the same number as the section score, by coincidence). The forward-looking label, also a three-way choice, matches on 27.7%. The one exception is specificity, a two-way choice, where the base model reaches 74.5%. The JSON parses perfectly, and most of the labels inside are close to random. For a retrieval index, facets like these are worse than no facets, because filters silently return the wrong documents.

Your next idea is probably the standard one: put a few worked examples in the prompt. Few-shot prompting is the usual fix when a model follows the task poorly, and it costs nothing but prompt tokens. Here is what it did at 0.6B. Valid JSON fell from 95.9% to 42.9%. Faithfulness collapsed from 0.83 to 0.29. Latency tripled, because every request re-reads the examples. Then I read the summaries, and found the most instructive failure of the whole experiment.

Here is a real pair from the runs. The document is a bank's risk factors section. The summary the model returned describes the chemicals company from the worked example, including the businesses it sold.

ITEM 1A. RISK FACTORS
The material risks and
uncertainties that management
believes affect us are described
below... [a regional bank's
risk factors]

"PQ Group Holdings Inc. operates
as an integrated global supplier
of specialty catalysts and
chemicals. The company recently
sold two business segments..."

Why does this happen? A 0.6B model struggles to keep two things separate: the examples it should imitate, and the document it should process. Inside its context window, the example already looks like a perfect answer, so under pressure the model copies it. And this is not bad luck in a single run. I ran the experiment twice, with two completely different pairs of examples, and the leak followed the examples both times, word for word. To be clear, this run does not test larger models; on those, few-shot examples are widely reported to work as intended. But at this scale, the technique quietly destroyed the one property a retrieval index cannot give up: the summary must describe the right document.

So prompting cannot rescue the small model on this task: zero-shot classifies at chance, and few-shot poisons the summaries too. The examples need to move out of the context window and into the weights. That is precisely what distillation does.

The idea is simple. Instead of showing the small model examples at inference time, you have the big model write a training set once, and you fine-tune the small model on it. The teacher's behavior gets compressed into the student's weights. At inference time the student sees only the document, with no examples left to leak. The diagram below shows the whole pipeline; each label in it gets its own brick in the walkthrough that follows.

Brick 1: the teacher writes the answers. I ran deepseek-r1:8b at temperature 0 over the training documents. For each document, it writes the exact JSON object I want the student to produce. Temperature 0 matters: it forces the model to give its most confident answer every time, so the dataset is the teacher at its most reliable. The reasoning trace is removed; only the final JSON is kept as the training target. The few outputs that broke the schema (3 of 250) were dropped rather than taught.

Brick 2: the dataset. Each document and its JSON answer become one training example: the question (the instructions plus the document) and the answer (the teacher's JSON). After dropping the teacher's few broken outputs, the corpus splits into 173 training examples, 25 for validation, and 49 held out for testing. Two details decide whether you can trust the result.

The first detail: grade only the answer. It rests on one fact about how fine-tuning works. The model learns by predicting its training text one token at a time, and by default every token in the example counts in its score, the document included. Here each document runs about 4,000 characters and the JSON answer about 450, so roughly 88% of the training effort would go into learning to rewrite document text nobody asked for. One training flag fixes this: it masks the document tokens out of the score. The model still reads the document; those tokens just stop counting. Every bit of learning goes into the 450 characters that matter. The visual below shows where the effort goes.

The second detail: never test on a company you trained on. Split the data by company, not by document. If any part of Apple's report is in the training set, none of Apple's report can be in the test set, so the student only proves itself on companies it has never seen. My first split got this wrong: 45 of 50 test documents came from companies already seen in training, which quietly inflates the score. The numbers below use the corrected split.

{"messages": [
  {"role": "user", "content": "Summarize and label this document as one JSON object. <instructions> <document text>"},
  {"role": "assistant", "content": "{\"summary\": \"...\", \"section_type\": \"risk_factors\", \"specificity\": \"...\", ...}"}
]}

Brick 3: QLoRA on the 4-bit student. Full fine-tuning, even for a 0.6B model, is more than one narrow task needs. QLoRA takes a lighter route. In plain terms: the student's general knowledge stays frozen, and a thin trainable layer, added alongside the existing weights, learns what changes for this one task. Technically, the base model stays in its compressed 4-bit form and training only touches small adapter matrices. Their capacity is set by a dial called the rank: a higher rank gives the adapter more room to change the model's behavior. I used 32, a common middle setting. On my Mac, with MLX, the whole run takes about twelve minutes and peaks at 4.1 GB of memory. It trains comfortably while you make coffee, on the machine you already own.

# 1. Build the corpus (streams a public HF dataset, ~2 min)
python3 scripts/build_corpus.py 250

# 2. Teacher generation (resume-safe, ~10 s/document for the reasoning teacher)
python3 scripts/gen_teacher.py deepseek-r1:8b teacher_r1

# 3. Dataset (chat JSONL, split grouped by company) + QLoRA training (~12 min)
python3 scripts/prepare_data.py
bash scripts/train.sh   # mlx_lm lora, rank 32, --mask-prompt, seed 42

# 4. Evaluation: schema validity, ground-truth accuracy, label agreement, latency
uv run python scripts/evaluate.py

Note what we did NOT do: reinforcement learning, preference data, synthetic augmentation, or a cloud GPU. One teacher pass and one masked supervised fine-tune. The question is what something this simple actually transfers.

Every setup below ran on the same 49 held-out documents, all from companies the student never saw during training. Three metrics need a short definition first:

The judge also checks length and the absence of hype. Every setup passed those two nearly perfectly, so the table leaves them out.

Start with what the student got right: 100% valid JSON, and 100% ground-truth section accuracy, on documents from companies it has never seen. That last number is even better than its own teacher's. On these same 49 documents, the teacher misreads 2 and scores 95.9%.

How can a student beat its own teacher? Across the whole corpus the teacher misreads only 4 of 250 documents, and those mistakes are scattered noise, not a repeated pattern. Trained on 173 of the teacher's answers, the student absorbs the dominant pattern and skips the scattered exceptions, so on the test documents it gets right the 2 the teacher misses. A nudge rather than a leap, but a real one.

Two more facets tell the same story. On each facet we measure how often the student's label matches the teacher's, and on two of them that match rate jumped from near chance to strong. On specificity the student now agrees with the teacher 83.7% of the time, against 74.5% for the untrained base model. On numeric density it reaches 81.6%, against just 36.2% for the base model. Training moved both a long way.

Two other facets did not follow, and they belong in the same picture. Forward-looking only climbed to 49%, and audience level actually dropped below the base model's score. What moved into the weights is most of the structured judgment, not all of it. That is still the part no prompt could give the base model. And the student does all this at 1.24 seconds per document, 8 times faster than the teacher. In the latency plot below you will also spot a second student, which we come back to in the next section.

One number in the table cuts against the rest, and it is worth understanding. The student's faithfulness is 0.73, while the plain base model scored 0.83. How can the untrained model write MORE faithfully? Because the base model, when it summarizes at all, writes short, careful sentences that say little. And sentences that say little are hard to get wrong. The student inherited its teacher's richer, more confident style, and on dense financial text every extra claim is a chance to slip. The judge caught it writing "operating profit" where the filing says operating revenue. It also called an operating ratio "higher" when the ratio improved by going down. Distillation transferred the teacher's writing behavior, including its habit of making strong claims. It did not transfer the teacher's knowledge, or its ability to check what it writes.

That faithfulness trade raises a question worth an experiment of its own. If the student inherits its teacher's writing behavior, then a different teacher should produce a different student, even at the same model size. A recent study of on-device distillation measured exactly this effect at larger scale, and this article's protocol is modeled on it. So I replicated the move. I trained a second student, identical in every way: same 0.6B base, same QLoRA recipe, same corpus. The only change was the teacher. This time the training data was written by llama-3.1-8b-instruct, an 8B model with no reasoning traces and a more careful writing style. One consequence of rebuilding the dataset per teacher: each student gets its own train/test split, and the two test sets share only 7 of their 49 documents. Comparisons across the two students therefore hold the protocol constant, not the documents.

The result surprised me: the student of the instruct teacher scored 0.98 on faithfulness. That matches the level the 8B reasoning teacher itself reaches, from a model 13 times smaller, on unseen companies. Its summaries are shorter and plainer than the reasoning student's, and they almost never contain an error. Here are the two students on the SAME test document, a customer experience company's business overview:

"Sykes Enterprises provides
global customer experience
management services, including
full lifecycle solutions and
digital transformation
capabilities across various
industries. The company
operates..."

"Sykes Enterprises is a leading
global provider of customer
experience management services,
multichannel demand generation,
and digital transformation."

So the careful teacher wins? Look at the labels before you decide. The instruct teacher turns out to be a lazy labeler. On all 250 documents, it assigned the same audience level every single time, and it labeled 249 of 250 documents as specific to their company. Its own ground-truth section accuracy is 72.4% across the 250 documents, far below the reasoning teacher's 98.4% on the same set. The student copied this collapse faithfully. It scores a perfect 100% agreement with its teacher on those two constant fields, but agreeing with a constant is not a skill. On the one facet we can score objectively, it reaches only 68.1%. The caution and the laziness were inherited together, exactly like the richness and the errors were in the other student.

For the summary side of the pipeline, one composite number matters most: what fraction of documents yields an output that is both parseable and faithful?

A result you cannot reproduce is an opinion. Every number in this article comes from the exact configuration below, and the scripts in the repository rebuild the whole thing from a public HuggingFace dataset, so there is nothing private to request.

Two protocol choices deserve a word. First, the summary judge is itself an LLM, which should make you suspicious by default. The negative control is the answer to that suspicion. I fed the judge 15 deliberately mismatched pairs (a real document with a summary of a different document), and it rated 0 of 15 as faithful. Second, the ground-truth section check exists because agreement with the teacher can reward a student for imitating a broken teacher, as the lazy labeler finding shows. Where an objective anchor is available, use it.

Distillation is not a general replacement for a big model. It is a compression of ONE task into a small one, and this experiment sharpens where it pays. If your task is summarizing clean, formal documents, measure the base model first: it may already be good enough, and the simplest option is to add no training at all. The picture changes the moment your pipeline needs structured judgment: facets, classifications, anything a filter will trust. There, the base model is a coin flip, and few-shot prompting makes it worse. Distillation is the cheapest reliable fix: twelve minutes of training moved section accuracy from 36% to 100% here.

That final box, choosing the teacher, deserves a word. If faithfulness is what matters, distill a careful instruct teacher that sticks to the source; if classification is what matters, distill the teacher that actually classifies well. Audit the teacher's own habits on your labels before you spend the training run, because the student will copy the habits, not the intent. Measure the composite that matters to YOUR pipeline before you pick.

The whole experiment ran on one laptop, and that was the point: teacher generation, QLoRA training, evaluation, judging and controls, all local, all reproducible, about an afternoon end to end. You can point the corpus builder at your own documents and have a student model of your own by tonight. The same logic extends to richer enrichment: extracting entities and relations to feed a knowledge graph is the same shape of task, high volume and fixed format, and it is the natural next experiment.

Everything in this article is reproducible from the GitHub repository: corpus building, teacher generation, dataset preparation with the split grouped by company, QLoRA training, evaluation, and the judge pass with its negative control.