
A thread on LINUX DO recently asked the question with characteristic bluntness: 怎么蒸馏GPT — "how do you distill GPT?" No preamble, no context, just the raw desire behind it: take the smartest model you can reach and bottle its ability into something small enough to run on your own hardware.
The short answer: you don't shrink the big model; you train a small one to imitate it. A large "teacher" generates answers — sometimes including its step-by-step reasoning — and a smaller "student" is fine-tuned on those outputs until it behaves like the teacher on the tasks you care about. You need no access to the teacher's weights or internals; API outputs alone are enough, which is why the question keeps resurfacing. Below is the practical version: what the technique is, the two DIY routes, a minimal recipe, and the legal line to know before you start.
The idea predates the LLM boom by a decade. In 2015, Hinton, Vinyals, and Dean published "Distilling the Knowledge in a Neural Network" (arXiv:1503.02531), describing a simple trick: train a small student model not just on the correct answers (hard labels), but on the teacher model's full probability distribution over possible answers (soft targets).
Why does that help? A hard label says "the answer is 7." A soft target says "the answer is 7, but 1 and 9 were plausible runners-up." That runner-up information — Hinton called it "dark knowledge" — teaches the student which mistakes are reasonable, so it learns the shape of the teacher's judgment, not just its verdicts. Because raw probabilities are usually too sharp to be informative, the teacher's outputs are softened with a temperature parameter T before training:
student_loss = KL( softmax(teacher_logits / T), softmax(student_logits / T) )
+ cross_entropy(student_logits, true_labels)
In plain terms: the student is penalized both for disagreeing with the teacher's distribution and for getting the actual answer wrong. That two-part loss is the entire classical recipe, and every modern variant is a variation on it.
When forum users say "distill GPT," they almost always mean the second of these two routes. It matters which one you pick, because they differ in what you need, what they cost, and what legal exposure they carry.
Route A: logit distillation (white-box). This is the original Hinton setup — you need the teacher's logits or probability distributions, which means you need the teacher's weights running locally, or an API that exposes token probabilities. A few commercial APIs expose top token log-probabilities on request, but full distributions are essentially never available. In practice this route is for researchers distilling one open model into another, not for distilling a closed frontier model.
Route B: sequence-level distillation (black-box). Forget the internals. You prompt the teacher, collect its generated outputs, and fine-tune the student on those prompt→response pairs as ordinary training data. No logits, no special loss — just supervised fine-tuning where the teacher wrote the textbook. This is the route with real-world proof behind it: DeepSeek's R1 distills were built exactly this way. Per the project's own README, the six distilled checkpoints (1.5B through 70B parameters, based on Qwen2.5 and Llama 3) are "fine-tuned based on open-source models, using samples generated by DeepSeek-R1" — the 671B teacher's reasoning traces, poured into students small enough for a single GPU. The 32B version reportedly beats OpenAI's o1-mini on math, coding, and science benchmarks.
| Route A: logit distillation | Route B: sequence-level distillation | |
|---|---|---|
| What you need | Teacher weights or full probability outputs | Just API access to the teacher |
| Training signal | Soft targets + hard labels | Teacher's generated text as labels |
| Fidelity | Higher (richer signal per example) | Lower per example, compensated with more data |
| Works on closed models? | Effectively no | Yes — this is the "distill GPT" people ask about |
| Legal exposure | Depends on the teacher's license | Depends on the teacher's terms of service |
If you are asking "how do I distill GPT," Route B is your route. The rest of this article assumes it.
Here is the smallest version that works, in four stages.
1. Narrow the task ruthlessly. Distillation transfers a skill, not general intelligence. "Answer questions about our internal API" distills well; "be as smart as GPT" does not. Pick the one capability you need and define how you'll measure it — accuracy on a few hundred held-out examples is the minimum.
2. Generate the teacher dataset. Prompt the teacher with diverse, realistic inputs for your task and save the prompt→response pairs. Diversity matters more than volume: cover edge cases, rephrase the same intent several ways, and include inputs where you'd expect the teacher to struggle. Then curate — read a random sample and drop the garbage, but fix nothing by hand (hand-fixing teaches the student your corrections are teacher behavior, which they aren't). A few thousand clean pairs beat fifty thousand sloppy ones. If your task benefits from reasoning, ask the teacher to show its work and keep the traces; DeepSeek's distills demonstrate that reasoning traces transfer surprisingly well.
3. Fine-tune the student. Pick a small open base model in the 1B–14B range — the same families DeepSeek used (Qwen, Llama) are the community defaults for a reason: good instruction-following starting points with broad tokenizer coverage. Then it's standard supervised fine-tuning on your teacher-generated pairs. With Hugging Face's TRL library the training loop is a few lines:
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
dataset = load_dataset("json", data_files="teacher_outputs.jsonl")
args = SFTConfig(
output_dir="student-distill",
num_train_epochs=3,
per_device_train_batch_size=4,
learning_rate=2e-5,
)
trainer = SFTTrainer(
model="Qwen/Qwen2.5-7B-Instruct", # your chosen student base
args=args,
train_dataset=dataset["train"],
)
trainer.train()
Three epochs at a small learning rate is the conventional starting point. If training loss keeps dropping while evaluation stalls, the student is memorizing — gather more diverse teacher data.
4. Evaluate against the teacher, not against vibes. Run both on your held-out set and compare task accuracy against cost and latency. A student scoring 92% of the teacher at 5% of the cost is a success; 60% means your task was too broad or your data too thin.
Once it works, the student is just a small model you own: quantize it and serve it behind a single endpoint (the self-hosted gateway pattern fits naturally), and stop paying per-token for the narrow skill. A cheap always-on box serves a 7B model fine — the homelab playbook on this site covers that tier of machine.
Distillation is one of four standard answers to "the big model is too expensive for this job." The decision table:
| Approach | Best when | Watch out for |
|---|---|---|
| Distill a small model | One narrow skill, needed cheaply, privately, or offline | Student inherits the teacher's blind spots; evaluation is on you |
| Quantize an existing open model | An open model is already good enough at the task | Quantization shrinks size, not capability gaps |
| Just call the API | You need generality or the task keeps changing | Per-token cost and latency at scale; data leaves your network |
| Fine-tune on human data | You have domain data no teacher can generate | Expensive to collect; still needs a capable base model |
The rule of thumb from practitioners: distill when the skill is narrow, the volume is high, and the teacher is overkill. If any of those three isn't true, one of the other rows is probably cheaper.
Model providers write their terms of service with distillation specifically in mind. OpenAI's Terms of Use prohibit users from using "Output to develop models that compete with OpenAI" and from "automatically or programmatically extract[ing] data or Output" — quoted verbatim in a public mirror of the terms, since OpenAI's own policy pages block automated fetching. Anthropic's commercial terms carry a parallel restriction on training competing models from Claude outputs.
What counts as "competing" is deliberately undefined, which is why you should think before generating 100,000 teacher outputs. The clean way to sidestep the question is to distill from an open-weight teacher whose license permits it — which is what DeepSeek did by publishing its R1 distills openly. If the teacher is open and permissively licensed, the ToS problem evaporates; if it's a closed commercial API, read the terms as carefully as the training docs. This isn't legal advice, but it's the most expensive line to skip.
Honest limits, since the hype around "free GPT in 7B" deserves deflating. A student cannot systematically outperform its teacher on the distilled distribution — the training signal is the teacher's behavior, ceiling included. It inherits the teacher's biases and failure modes, laundered into a smaller package that looks deceptively authoritative. Low-quality teacher outputs don't average out; they become gospel, which is why curation dominates training in determining the outcome. And none of this replaces evaluation: a distillation without a held-out test set is a superstition with extra steps.
Knowledge distillation is how the frontier's capabilities actually reach ordinary hardware — the R1 distills running on single GPUs today are the proof. The forum question has a real answer, and it's refreshingly unmystical: pick a narrow skill, have the teacher write the textbook, train the student on it, and measure what you got. The magic, such as it is, is all in the data.