Knowledge Distillation Compression Calculator
Size/speed/quality trade of distilling a teacher into a student — compression ratio, capacity gap and serving gains.
Retention figures summarize the 2024–25 distillation wave (Gemma↔Gemini, Llama-3.1-405B→70B/8B, DeepSeek-R1→Qwen/Llama students): 3–6× compressions routinely keep 93–97% of capability ON DISTRIBUTION; out-of-distribution gaps are larger.
Formula
About Knowledge Distillation Compression Calculator
Distillation is how the industry ships frontier quality at edge prices: train a small student on the big teacher's outputs (soft logits, reasoning traces, or curated generations) and keep most of the capability at a fraction of the serving cost. This calculator frames the deal quantitatively — compression ratio, the near-proportional decode speedup that follows from bandwidth-bound inference, and an evidence-based estimate of score retention drawn from the recent distillation wave (R1→Qwen students, 405B→8B). Use it to set expectations before committing to a student size.
How to use Knowledge Distillation Compression Calculator
- 1Enter your values into Knowledge Distillation Compression Calculator — sensible, domain-typical defaults are pre-filled so you see a real result immediately.
- 2The result recomputes live using the formula shown on the page; there is no button to press.
- 3Adjust any input to compare scenarios, then read the worked example to see the substituted numbers.
Why use Knowledge Distillation Compression Calculator?
- ✓Computes Knowledge Distillation Compression instantly in your browser — no sign-up, no upload, no server round-trip.
- ✓100% free and unlimited, with the exact formula shown: compression = teacher_P ÷ student_P.
- ✓Runs entirely client-side, so every value you enter stays private on your device.
- ✓Live recompute as you type, with a worked example and authoritative references for trust.
Frequently asked questions
What does the student actually train on?+
Three regimes: classic soft-label KD (match the teacher's full output distribution — needs logits access), sequence-level KD (fine-tune on teacher GENERATIONS — what API-based 'distillation' means), and trace distillation (R1-style reasoning chains). Modern open-model distills mostly use the latter two.
Why do students lose more out-of-distribution?+
The student inherits the teacher's behavior only where training data exercised it; rare domains, adversarial phrasing and long-tail knowledge get compressed away first. On-benchmark retention of 95% can coexist with much larger gaps on YOUR niche — always evaluate in-domain.
Distillation vs quantization — which first?+
They stack: distill to the right capability size, then quantize the student for deployment (a 8B Q4 ≈ 4.8 GB serves what once needed a 70B). Quantization preserves the same model's behavior cheaply; distillation actually changes the capability/size point. Different tools.
Can distillation make a student BETTER than its size class?+
Yes — that is its entire point. R1-distilled 7–8B models beat same-size baselines by wide margins on reasoning because teacher traces are far higher-quality training signal than web text. The student exceeds its class, not its teacher.
Related tools
Related ML & AI tools
ROC-AUC Calculator (from TPR/FPR points)
Trapezoidal area under the ROC curve from your (FPR, TPR) operating points — the threshold-independent ranking score.
● LiveClassification Threshold Cost Calculator
Find the probability cutoff that minimizes expected cost given your false-positive and false-negative penalties.
● LiveSilhouette Score Calculator
Cluster cohesion vs separation for one point — the building block of the silhouette metric for choosing K.
● Live