At a glance
A review by Anoushka Nayak and colleagues at RV Institute of Technology and Management in Bengaluru reports that knowledge distillation preserves 90 to 97 percent of a teacher model's accuracy while cutting size by 50 to 70 percent and speeding up inference two to five times.
Strip a neural network down to half its size and it can still get roughly nine answers in ten right. It will also respond two to five times faster than the model that taught it. Those are the operating numbers in a review published in April in the International Journal of Creative Research Thoughts, written by Arfa Yunus, Anoushka Nayak, Aisiri S, Aishwarya R and Hema M S, all in the Department of Computer Science Engineering at RV Institute of Technology and Management in Bengaluru.
The timing matters. The systems that set the pace in machine learning keep getting bigger, while the places that could use them keep getting smaller. A warehouse sensor, a hearing aid, a phone in a clinic with intermittent power: none of them can carry a data-centre GPU or the electricity bill that comes with one. Knowledge distillation is the bridge the Bengaluru team examines, a training method that folds the behaviour of a large model into a small one so the small one performs above its weight class.
The paper is a synthesis rather than a bench experiment. It traces the technique from its 2015 formulation, works through multi-stage variants, transformer distillation and TinyML deployments, then gathers published results into ranges. Its verdict is deliberately measured: distillation narrows the disparity between highly complex models and environments with fewer computational resources, cutting energy use and enabling deployment on low-power hardware without a large loss in performance.
An apprentice that copies the reasoning, not only the answers
The standard setup has three moving parts. A teacher: large, pre-trained, accurate, expensive. A student: small, cheap to run, designed around the memory and power it will actually have. And an objective function that tells the student what to imitate. The student trains on the labelled dataset as any model would, and simultaneously on the teacher's outputs.
That second signal carries the value. A hard label says a photograph is a seven. The teacher's probability distribution says it is probably a seven, plausibly a one, and almost certainly not a four. The ranking of near-misses is information a single label cannot convey. The review calls these soft targets, and the comparison holds: a chess coach who says a move is strong but slightly risky teaches more than a coach who only marks it right or wrong.
Temperature scaling is the dial that controls how much of that texture reaches the student. Before the teacher's raw scores are converted into probabilities, they are divided by a temperature value. Raise it and the distribution flattens, lifting small probabilities and exposing which wrong answers the teacher considered plausible; lower it and the teacher looks certain about everything, which is precisely the signal you don't want copied. During training the student minimises a blend of two terms, one measured against the true labels, the other against those softened outputs.
What the review puts on the table
Section four of the paper converts scattered published results into a compact set of ranges. The authors report that distilled models keep about 90 to 97 percent of their teacher's accuracy in most instances, that model size can fall by roughly 50 to 70 percent, and that inference can run two to five times faster.
| Outcome | Range reported in the review |
|---|---|
| Accuracy retained against the teacher model | 90 to 97 percent |
| Reduction in model size | 50 to 70 percent |
| Gain in inference speed | 2 to 5 times |
DistilBERT is the clearest single case in the text. Sanh and colleagues reported that a slimmed-down version of BERT reached about 97 percent of the original's performance while being considerably smaller and faster to run, which demonstrated that distillation works on transformer architectures and not just on older convolutional networks. The authors pair those figures with a list of what a fair evaluation should actually measure: prediction accuracy, memory footprint and model size, inference latency, and energy efficiency in deployment. Measured together, the four give a fuller picture of both performance and resource use than accuracy alone.
Photo by Marija Zaric on Unsplash
Why a smaller model is also a cleaner one
The environmental argument runs through the paper as a second thread. Training large-scale machine learning models means long runs on power-hungry accelerators, and that translates into energy demand and carbon emissions. The review points to published work estimating the carbon footprint of BLOOM, a 176-billion-parameter language model, as an illustration of what scale costs. Its claim for distillation is narrower than the headlines that usually surround Green AI: less intensive GPU use, shorter training runs, lower total emissions.
One thing the paper's own framing makes clear is that the teacher still had to be trained. Distillation reduces the bill for everything that comes afterwards, including the inference performed millions of times on cheap hardware, but it does not refund the cost of building the original model. That distinction matters for anyone tempted to treat compression as a substitute for efficiency in the first place. The full text is available open access under a CC BY 4.0 licence through the journal's published PDF.
Where the memory budget is counted in kilobytes
TinyML is the paper's term for machine learning that runs on devices with very little memory and compute, and it is where the arithmetic gets unforgiving. The review cites MicroNet, presented at CVPR 2024, as an example of designs built to be extremely compact. It also notes that distillation can produce lightweight variants of long-standing architectures such as LSTMs and convolutional networks, which then run with faster inference and lower power draw on real edge hardware.
The applications the authors name are the ordinary infrastructure of a connected economy: internet-of-things sensors, smartphones, embedded controllers. For anything that has to respond in real time on a battery, latency and wattage matter as much as accuracy, and a model that is 60 percent smaller is often the difference between a feature that ships and one that never leaves the lab. This is the paper's central practical claim, and it is a claim about feasibility as much as about performance.
Copying the working, not only the conclusion
Basic distillation transfers knowledge at the output layer. The paper spends more space on multi-stage approaches, which also transfer hidden-to-visible feature representations and the internal activations a teacher produces along the way. The student then learns structural patterns rather than a final answer key, which the authors argue leads to deeper and more durable learning.
They cite multi-stage work presented at NeurIPS 2023 that copies intermediate reasoning patterns between teacher and student, allowing a small model to reproduce behaviours closer to chain-of-thought problem solving. A related strand the review follows concerns distribution shift. In autoregressive settings, students that generate their own outputs during inference can drift away from what the teacher would have produced, and work published in 2024 on generalised distillation in autoregressive models addresses that instability directly. These are the parts of the field where the interesting failures live, and the paper treats them as open problems rather than settled ones.
What the review leaves open
Read closely, the paper is a map of published evidence, and its numbers are aggregates across many architectures, datasets and hardware platforms rather than outputs from a single controlled experiment. That is useful for orientation and weak for prediction. A 50 to 70 percent size reduction on one family of models does not guarantee the same on another, and the review does not report a standardised benchmark, a hardware testbed or measured energy figures from its own trials.
Sceptics in the field raise a harder point. Distillation compresses what a teacher knows; it cannot add capability the teacher never had, and a student with a fraction of the parameters has a ceiling on how much internal reasoning it can hold. Accuracy parity claims are also task-dependent: a distilled classifier that matches its teacher on clean benchmark images may fall apart on noisy field data, and the review itself flags distribution shift as a live concern. There is a training-side cost as well, since every iteration of distillation needs a forward pass through the teacher, which lengthens training runs even as it shortens inference.
The paper lists no industry partner and relies entirely on published literature, which keeps it independent but also means no new hardware was tested. The record of the article, including its volume, issue and page range, is indexed on the journal's article page.
Next: reasoning, pruning and microwatts
The authors close with a short agenda. They want better transfer of higher-order reasoning skills, so that small models inherit more than surface accuracy. They want distillation combined with pruning, which removes weights outright, on the theory that the two compression methods attack different kinds of redundancy. And they want design methods built specifically for very low-power, resource-constrained devices, rather than scaled-down versions of methods meant for servers.
The surrounding literature they survey shows how fast that agenda is moving. MiniLLM and LLM-Pruner, both cited in the paper, apply distillation and structural pruning to large language models. A 2026 survey of distillation for large language models and recent preprints on feature alignment and cross-modal transfer suggest the frontier has shifted from shrinking image classifiers to shrinking systems that write and reason.
Where that leads over the next five to ten years is easier to describe than to guarantee. A phone that runs a competent assistant offline, a field sensor that diagnoses a crop disease without a network connection, a clinic in a low-bandwidth region running triage on a device the size of a matchbox: each becomes plausible when a model that once needed a server rack spends its life in a few hundred kilobytes of memory. The Bengaluru review does not promise that future. It puts credible numbers on the trade that would make it possible, and is candid about which parts of the trade remain unmeasured.
- edge-computing
- neural-networks
- knowledge-distillation
- tinyml
- green-ai

