Academic Jobs - Home of Higher Ed Logo

Research spotlight: Anoushka Nayak on shrinking AI to fit tiny, low-power devices

8 min read0 views

At a glance

A review by Anoushka Nayak and colleagues at RV Institute of Technology and Management in Bengaluru reports that knowledge distillation preserves 90 to 97 percent of a teacher model's accuracy while cutting size by 50 to 70 percent and speeding up inference two to five times.

trees with be known text overlay
Photo by Griest Projects on Unsplash

Strip a neural network down to half its size and it can still get roughly nine answers in ten right. It will also respond two to five times faster than the model that taught it. Those are the operating numbers in a review published in April in the International Journal of Creative Research Thoughts, written by Arfa Yunus, Anoushka Nayak, Aisiri S, Aishwarya R and Hema M S, all in the Department of Computer Science Engineering at RV Institute of Technology and Management in Bengaluru.

The timing matters. The systems that set the pace in machine learning keep getting bigger, while the places that could use them keep getting smaller. A warehouse sensor, a hearing aid, a phone in a clinic with intermittent power: none of them can carry a data-centre GPU or the electricity bill that comes with one. Knowledge distillation is the bridge the Bengaluru team examines, a training method that folds the behaviour of a large model into a small one so the small one performs above its weight class.

The paper is a synthesis rather than a bench experiment. It traces the technique from its 2015 formulation, works through multi-stage variants, transformer distillation and TinyML deployments, then gathers published results into ranges. Its verdict is deliberately measured: distillation narrows the disparity between highly complex models and environments with fewer computational resources, cutting energy use and enabling deployment on low-power hardware without a large loss in performance.

An apprentice that copies the reasoning, not only the answers

The standard setup has three moving parts. A teacher: large, pre-trained, accurate, expensive. A student: small, cheap to run, designed around the memory and power it will actually have. And an objective function that tells the student what to imitate. The student trains on the labelled dataset as any model would, and simultaneously on the teacher's outputs.

That second signal carries the value. A hard label says a photograph is a seven. The teacher's probability distribution says it is probably a seven, plausibly a one, and almost certainly not a four. The ranking of near-misses is information a single label cannot convey. The review calls these soft targets, and the comparison holds: a chess coach who says a move is strong but slightly risky teaches more than a coach who only marks it right or wrong.

Temperature scaling is the dial that controls how much of that texture reaches the student. Before the teacher's raw scores are converted into probabilities, they are divided by a temperature value. Raise it and the distribution flattens, lifting small probabilities and exposing which wrong answers the teacher considered plausible; lower it and the teacher looks certain about everything, which is precisely the signal you don't want copied. During training the student minimises a blend of two terms, one measured against the true labels, the other against those softened outputs.

What the review puts on the table

Section four of the paper converts scattered published results into a compact set of ranges. The authors report that distilled models keep about 90 to 97 percent of their teacher's accuracy in most instances, that model size can fall by roughly 50 to 70 percent, and that inference can run two to five times faster.

OutcomeRange reported in the review
Accuracy retained against the teacher model90 to 97 percent
Reduction in model size50 to 70 percent
Gain in inference speed2 to 5 times

DistilBERT is the clearest single case in the text. Sanh and colleagues reported that a slimmed-down version of BERT reached about 97 percent of the original's performance while being considerably smaller and faster to run, which demonstrated that distillation works on transformer architectures and not just on older convolutional networks. The authors pair those figures with a list of what a fair evaluation should actually measure: prediction accuracy, memory footprint and model size, inference latency, and energy efficiency in deployment. Measured together, the four give a fuller picture of both performance and resource use than accuracy alone.

a set of stairs with a poem written on them

Photo by Marija Zaric on Unsplash

Why a smaller model is also a cleaner one

The environmental argument runs through the paper as a second thread. Training large-scale machine learning models means long runs on power-hungry accelerators, and that translates into energy demand and carbon emissions. The review points to published work estimating the carbon footprint of BLOOM, a 176-billion-parameter language model, as an illustration of what scale costs. Its claim for distillation is narrower than the headlines that usually surround Green AI: less intensive GPU use, shorter training runs, lower total emissions.

One thing the paper's own framing makes clear is that the teacher still had to be trained. Distillation reduces the bill for everything that comes afterwards, including the inference performed millions of times on cheap hardware, but it does not refund the cost of building the original model. That distinction matters for anyone tempted to treat compression as a substitute for efficiency in the first place. The full text is available open access under a CC BY 4.0 licence through the journal's published PDF.

Where the memory budget is counted in kilobytes

TinyML is the paper's term for machine learning that runs on devices with very little memory and compute, and it is where the arithmetic gets unforgiving. The review cites MicroNet, presented at CVPR 2024, as an example of designs built to be extremely compact. It also notes that distillation can produce lightweight variants of long-standing architectures such as LSTMs and convolutional networks, which then run with faster inference and lower power draw on real edge hardware.

The applications the authors name are the ordinary infrastructure of a connected economy: internet-of-things sensors, smartphones, embedded controllers. For anything that has to respond in real time on a battery, latency and wattage matter as much as accuracy, and a model that is 60 percent smaller is often the difference between a feature that ships and one that never leaves the lab. This is the paper's central practical claim, and it is a claim about feasibility as much as about performance.

Copying the working, not only the conclusion

Basic distillation transfers knowledge at the output layer. The paper spends more space on multi-stage approaches, which also transfer hidden-to-visible feature representations and the internal activations a teacher produces along the way. The student then learns structural patterns rather than a final answer key, which the authors argue leads to deeper and more durable learning.

They cite multi-stage work presented at NeurIPS 2023 that copies intermediate reasoning patterns between teacher and student, allowing a small model to reproduce behaviours closer to chain-of-thought problem solving. A related strand the review follows concerns distribution shift. In autoregressive settings, students that generate their own outputs during inference can drift away from what the teacher would have produced, and work published in 2024 on generalised distillation in autoregressive models addresses that instability directly. These are the parts of the field where the interesting failures live, and the paper treats them as open problems rather than settled ones.

selective focus photography of Productivity printed book

Photo by kris on Unsplash

What the review leaves open

Read closely, the paper is a map of published evidence, and its numbers are aggregates across many architectures, datasets and hardware platforms rather than outputs from a single controlled experiment. That is useful for orientation and weak for prediction. A 50 to 70 percent size reduction on one family of models does not guarantee the same on another, and the review does not report a standardised benchmark, a hardware testbed or measured energy figures from its own trials.

Sceptics in the field raise a harder point. Distillation compresses what a teacher knows; it cannot add capability the teacher never had, and a student with a fraction of the parameters has a ceiling on how much internal reasoning it can hold. Accuracy parity claims are also task-dependent: a distilled classifier that matches its teacher on clean benchmark images may fall apart on noisy field data, and the review itself flags distribution shift as a live concern. There is a training-side cost as well, since every iteration of distillation needs a forward pass through the teacher, which lengthens training runs even as it shortens inference.

The paper lists no industry partner and relies entirely on published literature, which keeps it independent but also means no new hardware was tested. The record of the article, including its volume, issue and page range, is indexed on the journal's article page.

Next: reasoning, pruning and microwatts

The authors close with a short agenda. They want better transfer of higher-order reasoning skills, so that small models inherit more than surface accuracy. They want distillation combined with pruning, which removes weights outright, on the theory that the two compression methods attack different kinds of redundancy. And they want design methods built specifically for very low-power, resource-constrained devices, rather than scaled-down versions of methods meant for servers.

The surrounding literature they survey shows how fast that agenda is moving. MiniLLM and LLM-Pruner, both cited in the paper, apply distillation and structural pruning to large language models. A 2026 survey of distillation for large language models and recent preprints on feature alignment and cross-modal transfer suggest the frontier has shifted from shrinking image classifiers to shrinking systems that write and reason.

Where that leads over the next five to ten years is easier to describe than to guarantee. A phone that runs a competent assistant offline, a field sensor that diagnoses a crop disease without a network connection, a clinic in a low-bandwidth region running triage on a device the size of a matchbox: each becomes plausible when a model that once needed a server rack spends its life in a few hundred kilobytes of memory. The Bengaluru review does not promise that future. It puts credible numbers on the trade that would make it possible, and is candid about which parts of the trade remain unmeasured.

  • edge-computing
  • neural-networks
  • knowledge-distillation
  • tinyml
  • green-ai
Portrait of Jarrod Kanizay
About the author

Jarrod KanizayView author

Academic Jobs In House Author

Visionary leader transforming academic recruitment with 20+ years in higher education.

Discussion

Sort by:

Be the first to comment on this article!

You

You’ll be asked to sign in before your comment is posted.

Browse by Faculty

Browse by Subject

Frequently Asked Questions

🎓What is knowledge distillation in simple terms?

Knowledge distillation is a training method in which a small model, called the student, is taught to imitate a large pre-trained model, called the teacher. The student learns from the true labels in the dataset and also from the teacher's probability outputs, which reveal how the teacher ranked plausible answers to each question.

The paper defines it as a way to transfer knowledge from a complex teacher model to a smaller student so the student achieves similar performance with far lower computational cost.

📄Who carried out this research and where was it published?

The authors are Arfa Yunus, Anoushka Nayak, Aisiri S, Aishwarya R and Hema M S, all affiliated with the Department of Computer Science Engineering at RV Institute of Technology and Management in Bengaluru, India.

The paper, titled Efficient Knowledge Distillation For Resource-Constrained Neural Networks, appeared in the International Journal of Creative Research Thoughts in April 2026, volume 14, issue 4, pages i544 to i550, under a CC BY 4.0 open access licence.

📉What accuracy trade-off does the review report?

The paper reports that distilled models retain roughly 90 to 97 percent of the accuracy of their original teacher models in most instances, while being substantially smaller and faster.

Those ranges are compiled from previously published studies across different architectures and tasks, so they describe the general pattern in the literature rather than a result measured in a single new experiment.

⚡How much smaller and faster can a distilled model be?

According to the review, model size can be reduced by about 50 to 70 percent, and inference can run two to five times faster than the larger teacher model.

DistilBERT, a distilled version of BERT discussed in the paper, was reported by its original authors to reach about 97 percent of BERT's performance while being much smaller and quicker to run.

📱What is TinyML and why does it matter here?

TinyML refers to running machine learning models on devices with very limited memory and computing power, such as microcontrollers and embedded sensors. The paper explains that distillation produces lightweight models suitable for internet-of-things devices, smartphones and embedded systems where low latency and low power draw are essential.

The review cites MicroNet, presented at CVPR 2024, as an example of work aimed at extremely compact network designs.

🌱How does knowledge distillation relate to Green AI?

Green AI is the effort to make machine learning more energy efficient and environmentally sustainable. The paper argues that distillation supports these goals by reducing intensive GPU use, shortening training time and lowering total carbon emissions.

It cites published research estimating the carbon footprint of BLOOM, a 176-billion-parameter language model, to illustrate the environmental weight of very large training runs.

🔍Does the paper report new experiments of its own?

No. The article is a review that surveys background research and recent advances, then consolidates reported results into ranges. It describes the distillation framework, multi-stage approaches and efficiency metrics, but it does not present a new benchmark, hardware testbed or original energy measurement.

Readers should therefore treat its figures as a summary of the field rather than as independent verification of any specific deployment.

🌡️What is temperature scaling in distillation?

Temperature scaling is a parameter applied to a model's raw output scores before they become probabilities. Raising the temperature softens the distribution, making the teacher's smaller probabilities visible and showing which incorrect answers it considered plausible.

The paper describes this as part of the original formulation by Hinton and colleagues in 2015, where soft targets produced better generalisation than hard labels alone.

⚠️What are the known limitations of knowledge distillation?

A student model cannot exceed the capability of the teacher or of its own parameter budget, and matching accuracy on a clean benchmark does not guarantee matching accuracy on noisy real-world inputs. The review also points to distribution shift, where students that generate their own outputs during inference drift from the teacher's behaviour.

On the training side, each distillation step requires a forward pass through the teacher, which extends training time even as it reduces inference cost.

🧠Can distillation transfer reasoning abilities, not just answers?

That is an active research question. The paper describes multi-stage distillation, in which students learn from intermediate activations and internal representations rather than only final outputs, and cites NeurIPS 2023 work that transfers patterns closer to chain-of-thought reasoning.

The authors list better transfer of higher-order reasoning skills as a priority for future work, which implies it remains incomplete.

🧭What future directions do the authors recommend?

The conclusion names three priorities: improving the transfer of higher reasoning skills, combining distillation with pruning and related compression methods, and developing techniques designed specifically for extremely low-power and resource-constrained devices.

The paper also surveys work applying distillation to large language models, including MiniLLM and LLM-Pruner, suggesting that language systems are the next major target.

🔗Where can I read the full paper?

The article is open access under a Creative Commons Attribution 4.0 licence and is hosted by the International Journal of Creative Research Thoughts. The complete text is available as a PDF on the journal site.

Its citation is: Arfa Yunus, Anoushka Nayak, Aisiri S, Aishwarya R, Hema M S, Efficient Knowledge Distillation For Resource-Constrained Neural Networks, IJCRT, volume 14, issue 4, pages i544 to i550, April 2026.