AI Distillation: How DeepSeek Used It to Reshape the AI Landscape

May 6 / AI Degree
The world of Artificial Intelligence (AI) is undergoing a seismic shift, triggered in large part by a powerful, yet seemingly simple, technique known as AI distillation. This method, brought into the mainstream spotlight by the Chinese startup DeepSeek, is challenging the long-held dominance of tech giants like OpenAI and Google, making advanced AI models more accessible and affordable than ever before.

You can listen to this podcast:

What is AI Distillation?

At its core, distillation is a technique where a smaller team with limited resources can create an advanced AI model by extracting knowledge from a larger, more complex one. Geoffrey Hinton, often referred to as the "godfather of AI," first coined the term in a 2015 paper while working at Google, describing it as a way to "transfer the knowledge from a large, cumbersome model to a small model that's more suitable for deployment".

Here's how it generally works:

  • A leading tech company invests years and significant capital to develop a top-tier AI model, training it on massive datasets with substantial computing power.
  • A smaller team then "swoops in". Instead of starting from scratch, they apply knowledge distillation by repeatedly querying the large, advanced model and using its reasoning and responses to train their own smaller, more specialised model.
  • The result is a model that is nearly as capable but significantly faster, more efficient, and far less resource-intensive. This technique is described as "extremely powerful and so extremely cheap," making it "available to anyone".

DeepSeek's Breakthrough and its Market Impact

While Google was actually a pioneer in distillation thanks to Hinton's research, and was already using it to optimise lightweight versions of its Gemini models, it was DeepSeek that "woke up the AI world to its disruptive potential" and "turned it into the story". DeepSeek demonstrated to Wall Street just how effective distillation could be.
DeepSeek achieved impressive results, seemingly mimicking and even surpassing OpenAI's advancements in just two months, spending less than $6 million on the final training phase. While its success isn't solely attributed to "copying" – DeepSeek also applied "clever innovations" and made "fundamental improvements" – distillation was a major factor in its rapid ascent, paving the way for other less-capitalised startups and research labs to compete at the cutting edge faster than ever.
Examples of this cost-effective innovation include:
  • Berkeley researchers created models "almost as smart as" OpenAI's O1 reasoning model for just $450 in 19 hours using only eight Nvidia H100 chips.
  • Stanford and the University of Washington created their S1 reasoning model in just 26 minutes, using less than $50 in compute credits.
  • Hugging Face, an open-source AI platform, hosted a 24-hour challenge that resulted in an open-source AI research agent, Open Deep Research.
These developments raise a critical question: why are big tech firms still investing billions to push the frontier when others can distill their work for significantly less time and money?

The Dawn of a New Open-Source Order

The rise of distillation has accelerated a "paradigm shift" towards a new open-source order, where transparency and accessibility are believed to drive innovation faster than closed-door research. As one expert joked, "everybody's model is open source. They just don't know it yet... because it's so easy to distill them".
DeepSeek itself open-sourced its V3 and R1 models, providing a blueprint for anyone to download, use, and modify for free. University and Hugging Face models followed suit, putting pressure on OpenAI's once-secure closed-source strategy, which now looks "more like a target".
This shift has been acknowledged even by OpenAI's CEO, Sam Altman, who notably stated on Reddit that OpenAI has been "on the wrong side of history" regarding its open-source strategy and needs to "figure out a different open source strategy". This is a remarkable admission from a leader who previously championed the closed-source approach for safety, competitive dynamics, and monetisation.
The benefits of this open-source momentum are significant:
  • Cost Reduction: Distilled models can be created cheaply and are often given away for free by universities and startups. This "keeps the proprietary players in check from a pricing and performance standpoint".
  • Developer Empowerment: Developers can run open-source models on their own infrastructure, reducing the pricing power of proprietary providers. For example, DeepSeek R1 costs $2.19 per million tokens, while OpenAI's comparable O1 costs $60. This drastic cost difference means the "script has now flipped" for AI application makers, giving them a significant advantage.
  • Increased Innovation & Use Cases: Making AI more efficient leads to "a dramatic increase in more use cases". It's a win for developers, those building at the AI application layer, and the long-term AI ecosystem due to the "deflationary effect of the cost of these models".
  • Commoditisation of Models: AI model builders who were on top are becoming "more and more commoditized". The market is transitioning from a few dominant model builders to "many, many more" comparable models at much lower training costs.

Implications for Big Tech and the Pursuit of AGI

Despite these disruptive trends, AI distillation has not fundamentally changed the calculus for the biggest players in AI. Companies like Microsoft, Google, Amazon, Meta, and OpenAI are still investing billions and even ramping up their capital expenditures.

This continued investment is driven by the "AGI at all costs strategy" – the relentless pursuit of Artificial General Intelligence (AGI), which aims to surpass human intelligence in all aspects, including creativity and problem-solving. While distillation offers cost-effective performance gains, it is understood that it "does not drive the revolutionary breakthroughs needed to reach AGI or ASI (Artificial Superintelligence)". For these frontier players, reaching AGI first is worth any investment, highlighting that the "race has never been more urgent".

However, the plunging cost of running AI applications means that faster, smaller, and more targeted distilled models are increasingly "good enough" for enterprises integrating AI into their businesses. Many Fortune 100 companies are now questioning the premium they pay for proprietary APIs from OpenAI and Anthropic.

The "real power of LLMs has always been reasoning," and these capabilities are on the rise. This allows for the automation of more complex business processes within enterprises. While enterprises initially prioritise building an agent that works, cost considerations become increasingly important for scaling. This will likely lead to pricing pressure on proprietary models and a continued shift towards more competitive, commoditised AI offerings.

In conclusion, AI distillation, spearheaded into public awareness by DeepSeek, has democratised access to powerful AI models, fostered a new open-source order, and dramatically lowered the cost of running AI applications. This transformative technique is driving unprecedented innovation and competition, compelling even industry leaders to re-evaluate their strategies in an AI landscape that is evolving faster than ever before.

Your Role in This New Era of AI

AI distillation has made cutting-edge AI more accessible than ever—lowering costs, accelerating innovation, and opening the field to smaller teams and individuals.
If you’re inspired to build in this new wave, AI Degree offers the skills you need. Learn practical techniques like distillation, LLM evaluation, and agent design through expert-led courses and hands-on projects.

Start building smarter AI today at aidegree.org.

Start Your AI Journey Today!

If you’re ready to become an elite AI developer, now is the time to take action.
To truly harness the power of AI, you need more than just curiosity—you need expertise. The AI Degree program offers a comprehensive, flexible curriculum that lets you learn at your own pace.

From foundational topics to advanced AI development, you’ll gain the skills needed to excel in this dynamic field. Scholarships make it accessible to everyone, and optional ECTS credits provide global recognition.

Start your journey today. Explore free courses, apply for scholarships, and begin building the future of AI—your future. Learn More Here.