Build a DeepSeek Model (From Scratch) (From Scratch)

Get a free eBook (PDF or ePub) from Manning as well as access to the online liveBook format (and its AI assistant that will answer your questions in any language) when you purchase the print book.

When DeepSeek started making waves in January 2025, it sounded too good to be true. How could a generative AI model get such incredible performance with such low training and operation costs? By creatively blending a variety of strategies and innovations like Mixture of Experts, Latent Attention, Multi-token Prediction, model distillation, and efficient parallelization, DeepSeek set a new standard for what’s possible in an open LLM.

Now, in this book you can recreate a laptop-scale version of this cutting-edge model yourself! Learn how to build the features that set DeepSeek apart from other top LLMs!

In Build a DeepSeek Model (From Scratch) you will learn how to:

• Implement DeepSeek’s core architectural innovations, including Multi-Head Latent Attention and Mixture-of-Experts layers
• Build a production-ready training pipeline with Multi-Token Prediction and FP8 quantization for efficiency and speed
• Maximize hardware utilization with parallelism strategies like DualPipe
• Apply post-training methods such as supervised fine-tuning and reinforcement learning to unlock reasoning capabilities
• Compress and distill large models into smaller, deployable versions for real-world use

In Build a DeepSeek Model (From Scratch) you’ll build your own DeepSeek clone from the ground up. First, you’ll quickly review LLM fundamentals, with an eye to where DeepSeek’s innovations address the common problems and limitations of standard models. Then, you’ll learn everything you need to create your own DeepSeek-inspired model, including the innovations that put DeepSeek on the map: Multihead Latent Attention (MLA), Multi-Token Prediction (MTP), Mixture of Experts (MoE), model distillation, and reasoning.

About the book

Build a DeepSeek Model (From Scratch) uses intuitive visualizations, code walkthroughs, and a problem-solution narrative to transform complex concepts into practical skills. You will start by coding a DeepSeekAttention module, progress to building a fully functional MoE layer, and set up a high-efficiency training pipeline. By the end of the book, you will have a fully operational mini-DeepSeek that runs on your laptop, along with the skills to extend and optimize it for your own research or production applications.

About the reader

For intermediate-to-advanced ML engineers, AI researchers, and graduate students who want to go beyond prebuilt models. You'll need to know deep learning and Python programming.

About the author

Dr. Raj Abhijit Dandekar is a computer scientist and co-founder of Vizuara AI Labs, an online education platform that has trained over 50,000 students globally. He holds a PhD from MIT and is the lead instructor of the popular YouTube series Build DeepSeek from Scratch.

Dr. Rajat Dandekar, PhD in Mechanical Engineering from Purdue University, specializes in applying machine learning to complex physical systems. He co-founded Vizuara AI Labs.

Naman Dwivedi is an AI researcher at Vizuara AI Labs, specializing in turning advanced deep learning concepts into hands-on, practical code.

Dr. Sreedath Pana holds a PhD from MIT and is a co-founder of Vizuara AI Labs. He is an inventor and AI engineer known for creating self-cleaning AI-powered solar technology.