CUDA for LLMs

Get the eBook free when you register your print book at Manning.

CUDA (Compute Unified Device Architecture) provides a powerful parallel programming model AI engineers can use to tap the massive processing power of NVIDIA GPUs. CUDA delivers direct control, debugging power, and acceleration at the GPU level that can’t be matched by other types of optimizations.

This book shows you how to work within the CUDA ecosystem, from your first kernel to implementing advanced LLM features like Flash Attention. You’ll learn to profile with Nsight Compute, identify bottlenecks, and understand why each optimization works. By solving problems at multiple levels of abstraction, you’ll develop a deep understanding of CUDA, along with a practical mastery of kernel-building skills. Written for the latest NVIDIA hardware, the book builds a deep understanding of CUDA fundamentals that will stay relevant as chips upgrade and evolve.

What's inside

• 56 kernels to utilize in your models
• PyTorch C++ extension pipeline for integrating custom kernels
• Exploit advanced NVIDIA GPU features (Ampere, Hopper, Blackwell)
• Build backpropagation from scratch, ending with a single-file MNIST MLP

About the reader

For software and AI engineers comfortable with C/C++. No prior CUDA experience required.

About the author

Elliot Arledge created the 12-hour CUDA course and the 6-hour LLM from Scratch course for FreeCodeCamp, and consults on deep learning performance.