Build a Large Language Model (From Scratch)

Sebastian Raschka's 368-page Manning book (September 2024, ISBN 9781633437166) is structured as seven chapters plus five appendices, each mapped to runnable notebooks in the companion repository rasbt/LLMs-from-scratch. Chapter 1 frames what an LLM is and ships no code; Chapter 2 covers working with text data — tokenization, byte-pair encoding and a sliding-window data loader; Chapter 3 codes attention mechanisms from a single dot product up to causal multi-head attention; Chapter 4 assembles the full GPT architecture; Chapter 5 pretrains it on unlabeled text and adds temperature and top-k sampling plus loading OpenAI's public GPT-2 weights; Chapter 6 finetunes it as a text classifier; Chapter 7 does instruction finetuning. Appendix A teaches PyTorch from zero including distributed data parallel, Appendix C gives full exercise solutions, Appendix D adds training refinements, and Appendix E implements LoRA. The repository has 103.7k GitHub stars and 1,062 commits, is CI-tested on Linux, Windows and macOS, and carries bonus material far beyond the printed book: standalone BPE tokenizer implementations, efficient-attention comparisons, KV caching, grouped-query, multi-head-latent and sliding-window attention, DeltaNet, mixture-of-experts, learning-rate schedulers, hyperparameter tuning, Direct Preference Optimization, and ports of the same code to the Llama 3.2, Qwen3, Gemma 3 and Olmo 3 architectures. Raschka holds a PhD, works as an LLM research engineer, was previously a senior staff engineer at Lightning AI, taught statistics at the University of Wisconsin-Madison, and wrote the bestselling Machine Learning with PyTorch and Scikit-Learn. Everything is deliberately sized to run on a conventional laptop, using a GPU automatically if one is present.

Topic

ML

Level

Intermediate

Format

Book

Cost

Paid

Duration

368 pages, 7 chapters + 5 appendices; ~30-40 hours working through the notebooks

Provider

Manning / Sebastian Raschka

Hands-on

Yes — code/exercises

Certificate

None

• ✓Tokenize raw text and build a byte-pair-encoding sliding-window data loader for pretraining
• ✓Implement self-attention, causal masking and multi-head attention in PyTorch from first principles
• ✓Assemble a complete GPT-style transformer with layer norm, GELU and residual connections
• ✓Pretrain the model on unlabeled text and implement temperature and top-k sampling
• ✓Load OpenAI's public GPT-2 weights into your own implementation and verify the outputs match
• ✓Finetune the model into a spam classifier by replacing and retraining the output head
• ✓Run instruction finetuning on an Alpaca-style dataset and evaluate the generated responses
• ✓Add parameter-efficient LoRA finetuning using the appendix E reference implementation

• •The companion repo rasbt/LLMs-from-scratch has 103.7k stars and 1,062 commits — one of the most-starred learning repositories on GitHub
• •Every example is sized to run on a conventional laptop; no GPU cluster, cloud account or API key is required
• •The code is CI-tested on Linux, Windows and macOS, so the notebooks still execute years after the print date
• •Bonus material goes well past the book: KV caching, grouped-query and multi-head-latent attention, DeltaNet, mixture-of-experts and DPO
• •Includes side-by-side ports of the same architecture to Llama 3.2, Qwen3, Gemma 3 and Olmo 3

This page was written from 2 sources, 1 on domains other than manning.com.

1. 1.manning.com — build a large language model from scratchvendor
2. 2.github.com — LLMs from scratch