LEARN
Understand LLMs from Zero
These pages go with the 100-episode video series. After each video, run the same mechanism yourself and type in your own words to see what happens.
Part 1The Big Picture
- #1What Is an LLM? Large Language Models Predict the Next WordOpen
- #2How LLMs Generate Text: Autoregressive, One Token at a TimeOpen
- #3LLM Output Is a Probability Distribution: Next-Token PredictionOpen
- #4Bigram Language Model: The Simplest Language Model, Built by CountingOpen
- #5The Limits of N-gram Language Models: Why LLMs Need Neural NetworksOpen
- #6How LLMs Turn Text into Numbers: Token IDs ExplainedOpen
- #7What Are LLM Parameters? A Model Is One Giant FunctionOpen
- #8The Full LLM Pipeline: Tokenization, Transformer, GenerationOpen
Part 2Tokenization
- #9What Is Tokenization? Characters vs Words vs Subword TokensOpen
- #10Byte Pair Encoding (BPE) Part 1: Merging the Most Frequent PairOpen
- #11BPE Part 2: Training a Tokenizer VocabularyOpen
- #12Tokenizer Encode and Decode: From Text to Token IDs and BackComing soon
- #13Why Token Counts Differ by Language: LLM Tokenization ComparedOpen
- #14Special Tokens and Chat Templates: How an LLM Reads a ConversationComing soon
- #15What Is a Context Window? How Much an LLM Can Read at OnceComing soon
- #16Why LLMs Fail at Spelling and Arithmetic: Tokenization PitfallsComing soon
Part 3Vectors and Embeddings
- #17What Is a Vector? How LLMs Represent Words as NumbersComing soon
- #18Dot Product and Cosine Similarity: How LLMs Measure MeaningComing soon
- #19One-Hot Vectors and Their Limits: Why LLMs Need EmbeddingsComing soon
- #20What Is an Embedding? The Token Embedding Table ExplainedComing soon
- #21Word Embedding Space: Why Similar Words Sit Close TogetherComing soon
- #22Word Vector Arithmetic: King − Man + Woman = Queen (word2vec)Coming soon
- #23What Is a Matrix? The Linear Transformations Behind Every LLMComing soon
- #24High Dimensions: Why LLM Embeddings Have Hundreds of DimensionsComing soon
- #25How Does an LLM Know Word Order? The Position ProblemComing soon
Part 4How Neural Networks Learn
- #26What Is a Neuron? Weights and Biases in a Neural NetworkComing soon
- #27Activation Functions (ReLU, GELU): Nonlinearity in Neural NetworksComing soon
- #28Multilayer Perceptron (MLP): Stacking Layers in a Neural NetworkComing soon
- #29The Softmax Function: Turning Logits into ProbabilitiesComing soon
- #30Loss Function and Cross-Entropy: Measuring an LLM's MistakesComing soon
- #31What Is a Gradient? The Direction of Neural Network LearningComing soon
- #32Gradient Descent by Hand: The Core Algorithm of Machine LearningComing soon
- #33Backpropagation Part 1: The Chain RuleComing soon
- #34Backpropagation Part 2: Computation Graphs and AutogradComing soon
- #35Mini-Batches and Learning Rate: Step Size in Neural Network TrainingComing soon
- #36Overfitting and Generalization: When Machine Learning MemorizesComing soon
- #37Building a Neural Language Model: Embeddings + MLPComing soon
Part 5Attention
- #38What Is Attention? How LLMs Read ContextComing soon
- #39Query, Key, Value: The Three Roles in the Attention MechanismComing soon
- #40Attention by Hand: Scaled Dot-Product Attention Step by StepComing soon
- #41Attention Scaling: Why Divide by √d Before SoftmaxComing soon
- #42Causal Mask: How GPT Avoids Looking at Future TokensComing soon
- #43Self-Attention as Matrix Math: The Core of the TransformerComing soon
- #44Multi-Head Attention: Reading Context from Several ViewpointsComing soon
- #45What Do Attention Heads Look At? Visualizing LLM AttentionComing soon
- #46Positional Encoding: Transformer Positions with Sine and CosineComing soon
- #47RoPE (Rotary Position Embedding): How Modern LLMs Encode PositionComing soon
- #48Attention Complexity: Why Long Context Makes LLMs SlowComing soon
- #49Building an Attention-Only Language Model: A One-Layer TransformerComing soon
Part 6Building the Transformer
- #50Residual Connections: Why Deep Transformers Can Be TrainedComing soon
- #51Layer Normalization: Keeping the Transformer StableComing soon
- #52The Feed-Forward (MLP) Layer: The Other Half of a TransformerComing soon
- #53Building a Transformer Block: Attention + MLP + ResidualsComing soon
- #54Stacking Transformer Layers: What Depth Gives an LLMComing soon
- #55The Residual Stream: How Information Flows Through a TransformerComing soon
- #56Logits and the Unembedding Layer: From Vectors Back to TokensComing soon
- #57Counting LLM Parameters: Where Are GPT's Weights?Coming soon
- #58How GPT Works, End to End: From Input to Next-Token PredictionComing soon
Part 7Training
- #59LLM Training Data: Self-Supervised Next-Token PredictionComing soon
- #60Reading a Loss Curve: Diagnosing LLM TrainingComing soon
- #61Optimizers for LLM Training: From SGD to Adam and AdamWComing soon
- #62Learning Rate Schedules and Warmup in LLM TrainingComing soon
- #63Weight Initialization: Avoiding Vanishing and Exploding GradientsComing soon
- #64Dropout and Weight Decay: Regularizing Neural NetworksComing soon
- #65Train a Mini GPT in Your Browser: A Transformer from ScratchComing soon
- #66What Does an LLM Learn During Training? Stages of LearningComing soon
- #67Scaling Laws: Why Bigger LLMs Are SmarterComing soon
- #68Perplexity: Measuring Language Model QualityComing soon
Part 8Generation
- #69Greedy Decoding: The Simplest Way an LLM Generates TextComing soon
- #70What Is Temperature? Controlling LLM RandomnessComing soon
- #71Top-k and Top-p (Nucleus) Sampling in LLMsComing soon
- #72Repetition Penalty: Stopping an LLM from Repeating ItselfComing soon
- #73Beam Search: LLM Text Generation That Looks AheadComing soon
- #74KV Cache: How LLM Inference Gets FastComing soon
- #75EOS Token and Max Tokens: When Does an LLM Stop?Coming soon
- #76Structured Output and JSON Mode: Constraining an LLM with a GrammarComing soon
Part 9From Predictor to Assistant
- #77What Is Pre-training? How an LLM Foundation Model Is MadeComing soon
- #78What Is Fine-Tuning? Teaching an LLM Something NewComing soon
- #79Instruction Tuning: How an LLM Becomes a Chat AssistantComing soon
- #80RLHF: Reinforcement Learning from Human Feedback and LLM AlignmentComing soon
- #81What Is LoRA? Fine-Tuning an LLM with Few ParametersComing soon
- #82System Prompts and In-Context Learning in LLMsComing soon
- #83Few-Shot Prompting: Why Prompt Engineering WorksComing soon
- #84Chain of Thought and Reasoning Models: LLMs That Think FirstComing soon
Part 10Looking Inside
- #85Logit Lens: Reading an LLM's Middle Layers (Interpretability)Coming soon
- #86Can You Trust Attention Maps? LLM InterpretabilityComing soon
- #87Superposition and Features in LLMs: An Intro to Sparse AutoencodersComing soon
- #88LLM Circuits: Tracing Computation with Mechanistic InterpretabilityComing soon
- #89Why Do LLMs Hallucinate? An Explanation from the MechanismComing soon
- #90Memorization vs Generalization in LLMs: GrokkingComing soon
Part 11Efficient and Practical
- #91What Is Quantization? Making LLMs Smaller and FasterComing soon
- #92Knowledge Distillation: How Small Language Models (SLMs) Are MadeComing soon
- #93Running a Local LLM on CPU: How llama.cpp and GGUF WorkComing soon
- #94How RAG Works: Retrieval-Augmented Generation for LLMsComing soon
- #95AI Agents and Tool Use: How LLM Function Calling WorksComing soon
- #96Multimodal LLMs: Turning Images and Audio into TokensComing soon
- #97Long Context: How LLMs Handle Very Long InputsComing soon
Part 12Wrap-Up
- #98LLM Evaluation and Benchmarks: How to Measure and Read ThemComing soon
- #99LLM Limitations and AI Safety: What the Mechanism Tells UsComing soon
- #100Build GPT from Scratch, the Finale: Every Step of an LLM by HandComing soon