If you want to truly understand how LLM is trained, CS336 is the most worthy set of open courses to learn the system.
Building a modern LLM engineering stack from scratch covers not only Transformer, but also tokenization, GPU, kernel, parallelism, scaling law, data, evaluation, and alignment.
Links to the 17 chapters are here!
Learn it, mark and save it!
01｜Overview & Tokenization｜Start from the language model panorama and talk about how text becomes token.
02｜PyTorch & Resource Accounting｜Train LLM.
First learn to calculate parameters, video memory, FLOPs and throughput.
03｜Architectures & Hyperparameters｜Transformer structure, training hyperparameters, model design choices.
04｜Mixture of Experts｜The core of MoE: trade sparse activation for larger model capacity.
05｜GPUs｜Only by understanding GPU can we understand why LLM projects look like this.
06｜Kernels & Triton｜From operators to Triton, entering into the underlying performance optimization.
07｜Parallelism 1｜After a single card is not enough, model training enters the distributed world.
08｜Parallelism 2｜Continue to remove training bottlenecks: data/tensor/pipeline parallelism.
09｜Scaling Laws 1｜Why large models “get stronger according to rules”.
10｜Inference｜After training, it is just the beginning.
Inference efficiency determines the product cost.
11｜Scaling Laws 2｜In-depth look at the optimal ratio of compute, data, and model size.
12｜Evaluation｜The key to whether a model is good or not lies in how it is evaluated.
13｜Data 1｜Data is the fuel for pre-training: source, cleaning, quality control.
14｜Data 2｜Continue to talk about data engineering: deduplication, filtering, mixing, and data matching.
15｜Alignment - SFT/RLHF｜From base model to assistant: SFT and RLHF main lines.
16｜Alignment - RL 1｜Introducing reinforcement learning into LLM alignment.
17｜Alignment - RL 2｜Continue RL alignment: training stability, objective function, engineering details.