LLMs from Scratch

Программа курса

1. Tokenizers: BPE, WordPiece, SentencePiece
2. Building a Tokenizer from Scratch
3. Data Pipelines for Pre-Training
4. Pre-Training a Mini GPT (124M Parameters)
5. Scaling: Distributed Training, FSDP, DeepSpeed
6. Instruction Tuning (SFT)
7. RLHF: Reward Model + PPO
8. DPO: Direct Preference Optimization
9. Constitutional AI and Self-Improvement
10. Evaluation: Benchmarks, Evals, LM Harness
11. Quantization: Making Models Fit
12. Inference Optimization
13. Building a Complete LLM Pipeline
14. Open Models: Architecture Walkthroughs
15. Speculative Decoding and EAGLE-3
16. Differential Attention (V2)
17. Native Sparse Attention (DeepSeek NSA)
18. Multi-Token Prediction (MTP)
19. DualPipe Parallelism
20. DeepSeek-V3 Architecture Walkthrough
21. Jamba — Hybrid SSM-Transformer
22. Async and Hogwild! Inference
23. Speculative Decoding and EAGLE
24. Gradient Checkpointing and Activation Recomputation
25. Итоговое задание