Infrastructure & Production

Программа курса

1. Managed LLM Platforms — Bedrock, Vertex AI, Azure OpenAI
2. Inference Platform Economics — Fireworks, Together, Baseten, Modal, Replicate, Anyscale
3. GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler, Gang Scheduling
4. Serving Engine Internals — PagedAttention, Continuous Batching, Chunked Prefill
5. EAGLE-3 Speculative Decoding in Production
6. Prefix-Cache Serving — RadixAttention and KV Reuse
7. Hardware-Specialized Inference Compilation — FP8 and NVFP4 on Blackwell
8. Inference Metrics — TTFT, TPOT, ITL, Goodput, P99
9. Production Quantization — AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4
10. Cold Start Mitigation for Serverless LLMs
11. Multi-Region LLM Serving and KV Cache Locality
12. Edge Inference — Apple Neural Engine, Qualcomm Hexagon, WebGPU/WebLLM, Jetson
13. LLM Observability Stack Selection
14. Prompt Caching and Semantic Caching Economics
15. Batch APIs — the 50% Discount as Industry Standard
16. Model Routing as a Cost-Reduction Primitive
17. Disaggregated Prefill/Decode — NVIDIA Dynamo and llm-d
18. Production Serving Stack — KV Offloading and Cache-Aware Routing
19. AI Gateways — LiteLLM, Portkey, Kong AI Gateway, Bifrost
20. Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs
21. A/B Testing LLM Features — GrowthBook, Statsig, and the Vibes Problem
22. Load Testing LLM APIs — Why k6 and Locust Lie
23. SRE for AI — Multi-Agent Incident Response, Runbooks, Predictive Detection
24. Chaos Engineering for LLM Production
25. Security — Secrets, API Key Rotation, Audit Logs, Guardrails
26. Compliance — SOC 2, HIPAA, GDPR, PCI-DSS, EU AI Act, ISO 42001
27. FinOps for LLMs — Unit Economics and Multi-Tenant Attribution
28. Self-Hosted Serving Selection — Matching Engine to Hardware and Scale
29. Итоговое задание