[https://image.xyzcdn.net/FkZ-gquke6nKJCnJ4di5kQ0lEfAc.png] 今天的节目是一集学习播客，学习 Kimi K3 技术报告，希望和大家一起领略 “技术之美”。 K3 是有效扩展到 2.8T 总参数并全量开源的 MoE 模型。大家可能注意到，这集技术报告的领读距离模型发布已经过去一段时间。期间，我们试图寻找一位合适的嘉宾，我们希望这位嘉宾的学术和工作背景非常适合讲 K3。最后找到孙宇涛。 宇涛目前是清华大学计算机系博士候选人，上海创智学院璞锐学者。他从博士开始的研究方向是 LLM 架构、预训练，架构创新一直是他的兴趣点，这正好是 K3 亮点之一。 宇涛透过领读 K3 技术报告也串联讲解了十多篇相关论文。他的语速非常快 —— 前方语速预警。 OUTLINE: 02:00 宇涛的自我介绍和研究经历，为什么对架构创新最感兴趣？ 16:05 从 Kimi K3 论文出发，论文讲解的框架与脉络 19:02 宇涛开始领读论文： 1 导读 Kimi K3 将设计概括为沿 sequence、depth 和 width 三个维度扩展信息流；其中 depth 与 width 本质上仍是模型容量扩展的两种组织⽅式，这⼀ “三维 scaling” 更多是贯穿论⽂的叙事框架。 2 Model Architecture 2.1 线性注意⼒的前世今⽣ [Microsoft Research] RetNet [https://www.microsoft.com/en-us/research/publication/retentive-network-a-successor-to-transformer-for-large-language-models/]：最初的 data-independent decay 与 chunk-recurrent 递归形式 [NVIDIA] Gated DeltaNet [https://research.nvidia.com/publication/2025-04_gated-delta-networks-improving-mamba2-delta-rule]：引⼊ gated delta rule [Moonshot AI] Kimi Linear [https://arxiv.org/abs/2510.26692]：fine-grained decay 及其带来的 infra 变化 2.2 Gated MLA [Alibaba Qwen] Gated Attention for Large Language Models [https://arxiv.org/abs/2505.06708]：attention gating 与训练稳定性 2.3 Attention Residuals [Microsoft Research] On Layer Normalization in the Transformer Architecture [https://arxiv.org/abs/2002.04745]：Pre-LN、Post-LN 与训练稳定性 [ByteDance Seed] Hyper-Connections：残差连接的新维度 [Moonshot AI] Attention Residuals [https://arxiv.org/abs/2603.15031]：以跨深度 attention 取代固定残差累积，使各层选择性聚合此前表⽰ 2.4 Stable LatentMoE 与 SiTU-GLU [NVIDIA] LatentMoE [https://arxiv.org/abs/2601.18089]：通过 latent expert space 降低 MoE 通信与权重访问成本 [OpenAI] GPT - OSS [https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf]：以 clamped SwiGLU 控制激活值 2.5 Muon [Moonshot AI] Moonlight / Muon is Scalable for LLM Training [https://arxiv.org/abs/2502.16982]：更好的优化表现及随之突出的 activation outlier 问题 2.6 Quantile Balancing [科学空间] 《MoE 环游记：6、最优分配促均衡》 [https://kexue.fm/archives/11619]：从最优分配推导 Quantile Balancing 2.7 Native Vision 3 Pre-Training 3.1 Scaling Law [OpenBMB] MiniCPM [https://arxiv.org/abs/2404.06395]：WSD 的训练预算扩展、稳定阶段复⽤、继续训练与提前停⽌ 3.2 Long-Context Extension [Cohere] RNoPE [https://arxiv.org/abs/2501.18795]：交替使⽤ RoPE 与 NoPE，兼顾位置建模与长上下⽂检索 4 Post-Training 4.1 Post-Training Pipeline 4.2 Reinforcement Learning [Moonshot AI] Kimi K1.5 [https://arxiv.org/abs/2501.12599]：提出 partial rollout，通过复⽤未完成轨迹降低长 CoT rollout 开销 [Moonshot AI] Kimi K2.5 [https://arxiv.org/abs/2602.02276]：reasoning-effort budget control 与 Agentic Generative Reward Model 4.3 Multi-Teacher On-Policy Distillation [Microsoft Research] MiniLLM [https://arxiv.org/abs/2306.08543]：基于 student-generated samples 的 on-policy distillation 4.4 Deployment-Aware Post-Training 4.5 Draft Model Fine-Tuning 5 Infrastructure 5.1 Pre-Training 5.1.1 KDA Kernel [FLA] Flash Linear Attention [https://github.com/fla-org/flash-linear-attention]：KDA kernel 与线性注意⼒⾼效实现 5.1.2 Distributed Training [DeepSeek-AI] DeepSeek-V3 Technical Report [https://arxiv.org/abs/2412.19437]：DualPipe 与细粒度 MoE communication–computation overlap MoE overlap 的资源 trade-off 与 cross-PP activation transfer 5.1.3 Perfectly Balanced Expert-Parallel MoE Training 5.1.4 Memory-Efficient Training [Moonshot AI] Mooncake [https://arxiv.org/abs/2407.00079]：Mooncake Transfer Engine 与 cross-PP activation remote offload 5.1.5 Multimodal Encoder Optimization 5.2 RL 5.2.1 Long-Context RL Infrastructure 5.2.2 Sandbox Infrastructure 5.3 Inference 5.3.1 KDA-Aware Prefix Cache Management 5.3.2 High-Performance Kernels 5.3.3 Fleet-Level Scheduling LINKS： 我们的播客在小宇宙 [https://www.xiaoyuzhoufm.com/podcast/626b46ea9cbbf0451cf5a962]、Apple Podcast [https://podcasts.apple.com/cn/podcast/% E5% BC% A0% E5% B0%8F% E7%8F% BAj% C3% B9n-% E5%95%86% E4% B8%9A% E8% AE% BF% E8% B0%88% E5% BD%95/id1634356920]、Spotify 等全音频平台播出； 我们的视频播客在小宇宙 [https://www.xiaoyuzhoufm.com/podcast/626b46ea9cbbf0451cf5a962]、Bilibili [https://b23.tv/FZ0s2Q1]、小红书 [https://xhslink.com/m/4clvm6edHk1]、视频号、抖音等全视频平台播出； 如果你想服用文字版，请搜索我们工作室的公众号：语言即世界 language is world。 DISCLAIMER: 本内容不作为投资建议。 CONTACT: xiaojunzhang@lisw.ai Jump into the new world-and explore with us!😉