{"speakers":{"0":"张小珺"},"segments":[{"speaker":0,"sentences":[{"text":"Hello，大家好，我是小珺。","start":3.47,"end":5.33,"timestamps":[3.47,4.07,4.183333333333334,4.296666666666667,4.41,4.65,4.89,5.17]},{"text":"今天的节目是一集学习播客，学习 Kimi K3 的技术报告，希望和大家一起领略技术之美。","start":5.75,"end":12.65,"timestamps":[5.75,5.883333333333334,6.016666666666667,6.15,6.37,6.55,6.73,6.89,6.99,7.21,7.43,7.59,7.85,8.29,8.51,8.85,9.19,9.35,9.49,9.61,9.77,9.93,10.21,10.49,10.65,10.8,10.95,11.14,11.33,11.65,11.87,12.17,12.35,12.49]},{"text":"K3 是有效扩展到 2.8T 总参数，并全量开源的 MoE 模型。","start":12.65,"end":18.35,"timestamps":[12.65,13.61,13.77,13.93,14.09,14.31,14.47,14.67,15.03,15.47,15.69,15.95,16.15,16.41,16.63,16.87,17.09,17.33,17.55,18.01,18.13]},{"text":"大家可能注意到，这集技术报告的领读播客，距离模型发布已经过去了一段时间。","start":18.35,"end":23.87,"timestamps":[18.35,18.64,18.93,19.09,19.25,19.35,19.45,19.67,19.87,20.03,20.21,20.33,20.47,20.59,20.73,20.87,20.97,21.13,21.39,21.75,21.91,22.11,22.33,22.55,22.75,22.86,22.97,23.13,23.21,23.29,23.43,23.51,23.69]},{"text":"期间我们试图寻找一位合适的嘉宾，我们希望这位嘉宾，他的工作和学术背景非常适合地来讲 K3。","start":23.87,"end":32.01,"timestamps":[23.87,24.63,24.81,24.979999999999997,25.15,25.41,25.61,25.91,26.07,26.21,26.35,26.61,26.87,27.01,27.17,27.41,27.630000000000003,27.85,27.97,28.09,28.25,28.35,28.51,28.69,28.85,29.01,29.21,29.41,29.75,29.93,30.07,30.23,30.37,30.53,30.69,30.97,31.15,31.29,31.45,31.61]},{"text":"最后找到了孙宇涛，宇涛目前是清华大学计算机系博士候选人，上海创智学院璞锐学者。","start":32.01,"end":40.27,"timestamps":[32.01,32.47,32.61,32.79,32.91,33.03,33.21,33.35,33.57,34.27,34.41,34.59,34.77,35.17,35.37,35.53,35.67,35.87,36.11,36.29,36.51,36.75,37.01,37.23,37.41,37.63,37.85,38.35,38.55,38.81,38.97,39.19,39.45,39.65,39.83,40.05]},{"text":"他从博士开始的研究方向是 LLM 架构，预训练架构创新一直都是他的兴趣所在。","start":40.27,"end":47.37,"timestamps":[40.27,40.83,40.99,41.13,41.25,41.4,41.55,41.75,41.89,42.01,42.21,42.41,42.67,43.45,43.79,44.03,44.61,44.81,45.01,45.47,45.57,45.79,45.93,46.04,46.15,46.3,46.45,46.57,46.69,46.87,46.99,47.15]},{"text":"这正好是 K3 的亮点之一。","start":47.37,"end":49.41,"timestamps":[47.37,47.63,47.77,47.93,48.15,48.55,48.75,48.93,49.07,49.23]},{"text":"那宇涛透过领读 K3 的论文，也串联讲解了十多篇相关的论文。","start":49.41,"end":55.71,"timestamps":[49.41,49.99,50.17,50.49,51.11,51.33,51.51,51.73,52.11,52.29,52.53,52.75,53.17,53.41,53.63,53.85,54.09,54.29,54.57,54.71,54.87,55.05,55.25,55.37,55.51]},{"text":"他的语速非常快，所以前方语速高能预警。","start":55.71,"end":59.41,"timestamps":[55.71,56.03,56.35,56.55,56.73,56.87,57.01,57.33,57.5,57.67,57.95,58.19,58.37,58.57,58.81,58.99,59.17]},{"text":"期待 2026 年，我们和 AI 共同进步。","start":59.87,"end":63.29,"timestamps":[59.87,60.31,60.43,60.91,61.19,61.4,61.61,61.85,62.19,62.73,62.93,63.13]},{"text":"当然了，这块我觉得可能还有一些别的 concern，比如说我们发现。","start":63.65,"end":66.89,"timestamps":[63.65,63.81,63.97,64.11,64.23,64.29,64.39,64.49000000000001,64.59,64.76,64.93,65.07,65.13,65.19,65.35,65.51,65.61,65.85,66,66.15,66.29,66.38,66.47,66.69]},{"text":"其实如果用命的话，它其实是会更容易出现 outlier 的。","start":67.23,"end":70.49,"timestamps":[67.23,67.39,67.55,67.65,67.75,67.95,68.05,68.14,68.23,68.45,68.56,68.67,69.11,69.29,69.45,69.56,69.67,69.83,69.95,70.29]},{"text":"你控制中间的激活值，总不是一个比较好的一个现象。","start":70.83,"end":74.29,"timestamps":[70.83,71.15,71.37,71.49,71.63,71.73,71.87,71.97,72.09,72.27,72.55,72.67,72.79,72.9,73.01,73.23,73.31,73.43,73.69,73.8,73.91,74.13]},{"text":"当然如果我说，这个命更容易来 outlier，这个苏建林一定会反对。","start":74.29,"end":78.05,"timestamps":[74.29,74.57,74.85,74.94999999999999,75.05,75.21,75.39,75.49000000000001,75.59,75.75,76.03,76.16,76.29,76.53,76.93,77.02000000000001,77.11,77.25,77.35,77.47,77.56,77.65,77.79,77.91]},{"text":"他肯定会说，对于任何优化器来说，它都一定会出现 outlier。","start":78.05,"end":81.55,"timestamps":[78.05,78.25,78.41,78.45,78.57,78.75,79.11,79.21,79.31,79.53,79.83,79.91,80.01,80.11,80.27,80.57,80.69,80.78,80.87,81.01,81.13,81.23]},{"text":"因为你从模型架构上是控制不了的，所以说如果想严格控制的方法，它一定是直接从模型架构下手。","start":81.55,"end":87.29,"timestamps":[81.55,81.67,81.79,81.97,82.13,82.25,82.35,82.49,82.57,82.69,82.83,82.95,83.05,83.11,83.17,83.45,83.71000000000001,83.97,84.11,84.21000000000001,84.31,84.47,84.67,84.79,84.93,85.03,85.13,85.25,85.39,85.51,85.64,85.77,86.01,86.11,86.21,86.43,86.59,86.69,86.83,86.93,87.07]},{"text":"这个学术上还有一个对应的概念，就是忒修斯之船，就是你刚开始 2017 年这个船，刚开始行驶的时候，是 Transformer，对，但是这个船，刚开始是怎么拼起来的？","start":88.27,"end":98.45,"timestamps":[88.27,88.43,88.59,88.77,88.91,89.07,89.19,89.31,89.39,89.47,89.67,89.85,89.93,90.03,90.19,90.28,90.37,90.61,91.11,91.23,91.33,91.53,91.69,91.85,92.07,92.27,92.41999999999999,92.57,92.95,93.39,93.47,93.55,93.77,93.97,94.09,94.21,94.37,94.53,94.67666666666666,94.82333333333334,94.97,95.37,96.33,96.59,96.65,96.71,96.82,96.93,97.13,97.29,97.41,97.53,97.71,97.82,97.93,98.05,98.19,98.33]},{"text":"它是 Attention 加 Residual。","start":98.45,"end":100.29,"timestamps":[98.45,98.97,99.09,99.49,99.83]},{"text":"行驶的过程中，我们可能会慢慢地，把某些部分换了，某些部分换了。","start":100.29,"end":104.61,"timestamps":[100.29,100.45,100.51,100.61,100.73,100.83,100.99,101.21000000000001,101.43,101.7,101.97,102.17,102.30000000000001,102.43,102.71,102.93,103.15,103.27,103.35,103.43,103.63,103.77,103.93,104.07,104.13999999999999,104.21,104.47]},{"text":"其实跟 Transformer 已经，我觉得相似已经很低很低，你是否还把这个东西，叫做 Transformer？","start":104.61,"end":109.97,"timestamps":[104.61,104.75999999999999,104.91,105.07,105.47,105.57,105.67,105.84333333333333,106.01666666666667,106.19,106.39,106.57,106.68,106.79,106.95,107.03,107.15,107.27,107.43,107.59,108.05,108.19,108.31,108.39,108.47,108.59,108.71,108.89,109.05]},{"text":"我说一下我的判断，我的判断比较暴论，我的暴论就是。","start":109.97,"end":113.85,"timestamps":[109.97,110.61,110.73,110.78999999999999,110.85,111,111.15,111.29,111.43,111.59,111.75,111.89,111.97,112.13,112.19,112.33,112.99,113.15,113.31,113.47,113.59,113.72]},{"text":"大模型可能没有太本质的创新，后面都是一些，改良性的渐进式进步。","start":113.85,"end":117.57,"timestamps":[113.85,114.47,114.59,114.73,114.83000000000001,114.93,115.05000000000001,115.17,115.29,115.39,115.51,115.61,115.69,115.83,116.13,116.23,116.33000000000001,116.43,116.53,116.63,116.75,116.87,116.99,117.09,117.21,117.31,117.41]},{"text":"好的，那我们的论文学习。","start":120.04,"end":121.84,"timestamps":[120.04,120.2,120.36,120.62,120.76,120.9,121.12,121.3,121.44,121.68]},{"text":"这次我邀请了清华大学的 PhD Candidate，上海创智学院璞锐学者孙宇涛，将由他带领大家一起来学习和阅读。","start":122.64,"end":134.02,"timestamps":[122.64,123.08,123.28,123.44,123.64,123.84,124.04,124.26,124.54,124.76,125.06,125.3,126.2,126.9,127.38,127.58,127.8,128.02,128.3,128.62,129.12,129.32,129.58,129.82,130.04,130.18,130.48,130.92,131.06,131.26,131.5,131.62,131.76999999999998,131.92,132.06,132.2,132.56,132.98,133.26,133.54,133.74]},{"text":"最近大家关注度非常高的 Kimi K3 的论文，但是宇涛会带领大家读一系列的论文，那宇涛你能不能先给大家做一个自我介绍，并且介绍一下你的研究方向？","start":134.02,"end":145.76,"timestamps":[134.02,134.23000000000002,134.44,134.63,134.82,135,135.14,135.26,135.39,135.52,135.78,135.94,136.32,136.82,137.02,137.18,137.4,137.69,137.98,138.14,138.28,138.46,138.6,138.68,138.81,138.94,139.08,139.26,139.48,139.6,139.74,139.86,140.04,140.62,140.98,141.18,141.34,141.44,141.51999999999998,141.6,141.82,141.96,142.14,142.32,142.58,142.75,142.92,143.18,143.26,143.4,143.62,143.88,144.04,144.26,144.46,144.60000000000002,144.74,144.9,145.06,145.28,145.42,145.6]},{"text":"好的，各位观众朋友大家恰不好？","start":146.94,"end":149.52,"timestamps":[146.94,147.16,147.72,147.79000000000002,147.86,148.2,148.38,148.55,148.72,148.9,149.08,149.26,149.34]},{"text":"然后我是来自清华大学的孙宇涛。","start":149.52,"end":151.86,"timestamps":[149.52,149.78,150.04,150.23,150.42,150.66,150.82,150.98,151.08,151.22,151.36,151.5,151.64,151.7]},{"text":"我在这个博士期间的，主要的研究方向是，大模型的架构和预训练，然后这个博士课题呢？","start":151.86,"end":158.22,"timestamps":[151.86,152.15,152.44,152.59,152.74,152.94,153.08,153.26,153.44,153.6,153.8,153.92,154.06,154.26,154.36,154.52,154.66,154.86,155.06,155.18,155.3,155.54,155.86,156,156.16,156.26,156.36,156.5,156.82999999999998,157.16,157.31,157.46,157.68,157.82,157.94,158.08]},{"text":"我是从二十三年，从那个时候开始，我们主要去研究，这个大模型的模型架构的，一系列的工作。","start":158.22,"end":164.34000000000003,"timestamps":[158.22,158.4,158.58,158.86,159.08,159.18,159.34,159.5,159.7,159.8,159.9,159.97,160.04,160.18,160.32,160.64,160.96,161.14,161.24,161.44,161.68,161.8,161.89,161.98,162.36,162.48,162.6,162.74,162.94,163.1,163.26,163.42,163.6,163.8,163.94,164.04,164.14,164.24]},{"text":"然后在那个时间点，应该是 ChatGPT 刚刚发布一段时间吧，但是我们的架构研究，其实从 ChatGPT 发布之前，就开始。","start":164.34,"end":173.42,"timestamps":[164.34,164.48000000000002,164.62,164.84,164.94,165.04,165.14999999999998,165.26,165.4,165.64,165.74,166.08,166.92,167.18,167.38,167.62,167.84,168.24,168.36,168.48000000000002,168.6,168.76,169.01999999999998,169.28,169.41,169.54,170.04,170.26,170.34,170.46,170.58,170.9,171.22,171.42,172,172.2,172.36,172.53,172.7,172.96,173.19]},{"text":"对，差不多就开始了。","start":173.42,"end":174.88,"timestamps":[173.42,174.1,174.19333333333333,174.28666666666666,174.38,174.52,174.62,174.72]},{"text":"然后我这两三年的，主要的一个研究方向，主要是去尝试解决。","start":175.34,"end":180.96,"timestamps":[175.34,175.51999999999998,175.7,175.92,176.16,176.48,176.7,176.85,177,177.22,177.38,177.52,177.68,177.84,178.48,178.58,178.74,178.84,179.46,179.62,179.8,180,180.22,180.44,180.74]},{"text":"然后大模型在推理这个环节的一些低效性，所以说我们主要的一些研究方向，都是围绕着推理高效性来去展开的。","start":180.96,"end":190.3,"timestamps":[180.96,181.09,181.22,181.4,181.54,181.68,181.96,182.66,182.78,182.86,182.94,183.12,183.26,183.4,183.53,183.66,184.54,184.68,184.82,184.97,185.12,185.44,185.55,185.66,185.86,186,186.12,186.28,186.44,187.02,187.08,187.22,187.32,187.45,187.58,187.82,187.92,188.04,188.18,188.32,188.46,188.64,188.84,189.1,189.56,189.96,190.14]},{"text":"然后其实大家也可以注意到说，就是在模型架构这个方面。","start":190.3,"end":194.2,"timestamps":[190.3,190.53,190.76,190.89,191.02,191.18,191.34,191.45333333333335,191.56666666666666,191.68,191.81,191.94,192.14,192.4,192.53,192.66,192.94,193.12,193.26,193.44,193.62,193.75,193.88,194.04]},{"text":"整个业界的研究方向，也是逐渐从偏这个性能为导向，逐渐的变得是以效率为导向。","start":194.2,"end":201.36,"timestamps":[194.2,194.56,194.86,195.2,195.32,195.44,195.56,195.64,195.82,195.98,196.19,196.4,196.9,197.08,197.34,197.72,197.87,198.02,198.22,198.4,198.58,198.7,198.88,199.1,199.3,199.5,199.7,199.86,200,200.64,200.82,200.92,201.04,201.18]},{"text":"然后在这个方面，其实是有一些范式上的变化的。","start":202.02,"end":205.42,"timestamps":[202.02,202.2,202.38,202.74,202.87,203,203.13,203.26,203.39999999999998,203.54,203.84,203.95,204.06,204.36,204.72,204.82,204.94,205.04,205.16,205.3]},{"text":"因为可能在 Vision 时代，或者说在 ImageNet 时代，大家还是尝试去，做到更精妙的一些架构，然后去提升模型本身的一个表现。","start":205.42,"end":213.86,"timestamps":[205.42,205.52999999999997,205.64,205.82999999999998,206.02,206.16,206.42,206.56,206.7,206.8,206.9,207.1,207.28,207.7,207.88,207.94,208.07,208.2,208.35,208.5,208.86,209,209.78,210.06,210.2,210.42,210.72,210.92,211.04,211.16,211.28,211.5,211.66,211.82,211.98,212.44,212.62,212.78,212.88,212.98,213.12,213.22,213.34,213.41,213.48,213.68]},{"text":"但是在大模型时代，然后我们其实逐渐发现。","start":214.32,"end":217.02,"timestamps":[214.32,214.45999999999998,214.6,214.76,214.86,214.96,215.08,215.22,215.38,215.56,215.74,215.95,216.16,216.28,216.4,216.56,216.68,216.84]},{"text":"对于模型的性能而言，模型的参数量原来是一个，最主要的一个因素。","start":217.02,"end":221.52,"timestamps":[217.02,217.32,217.44,217.58,217.76,217.94,218.16,218.28,218.4,218.56,219.06,219.18,219.3,219.44,219.54,219.66,219.78,219.88,220.02,220.15,220.28,220.52,220.72,220.86,221,221.1,221.2,221.34]},{"text":"然后在模型架构本身的一些，改进性的变化。","start":221.52,"end":225.28,"timestamps":[221.52,221.72,221.92,222.18,222.92,223.04,223.18,223.26,223.46,223.64,223.88,224.07,224.26,224.72,224.84,224.94,225.04,225.14]},{"text":"其实相对于模型参数本身的提升，对带来的性能提升是相当微小的。","start":225.28,"end":230.04,"timestamps":[225.28,225.47,225.66,225.82,225.96,226.1,226.24,226.38,226.58,226.68,226.8,226.94,227.08,227.3,227.5,227.92,228.14,228.26,228.38,228.86,229,229.16,229.3,229.44,229.56,229.66,229.78,229.94]},{"text":"但是其实不同的架构，在模型的推理方面，然后性能差异是巨大的，而且这个也是主要决定了，模型最后的一个部署的价格。","start":230.48,"end":237.5,"timestamps":[230.48,230.62,230.76,230.89,231.02,231.14,231.26,231.42,231.54,231.66,231.84,232.16,232.26,232.36,232.5,232.62,232.76,232.9,233.03,233.16,233.42,233.54,233.66,233.76,233.86,234,234.07999999999998,234.16,234.26999999999998,234.38,234.48000000000002,234.58,234.72000000000003,234.86,235.12,235.4,235.86,236,236.12,236.24,236.38,236.52,236.64,236.74,236.82999999999998,236.92,237.08,237.16,237.24,237.38]},{"text":"然后其实这两年大家，主要在这个方向去努力。","start":237.5,"end":240.74,"timestamps":[237.5,237.6,237.7,237.82,237.94,238.22,238.38,238.5,238.65,238.8,239.52,239.6,239.76,239.86,239.96,240.14,240.24,240.44,240.6]},{"text":"然后对于我而言的话，其实我们是在 23 年开始，然后探索这个线性注意力，这样的一个方向的一个工作。","start":240.74,"end":247.96000000000004,"timestamps":[240.74,241.05,241.36,241.54,241.62,241.86,242.04,242.16,242.26,242.36,242.5,242.64,242.81,242.98,243.34,243.78,243.98,244.18,244.32,244.46,244.57999999999998,244.7,244.94,245.06,245.16,245.26,245.4,245.48,245.6,246.06,246.24,246.42,246.74,246.86,246.98,247.26,247.4,247.5,247.56,247.62,247.79000000000002]},{"text":"然后我们这篇 RetNet 的工作。","start":247.96,"end":249.51999999999998,"timestamps":[247.96,248.14,248.32,248.48,248.64,248.8,248.9,249.22,249.38,249.45]},{"text":"刚开始主要是解决了两个问题的，我觉得从现在的这个阶段，主要解决了两个问题。","start":249.52,"end":254.84,"timestamps":[249.52,249.78,249.92000000000002,250.06,250.22,250.32,250.48,250.7,250.84,250.98,251.14,251.26,251.38,251.5,251.66,251.77333333333334,251.88666666666666,252,252.16,252.26999999999998,252.38,252.72,252.81,252.9,253.26,253.42,254.08,254.18,254.3,254.42,254.52,254.58,254.66,254.75]},{"text":"第一个问题就是线性注意力。","start":254.84,"end":256.02,"timestamps":[254.84,254.93,255.02,255.12,255.22,255.32,255.42000000000002,255.52,255.74,255.84,255.94]},{"text":"怎么能尽量的去？","start":256.02,"end":257.24,"timestamps":[256.02,256.15,256.28,256.58,256.78,256.92,257.04]},{"text":"在一些比较 general 的场景下，能逼近这个全注意力的，这样的一个性能。","start":257.24,"end":262.1,"timestamps":[257.24,257.7,257.81,257.92,258.1,258.16,258.34,258.6,258.78,258.92,259.08,259.24,259.42,259.64,259.75,259.86,260.6,260.74,260.88,261.1,261.23,261.36,261.5,261.61,261.72,261.94]},{"text":"因为 linear 从刚开始提出，主要是为了去用一个，比较 linear kernel。","start":262.1,"end":266.68,"timestamps":[262.1,262.33000000000004,262.56,262.8,263.1,263.4,263.53999999999996,263.68,263.86,264.04,264.18,264.28,264.4,264.52,264.64,264.86,265.14,265.28999999999996,265.44,265.68,265.78,266.4]},{"text":"然后去一个 kernelized 的方式，去拟合。","start":266.68,"end":268.96,"timestamps":[266.68,266.82,266.96,267.16,267.31,267.46,267.94,268.1,268.18,268.36,268.58,268.76]},{"text":"然后二次复杂度的，全注意这样的一个 pattern。","start":268.96,"end":271.66,"timestamps":[268.96,269.06,269.16,269.3,269.48,269.84,269.96,270.08,270.22,270.58,270.66999999999996,270.76,270.93,271.1,271.24,271.3,271.36]},{"text":"所以在之前那个时代，其实对于这个 linear attention 而言，它的位置信息，它其实是不够 sensitive 的，然后其实在那个时代，这个 linear attention 和这个。","start":271.66,"end":280.76,"timestamps":[271.66,271.79,271.92,272.88,273.07,273.26,273.35,273.44,273.56,273.66,273.79,273.92,274.22,274.44,274.53,274.62,274.92,275.14,275.36,275.48,275.61,275.74,275.94,276.02,276.16,276.32,276.48,276.6,276.72,277.02,277.36,277.46,277.76,278.02,278.24,278.46,278.57,278.68,278.88,279.09000000000003,279.3,279.44,279.52,279.62,279.72,279.96,280.22,280.54,280.65]},{"text":"和 RNN 只是在形式上比较相似。","start":280.76,"end":283.04,"timestamps":[280.76,281.26,281.56,281.65999999999997,281.76,282.14,282.28,282.42,282.56,282.7,282.78,282.92]},{"text":"但是从这个建模的特性上来讲，RNN 其实还是比较强调，一些 locality 的一些表示。","start":283.04,"end":289.03999999999996,"timestamps":[283.04,283.14,283.24,283.6,283.71000000000004,283.82,283.96,284.12,284.36,284.86,284.98,285.1,285.2,285.34,285.76,285.88,286,286.14,286.28,286.52,286.58,286.74,287.44,287.62,287.8,288.42,288.6,288.68,288.76,288.9]},{"text":"所以说在那个时代，我们在 RetNet 上是，比较早的引入了一个，衰减的一个机者。","start":289.04,"end":294.28,"timestamps":[289.04,289.29,289.54,289.72,289.9,289.99,290.08,290.22,290.28,290.4,290.52,290.72,290.98,291.22,291.5,292,292.12,292.46,292.58,292.72,292.84,292.96,293.09000000000003,293.22,293.7,293.8,293.92,293.99,294.06,294.16]},{"text":"这样的话可以让这个，线性注意力在一个有限的这个状态空间里。","start":294.28,"end":298.2,"timestamps":[294.28,294.37,294.46,294.56,294.66,294.8,294.94,295.16,295.26,295.36,295.56,295.66,295.8,295.88,296.06,296.15,296.24,296.42,296.58,296.8,296.93,297.06,297.68,297.76,297.92,298.08]}]}]}