零阶预训练首次追上反向传播
推荐指数 72.0 NO. 008 · 2026.10.06
发布2026/10/05Score67Comments4
为什么值得看
Dust 提出首个能匹敌反向传播的零阶优化方法，通过逐 token 独立扰动激活实现并行评估。在算力充裕时甚至超越传统反向传播，为无梯度训练开辟新路径。
编辑判断
反向传播是深度学习的基础设施，但内存瓶颈和生物不可行性一直是其软肋。零阶方法过去只在小模型或微调场景有效，Dust 的突破在于把"逐 token 扰动"变成了隐式并行种群评估——本质上是用前向算力换反向传播的内存和图依赖。
不过论文提到的"算力充裕时超越 backprop"需要警惕：这里的超越是在特定设置下，且零阶方法需要大量前向传递，实际训练成本可能数倍于反向传播。真正值得跟踪的是它能否解锁当前 backprop 做不到的事，比如超大上下文或动态图场景。
代码和具体超参数尚未公开，建议等开源后重点测试其在长序列和边缘设备上的实际效率，而非仅看准确率数字。
社区反馈
意见分歧 4 条评论
核心争论：零阶优化是否值得为并行性牺牲计算效率，还是反向传播的协调成本已被现代硬件充分吸收
It sounds like this is less computationally efficient than backprop, but more easily parallelizable. Is that fair?
Not necessarily, backprop is highly parallelizable since it is just a bunch of matrix mults. Something like Dust skips the backward pass on backprop. But other techniques like Neural Predictive Coding can be completely asynchronous, each "weight" can fire independent of those far away from it. Inno
The reason why I don't see the promise for ML-only applications is that the coordination backprop requires comes very cheap to us. "Much easier given the right device" - the "right" there just isn't shaped like the devices we actually build. And the price of "not having backprop" is usually expendin