自蒸馏破解灾难性遗忘
推荐指数 82.0 NO. 011 · 2026.05.18
发布2026/05/17Score89Comments21
为什么值得看
论文提出用模型自身输出作为软标签进行蒸馏，无需存储旧数据即可实现持续学习。这对需要在线更新模型的生产环境意义重大，终于可能摆脱"重训全量数据"的昂贵流程。
媒体预览
编辑判断
持续学习的核心矛盾一直是"学新忘旧"，之前的主流解法要么存旧数据 replay（隐私和存储成本爆炸），要么用正则化约束重要参数（效果差）。这篇的关键洞察是：模型自己生成的软概率分布比硬标签包含更多结构信息，蒸馏自己反而比外部教师更稳定。
实验显示在标准 benchmark 上接近甚至超过 replay 方法，且没有隐私泄露风险。如果复现稳健，推荐系统、金融风控这些必须在线更新但受合规限制不能留用户数据的场景会最先受益。建议等代码开源后优先测自己的任务分布漂移程度。
社区反馈
意见分歧 17 条评论
核心争论：标题是否过度自信，以及"policy"术语是否让非RL读者困惑
Both title and abstract feel a little too confident, which ironically makes me more skeptical rather than less. I find the choice of the words "enable" in the title and "establishing" at the end of the abstract to be particularly jarring.
From Jan 2026. This is very interesting: "Empirical Validation. While we cannot verify these theoretically, we evaluate each empirically. We use the Qwen-2.5-7B-Instruct model (Hui et al., 2024) as the base policy and the ToolAlpaca dataset (Tang et al., 2023). In this benchmark, the model receives
Wtf is a policy? Is this some sort of RL thing that I'm too ML to understand? Gemini tells me it's the probability of the next token for an LLM. Okay then.