数学竞赛基准测试方法论革新
值得看指数 71.0 NO. 014 · 2026.06.07
发布2026/06/06Score91Comments39
为什么值得看
莱比锡团队提出了一套新的数学竞赛基准测试框架，解决了现有 benchmark 容易被模型过拟合、区分度下降的问题。对做模型评估和数学推理的团队有参考价值，尤其是需要设计抗作弊评测方案的场景。
编辑判断
当前数学推理 benchmark 如 GSM8K、MATH 已被大量模型刷到饱和，数据污染问题严重但很难检测。这篇工作的价值在于把'竞赛题'作为天然抗污染数据源——竞赛题在训练截止日期前不会公开，且每年更新。
不过论文作者全是纯数学背景，没有 ML 系统经验，工程实现细节缺失。代码未开源，复现成本不明。建议关注其核心方法论（动态题库+时间隔离验证），而非直接照搬具体指标。
如果你在做 LLM 数学能力评测，可以借鉴这个思路自建私有题库，比花钱买商业 benchmark 更靠谱。
社区反馈
意见分歧 31 条评论
核心争论：LLM解决的是文献综合还是真正原创推理，benchmark是否因训练数据污染而失效
"...Between April 1 and May 15, 2026, a group of 49 mathematicians compiled a dataset of research-level mathematics questions with known answers... We present the resulting collection of 100 questions....We evaluated these questions in three stages: a single attempt by five state-of-the-art LLMs....
mathematics questions with known answers... ... that are therefore liable to be in the training data?
I had the same thought, because even if the exact solution doesn't appear there's a notable difference between performing a literature search versus solving something de novo. But I think perhaps this benchmark wasn't meant to exclude the former and that the point may have been to test the ability o