英文标题：Tool Retrievers Are Underestimated: Annotation Expansion Reveals True Capability
作者：Yanyu Zhu, Chenheng Zhang, Shaoshen Chen, Hoilam Pao, Yufei zhang, Jiajun Chai, Dongnian Wang, Zhaoyu Hu, Guojun Yin, Wei Lin, Hai-Tao Zheng
arXiv ID：2609.08327 | 分类：cs.SE | 发表：2026-09-08
许可：CC-BY
摘要 在拥有海量且不断演化的工具库的开放世界场景中，工具增强的大语言模型依赖检索器为给定查询找出相关工具。由于此类工具库中往往包含许多实现相同功能的工具，单个查询通常可以由若干不同但功能等价的工具组合来解决，使得自然的查询到工具映射本质上是一对多的。然而，现有的工具检索基准为每个查询仅标注一个相关工具组合，将这种一对多映射压缩为僵化的一对一标注，导致有效的检索工具被误判为失败。{{NL}}为解决这一问题，我们提出 ToolEX（Tool Equivalent eXpansion，工具等价扩展），{{NL}}一个自动发现并标注与已标注工具组合功能等价的工具组合的框架。{{NL}}将其应用于包含
In open-world scenarios with massive and evolving tool repositories, tool-augmented large language models rely on a retriever to surface relevant tools for a given query. Because such repositories often contain many tools that implement the same functionality, a single query can often be resolved by several distinct but functionally equivalent tool combinations, making the natural query-to-tool mapping inherently one-to-many. However, existing tool retrieval benchmarks annotate each query with a single relevant tool combination, collapsing this one-to-many mapping into a rigid one-to-one annotation and causing valid retrieved tools to be misjudged as failures. To address this, we propose ToolEX (Tool Equivalent eXpansion), a framework that automatically discovers and annotates the tool combinations functionally equivalent to the labeled ones. Applied to the 7,360-query Tool-DE benchmark, ToolEX finds that 67.9% of sub-queries admit equivalent alternatives, expanding the singular ground truth to an average of 5.3 valid combinations per query. Using the expanded benchmark ToolEQ, we re-evaluate eight base retrievers and two fine-tuned variants; metrics on ToolEQ rise substantially over Tool-DE, showing that one-to-one annotation systematically underestimates retrievers and that 30--47% of the reported fine-tuning gain is an evaluation artifact rather than genuine improvement. Applying the same pipeline to skill retrieval on SkillRet further confirms that the one-to-one problem extends beyond tool retrieval.
查看完整双语翻译 →
正在跳转到翻译阅读页… 如果没有自动跳转，请点击这里。