AIDB Daily Papers
LLMエージェントの信頼性は能力ではなく「ハルシネーション感受性」で決まる:階層間の非単調性を解明
※ 日本語タイトル・ポイントはAIによる自動生成です。正確な内容は原論文をご確認ください。
ポイント
- LLMエージェントの信頼性向上には、能力階層とハーネス(制御構造)の複雑さの間に単調な逆相関があるという仮説を検証した。
- 最先端モデルでもハーネスの複雑化が信頼性を低下させ、推論モデルでは厳格なハーネスが最も高い成功率と低い遅延を示した。
- モデルのタイプ(チャットか推論か)によってハーネス感受性が非単調に変化し、失敗原因もモデル階層によって異なることを発見した。
Abstract
A prevalent assumption in LLM agent deployment holds that more structured harnesses universally improve reliability, and that higher-capability models need proportionally less structural guidance -- together implying a monotone inverse relationship between model capability tier and optimal harness complexity. We test this hypothesis through a controlled 432-run experiment crossing six models across four capability tiers with three harness conditions (light, balanced, strict) on HEAT-24, a 24-task synthetic benchmark with git-based workspace verification. Our results refute the monotone inverse relationship on two fronts. First, for the frontier chat model evaluated (Gemini 2.5 Flash), increased harness verbosity lowers VTSR by 29-38 percentage points -- a harness-complexity paradox. Second, for the frontier reasoning model evaluated (Qwen3.5-122B, extended thinking enabled), strict harness achieves the highest VTSR (91.7%) and the lowest latency, the opposite of the prediction. Within the constrained tier, a 2B model (Gemma4:e2B) matches strong-open-tier stability at 91.7% across all harnesses. Because each tier is represented by a single model in this study, these results should be interpreted as model-specific observations; harness sensitivity appears non-monotone across the models evaluated, and depends critically on model type (chat vs. reasoning). We introduce a six-label failure taxonomy showing that format_violation dominates capable-model failures while wrong_file dominates low-capability failures, and we derive practical tier-aware harness selection guidelines.
Paper AI Chat
この論文のPDF全文を対象にAIに質問できます。
質問の例: