AIDB Daily Papers
AIコーディングツールはなぜバグる? Claude Code、Codex、Gemini CLIの落とし穴
※ 日本語タイトル・ポイントはAIによる自動生成です。正確な内容は原論文をご確認ください。
ポイント
- Claude Code、Codex、Gemini CLIにおける3800件以上のバグを分析し、AIコーディングツールの課題を調査しました。
- 従来のソフトウェアエンジニアリング、AIシステム設計、HCIが交差する領域で、特有の課題が頻発している点が重要です。
- 機能不全が67%以上を占め、API、統合、設定エラーが主な原因で、APIエラー、ターミナル問題、コマンド失敗が頻発しています。
Abstract
The rapid integration of Large Language Models (LLMs) into software development workflows has given rise to a new class of AI-assisted coding tools, such as Claude-Code, Codex, and Gemini CLIs. While promising significant productivity gains, the engineering process of building these tools, which sit at the complex intersection of traditional software engineering, AI system design, and human-computer interaction, is fraught with unique and poorly understood challenges. This paper presents the first empirical study of engineering pitfalls in building such tools, on a systematic, manual analysis of over 3.8K publicly reported bugs in the open-source repositories of three AI-assisted coding tools (i.e., Claude-Code, Codex, and Gemini CLIs) on GitHub. Specifically, we employ an open-coding methodology to manually examine the issue description, associated user discussions, and developer responses. Through this process, we categorize each bug along multiple dimensions, including bug type, bug location, root cause, and observed symptoms. This fine-grained annotation enables us to characterize common failure patterns and identify recurring engineering challenges. Our results show that more than 67% of the bugs in these tools are related to functionality. In terms of root causes, 36.9% of the bugs stem from API, integration, or configuration errors. Consequently, the most commonly observed symptoms reported by users are API errors (18.3%), terminal problems (14%), and command failures (12.7%). These bugs predominantly affect the tool invocation (37.2%) and command execution (24.7%) stages of the system workflow. Collectively, our findings provide a critical roadmap for developers seeking to design the next generation of reliable and robust AI coding assistants.
Paper AI Chat
この論文のPDF全文を対象にAIに質問できます。
質問の例: