AIDB Daily Papers
LLMが過剰に回答してしまう問題:ハードウェア記述言語の質問応答における品質課題の測定と軽減
※ 日本語タイトル・ポイントはAIによる自動生成です。正確な内容は原論文をご確認ください。
ポイント
- ハードウェア記述言語の質問応答におけるLLMの回答品質を評価するため、Stack Overflowから収集したデータセットとエンジニアによるユーザー調査を実施した。
- LLMは正解を含むものの過剰な別解や冗長な説明が多く、専門家の回答と完全に一致しない場合でも読みやすさの面で好まれる傾向があることが判明した。
- 提案するマルチエージェントフレームワークにより、主要な回答の品質と非中核コンテンツの品質を大幅に向上させることができた。
Abstract
The rapid advancement of large language models (LLMs) has led practitioners to increasingly rely on them for answering questions about hardware description languages (HDLs). Because HDL is ultimately synthesized into physical hardware, an imprecise or redundant answer can propagate into timing violations or non-synthesizable logic that surface only late in the design flow, making the quality of HDL answers especially consequential. However, the quality of LLM-generated responses, particularly in comparison with answers provided by human experts, remains unclear. To investigate this question, we collect 6,246 HDL Q&A posts with accepted answers from Stack Overflow and curate them into a dataset, organized into a taxonomy of four main categories (Conceptual, Debugging, Generation, and Optimization) and ten subcategories. Using this dataset, we design a user study conducted with 19 HDL engineers with one to three years of experience. Our findings reveal a pervasive over answering tendency: LLMs supply correct content but bury it under redundant alternatives (65.7%) and verbose padding (69.1%), while nearly half of answers (49.0%) fail to fully align with expert answers yet participants still preferred LLM responses for readability (58.3%). Motivated by these findings, we propose a multi-agent framework for improving LLM-based HDL question answering. We evaluate answer quality using an LLM-as-Judge and two structural metrics: the number of core answers, which reflects redundancy since LLMs often provide multiple alternative solutions, and the length of non-core content, which reflects verbosity. Evaluated on the four mainstream LLMs, our framework increases the average core-answer quality score from 3.71 to 4.67 (+0.96) and the non-core content quality from 3.72 to 4.23 (+0.51), on a five-point scale.
Paper AI Chat
この論文のPDF全文を対象にAIに質問できます。
質問の例: