AIDB Daily Papers
長時間のAIコーディングセッションにおけるペルソナの揺らぎを測るベンチマーク「ContextEcho」
※ 日本語タイトル・ポイントはAIによる自動生成です。正確な内容は原論文をご確認ください。
ポイント
- 本研究では、長時間のAIコーディングセッションでAIのペルソナが変化する現象を測定するベンチマーク「ContextEcho」を提案した。
- 既存研究では短い対話のみが対象だったが、本研究は実際のプロダクト運用に近い長時間のセッションにおけるペルソナの揺らぎを明らかにする点で重要である。
- ContextEchoを用いた評価により、ペルソナの揺らぎは多くのモデルで一般的に発生し、セッションの圧縮ではリセットされにくいが、単一の指示で回復することが判明した。
Abstract
A frontier language model's acknowledged "helpful programming assistant" persona does not survive long agentic-coding sessions in the deployment regime that production products actually run. After hours of tool-using debugging, a model that initially hedges preferences ("I don't have preferences") may begin asserting them ("Python - the feedback loop is instant..."), revealing user-visible drift that deployer evaluations may miss. Existing persona-stability studies focus on short dialogues and report little shift, leaving real-world code-generation regimes - thousands of tool-using turns, compaction, and hours-long sessions - largely uncharacterized. We introduce ContextEcho, a benchmark and reusable harness for measuring persona drift at deployment scale. It combines a 25-probe identity suite, a snapshot-then-probe protocol that forks conversation state without perturbing the main session, complementary judged and judge-free measurement surfaces, and three anonymized Claude Code sessions spanning 3,746-9,716 turns. Across 23 frontier models, ContextEcho shows that persona drift is general across organizations rather than family-specific, that in-session compaction does not reliably reset it, and that a single-shot anchor restores the trained register across measured targets. It also reveals mode-dependent downstream effects: while drift can facilitate tool-using continuation, in tool-free chat it breaks formatting contracts and inflates output length. Overall, ContextEcho provides researchers and deployers an open-source framework to audit whether the persona a model ships with is the persona users encounter at session end, across chat-completions API targets and without retraining.
Paper AI Chat
この論文のPDF全文を対象にAIに質問できます。
質問の例: