評価(Eval)とは何か — 「最後のチェック」ではなく「毎日良くするエンジン」

When you change an AI system, the real question is whether it actually got better — and at scale, "vibes" are not enough. A good setup rests on two pillars. The first is offline eval: a custom eval suite run before shipping, which stops a release if there is a major regression. The second is online eval: every session in production leaves a trace, and millions of them show what users really do. Because one trace is hard to debug, the key move is trace clustering — grouping similar failures so a recurring failure mode stands out. Important or controversial fixes can then be checked with AB testing when needed, since results are rarely clean and human judgment still decides. This is hard because vibe coding starts from a green field with no tests, so grading cannot just run a fixed test set. The takeaway: do not treat evaluation as a last check before shipping; treat it as an engine that ships a better, production quality agent every day.