PsychoHAistoryResearch Initiative

Journal · No. 10

The Challenge of Measuring AI Agent Behavior

Before predicting agents, we need measurements of them that repeat.

Measurement comes before prediction. If an agent’s retry rate swings widely between runs of the same task, no model built on it will predict anything.

Candidate measurements

Retry tendency, verification habits, tool-use breadth, escalation rate and handoff quality are all observable in execution traces. The first question is reliability: do repeated runs agree?

Moving targets

Agents change when their underlying models change. A measurement may have a short shelf life, which itself is worth quantifying. Track R01 plans to report not just the traits, but how long they stay valid.

This is a concept note. It reports no experiments and no results.