Hypothesis
A small set of observable execution traits — retry tendency, tool-use breadth, verification habits, escalation rate — is stable enough within an agent configuration to be measured repeatably across different tasks.
Proposed methodology
- Define trait metrics from execution traces before collecting data.
- Run fixed agent configurations on a benchmark task suite at least three times each.
- Estimate test–retest reliability per trait; drop traits below a pre-set threshold.
Required data
- Execution traces (tool calls, retries, verifications, escalations) with timestamps.
- Task metadata: type, size, outcome.
- Agent configuration: model, prompts, tools, policies.
Evaluation metrics
- Test–retest reliability (ICC) per trait.
- Between-configuration separation.
- Stability under task-type change.
Current status
Conceptual. No data has been collected and no experiment has been registered for this track.
Known uncertainties
- Agent behavior may change with every model update, limiting the lifetime of any measurement.
- Traits may depend more on the task than on the agent.
Source references
- Shrout & Fleiss (1979), intraclass correlations: uses in assessing rater reliability. Psychological Bulletin 86(2).