TraceTrial-Bench: A benchmark for evaluating AI agent harness reliability

Researchers introduce TraceTrial-Bench, a benchmark designed to assess the reliability of AI agent infrastructures, a dimension largely overlooked by existing benchmarks. According to the announcement, the benchmark breaks down reliability into seven components: context, execution, governance, lifecycle, observability, tools, and verification. It offers 95 expert-level tasks with executable or state-based ground truths to compare harnesses without altering their native behavior. A Trajectory Tracing module and a two-layer evaluation framework allow separate analysis of agent performance and harness reliability.

TraceTrial-Bench also includes a Multi-source Evidence Acquisition module and a Failure Attribution module to identify root causes at the execution level. Results from 3,420 runs show harness reliability scores ranging from 59.8% to 72.0%, with 31.1% of successful runs failing at least one harness criterion. The benchmark provides an auditable and reproducible foundation for comparing harnesses, diagnosing failures, and studying their interaction with models.