EdgeBench claims a near‑perfect log‑sigmoid fit (R² = 0.998) across roughly 38, 000 agent‑hours, and that needs scrutiny
The EdgeBench project (ByteDance‑Seed) reports that agent performance as a function of interaction time follows a log‑sigmoid curve with R² = 0.998 based on ≈38, 000 agent‑hours. That claim appears in the EdgeBench README and the project tech report (see the project pages below). A MARKTECHPOST Colab tutorial reproduces and inspects that fit programmatically: it downloads the public snapshot, parses task JSONs and README leaderboards, canonicalizes model names, rescales raw judge outputs, reshapes scores across time budgets, and fits log‑sigmoid curves to score‑vs‑time trajectories.
“In this tutorial, we explore EdgeBench as a practical benchmark for evaluating advanced AI agents across diverse task categories, runtime environments, and interaction-time budgets.”, MARKTECHPOST notebook
Links to primary sources:
- EdgeBench GitHub
- EdgeBench docs / SForge guide
- EdgeBench on Hugging Face
- EdgeBench tech report (arXiv)
Why this matters for decision makers
EdgeBench is one of the first public benchmarks to tie agent performance directly to interaction time with a simple curve. If the R² = 0.998 fit is real, it changes how you plan experiments, buy compute, and judge model improvements. A predictable curve means you can forecast gains from extra interaction time, compare agents more consistently, and set clearer time‑budget targets for deployments. It also raises questions: how robust is the fit across tasks, judges, and model families, and how much of the signal comes from preprocessing and score rescaling versus real agent learning?
Further reading
For a practical walkthrough of running and scaling experiments that mirror the type of reproductions we discuss above: