(a) Sequential test-time scaling

(b) Parallel test-time scaling

• Sequential scaling hits a context ceiling: performance initially improves with more interaction turns, but then plateaus and even declines as the growing context destabilizes the agent.
• Parallel scaling suffers from a verifiability gap: while pass@K grows steadily with more samples, self-choice accuracy remains nearly flat — agents cannot reliably identify the correct trajectory among candidates.

• We introduce General AgentBench, a benchmark that provides a unified framework for evaluating general LLM agents across search, coding, reasoning, and tool-use domains. Evaluation of ten leading LLM agents reveals a substantial performance degradation when moving from domain-specific evaluations to this general-agent setting.

Performance comparison between specialized-agent and general-agent settings.
Top: Absolute performance. Bottom: Relative performance degradation under the general-agent setting.

• Using General AgentBench, we systematically study test-time scaling behaviors under sequential scaling (iterative interaction) and parallel scaling (sampling multiple trajectories). We find that neither scaling methodology yields effective performance improvements in practice, due to two fundamental limitations: context ceiling in sequential scaling and verification gap in parallel scaling.

• Run the general agent system: see general_agent/scripts/ for experiment scripts and general_agent/README.md for details.
• Run individual benchmarks independently: see benchmarks/instructions/ for per-benchmark setup and usage guides.

General AgentBench may call external APIs for (1) LLM inference and (2) benchmark tools.

We use LiteLLM-style model strings (a.k.a. “model routes”) to specify which API/provider a run uses. If a single model has multiple routes listed, it means we have used all of those routes in different runs/stages.

For the exact LiteLLM model routes used in experiments, see general_agent/scripts/models.py.

If you find this work or code useful, please consider citing:

coming soon

This project is released under the MIT License. See LICENSE for details.