Marginal tool utility in agentic coding
Measuring whether each tool call helped an agent solve the task, not merely whether the tool was used.
TL;DR
We evaluated 150 agent trajectories on APEX-SWE Observability. Removing Mattermost and Plane preserved task accuracy while sharply increasing the proportion of tool calls that helped solve the task. Grafana/Loki was the only read-only tool with consistently positive aggregate utility.
+100%
GPT-5.3-Codex
Tool efficiency increased from 0.359 to 0.720.
+62%
Gemini 3.1 Pro
Tool efficiency increased from 0.367 to 0.593.
0pp
Accuracy lost
Neither model lost accuracy after removing noise.
- Marginal tool utility measures whether one call increased the probability of solving the task.
- Tool efficiency is the share of calls with positive marginal utility.
- The result is not that agents always need fewer tools. Tool suites need direct measurement.