← All research

Bug fixing accuracy from observability data

A public benchmark for the question every debugging agent should answer: what caused the production failure?

We published a benchmark of 22 real production incidents, each with a human-written answer key. An agent reads the codebase, investigates, and writes a root cause. A GPT-4o judge marks it correct only if it names the same incident, the same precise cause, and would lead an engineer to the same fix.

This is not a log summarization task, and it is not a test of whether an agent can name a nearby service. The expected answer is one an engineer can act on: the failing behavior, the relevant signal, and the fault that explains both.

81.8%
Foam agent (18/22)
+9pp
From adding Foam MCP to Cursor
22
Production incidents
RCA accuracy on April 4, 2026 Cursor with Sentry MCP scores 40.9 percent, Cursor alone 54.5, Cursor with Foam MCP 63.6, and the Foam agent 81.8. An April 12 update to the Foam agent reached 86.4 percent. Cursor + Sentry MCP 40.9% (9/22) Cursor alone 54.5% (12/22) Cursor + Foam MCP 63.6% (14/22) Foam agent 81.8% (18/22) | 86.4% (19/22) on April 12 Cursor runs opus-4.6 in every setup; only the tools change.

Measured against Cursor with no extra tools, Sentry MCP costs three incidents, Foam MCP gains two, and the Foam agent gains six. An update to the Foam agent on April 12 took it to 19 of 22.

Net change in incidents solved versus Cursor alone Sentry MCP solves 3 fewer incidents than Cursor alone, Foam MCP 2 more, and the Foam agent 6 more. Cursor alone (12) -3 Sentry MCP +2 Foam MCP +6 Foam agent Net incidents solved versus Cursor with no extra tools.

Net numbers hide how much each setup changes. Sentry MCP solved three incidents Cursor alone missed but lost six it had solved. Foam MCP solved four new ones and lost two. The Foam agent solved seven new ones and lost one.

Incidents each setup solved, split by whether Cursor alone also solved them Sentry MCP adds 3 incidents Cursor alone missed, Foam MCP adds 4, and the Foam agent adds 7. also solved by Cursor alone missed by Cursor alone Cursor alone 12/22 Cursor + Sentry MCP 9/22 +3 new Cursor + Foam MCP 14/22 +4 new Foam agent 18/22 +7 new
The difference is not only model quality. It is the shape of the context. Agents are materially better when telemetry is presented as an investigation surface instead of as raw exhaust.

We use this benchmark as a product instrument. If Foam improves here, it means the system is getting better at the work customers actually ask it to do: isolate the fault, explain the evidence, and reduce the time between incident and fix.

Foam Research · April 4, 2026