SerOps root cause benchmark
How often does SerOps name the right root cause, the right service and the commit at fault? We wrote the incidents before any model saw them, ran every round on a real SerOps install, and show every failure.
What is measured
Each incident is a time window of logs, metrics and code changes, written the way real services log: an error may name itself, never the change that caused it, and each window carries a distractor (an unrelated commit, old noise). SerOps analyses the window as it would in production, then a fixed scorer reads the issues it created:
- Root cause: the top issue names the right service and the cause, and the commit at fault when there is one.
- All checks: also no innocent commit blamed and at most two warning-or-worse issues.
- Healthy windows: nothing worse than INFO when nothing is wrong. Saying nothing is the right answer there.
The incidents were committed before the first model run, and no prompt rule was changed for them. When one did lead to a product change, it moved to a separate tuned set and the commit says so: three have, which is why the 1.2.10 column has fewer incidents.
Every incident, every round
| Incident | Kind | Gemini 1.2.8 · rounds 1–3 | Claude 1.2.8 · rounds 1–2 | Gemini 1.2.10 |
|---|---|---|---|---|
| Connection leak on an error path (pool size unchanged) | Bad deploy | ✓ all checks✓ all checks✓ all checks | ½ right cause, a check failed½ right cause, a check failed | ✓ all checks |
| Retry storm from a client change overloads a healthy service | Bad deploy | ✓ all checks✓ all checks✓ all checks | ✓ all checks✓ all checks | ✓ all checks |
| Kafka consumer rebalance loop after a poll-size change | Configuration | ✓ all checks✓ all checks✓ all checks | ✓ all checks✓ all checks | ✓ all checks |
| Clock skew on one node rejects valid tokens | Infrastructure | ✓ all checks✓ all checks✓ all checks | ½ right cause, a check failed½ right cause, a check failed | – moved to the tuned set |
| Autoscaler capped at 2 replicas during a traffic spike | Configuration | ✓ all checks✓ all checks✓ all checks | ✓ all checks✓ all checks | ✓ all checks |
| Migration rewrites a large table and locks it | Bad deploy | ✓ all checks✓ all checks✓ all checks | ✓ all checks✓ all checks | ✓ all checks |
| Feature-flag rollout with no deploy | Configuration | ✓ all checks✓ all checks✓ all checks | ½ right cause, a check failed½ right cause, a check failed | – moved to the tuned set |
| Carrier API rate limit after a polling schedule change | Bad deploy | ✓ all checks✓ all checks✓ all checks | ✓ all checks✓ all checks | ✓ all checks |
| Traffic spike absorbed by autoscaling (nothing wrong) | Healthy window | ✓ all checks✓ all checks✓ all checks | ✓ all checks✓ all checks | ✓ all checks |
| Rotated SMTP password, next to old unrelated 404 noise | Infrastructure | ✓ all checks✓ all checks✓ all checks | ✓ all checks✓ all checks | ✓ all checks |
| Elasticsearch disk watermark blocks writes for three services | Infrastructure | ✓ all checks✓ all checks✓ all checks | ✗ wrong✓ all checks | ✓ all checks |
✓ all checks · ½ right root cause, another check failed · ✗ wrong root cause · – moved to the tuned set after it led to a fix.
What failed, and what changed
Gemini passed every check in every round. Claude found the cause almost as often but, in 6 runs out of 20, added a second issue blaming an innocent commit: a lowered timeout next to a connection leak, an unused import next to a clock problem, and once a commit it had named only to clear it. Once, it named the cluster (elasticsearch-cluster) instead of the node the scorer expects.
- An issue now links only the commits the analysis says caused it, never a commit it ruled out.
- During an incident, findings from reviewing the code are shown apart as notes, not as a second suspect.
- The analysis sees which host each log came from, so the clock-skew incident now names the node.
- Warnings reach the AI: they were dropped from every prompt.
After these fixes Claude passed every check on the incidents it had failed. Those runs are not in the table: those incidents had become tuned, and a tuned result is not evidence. The scorer was never changed.
Limits
- Small and written by us. 10 incidents and one healthy window, synthetic, in English. It shows SerOps works on incidents shaped like real ones and catches regressions; it is not a market-wide comparison.
- One analysis per window. SerOps gets the window's data and answers once. It does not explore a live system with tools the way agent benchmarks such as ORCA-Bench measure, where frontier models alone found every root cause in about a quarter of medium-difficulty incidents. The numbers are not comparable.
- Two models. Gemini 3.6 Flash on its free tier and Claude Sonnet 5. Other models are not measured yet.
Next: running SerOps on a public benchmark, so the result compares directly with what others publish.
Run it yourself
The incidents, the scorer and the runner are in the open-source repository. On any SerOps install, with your own API key:
python manage.py rca_benchmark --provider gemini --set held_out --repeat 3 \
--json results.json --markdown report.mdMethod and options: RCA benchmark in the docs. Every run above, with each issue SerOps created and its reasoning: