BizBench preprint: Claude Opus 5 hits 64.0% strict success on payments-company tasks
Researchers built 50 tasks from a leading payments company's procedures, with 789 documents and 601 tools. GPT-6 Sol scored 61.3%. Qwen3.8-27B reached 42.0% goal attainment but 32.7% strict success.
Why it matters: Outcome checks can hide rule-breaking along the way, the authors argue. They also report DeepSeek-V4.1-Flash averaged 191.3 tool calls per run, about 70% more than reference runs.
From Drop #033 · Mon, Oct 19 · Morning