When 500 investment bankers reviewed AI outputs from top models like GPT-5.4 and Claude Opus 4.6, they delivered a unanimous verdict: not a single deliverable was ready to send to a client. The BankerToolBench study, released by researchers at Handshake AI and McGill University, tested nine frontier AI models on the actual work junior investment bankers do daily—Excel financial models, PowerPoint decks, and Word memos. While 68% of AI-generated work might serve as a starting point, 41% required major rework and 27% was deemed completely unusable. For CIOs and CFOs evaluating AI investments, this benchmark exposes a critical gap between lab performance and production readiness.

This isn't about models lacking intelligence—it's about AI failing at the invisible expertise that separates draft work from deliverable output. The research team enlisted bankers from Goldman Sachs, JPMorgan, Evercore, Morgan Stanley, and Lazard to design 100 realistic tasks. Each task took human bankers an average of 5 hours, with some running up to 21 hours. AI agents had to navigate real data rooms, pull from FactSet and Capital IQ, parse SEC filings, and produce deliverables graded against 150 criteria covering technical correctness, client readiness, compliance, auditability, and consistency. The verdict: even the best-performing model, GPT-5.4, cleared every critical criterion on just 2% of tasks.

The Hardcoded Values Problem: When Polished Looks Hide Broken Logic

Claude Opus 4.6's outputs looked professional at first glance, but the Excel models revealed a fundamental flaw that made them useless for actual banking work: hardcoded values instead of formulas. According to the original research report from The Decoder, this dealbreaker means changing one variable—like the purchase price in a merger model—doesn't update the rest of the spreadsheet. In investment banking, scenario analysis is the entire point of building financial models. A model that can't dynamically recalculate when assumptions change is worthless, no matter how cleanly formatted the output appears.

This is a pattern we're seeing across enterprise AI deployments: models optimized for surface-level correctness fail at structural requirements that domain experts take for granted. Claude Opus 4.5 had the identical flaw. The issue isn't that Anthropic's engineers don't understand Excel—it's that the training data and reinforcement learning optimization prioritize "looks right" over "works right." For CFOs and COOs evaluating AI tools, this is the key question: does your vendor's benchmark measure cosmetic correctness or functional reliability under real-world use?

GPT-5.4 Leads, But 16% Usability Is Still a Failing Grade

GPT-5.4 topped the leaderboard with an overall score of 58.1 out of 100—better than Claude Opus 4.6's tie with Gemini 3.1 Pro, but still failing nearly half of grading criteria. Just 16% of GPT-5.4's outputs were rated as a "useful starting point" by bankers. When the researchers required three consistent runs (to test reliability), that figure dropped to 13%. For enterprise leaders, this consistency gap is critical: an AI agent that produces usable output 16% of the time isn't automating work—it's creating quality control overhead.

The research identified four recurring failure modes in GPT-5.4's agent trajectories: 41% were code and formula bugs (calling non-existent Python-PPTX functions, then deleting broken lines instead of fixing them); 27% were business logic errors (adding cost synergies to the revenue line); 18% were aborted data queries; and 13% involved fabricating missing numbers and passing them off as sourced. That last category is a compliance nightmare for any regulated industry. When an AI agent invents clinical trial data for a pharma competitive analysis or fabricates revenue figures, it's not just wrong—it's a legal liability.

Here's the cost reality for enterprise AI at this performance level: GPT-5.4 costs $2.50 per million input tokens and $15 per million output tokens. Claude Opus 4.6 runs $5.00 input / $25.00 output. A single BankerToolBench task triggered up to 539 calls to the language model, with 97% tied to tool use or code execution. If you're paying API costs for hundreds of LLM calls per task and still getting output that requires 5 hours of human rework, the ROI math doesn't close. For technical leaders evaluating build-vs-buy, this benchmark suggests purchasing specialized AI solutions from domain-focused vendors (67% success rate) beats internal builds (33% success rate).

Subtle Errors That Slip Through: The QA Nightmare

The examples in the research paper illustrate how easily these failures slip past surface-level review. One generated PowerPoint deck showed a revenue figure of $189.5 billion on one slide and $201.0 billion on the next slide—both supposedly covering the same period. Another agent used Netflix red as an accent color despite the bank's style guide mandating uniform blue. These aren't catastrophic failures that trigger immediate red flags—they're subtle inconsistencies that erode client trust and require human QA to catch.

The researchers built an AI verifier called "Gandalf" (based on Gemini 3 Flash Preview) to grade outputs against banker-designed rubrics. Gandalf agreed with human reviewers 88.2% of the time, slightly above the 84.6% agreement rate between two human reviewers. That 11.8% disagreement rate is the problem: when AI grading itself can't reliably catch AI mistakes, you're still paying for human oversight. For VPs of Engineering and CTOs, this is the production deployment reality—you can't eliminate human review, so the value proposition shifts from "automation" to "augmentation with quality control overhead."

Where AI Performs Better (And Worse): Task-Level Breakdown

The models generally performed better on PowerPoint tasks than Excel work, with the toughest challenges in debt capital markets, merger models, and capital structure tables. The research team attributed some failures to missing domain knowledge—when tasks were enriched with context bankers take for granted, scores rose significantly. This aligns with broader enterprise AI failure data: 85% of AI projects fail due to poor data quality or lack of relevant context, and 70-90% of enterprise AI initiatives never reach stable production deployment.

For business leaders evaluating AI pilots, this is the key insight: lab benchmarks measure narrow task completion, not the messy reality of incomplete data, ambiguous instructions, and evolving business logic. MIT's Project NANDA (July 2025) found that 95% of organizations deploying generative AI saw zero measurable return. Deloitte's 2026 "State of AI in the Enterprise" survey noted that only 25% of organizations have moved even 40% of their AI experiments into production. The gap between pilot success and production viability is structural, not technical.

What This Means for Enterprise AI Strategy

BankerToolBench is one of the most detailed tests yet of whether AI agents can handle demanding knowledge work. For now, the answer is no—but with important caveats. Over half of bankers said they'd use AI output as a starting point, and reinforcement learning experiments boosted benchmark performance by 5-13x (though from very low baselines). The technology is improving, but current frontier models aren't ready for autonomous client-facing work.

Here's what works in production today: Simple, tightly controlled workflows with few steps and human oversight at decision points. UC Berkeley research concluded that teams getting agents to work in production rely on narrow, deterministic setups—not the ambitious end-to-end automation vendors promise. For CIOs and CTOs, this means treating AI as a productivity tool for junior analysts, not a replacement for domain expertise. The ROI case shifts from headcount reduction to faster iteration and higher-quality first drafts.

The investment banking benchmark also exposes a broader enterprise AI challenge: most agent development has focused on coding tasks, leaving economically important fields like management, law, and finance largely absent from benchmarks. Carnegie Mellon and Stanford researchers argue this creates a false sense of AI readiness when applied to non-coding knowledge work. For enterprise leaders, the lesson is clear: don't trust vendor benchmarks that measure different tasks than your actual workflows.

The Vendor Response: What's Coming Next

AI labs are already working on the exact weaknesses BankerToolBench exposes. Anthropic recently introduced a feature that lets Claude switch independently between Excel and PowerPoint, and Cowork plugins now pipe market data services like FactSet, MSCI, and LSEG directly into workflows. OpenAI positions GPT-5.4 as a "finance powerhouse," citing 87.3% accuracy on internal investment banking analyst benchmarks—but BankerToolBench's independent evaluation shows the gap between controlled tests and real-world complexity.

For procurement and vendor evaluation, the critical question isn't "what's your benchmark score?"—it's "show me three production deployments with documented ROI." BankerToolBench lines up with other recent research: Vals.ai found OpenAI's o3 hit just 48.3% accuracy on financial analysis tasks. The pattern is consistent: frontier models struggle when moved from lab conditions to real workflows with incomplete data, evolving business logic, and strict compliance requirements.

The research is open source—full benchmark, data, rubrics, and verifier available on GitHub. For enterprise AI teams, this is a gift: an independent, domain-expert-designed evaluation framework that measures what actually matters in knowledge work. Use it to pressure-test vendor claims before deployment, not after.

---

Want to calculate your own AI ROI? Try our AI ROI Calculator — takes 60 seconds and shows projected savings, payback period, and 3-year ROI.

Continue Reading

Interested in more enterprise AI insights and benchmarks? Check out these related articles:

• How Enterprise AI Projects Actually Fail: The Data Quality Crisis – Why 85% of AI initiatives fail before reaching production
• The ROI Reality Check for Generative AI in Finance – What MIT's 95% zero-return finding means for your budget
• Vendor AI Benchmarks vs. Production Reality: What CTOs Need to Know – How to evaluate AI tools before expensive pilots fail