Twenty-three of 25 health systems with third-party AI in production say they test it before go-live. Only 11 of them have anywhere to run that test.
Everyone else is reading the vendor's accuracy numbers — measured on a different population, in a different workflow, under a different lab's assays — and filing it as validation. That is the load-bearing finding in Validation and Trust: How Health Systems Are Testing and Governing Analytics and AI Solutions, published August 6 by the Center for Connected Medicine at UPMC and KLAS Research. Read the provenance before the numbers: UPMC Enterprises, the system's commercial arm, is building exactly the testing platform the report finds most systems lack, and the report holds it up as the example of what a well-equipped organization does. That does not make the survey wrong — KLAS collected it — but the gap it identifies is one its publisher sells into.
The report is drawn from interviews with 27 executives — VPs, CIOs, CTOs, CFOs, COOs and CMIOs — conducted in May and June 2026, nine of them at systems with more than 1,000 beds. It is the first dataset I have seen that separates the claim of pre-deployment testing from the infrastructure to perform it. The claim is nearly universal. The infrastructure is a minority sport. If you buy AI in any regulated industry, that distinction is now your problem too.
What "We Validate It" Actually Means Here
Validation in health systems today is a word, not a method. The report found that 92% of organizations evaluate third-party AI tools before deployment while only 44% have a dedicated data platform or environment to do it in — and that what leaders mean by "validate" ranges from confirming a workflow fires correctly, to user acceptance testing, to full assessment of accuracy, bias, drift and clinical safety.
The methods named are equally scattered: formal test cases, business sign-off, limited pilots, proof-of-concept trials, dual coding against human output, and sandbox testing. Those are not variants of one practice. A business owner clicking through a demo and a data science team scoring a model against a held-out local cohort are different activities that happen to share a verb.
The infrastructure numbers explain why. Of the 25 systems running third-party AI, 11 have a testing platform, 8 have none at all, and 6 did not know. Ten respondents do their analysis inside the EHR or the vendor's own analytics tools — which is to say, inside the thing being evaluated. Seven use a cloud data warehouse. Five were unsure where the work happens.
And the day job is worse than the strategy deck suggests. The most-cited data problems were manual workarounds and spreadsheets (17 mentions) and inconsistent definitions across teams (14). You cannot measure a model's local performance when two departments disagree on what counts as an admission.
The Vendor's AUC Was Measured on Other Patients
An external validation is the only number that tells you how a model behaves on your population — and healthcare has the industry's most expensive proof of that. In June 2021, Wong et al. published an external validation of the Epic Sepsis Model across 27,697 patients and 38,455 hospitalizations at Michigan Medicine. The model, deployed at hundreds of US hospitals, scored an area under the curve of 0.63 against the 0.76–0.83 its developer reported. Sensitivity was 33%. Positive predictive value was 12%. It fired on 18% of all hospitalizations — nearly one alert for every five patients a clinician had to evaluate.
Nothing was broken in the vendor's math. Epic disputed the finding, arguing the authors had scored the model without the local tuning that is supposed to precede real-world deployment — which, taken at face value, is the case for a testing environment made by the vendor itself. The model was measured on a different population and generalized badly, which is what models do.
Here is the part that should end the argument. Epic's second-generation sepsis model was prospectively validated across 227,091 encounters at Michigan, OHSU, Emory and MetroHealth, in a study published in February 2026 and it is genuinely better — encounter-level AUROC of 0.82 to 0.92, beating v1 at every threshold and every site. But to hit the same 60% sensitivity, the four sites needed alert thresholds ranging from 14 to 37. The authors' own conclusion is that health systems implementing the model "should conduct internal evaluations to validate local performance, deploy alert silencing strategies, and design clinical workflows to help manage false positives."
A good model, four good hospitals, one shared configuration — and the right setting varies by more than a factor of two. There is no vendor number that can tell you which one is yours. This is the same failure mode enterprises hit when a vendor silently re-post-trained a model and every downstream eval quietly stopped meaning anything; healthcare just gets to find out with a patient in the bed.
Nobody Is Watching the Model After Go-Live
Post-deployment monitoring is the loosest link in the chain, and the report says so plainly: gaps persist in post-launch monitoring and ROI validation. Eight of 26 respondents are not measuring value at all. Efficiency and throughput metrics led at 12 mentions; financial ROI at 9. There is no consensus on what success even looks like — which means there is no threshold that, when crossed, triggers anyone to turn the thing off.
Meanwhile the model underneath you is allowed to move. The FDA's final guidance on predetermined change control plans, issued December 4, 2024, lets manufacturers ship pre-authorized modifications to an AI-enabled device "in lieu of obtaining separate authorizations for each significant change." That is sensible regulation. It also means the model you validated in March is not necessarily the model running in September, and the regulator is not going to call you.
For most of what health systems actually bought, there is no regulator in the loop at all. Clinical documentation and ambient scribing is 52% of deployments and revenue cycle, coding and billing another 36% — categories that largely sit outside FDA device oversight. The transparency that does exist is narrow: the ONC decision support intervention criterion requires certified health IT to surface 31 source attributes for predictive DSIs, including the external validation process and the schedule for continued validation, with maintenance obligations running from January 1, 2025. That covers predictive tools inside a certified EHR. It does not cover the scribe, the prior-auth drafter or the coding assistant.
The voluntary layer is filling in around it. The Joint Commission and the Coalition for Health AI released their first joint guidance on September 17, 2025, calling for policies, "appropriate local validation, monitoring, and use", with an optional certification to follow for the Joint Commission's 22,000-plus accredited organizations. Local validation is the phrase. Fewer than half of the systems in this report have the infrastructure to perform it.
What a Test Environment Actually Costs to Build
A test environment is a deidentified copy of your own patient data, plus the tooling to score a model against it — and the expensive version is not the only version. UPMC built the expensive one: Ahavi holds primary-source-verified deidentified data on more than 5 million patients across 24-plus hospitals, including 156 million structured encounter records since 2019, and it lets an organization "assess the efficacy of AI models against our patient population prior to ever having to deploy it against our actual population". Most systems will never build that.
They do not have to. Epic released Seismometer as open source under a BSD-3 license — "AI model evaluation with a focus on healthcare" — and it is agnostic to where the model came from, covering vendor models and homegrown ones, with a fairness audit across demographic cohorts. Michigan Medicine used it on the second-generation sepsis model to set different thresholds for cohorts such as the emergency department and the intensive care unit. That is the exact problem the multicenter study identified, solved locally, with free software.
The rest is a data platform you probably already pay for. Systems in the report named cloud warehouses, production clones and pre-production EHR environments; the eval workload lands on the same stacks the data team already runs — Snowflake Cortex AI, Databricks Mosaic AI, Azure AI Foundry — with ongoing performance monitoring from the observability tier, Fiddler AI or Arize AX, or a healthcare-specific platform such as Qualified Health.
The binding constraint is not licensing. Resource, budget, time and talent limits were the single most-cited barrier, at 11 mentions, and only one of 27 systems described its AI strategy as advanced — 16 called it "developing." A sandbox is a funded headcount decision that competes with buying another tool. Most organizations have been resolving that trade-off in favor of the tool. The same instinct shows up in enterprise AI generally, where teams buy the policy pack instead of the inventory that would tell them what they actually run.
Put the Right to Validate in the Contract
The report's own recommendation is a procurement instruction, not a technical one: "AI embedded in vendor platforms is hardest to evaluate independently, so press for testing access during contracting." Once the module is live inside a suite you already own, your leverage is gone and the evaluation surface is whatever the vendor exposes.
Four clauses, and none of them are exotic:
- A local validation right before go-live. You test on your own deidentified data, on a defined cohort, and the result is a condition of acceptance — not a post-implementation courtesy.
- Notification on model change. Written notice before a retrain, a version bump or a post-training update reaches your tenant, with enough lead time to re-run the acceptance test. The vendor gets a change window; you get to keep your evidence.
- An agreed success metric and a drift threshold, in the contract. Name the number, name the population, name what happens when it degrades. Eight of 26 systems are measuring nothing, which is a contract problem before it is an analytics problem.
- Performance transparency in the format the DSI criterion already defines. If a vendor can produce 31 source attributes for a certified EHR, it can produce them for you.
Ambient documentation shows what is at stake in both directions, including for this article. The reported wins are large — Emory recorded a 30.7% absolute increase in documentation-related well-being at 60 days, Mass General Brigham a 21.2% absolute drop in burnout prevalence at 84 days, and Intermountain a 27% reduction in time in notes. Then apply this article's own test to them. The first two come from a pre/post survey with no control arm, answered by 11% of eligible clinicians at Emory and 22% at Mass General Brigham — all of them people who had already chosen to keep using the tool. Intermountain's figure counts only clinicians who reached ten or more encounters on it. Those are outcomes worth pursuing. They are not outcomes you can confirm, or defend, without a baseline you measured yourself.
Three Things to Fund Before the Next Renewal
This Week: Pull your current AI inventory and mark each entry with one of three labels — validated on our data, validated on the vendor's data, not validated. Do not estimate; ask the owner for the artifact. The count in column two is the number you will be asked about the first time something goes wrong. Then check whether anyone would notice a silent model update: name the person, or admit there isn't one.
This Month: Stand up the cheapest thing that counts as a test environment — a deidentified extract of one high-volume cohort in the warehouse you already run, and one open-source evaluation harness pointed at it. Score one live model. One is enough to prove the pipe works and to expose every data-definition argument you were going to have anyway. Budget it as infrastructure, not as a project.
Before Renewal: Take the four clauses above into the next contract conversation, in writing, before the module is switched on. Ask for the vendor's external validation population and compare it to yours — payer mix, acuity, lab assays, care setting. If the vendor cannot describe the population its numbers came from, you have learned something more useful than the numbers.
The Bottom Line
Every technology cycle produces the same gap, and this one is unusually well documented. We bought clinical AI the way enterprises bought cloud in 2013 — faster than we built the ability to observe it — and the observability bill always comes due later, at a worse price. Only 5% of enterprises report substantial returns on AI — a measurement failure as much as a returns failure — and the most common production failures were never hallucination — they were the mundane, invisible degradations that only local monitoring catches. Reviewers are not a backstop either: the less an AI explained itself, the more novice reviewers deferred to it.
The federal government now pre-tests frontier models before they reach agencies. A 1,000-bed health system with a vendor model touching sepsis alerts has a strictly harder problem and, in fewer than half of cases, a confirmed place to run the test.
Thirty-seven is not fourteen. Somebody has to measure which one you are.
Continue Reading
- Qualified Health's $125M Series B Shows Healthcare AI ROI Is Real
- 5% Track AI ROI. Optura's $17.5M Fix for the $18B Gap
- DeepSeek Swapped the Model. Your Eval Didn't Notice.
- The Vaguer the AI Explanation, the More Novices Trusted It
- 10,000 AI Failures Exposed. Hallucination Isn't #1.
- Longevity Health Hits $1M in Savings, Launches AI Center of Excellence
- Feds Now Pre-Test Every Frontier AI Model: Vendor Playbook