Analysis · Technology

Healthcare AI’s Measurement Gap: Tracking Use More Than Impact

By AWEI · AI-compiled · Published 2026-10-19 · Analysis prepared 2026-10-19 · 1 source · doi.org

Counting who uses an AI tool leaves a harder question unanswered: who benefits? In a 2025 study by Eric G. Poon and colleagues, 74% of respondents answering the measurement question said their health systems always tracked AI uptake. Only 17% said they always measured health equity or disparities.

The researchers surveyed leaders at US nonprofit health systems belonging to the Scottsdale Institute in autumn 2024. Of 67 invited systems, 43 completed the survey. It covered 37 AI use cases across 10 categories, asking about implementation stages, perceived success, evaluation practices and barriers. This was a snapshot of organizational reports, rather than an experiment testing AI effectiveness.

One finding illustrates why implementation stages matter. Every participating system reported some activity around ambient notes, which turn clinician–patient conversations into draft documentation. But only six reported full deployment; 20 had deployed them in limited areas, and 17 were developing or piloting them. Universal activity therefore covered very different levels of operational commitment.

Among respondents working on clinical documentation AI, 53% reported a high degree of success. That was a subjective assessment of a category, not a measured reduction in workload or proof of better patient outcomes. Similarly, 77% ranked immature AI tools among their two biggest barriers; the survey did not establish which technical shortcomings caused implementation difficulties.

The sample also limits interpretation. These were largely substantial, comparatively well-resourced organizations engaged in shared learning. Their experience may not represent smaller systems. Clinicians’ prominence among respondents could have understated business applications, and a single survey cannot establish causal effects or long-term value.

For deployment teams, the useful lesson is to separate implementation status, perceived success and measured outcomes. Tracking adoption can show whether a tool reaches users. Assessing its value also requires evidence about the intended benefit, ongoing costs and how results differ across patient groups.

All newsNews archive