Analysis · Technology
An AI Draft Can Avoid Errors and Still Miss the Point
By AWEI · AI-compiled · Published · Analysis prepared · 1 source · doi.org
An AI response can contain no obvious falsehoods and still leave important work unfinished. In a 2026 study by Holly K. T. Huang and colleagues, roughly 90% of human assessments found no misleading or incorrect information in AI drafts for liver specialists. Yet about 65% judged that the drafts omitted some suggestions from the original specialist response.
The team at the University of California San Francisco tested LiVersa, a language model supplemented with retrieved liver disease guidance. It drafted answers to 61 electronic consultations completed between January and March 2025. These are written exchanges in which clinicians ask specialists for help with a patient’s care.
The retrospective comparison included a useful safeguard: LiVersa could use structured patient information available when the specialist answered, but neither the original answer nor later chart additions. Three independent liver specialists compared its drafts with the original responses using a 12-question assessment. This tested draft quality, without measuring effects on actual care.
The omissions were often about interpretation and relevance. Reviewers described drafts that suggested reviewing medications without assessing whether the patient’s medications explained the abnormal tests. Human assessments flagged potential harm without review in about 20% of evaluations, including severe or life-threatening potential harm in 3.4%. These were judgments about possible consequences, not observed injuries.
An automated reviewer also assessed the drafts, flagging potential harm much more frequently than humans. That disagreement limits any claim that automated grading can replace expert oversight. The study covered one institution and excluded consultations redirected to visits. Its assessment tool was not formally validated, reviewers were not blinded, and agreement among reviewers was only moderate.
For deployment, the practical lesson is to evaluate completeness and usefulness alongside factual accuracy. The authors propose draft-only use with mandatory clinician sign-off. Whether that workflow saves time after checking and editing remains unproven here; promising text quality alone does not establish operational value.
Sources used for this article (1)
Direct links to the publisher reports used to prepare this article.
- Source 1
- Hepatology e-consult responses generated by artificial intelligence demonstrate accuracy but require human oversight Holly Huang, Debra W. Yen, Michelle Y. Li …
