Epic Sepsis prediction model underperforms in deployment
A widely deployed clinical sepsis-prediction model was shown to miss most sepsis cases and generate frequent false alerts.
A 2021 JAMA Internal Medicine study evaluated Epic's Sepsis Prediction Model across roughly 28,000 patients at the University of Michigan and found it detected only 7% of sepsis cases that clinicians missed, with frequent false alarms contributing to clinician alert fatigue. The case is the canonical example of an AI clinical decision-support system that performed well on vendor benchmarks but degraded in real-world deployment, and is now cited in nearly every healthcare AI impact-assessment guideline.
What caused this
The Epic Sepsis Model, deployed at hundreds of US hospitals, was shown by independent researchers (JAMA Internal Medicine, 2021) to perform far worse than vendor claims — missing two-thirds of sepsis cases and firing thousands of false alarms, contributing to alert fatigue.
- Model trained and validated on one health system's data but generalised across very different patient populations.
- Performance claims were based on vendor-internal metrics that hospitals could not reproduce.
- No local recalibration or silent-mode evaluation before going live.
- Continuous monitoring of clinical impact was not contractually required.
How this could have been avoided
Clinical AI must be governed as a medical device, not a software upgrade.
- Local validation on the deploying hospital's own patient population before clinical use (ISO 42001 Annex A.6.2.5 verification and validation).
- Silent deployment phase — log predictions without acting on them, compare to ground truth, calibrate.
- Performance contracts with the vendor including independently reproducible metrics and right to audit.
- Continuous post-market monitoring for drift, alert burden and patient outcomes; predefined retirement criteria.
- Clinician override and feedback loop wired into the EHR, with regular case review by the AI governance committee.
Need AI assurance for your organisation?
Talk to AramGRC about ISO 42001 readiness, governance reviews, and incident response playbooks tailored to your sector.
Contact AramGRC for AI assurance →