Inside the Loop, Issue 30 | Daniel C. Pettus | 9 November 2026
For two years running, the patient safety organization ECRI has put artificial intelligence at the top of its list of health technology hazards. In 2025 it was AI-enabled health technology in general [1]. In 2026 it was the misuse of AI chatbots, which ECRI reports have suggested wrong diagnoses and, in one case, invented body parts [2]. If your company’s AI strategy is a chat window bolted onto the side of a device, the safety people have already graded it.
This one is for the people on this list who build, sell, buy and certify the equipment in an ICU. Most of you are not engineers, and you do not need to be. You need three things: what AI can actually do at the bedside today, how to tell when it is producing slop, and what the companies that win the next five years will do differently. I spent the second half of my career connecting infusion pumps to the electronic record, so I will start with the part that stings.
The data we never connected
Twenty years after the first smart pumps were wired to the record, a 2026 expert panel in the American Journal of Health-System Pharmacy reported that only 9 to 15 percent of U.S. hospitals had implemented pump-EHR interoperability as of 2021. The same panel found no known AI algorithms implemented in smart pumps at all [3]. Even where integration is live, it is not always used. A 2026 study of more than 1.3 million infusions across six hospitals found auto-programming compliance of 53 percent [4]. Where it was used, the gains were real: about half the programming steps and half the hard-limit alerts.
For a business leader, that is the headline. AI reasons over data. The richest real-time medication data in the hospital comes off the pump, and in most hospitals it still stops at the pump. No model, however clever, can reason over a data stream nobody connected. It was always about the data.

What AI can actually do in critical care
Strip away the demos and the evidence for critical care AI is thinner, and more interesting, than the marketing. Two sepsis tools tell most of the story.
At Johns Hopkins, a sepsis early-warning system called TREWS monitored 590,736 patients across five hospitals. When a clinician confirmed an alert within three hours, adjusted mortality fell 3.3 percentage points, an 18.7 percent relative reduction [5]. The study was not randomized, and the benefit depended on a person acting quickly. Notice where the value came from: the model found the patient, and a human did something about it inside three hours.
At Michigan Medicine, researchers tested the Epic Sepsis Model, which they described as widely implemented, on 38,455 hospitalizations. It missed 67 percent of the patients who developed sepsis and fired on 18 percent of all hospitalizations [6]. Same disease, very different results. For anyone selling into an ICU, the lesson is simple: performance on someone else’s patients is a claim. Performance on mine is evidence.
And the pump? The expert panel’s priorities are sober. Before anything fancy, make sure only clinically meaningful alarms fire, keep drug libraries current, catch IV incompatibilities, and reconcile what the pump did with what the record says [3]. None of that is a chatbot. All of it is reasoning over device data.
One more thing about those two pump papers: both were sponsored by Baxter [3][4]. The findings may well be right. But when the evidence for a product category comes mostly from the companies that sell it, buyers notice. In October a clinical AI company graded nine popular AI models on drug dosing, found serious or potentially lethal errors in about 2 of every 100 answers, and then reported 99.9 percent accuracy for its own product [7]. Everyone in this audience knows what an ICU nurse thinks of a vendor’s own scorecard.
Can you trust it? Ask for the receipts
Until last month, federal certification rules required AI built into certified health records to disclose basic facts about how it was built and tested. On 26 October, HHS finalized the removal of that requirement, over the objections of most of the people who commented, including the American Hospital Association [8]. In practice, the burden of vetting AI just moved onto your customers. Every health system now has to answer for itself: what was this trained on, how does it perform on our patients, who watches it, and who can turn it off.
That is a threat if you sell AI on promises. It is the opportunity of the decade if you sell it with receipts.
AI slop, and how to get rid of it
Slop is the word that stuck for AI output that is fluent, confident and unchecked. In an ICU it shows up as auto-generated summaries nobody reads closely, inbox drafts that sound right, and alerts that fire so often they become furniture. Think of the confident intern who never sleeps and never checks his work. Five rules remove most of it.
Receipts on every output. Every recommendation names its sources: which pump channel, which order, which lab, what time.
Say what it cannot see. No exam, no conversation with the family, a missing weight. A tool that cannot state its blind spots has no business near a patient.
Structure, not prose. A rate, a reason and a confidence beat three paragraphs of eloquence. Prose is where slop hides.
Quiet when normal. Pilots call it the dark cockpit: instruments stay silent until something is wrong. An AI that comments on every patient is just another alarm.
A named human with time, information and authority. Not a nurse clicking accept forty times an hour. Someone who can actually say no, and whose no is recorded.
My test of Yahoo’s new AI answer engine makes the same point from the consumer side: ten medication questions, ten correct answers, and almost none of the sources a pharmacist would accept [9]. Right is not enough when you cannot see why.
Six moves for the companies at the bedside
Name the decision before the model. Which bedside decision gets better, and which clinician owns it? If you cannot say it in one sentence, you have a feature, not a product.
Read before you write. Earn trust in order: AI that reads the pump and the record, then AI that recommends with receipts, then, much later, AI that acts within hard limits with a stop button. Skipping steps is how a company ends up explaining itself to the FDA.
Plan the regulatory path on day one. AI that changes therapy is a device function. The FDA’s final guidance on predetermined change control plans, issued in December 2024, lets a manufacturer get its model update plan authorized up front [10]. The companies that file early will update faster. Everyone on this list remembers what a forced resubmission costs: after the FDA required a new 510(k) for changes to Alaris in 2020, the system was largely off the market until clearance in July 2023 [11].
Put validation in the box. Your customers now carry the vetting burden. Ship the tools to test the model on their own patients and to watch it for drift, and you become the easy yes in procurement.
Design the dark cockpit. Fewer, better alerts. Treat alarm burden as a product metric, not a complaint.
Measure outcomes, not logins. Meaningful Use measured adoption and taught an industry to count clicks. Count harms avoided, minutes returned to nurses and infusions reconciled.
Save us or kill us?
Neither. Infusion pumps did not save us or kill us either. They made a narrow set of errors harder to make, in the hospitals that used them well. AI will follow the same pattern, faster and with more at stake. It will help where someone owns the decision, checks the receipts and can turn it off. It will hurt where it is bolted on, unchecked and unowned.
A disclosure, since I am asking everyone else for receipts: I have patent applications pending on AI-assisted medication oversight, and the idea of reasoning over device data is close to that work. Judge the argument on its evidence. It is all in the notes.
I have proposed a federal framework built on this same ladder, the Meaningful AI Act, at danielpettus.com/bill [12]. In January I plan to run a live session for executives called Beyond the Chat; reply here if you want a seat. And tell me in the comments: which bedside decision would you trust AI to help with first?
About me
Daniel Pettus spent forty years in medical device and health IT leadership at Alaris, CareFusion and BD, was a founding member of the iMetrikus team, and contributed to IHE Patient Care Device interoperability standards. He is co-inventor on US patent 7,375,647 and has filed five provisional applications on the reasoning method behind Ai MedAgent™. He writes Inside the Loop from Murrieta, California.
The serious version of the day job is a book. The Technology Was Never the Problem: forty years of trying to connect medical devices to the record, and what finally changed. pettusbook.com.
Ai MedAgent™ is a research demonstration of a patent-pending method. Advisory only. Not a medical device. Not FDA cleared. Synthetic patients and simulated feeds. See it at aimedagent.net. Ami is an unreleased experiment and is not available to download.
Notes
[1] ECRI. Top 10 Health Technology Hazards for 2025: #1, risks with AI-enabled health technologies. Reported by Becker’s Hospital Review, 4 December 2024. https://beckershospitalreview.com/healthcare-information-technology/top-health-tech-hazards-of-2025.html
[2] ECRI. Misuse of AI chatbots tops annual list of health technology hazards, 21 January 2026. https://home.ecri.org/blogs/ecri-news/misuse-of-ai-chatbots-tops-annual-list-of-health-technology-hazards
[3] Mamdani M, Walker K, Nelson SD, Willner M, Sowan A. Theoretical applications of artificial intelligence in smart infusion pump technology: expert panel insights. Am J Health-Syst Pharm, published online 27 March 2026. doi:10.1093/ajhp/zxag085. Panel sponsored by Baxter Healthcare; authors received honoraria.
[4] Afaq H, et al. Integration of smart infusion pumps with electronic medical records improves safety and staff productivity at a large academic healthcare system. Am J Health-Syst Pharm, advance article 28 June 2026. doi:10.1093/ajhp/zxag187. 1,333,056 infusions, 6 hospitals; auto-programming compliance 53.4%; 1.6 vs 3.3 programming steps; hard-limit alerts down 50.3%. Authors include Baxter International.
[5] Adams R, et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nature Medicine, published online 21 July 2022. doi:10.1038/s41591-022-01894-0
[6] Wong A, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine, 2021. doi:10.1001/jamainternmed.2021.2626
[7] Jill Hughes. Popular AI models get medication doses wrong, some dangerously: Doximity. Healthcare Dive, 29 October 2026. https://www.healthcaredive.com/news/popular-ai-models-get-medication-doses-wrong-some-dangerously-doximity/873207/
[8] HHS. HHS finalizes HTI-5 deregulatory rule, 26 October 2026, https://www.hhs.gov/press-room/hhs-finalizes-hti-5-deregulatory-rule.html. American Hospital Association comments, 27 February 2026, via Healthcare IT News, https://www.healthcareitnews.com/news/aha-and-ehra-support-proposed-hti-5-deregulation-air-concerns. Reed Smith, 30 October 2026: removal finalized although a “large majority of commenters” opposed it.
[9] Daniel C. Pettus. Right Answers, Wrong Receipts. Inside the Loop, Issue 29, November 2026.
[10] FDA. Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions, final guidance, 4 December 2024. Summary: Ropes & Gray, https://ropesgray.com/en/insights/alerts/2024/12/fda-finalizes-guidance-on-predetermined-change-control-plans-for-ai-enabled-device
[11] Susan Kelly. MedTech Dive, 24 July 2023: FDA clears BD Alaris after the agency required a new 510(k) in 2020; the device had been largely off the market. https://www.medtechdive.com/news/BDX-BD-Alaris-infusion-pump-FDA/688720/
[12] Daniel C. Pettus. The Meaningful AI Act, a citizen’s discussion draft, Rev 3, 27 October 2026. https://danielpettus.com/bill



