On Aug. 26, Dr. David Kirk, CMO of Regard, joined Dr. Brian Lichtenstein, CMIO at Sharp HealthCare, and Dr. Renee Allenbaugh, Regional CMO at Penn Highlands Healthcare, for a Scottsdale Institute webinar moderated by Dr. Harry Greenspun. The topic: what happens when AI stops helping a clinician think and starts thinking for them, and how a health system tells the difference before it costs a patient.

Key takeaways

1. Relief and surrender look identical on day one. The shift happens gradually, one accepted output at a time, with no single moment that marks the line being crossed.
2. The industry is watching the wrong failure. Hallucination gets the scrutiny. Omission is the more common risk in AI-generated clinical summaries.
3. Summarization invites passive acceptance. It restates a chart probabilistically, so there is nothing stable underneath for a clinician to challenge.
4. A deterministic tool makes challenge possible. It does not make challenge happen. Automated EKG readings have been reliable for decades, and clinicians still learned to defer to them anyway.
5. Keeping clinicians engaged is a design choice, not a training problem. It shows up in whether a tool surfaces evidence, whether findings require an explicit accept or reject, and whether anyone is watching what clinicians do with either.

Relief and surrender start out looking the same

Physicians can realistically review roughly 3% of a modern chart during a typical encounter. When an AI tool reads all of it in seconds, the first reaction is relief. Kirk put words to what that relief feels like: "Finally, I can more deeply engage in multidisciplinary rounds. I can go back to the bedside and spend time with my patient."

But that relief only holds up if it comes paired with vigilance, not instead of it. Lichtenstein likened it to driving with adaptive cruise control: "You cannot take your eyes off the wheel. You have to remain completely engaged." He called AI's pull "a siren song," especially in health systems already stretched thin on time, staffing, and margin, where a tool that promises relief gets adopted fast, and fast adoption without friction is exactly what lets automation bias take hold. A junior nurse's comment during a safety review stuck with him as the clearest sign of it – "If there was a problem, wouldn't the EMR have told me there was a problem?" That instinct, trusting the system to catch what a clinician misses, is where surrender actually starts.

The industry is watching the wrong failure

Most scrutiny on clinical AI has focused on hallucination, the fear that a tool invents something that was never in the chart. In practice, omission is the more common problem, and a missed diagnosis does more damage than a flagged one, because nothing prompts anyone to go looking for it.

The panel traced that risk back to a specific design choice, specifically summarization. A tool that restates a list of problems rather than evaluating it invites a clinician to accept a diagnosis because someone, or something, already wrote it down. Kirk gave a plain example, and shared, "If AI suggests acute kidney injury and you just say okay because that was on a problem list from two days ago, you really just took your hands off the wheel." He called it parroting, old content moved forward without anyone checking whether it still holds. A tool built to validate a diagnosis against the criteria that define it gives a clinician a reason to look twice instead of a reason to move on.

A case from Regard's own use made the stakes concrete. A patient was admitted with what looked like a cryptogenic stroke, a stroke with no clear cause. Regard found a mention of atrial fibrillation buried in an old EKG report that had never made it into the note. The stroke was not cryptogenic. It was embolic, caused by the AFib nobody had connected to it. That is the kind of finding sitting in the part of the chart a physician does not have time to read. It is why, Kirk said, he now hears clinicians frame the risk in reverse: "I believe it's unethical not to use AI because there's so much information that if I don't use it, I'm afraid I'm going to miss something."

Human judgment still has to do the work

The conversation turned to what responsible design actually looks like in practice, and Kirk offered a concrete example, saying that building "deterministic components where when AI brings a diagnosis forward, you can say this is absolutely there, you can query that information, query that logic." That gives a clinician something concrete to check. What it doesn't do is guarantee anyone checks it.

Allenbaugh watches for that gap on her own team, and the tell isn't technical, it's behavioral. Her advice was to, "take three to five seconds and ask yourself, do I just want to click this or does it generate a question?" The moment that worries her most isn't when a clinician disagrees with an AI finding. It's when they stop disagreeing at all.

Automated EKG interpretation is the cautionary example the panel kept returning to. It has been reliable and deterministic for decades, and clinicians still learned to defer to it wholesale. Allenbaugh said she still double-checks her own EKGs for exactly that reason. A reliable tool earns trust, and earned trust, left unmanaged, is what quietly turns into surrender.

Engagement is by design

"AI does not keep doctors from doing the wrong thing," Kirk said, framing where he sees responsibility actually sitting. A tool can create the conditions for good judgment, but it cannot supply the judgment itself. Allenbaugh puts that into practice by reviewing what her physicians accept, reject, and edit, and which pieces of supporting evidence they actually open.

Demand for these tools has flipped the usual pattern of resistance to new technology, and Lichtenstein said as much, stating,  "I don't think I've ever had physicians come up to us and say, can I have this new technology. This is much more of a you can pry it from my cold dead hands kind of conversation." Allenbaugh saw that demand play out the hard way, when she lost a hospitalist to another system, not over money or schedule. "He needed to be near his wife," she said, but the deciding factor between offers was an AI tool at the new health system: "I'm not willing to work without it."

That is the case for building tools clinicians want badly enough to leave for. It's also the reason the design underneath has to hold up once they do.