Health and Human Services Officials Visit Duke, Discuss Responsible Health Care AI

Duke Health hosted leaders from the U.S. Department of Health and Human Services, Microsoft, and UNC-Chapel Hill on May 23 to talk about next steps in making trustworthy health care artificial intelligence (AI) a reality.  

Introducing a panel discussion that he moderated, Michael Pencina, PhD, chief data scientist for Duke Health, said that AI poses a lot of exciting opportunities but also a lot of unknowns. “The question is — do I want the future my children will live in?”  he said. 

Panelists pointed out the potential for AI to transform health care by freeing up providers to do the most high-value work. They also discussed how to reduce risks such loss of jobs, privacy risks, and introducing unintended biases in algorithms used in clinical care. 

Panelists included: 

  • Andrea Palm, Deputy Secretary of the U.S. Department of Health and Human Services (HHS) 
  • Micky Tripathi, PhD, National Coordinator for Health Information Technology at HHS 
  • S. David McSwain, MD, MPH, System Chief Medical Informatics Officer, UNC Health, and Professor of Pediatric Critical Care Medicine, University of North Carolina School of Medicine 
  • Nicoleta Economou-Zavlanos, PhD, Director of AI Evaluation & Governance, Duke Health 
  • David Rhew, MD, Global Chief Medical Officer and Vice President of Healthcare, Microsoft.  

Rhew said that AI tools that already exist could solve a huge problem in health care:  the unsustainable rise in cost. By summarizing and compiling clinical visits, for example, AI tools can free up providers and improve access to care.

“Now is the chance to think about how to apply AI to reduce administrative burden and streamline care,” he said. “We need infrastructure and mechanisms to do that, and we need to put in place guardrails so we’re not having unintended consequences.”  

panel participants
From left: Micky Tripathi, PhD; Michael Pencina, PhD; Mary E. Klotman, MD; Andrea Palm; S. David McSwain, MD, MPH; David Rhew, MD; Nicoleta Economou-Zavlanos, PhD.

Economou-Zavlanos pointed out that mitigating discrimination and bias when using AI tools requires health systems to have a good grasp of the clinical AI algorithms that are in use and who is responsible for evaluating them. Duke’s Algorithm-Based Clinical Decision Support (ABCDS) Oversight framework does that with a registration process, she said. 

Panelists also pointed out that while large health systems such as Duke and UNC have already begun efforts to evaluate AI tools, smaller health systems don’t have the same resources and will need help. “That’s where assurance labs and third-party approaches can be helpful, to offload that burden from those smaller health systems and provide some consistency across multiple health systems,” said McSwain.   

Tripathi agreed that evaluating AI tools will require public-private partnership. “We need to find a way to do this together and make it so that everyone can take advantage of it,” he said, pointing out the importance of partnerships such as the Coalition for Health AI, of which Duke Health is a founding partner.  

Economou-Zavlanos and McSwain both touched on the need to demystify AI for patients and to listen to their feedback and concerns.  

“People with lower levels of health literacy and less trust in health care may opt out, leading to vastly different care for different populations,” McSwain said. “We need to pair opt outs with digital health literacy tools. We have to make sure the information we provide patients is understandable and actionable and accounts for their level of health literacy. If we don’t do that, we will exacerbate the digital divide.” 

While the federal government has issued broad guidelines about AI in health care, such as a final rule passed in April 2024, under Section 1557 of the Affordable Care Act, government and health systems will need to work together to implement such guidelines, Palm said. “We as an enterprise have more work to do certainly to put meat on the bones,” she said. 

Mary Klotman, MD, executive vice President for health affairs at Duke University and dean of the School of Medicine, opened and closed the event. She said she was reassured to see experts from government, academia, and industry focused on both the opportunities and risks of AI. “At the end of the day it is all about the people we serve, and that came across in all of your comments,” she said.