FDA AI tool Elsa raises safety concerns after producing fake medical research during drug approval testing. Experts warn that such errors could threaten patient safety and public trust in healthcare AI.
Why experts are concerned
Brooke Hartley Moy, CEO of AI fact-checking platform Infactory, said the incident is unsurprising. She noted that generative AI often hallucinates, oversimplifies nuance, and produces confident but inaccurate statements.
What happened with Elsa
Elsa, designed to speed up drug approvals, allegedly fabricated non-existent medical studies and misrepresented research. According to FDA sources, the tool’s recommendations were unreliable due to frequent factual errors. One insider warned that unverified AI output can be dangerously misleading when used in critical processes.
Risks in healthcare
Moy emphasized that in healthcare, even small errors can have life-threatening consequences. She explained that AI can process large volumes of data quickly but must be used only to support, not replace, human decision-making.
Misunderstanding large language models
Moy said that many organizations misunderstand the limits of large language models. She stressed that these models are poorly suited to tasks requiring high precision, accuracy, and trust. This misconception, she argued, contributed to the FDA’s overreliance on Elsa.
The need for safety mechanisms
She urged the adoption of stricter verification processes and more deterministic pathways in AI-assisted healthcare tools. Moy added that factual accuracy must always take priority over speed.
Conclusion
FDA AI tool Elsa raises safety concerns that underline the limits of generative AI in critical sectors. Human oversight, rigorous testing, and clear safety mechanisms remain essential to prevent dangerous mistakes and maintain public trust.
0 responses to “FDA AI tool Elsa raises safety concerns”