Experts caution against ‘snake oil’ claims in AI research

Many big tech firms are over-promising and under-delivering on AI, warned health data experts at Digital Health Rewired.

Speaking at the NEC Birmingham on 24 March, Dr Jessica Morley, postdoctoral researcher at Yale University Digital Ethics Center, said that many of the AI sector’s “big, hypey promises” are not deliverable, and urged healthcare providers to invest in people and infrastructure to deliver the best outcomes.

“Why are we giving our money to big private tech companies? And should we be building more in-house?

“I think NHS trusts are facing an enormous amount of pressure to adopt AI, but nobody really knows how to evaluate whether AI is the right solution.

“Smaller, more basic things are probably more useful than trying to buy some whizzy AI.

“It’s about feeling confident enough to say ‘actually, I think this is not the right solution for my particular hospital’,” Morley told Digital Health News.

Dr Joe Zhang, head of data science at the London AI Centre and Secure Data Environment, compared some of the AI used in health research to “snake oil” and said that despite all the hype they have yet to provide a real impact.

He highlighted that some AI used in research may have only been validated by the data it was built on, not been tested outside of a research setting, and is evaluated by the team that built it.

“We’re not getting the basics done right, whether that’s on a people and social level, for what we want to achieve,” Zhang said.

He also questioned whether the potential value of NHS data was being used effectively by tech firms.

“The value from the NHS flows into them.

“We can do a lot more to do basic things right on an infrastructure level, make the real value of data go up, and capture the value of that downstream by innovating to solve our own problems by partnering with tech, but not giving it to tech,” he added.

Morley and Zhang told the Rewired audience that some AI firms are claiming that their technology can provide increased efficiency, cost savings, improved patient outcomes, reduced inequities, and enhanced clinician wellbeing.

However, in slides shared at the conference, they said that many of these benefits are presented as an “automatic consequence of capability”, and the benefits as “immediately realisable regardless of the capabilities required”.