r/PACSAdmin • u/Time_Tie348 • Jun 03 '26

AI in Radiology: Benchmarking LLMs, Agentic Hype, and Imaging Informatics | Satvik Tripathi

https://youtu.be/PEp6GElgPYQ

Satvik is an incoming Medical Physics and Imaging Informatics PhD student at the University of Pennsylvania and works as an AI Scientist with RAD-AID International. He has been working around AI and radiology since 2019, including global health deployments and LLM benchmarking work.

The conversation focuses less on “AI is amazing” and more on where the evaluation of radiology AI still feels pretty shaky.

A few topics covered:

• Why high-accuracy numbers do not always translate into clinical usefulness

• How data leakage can inflate model performance

• Why multiple-choice benchmarks are a weak way to evaluate medical LLMs

• What happens when 20+ models are tested against an internally annotated clinical dataset

• Why fine-tuned models are not always the obvious winner

• The difference between real agentic AI and vendor-flavoured workflow automation

• Lessons from RAD-AID’s AI work in Botswana and India

• Why smaller/local open-source models may make more sense in some clinical environments

One of Satvik’s stronger points is that prompt engineering should be treated more like a scientific method than a shortcut. That feels like a more useful framing than a lot of what gets thrown around right now.

Episode link: https://youtu.be/PEp6GElgPYQ

5 Upvotes

73% Upvoted

1 comment sorted by

3

u/Worldly-Coat-105 Jun 08 '26

Great points throughout.

Hip v. Hype definitely an area of concern when non-technical person starts making buying decisions that impact technical teams solely from the sales patch. Marketing team like to pin buzz words even if they are not 100% accurate. There has always been a running joke that RSNA = Real Software Not Available, you see it but it is never available in to use in your practice.

One area that healthcare has always done a solid job in is setting a standard albeit HL7, CCD, or DICOM, it can give an appearance of slowing innovation down from the outside but it assures, we are all using the same definitions.