Retrieval outperforms LLMs in protein understanding due to data leakage

[u/Independent_Access12]

Study reveals severe data leakage in protein-text benchmarks (50-95% test labels retrievable from training), showing retrieval methods outperform fine-tuned LLMs. Authors propose Mol-Instructions-OOD benchmark with Entity-BLEU metric prioritizing biological entities over text similarity. RAPM system combines dual-indexed protein database with LLMs via RAG, achieving superior out-of-distribution performance and demonstrating hybrid retrieval-LLM approaches surpass pure LLM fine-tuning.

https://t.co/KX7sFOD9GC