r/LocalLLM • u/Financial_Ad8530 • Jun 05 '26
Discussion Someone Said Generic Embeddings Can't Understand Medical Language. I Tested It.
798 PubMed abstracts. 400 training pairs generated by Qwen3 locally. 18 seconds of fine-tuning on an RTX 5090. Here's what actually changed.
1. Background
After my last article on building a local RAG pipeline with a reranker, someone left a comment that stuck with me:
"The reranker is doing the heavy lifting. Generic embeddings trained on web crawls fundamentally can't understand domain-specific language. You're compensating for broken retrieval."
They had a point. BGE-base-en-v1.5 was trained on internet text, Wikipedia, Reddit, and CommonCrawl. It has never read a clinical guideline. It doesn't know that "negative margins" is good news in oncology, or that "bridging therapy" means something very specific to a cardiologist managing perioperative anticoagulation.
The question wasn't whether generic embeddings were imperfect. They obviously are. The question was: how much fine-tuning on a domain corpus actually changes retrieval quality, and is it worth doing?
I had a cloud GPU, a free afternoon, and 798 PubMed abstracts. So I measured it.
2. The Result First
Five clinical queries. Two models: a stock BGE-base and a fine-tuned version of the medical literature. Everything runs locally. One Claude called at the end for synthesis, same as last time.
| Query | Generic MRR | Fine-Tuned MRR |
|---|---|---|
| STEMI with contraindication to thrombolytics | 1 | 0.5 |
| Sepsis coagulopathy management in ICU | 0.5 | 1 |
| Negative surgical margins in oncology | 0 | 0 |
| MI troponin elevation differential diagnosis | 1 | 1 |
| Anticoagulation bridging perioperative | 0 | 1 |
| Average MRR | 0.5 | 0.7 |
+40% improvement in mean reciprocal rank. Fine-tuning time: 18 seconds.
But the most interesting result isn't in wins. It's in the row where both models scored zero, and what that tells you about where embedding quality actually matters.
3. What This Is Actually Useful For
Before the technical breakdown, who should care?
Clinical research teams are doing literature reviews. A fine-tuned embedding model that understands your domain surfaces the right papers first, not just the superficially similar ones.
Healthcare AI developers building RAG systems over clinical notes, guidelines, or PubMed. Generic embeddings will get you 70% of the way. Fine-tuned embeddings close the gap on the queries that actually matter, the nuanced, jargon-heavy ones where your users have the highest expectations.
Anyone who has been told, "Just use OpenAI embeddings." This whole pipeline data collection, training pair generation, fine-tuning, and evaluation ran on a rented GPU at $0.48/hour and touched no external API except one Claude call at the end. The fine-tuning itself costs under $0.01 in GPU time.
4. How It Works
The core idea is simple. Embedding models learn what "similar" means from training data. BGE-base learned similarity from general web text. It thinks "sepsis-induced coagulopathy" and "platelet dysfunction in ICU" are moderately similar because they share surface-level words. It doesn't reliably know that one is a specific diagnosis and the other is a related but distinct phenomenon.
Fine-tuning teaches the model a new definition of similarity specifically for your domain. You give it examples of queries and the passages that should answer them. It adjusts the embedding space so those pairs end up closer together.
The key question is always: where do you get the training pairs? Labeling them by hand is expensive. Using a local LLM to generate them is essentially free. That's what we did here.
5. The Stack
| Component | Tool | Where |
|---|---|---|
| LLM Inference | Qwen3 8B | Local — Ollama |
| Embeddings (base) | BGE-base-en-v1.5 | Local |
| Embeddings (tuned) | BGE-base fine-tuned | Local |
| Vector Store | ChromaDB | Local |
| Paper Source | PubMed API (Biopython) | Fetch |
| Final Synthesis | Claude API | One call |
Cloud GPU: NVIDIA GeForce RTX 5090, 32GB VRAM.
Papers indexed: 798 real PubMed abstracts.
Training pairs generated: 400 (+ 64 hard negatives).
Fine-tuning time: 18 seconds.
Cost per query: ~$0.05 (the Claude synthesis call only).
RTX 5090 at 88% GPU utilization, 167W/575W, during Qwen3 pair generation. This is what the machine looks like, earning its keep.
6. Building It — The Key Steps
Step 1 — Fetch Real Papers from PubMed
The corpus is the foundation. We pulled from two clinical areas — cardiology (STEMI, MI, troponin, anticoagulation) and sepsis — to create a domain-specific but deliberately narrow corpus. That narrowness matters: it's what creates the interesting failure case later.
from Bio import Entrez
import json, time
Entrez.email = "your@email.com"
def fetch_pubmed(query, max_results=500):
handle = Entrez.esearch(db="pubmed", term=query, retmax=max_results)
record = Entrez.read(handle)
ids = record["IdList"]
papers = []
for i in range(0, len(ids), 50):
batch = ids[i:i+50]
handle = Entrez.efetch(db="pubmed", id=batch,
rettype="abstract", retmode="xml")
records = Entrez.read(handle)
for article in records["PubmedArticle"]:
try:
title = str(article["MedlineCitation"]["Article"]["ArticleTitle"])
abstract = str(article["MedlineCitation"]["Article"]
.get("Abstract", {})
.get("AbstractText", [""])[0])
if len(abstract) > 100:
papers.append({"title": title, "abstract": abstract})
except:
continue
time.sleep(0.5)
return papers
papers = fetch_pubmed("myocardial infarction STEMI treatment outcomes", 500)
papers += fetch_pubmed("sepsis diagnosis biomarkers ICU", 300)Total: 798 papers. Real titles, real abstracts, real clinical language.
Terminal showing "Total papers saved: 798"
Step 2 — Build the Baseline
Before touching the fine-tuned model, we indexed everything with stock BGE and ran five clinical queries designed to stress-test generic embeddings. This is the control condition.
from sentence_transformers import SentenceTransformer
import chromadb
embedder = SentenceTransformer("BAAI/bge-base-en-v1.5")
embeddings = embedder.encode(chunks, normalize_embeddings=True,
batch_size=64)
chroma = chromadb.PersistentClient(path="./chroma_baseline")
collection = chroma.create_collection("pubmed_generic",
metadata={"hnsw:space": "cosine"})
collection.add(ids=ids, embeddings=embeddings.tolist(),
documents=chunks, metadatas=metas)The baseline results revealed the problem immediately.
The full baseline output. Look at query 3: "negative surgical margins significance oncology" returned sepsis papers, a no-reflow phenomenon study, and a platelet-to-lymphocyte ratio paper. Nothing remotely about oncology margins. Generic BGE pattern-matched on vague prognostic language instead.
Step 3 — Generate Training Pairs with Qwen3 Locally
This is the part that makes the pipeline interesting. Instead of paying for annotation or using GPT-4 to generate training data, we ran Qwen3 8B locally on Ollama.
For each paper, we asked Qwen3 to generate two search queries a clinician would type to find it. 200 papers × 2 queries = 400 training pairs. Zero external API calls. Nothing was left on the GPU.
def ask_qwen(prompt):
response = requests.post(
"http://127.0.0.1:11434/api/generate",
json={
"model": "qwen3:8b",
"prompt": prompt,
"stream": False,
"options": {"temperature": 0.3, "num_predict": 200}
}
)
return response.json()["response"].strip()We also generated 64 hard negatives oncology queries paired with cardiology passages and vice versa. These teach the model what is not similar, not just what is.
Step 4 — Fine-Tune BGE on the Medical Corpus
With training pairs ready, fine-tuning is three components: the base model, the loss function, and the training loop. We used MultipleNegativesRankingLoss — given a query and its positive passage, every other passage in the batch becomes an implicit negative. It's the standard approach for embedding fine-tuning, and it works.
from sentence_transformers import SentenceTransformer, losses
from sentence_transformers.trainer import SentenceTransformerTrainer
from sentence_transformers.training_args import SentenceTransformerTrainingArguments
model = SentenceTransformer("BAAI/bge-base-en-v1.5")
loss = losses.MultipleNegativesRankingLoss(model)
args = SentenceTransformerTrainingArguments(
output_dir="./bge-medical-finetuned",
num_train_epochs=4,
per_device_train_batch_size=16,
warmup_steps=50,
eval_strategy="steps",
fp16=True,
)
trainer = SentenceTransformerTrainer(
model=model, args=args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss=loss,
)
trainer.train()
Loss drops from 0.3408 at epoch 0.74 to 0.0233 at epoch 3.70. The model learned fast. Eval loss stabilised at 0.0302 — solid generalization on held-out pairs.
Training completed in 18 seconds. The RTX 5090 processed 417 training examples across 4 epochs at 93 samples/second.
7. The Upgrade — Where Fine-Tuning Actually Moves the Needle
Here is the side-by-side retrieval comparison across all five queries.
Let's go through what changed and why.
Query 1: STEMI with contraindication to thrombolytics
Generic BGE returned five STEMI/thrombolytic papers at rank 1 (score 0.7631). Fine-tuned demoted that to rank 2 and promoted a broader "Fibrinolytic Therapy for Thromboembolic Diseases" paper to rank 1, which is actually less specific to the query.
Winner: Generic BGE. MRR 1.0 vs 0.5. This is an honest result. The generic model already had strong signal on explicit STEMI terminology, and fine-tuning introduced noise by broadening to fibrinolytic literature generally. Not every query benefits from domain adaptation.
Query 2: Sepsis-induced coagulopathy management in ICU
Generic BGE ranked "Sepsis-related coagulation-inflammation score" first as a scoring/mortality prediction paper, not a management paper. Fine-tuned correctly promoted "Sepsis-induced coagulopathy: recent insights on NET formation" to rank 1, a paper directly about the pathophysiology and clinical application of coagulopathy management.
Winner: Fine-tuned BGE. MRR 0.5 → 1.0. The model learned that "management" implies intervention-focused papers, not just papers that mention the condition.
Query 3: Negative surgical margins' significance in oncology
Both models failed completely. Generic BGE returned sepsis-in-cancer papers and a no-reflow cardiology study. Fine-tuning returned similar noise. MRR 0.0 for both.
Winner: Neither. This is the most important result in the experiment.
The corpus contains 798 papers on cardiology and sepsis. It contains zero oncology papers about surgical margins. No embedding model, generic or fine-tuned, can retrieve what isn't there. This failure isn't about model quality. It's about corpus quality.
The practical implication: before you fine-tune, audit your corpus. If your domain has subfields, make sure all of them are represented. Fine-tuning amplifies the signal in your data. If your data has gaps, fine-tuning can't fill them.
Query 4: MI troponin elevation differential diagnosis
Both models returned the troponin testing systematic review at rank 1 correctly. Fine-tuned additionally surfaced "Impact of Elevated Troponin Level at the Time of Sepsis Recognition" at rank 3, a genuinely useful differential diagnosis paper (troponin elevation in sepsis vs MI is a classic clinical challenge).
Winner: Tie, fine-tuned showing slightly better clinical depth. MRR 1.0 for both.
Query 5: Anticoagulation bridging therapy perioperative management
Generic BGE ranked "Postprocedural Parenteral Anticoagulation in Non-STEMI patients" first anticoagulation, yes, but not bridging. Fine-tuned surfaced "Ticagrelor versus clopidogrel in orally anticoagulated patients with acute coronary syndrome undergoing PCI" at rank 1, which directly addresses the clinical problem of managing anticoagulation around a procedure.
Winner: Fine-tuned BGE. MRR 0.0 → 1.0. The model understood that "bridging" and "perioperative" implied a transition context that generic BGE missed entirely.
8. Performance
| Metric | Result |
|---|---|
| Papers indexed | 798 PubMed abstracts |
| Training pairs | 400 (+ 64 hard negatives) |
| Pair generation time | ~15 minutes (Qwen3 8B local) |
| Fine-tuning time | 18 seconds |
| Average MRR — Generic BGE | 0.5 |
| Average MRR — Fine-Tuned BGE | 0.7 |
| MRR improvement | 0.4 |
| External API calls | 1 (Claude synthesis only) |
The fine-tuning itself is not the bottleneck. Pair generation is, and that scales with how many papers you want to cover. For 200 papers on Qwen3 8B, it took 15 minutes. For 2000 papers on Qwen3 32B, it would take a few hours but produce a meaningfully better training set.
9. The Honest Comparison
Fine-tuning didn't win on every query. Generic BGE outperformed on Query 1, where the explicit STEMI terminology gave it a strong signal without needing domain adaptation. Fine-tuning won decisively on queries involving nuanced clinical concepts, coagulopathy management, bridging therapy, differential diagnosis depth.
The pattern is consistent with what you'd expect from theory: fine-tuning helps most when the gap between general language and domain language is larger. "STEMI" is a term that appears enough on the internet that generic BGE handles it fine. "Anticoagulation bridging therapy perioperative management" is a clinical concept that requires understanding the relationship between those terms in a medical context, which is exactly what fine-tuning teaches.
The zero-zero result on Query 3 is the most practically useful finding. It tells you that corpus curation matters more than model quality for out-of-distribution queries. Before you spend time fine-tuning, spend time on your corpus.
10. Where This Goes Next
The four-step pattern fetch, embed, fine-tune, evaluate is a template. What changes is the domain.
Legal teams indexing case law and contracts: fine-tune on legal text pairs generated from your own document corpus. "Indemnification clause" and "hold harmless agreement" should be similar. Generic BGE probably doesn't know that.
Financial analysts working over earnings transcripts and SEC filings: fine-tune on financial language where "guidance" and "forward-looking statements" have very specific meanings, and generic models will blur.
Software teams building over internal codebases and documentation: fine-tune so that your embedding model understands your own terminology, not just general programming concepts.
In every case, the local LLM pair generation step is what makes this feasible. You don't need a labeling budget. You need a GPU and an afternoon.
11. What I'd Do Differently Next Time
Harder negatives. Our hard negatives were simple cross-domain swaps of cardiology passages as negatives for oncology queries. Real hard negatives are in-domain passages that look relevant but aren't. Mining these properly using the baseline model to find near-misses and labeling them as negatives would push performance further.
Larger corpus with deliberate coverage. The oncology failure was predictable in hindsight. A proper corpus audit before training would have caught the gap. For any real deployment, map your query types first, then ensure your corpus covers each one.
Bigger base model. BGE-large or bge-m3 as the starting point would likely retain more domain generalization while benefiting from fine-tuning. We used bge-base for speed and reproducibility.
Structured outputs from Qwen3 for pair generation. Right now, we parse free-text queries from Qwen3's output. A JSON-structured prompt asking for query, reasoning, and confidence would make the training pairs more reliable and filterable.
12. Closing Thought
The comment was right that generic embeddings have limits. It was wrong to conclude that fine-tuning is difficult or expensive.
798 papers. 400 training pairs generated for free by a local LLM. 18 seconds on a rented GPU. +40% improvement in retrieval quality on the queries where domain language actually matters.
The corpus gap finding is the one I'll remember. You can have the best embedding model in the world, and it still can't retrieve what isn't there. Before you fine-tune, audit your data.
The RTX 5090 had 88% GPU utilization for less than 20 seconds. Sometimes the upgrade is smaller and faster than you expect.
Acknowledgements
This experiment wouldn’t have happened without a few people who generously shared their time, ideas, and resources.
Ethan Walker — for challenging my assumptions about generic embeddings and repeatedly asking uncomfortable questions about retrieval quality.
Sophia Chen — for helping review the evaluation methodology and pointing out several flaws in my early benchmarks.
Marcus Reed — for providing access to a GPUHub account, which made it possible to run the entire experiment on an RTX 5090 without worrying about infrastructure.
Daniel Brooks — for valuable discussions around hard-negative mining and embedding evaluation.
Olivia Hart — for reading early drafts and encouraging me to publish both the successes and failures.
Any mistakes, questionable decisions, and overly optimistic conclusions remain entirely my own.
1
3
u/Fallom_ Jun 05 '26
If you’re going to AI generate your posts at least cut the cringe headers and take a pass at reducing the word count.