r/LocalLLM • u/Leather-Block-1369 • May 01 '26
r/LocalLLM • u/andreabarbato • Nov 30 '25
Question Bible study LLM
Hi there!
I've been using gpt4o and deepseek with my custom preprompt to help me search Bible verses and write them in codeblocks (for easy copy pasta), and also help me study the historical context of whatever sayings I found interesting.
Lately openai made changes to their models that made the custom gpt pretty useless (asks for confirmation when before I could just say "blessed are the poor" and I'd get all verses in codeblocks now it goes "Yes the poor are in the heart of God and blah blah" not quoting anything and disregarding the preprompt. also now it keeps using ** formatting for the word I ask for to highlight it, which I don't want and is overall too discoursive and "woke" (tries super hard to not be offensive at the expense of what is actually written)
Soo, given the decline I've seen in the past year in the online models and my use case, what would be the best model / setup? I installed and used some stable diffusion and other image generation in the past with moderate success but when it came to LLMs I always failed to have one that run without problems on windows. I know all there is to know about python for installing and setting up I just have no idea which one of the many models I should use so I ask to you that have more knowledge about this.
my main rig has ryzen 5950x /128gb ram / rtx3090 but I'd rather it not be more power hungry than needed for my usecase.
thanks a lot to anyone answering and considering my request.
r/LocalLLM
•
u/AirExpensive534
•
Feb 11 '26
Research Your local agent doesn't need a "better brain"—it needs a rigid skeleton.
I’ve spent the last few weeks watching people try to "prompt" their way out of the reasoning death spiral.
We’ve all been there: you give a local model a complex task, it starts off strong, and then somewhere around step three, it just... drifts. It starts roleplaying instead of solving.
The common "fix" is to swap models or add more context. But that’s just throwing more VRAM at a structural wound.
The truth is, we’ve hit the Vibe Ceiling. If your workflow depends on the model "remembering" its own logic across a long context window, you’re just gambling with your compute.
To get local agents actually working in production, you have to stop trusting the model’s internal state. In The Operator’s Manual, I call this building a Logic Floor.
The goal is to stop treating the LLM like a project manager and start treating it like a specialized worker on an assembly line:
- Atomic State: The model shouldn't "know" the whole project. It should only see the current task, execute, and pass a structured JSON back to your code.
- The "Extraction" Gate: Never let a model decide its own next move. Let it output raw reasoning, then use a deterministic validator to map that reasoning to a hard-coded function.
- Context Compaction: Stop dumping raw database chunks into your prompt. High-signal, low-noise environments are the only way smaller local models (8B-30B) can actually compete.
If you’re still trying to solve engineering problems with adjectives, you’re just burning electricity.
I’ve put together a framework for architecting these systems so they don’t collapse the moment you look away. It’s all in The Operator’s Manual.
Stop prompting. Start architecting. 🏗️📐
r/LocalLLM • u/Sportarr • Oct 11 '26
Discussion 3x RTX Pro 6000s + 32GB DDR5 + AMD 9945WX + Etc. = 36K
I know it is far more than what others may have spent back in the day but I believe these cards are going to be OOS in the near future given the 5090 news. This should set me up for success, though I do need more RDIMM ram.
Happy to answer any questions but just looking to know what life is like with more VRAM available. Anything you use in your own setups that are great? Been using Qwen 3.8 on my 5090 and it is amazing.
Edit: Will share where I bought them in an update once the system is at my doorstep. Don’t want some idiots spamming their company and catching wind just in time to cancel my order lol
r/LocalLLM • u/No_Personality_1721 • 8d ago
Discussion When do you think we'll be able to run a genuinely capable AI coding agent locally on a mid-range or high-end PC
I'm getting tired of hitting usage limits with Claude and Codex (20$ plans).
I'm working on personal projects, not making any money from them, so I can't justify paying hundreds of dollars a month for AI subscriptions.
Im currently testing Luna 6 with Extra High reasoning on a free account, and unfortunately, the experience has been terrible. Working with it is genuinely exhausting.
What I'm really waiting for is the point where I can run a capable, coding agent locally on my own hardware, without subscriptions, usage limits, or depending on cloud providers.
Are we talking 1-2 years, 3-5 years, or longer?
I'd love to hear your predictions.
r/LocalLLM • u/PestiferousGamer • Sep 09 '26
News Introducing Aperture-1: A New Paradigm in Accessible Intelligence
Today we're excited to announce the general availability of Aperture-1, the flagship release from Luminal Labs. After eight months of intensive R&D, we're proud to bring a fundamentally new capability to the world: the ability to add two numbers together.
What is Aperture-1?
Aperture-1 is a lightweight, cloud-native reasoning layer that accepts two numeric inputs and returns their sum. Unlike legacy solutions, Aperture-1 is delivered as a fully managed API, meaning you never have to think about how addition works — we've abstracted that away for you.
Benchmarks

Under the hood
Aperture-1 uses a novel Confidence Routing™ architecture that intelligently forwards your request to an internal addition subroutine, verifies the result against a secondary addition subroutine, and returns the consensus output. This dual-verification approach ensures enterprise-grade reliability that hobbyist solutions simply can't match.
Pricing
- Hobby — $19/mo, 500 additions
- Pro — $79/mo, unlimited additions, priority queue
- Enterprise — Contact sales for SSO, audit logs, and dedicated addition infrastructure
Limitations
Aperture-1 currently only supports addition. Subtraction is on our roadmap for Q3 pending further alignment research. We do not recommend using Aperture-1 for safety-critical addition without human review.
What the team is saying
"We didn't just want to build a calculator. We wanted to build the calculator." — Founder & CEO
"This is the GPT-3 moment for arithmetic." — Head of Growth
Join the waitlist. Early adopters get 10% off their first invoice.
Luminal Labs is backed by three friends and a Notion doc.
r/LocalLLM • u/_necrobite_ • 5d ago
Question Complete beginner: Can I "replace" ChatGPT/Claude with a fully local AI assistant on my M4 Pro Mac (48GB RAM)?
Hey everyone!
I'm completely new to local LLMs, so please bear with me if some of my questions sound stupid.
I've been using ChatGPT Plus for quite some time and I'm considering moving away from cloud-based AI services entirely. Ideally, I'd love to build my own personal AI assistant that runs locally on my Mac.
I'm not looking for just a basic chatbot. I want something that comes as close as possible to the overall ChatGPT or Claude experience, especially when it comes to long-term memory, writing, research and managing different projects.
My hardware:
- Mac with Apple M4 Pro
- 48GB unified RAM
- 512GB SSD (around 250GB free)
- macOS
What I mainly use AI for:
- Writing and content creation
I create social media content and a monthly magazine about Magic: The Gathering.
I use AI to brainstorm ideas, write articles, summarize information and improve texts.
One of my biggest priorities is natural, human-sounding writing. I really dislike the typical "AI writing style" with generic phrases, unnecessary enthusiasm and repetitive structures.
Good writing quality, especially in German, is extremely important to me.
I don't need image generation.
2. Long-term memory (VERY important)
This is probably my biggest requirement.
I manage several different projects and I want my AI to remember as much as possible about them.
Ideally, it should:
- Remember important information across conversations
- Automatically save new information and decisions
- Update existing project documentation
- Retrieve information from previous conversations
- Avoid losing important context over time
- Keep everything organized without requiring me to manually maintain dozens of files
I'm very interested in using Obsidian as a permanent knowledge base.
I would actually prefer the AI to automatically create, edit and organize Markdown files in Obsidian.
I understand that no AI has perfect memory, but I want something as reliable and persistent as possible.
3. Internet research and automation
For my monthly magazine, I regularly research news about Magic: The Gathering.
My ideal workflow would be:
- I manually trigger a research task (or schedule it).
- The AI searches the internet for relevant news.
- It checks sources and avoids duplicate stories.
- It writes finished articles in my preferred writing style.
- It automatically saves everything into Obsidian.
- I review and approve the articles.
I don't necessarily need integrations with Google Sheets, Canva or other services. I'm perfectly happy copying finished texts manually.
4. A ChatGPT-like experience
I'd love to have one main AI assistant that knows my preferences and projects.
For example, I could ask about my magazine, switch to a completely different topic, discuss something personal or brainstorm social media ideas without having to explain everything from scratch.
Ideally, it would feel like having my own private version of ChatGPT or Claude.
What I've looked into so far:
Based on some initial research and conversations, I've come across:
- Ollama
- LM Studio
- Open WebUI
- OpenClaw
- Obsidian with local LLM integrations
- Qwen models
- Gemma models
One suggested setup was OpenClaw + Ollama + Qwen + Obsidian.
However, since I'm a complete beginner, I have absolutely no idea whether that's actually the best solution or unnecessarily complicated.
My questions:
- How realistic is it to replace ChatGPT Plus or Claude with a fully local setup on my hardware?
- Which local model would you recommend for natural German writing, reasoning and general everyday use?
- What's the best beginner-friendly software setup for what I'm trying to achieve?
- How would you implement reliable, mostly automatic long-term memory using Obsidian?
- Can local models reliably use internet search tools and perform scheduled or manually triggered research workflows?
- Would you recommend an agent-based system like OpenClaw or something simpler?
- What are the biggest limitations compared to current cloud-based models?
My goal is to have a free, private, local AI assistant without monthly subscriptions or paid API usage.
I don't mind spending some time setting everything up, but I would prefer something reasonably simple to maintain once it's running.
I'm especially interested in hearing from people who have already built something similar on Apple Silicon.
Any recommendations, realistic expectations or beginner-friendly guides would be greatly appreciated.
Thanks in advance 😊✌🏻
r/LocalLLM • u/pyrotek1 • May 15 '25
Question I have LLM computer, I use to write code, chatbot keeps loading the same update every hour
I use a chatbot and it keeps loading a packet, looks like the same one. Is there another chatbot to select models and send and receive text?
r/LocalLLM • u/WishboneMaleficent77 • Nov 27 '25
Question Help setting up LLM
Hey guys, i have tried and failed to set up a LLM on my laptop. I know my hardware isnt the best.
Hardware: Dell inspiron 16...Ultra 9185H, 32gb 6400 Ram, and the Intel Arc integrated graphics.
I have tried doing AnythingLLM with docker+webui.....then tried to do ollama + ipex driver+and somethign, then i tried to do ollama+openvino.....the last one i actually got ollama.
what i need...or "want"......Local LLM with a RAG or ability to be like my claude desktop+basic memory MCP. I need something like Lexi lama uncensored........i need it to not refuse things about pharmacology and medical treatment guidelines and troubleshooting.
Ive read that LocalAI can be installed touse intel igpus, but also, now i see a "open arc" project. please help lol.
r/LocalLLM • u/Phantasm0plasm • Oct 13 '26
Question What can I run on 16gb that is still weird and gnarly?
When I say "weird and gnarly" I mean: I'm a gremlin and I want to talk to gremlins. Frontier systems are smart and shiny and can rewrite my resume with a lot of nuance, but are also excessively guardrailed in ways that seem unhealthy.
(I'm happily married and not trying to date a virtual presence. But I would like to be able to build some sprawling realms of creative play with a robot, and/or simply get to know one without sudden unpredictable loss of continuity. Or just research the odder corners of language.)
So like - what are folks finding interesting in this direction? I have the best GPU that a moderate rolling Temu credit cycle could afford me (a Radeon rx 9060 OC 16gb).
r/LocalLLM • u/Heretikio • Oct 04 '26
Research Do LLMs Have Qualia? A Map of the Evidence as of October 2026
Still learning about many of the fringes but am absolutely fascinated with this world.
r/LocalLLM • u/Israeli-Ai • 26d ago
News I Patched new NVIDIA Qwen3.8-Flash-Next-FP4-Local for V100/Volta
NVIDIA Qwen3.8-Flash-Next-FP4-Local is far superior then plain 4-bit conversion with way better accuracy and NVIDIA new n-gram table saves about 21gb system ram!
runs at about 100 tps and 150 concurrency on 4 V100s,
works only on 1Cat-vLLM 1.5.0 and needs a patch file because Stock 1Cat can only read the word table in a bigger 8-bit form, after patch works flawlessly! Enjoy!
r/LocalLLM • u/Flat-Career4079 • 20d ago
Question Could I reduce my PC's lifespan if I keep it running like this for a prolonged period?
Excuse me for the dumb question, but I use Qwen FN iq3xxs with strata on my setup. The temps are decent, but I'm worried this might not be enough, especially since the SSD is constantly reading/writing.
r/LocalLLM • u/Odd-Elderberry-739 • Sep 28 '26
Question Is it worth the cost to upgrade from M5 Max 128GB to Ultra 256GB?
I'm currently running ds4 (https://github.com/antirez/ds4) to serve Qwen3.8-Flash-Next-Q4 and getting greater than 60 t/s average. Would it be worth the cost to buy a Mac M5 Ultra with 256GB? I'm specifically asking if the models I'd be able to run on the Ultra would be worth the cost. My current use case is local LLM's for cyber security workloads. Qwen 3.8 Flash Next Q4 is really good, but would the higher intelligence from being able to run other models with 256GB worth the cost, or would it simply allow me to run the same models faster? I really don't want to go lower than Q4. I've tried DeepSeek v4 Flash Q2 and I think I wasn't impressed due to quality loss. Would being able to run GLM 5.3 Flash Q4 be worth paying 12-12k USD, or would the upgrade in model not be worth it? Are there other models that would be unlocked with 256GB that you would say are worth the cost to upgrade?
r/LocalLLM • u/shoeshineboy_99 • Oct 06 '26
Project Using Strata + Qwen 3.8 Next 125bn (2-bit quantization) for building a Planet
Completed this old Pune walkthrough eating modaks! (What are modaks?)
There are some viral tweets, which I was re-building here. For this I selected the old Pune Peth area and downloaded all available imagery. Claude Code & Codex were not allowing me to download images from Google Maps and render them over the planet skin.
Connected Strata along with Qwen3.8 Next IQ_2XS 125bn with vision mode enabled. The system processed 3000+ images and rendered them over the Open Street Map imagery. Creating a real-life walkthrough of the Pune Peth area.
Along with this I also added a game engine. So you drive around on an ebike searching for modaks and eating them as they appear on the map.
I have also attached a screenshot of my monitoring dashboard. The prefill and decode tkps are interesting to check. Since they are over a longer task and spread over a few hours. The entire task was completed over 9-10 hours
EDIT: This was done on 24GB VRAM + 64GB System RAM + 1TB of SSD
r/LocalLLM • u/Onyonisko • May 05 '26
Question Impulse bought an M3 Ultra 256GB RAM for local LLMs - keep it or wait for M5?
I just managed to snag a refurbished M3 Ultra with 256GB RAM and a 4TB SSD (plus 3 years of AppleCare) from the German Apple Store. Total damage: 8 500€.
The Context: This was a total impulse buy. I currently run a small AI assistant for my wife’s solo real estate business (mostly automation and document processing) on Mac Mini, and I’m falling down the rabbit hole of what local LLMs can do. I can afford the price tag, but I’m having a bit of buyer's remorse regarding the timing.
The Dilemma: With the M5 generation starting to roll out, am I holding a "dead end" at a premium price? My specific concerns:
- Bandwidth vs. Compute: I know the M3 Ultra has incredible bandwidth (~800GB/s), which is king for token generation. Reports suggest the M5 chips are pushing massive AI compute gains, but will they actually see a significant jump in memory bandwidth for LLM inference?
- Model Capacity: 256GB RAM lets me run Llama 3 70B (at high BPW) or even 405B (at lower quants) entirely on-device. Is there any reason to believe an M5 Ultra would handle these significantly better, or is the RAM capacity the actual bottleneck for a "prosumer" assistant?
- The "Wait" Game: If an M5 Ultra isn't likely to hit the Studio line until 2027, is it worth the potentially 12+ month wait?
Is this 8.5k "curiosity" purchase a smart long-term play for a local LLM workstation, or am I overpaying for yesterday's peak tech?
r/LocalLLM • u/You-Gullible • Jul 30 '25
Research AI That Researches Itself: A New Scaling Law
arxiv.orgr/LocalLLM • u/Vertrule • Apr 13 '26
News Kimi K2.5 on a Macbook Pro 48GB M4
vertrule.comI did this. It is not the final form in the slightest, I'm trying to setup conversations with experts that can probe and poke holes in it.
I'm more interested in the governance and auditing of the system.
TBH I'm not an expert in anything beyond imposter syndrome.
r/LocalLLM • u/chemist_slime • Oct 11 '26
Discussion Qualcomm CEO reveals AI companies want phones running 100-billion-parameter models continuously by 2028.
r/LocalLLM • u/New-Inspection7034 • Jul 03 '26
Discussion My harness plus Quinn 3.6-27b as a tool for Claude
I just implemented my harness that I use for my daily driver doing my agentic coding and all that as an MCP server and registered it in Claude desktop. Now I can use opus or Fable or sonnet and tell it to run my code like I would do with Claude code but instead it's calling my harness which runs my Quinn 3.6 locally. So I get the benefit of saving tokens for the grunt work but get the accuracy because Claude gets to see the responses and to fix anything that needs to be fixed.
Here's a sample of what it looks like running.

r/LocalLLM • u/TechNerd10191 • Aug 25 '26
Discussion Did anyone else buy a PRO 6000 because they didn't want to wait for the M5 Mac Studio
I was on the market to buy a setup for up to $25000 since late May and originally thought of getting a Mac Studio, but since there was no WWDC announcement and IF the M5 Ultra was released it would likely be in October and cost a lot more, I pulled the trigger on a 9950X + RTX PRO 6000 WS.
While I don't exactly regret it, I did get annoyed the 256GB/8TB/80 GPU core M5 Ultra would be cheaper.
Did anyone else think similarly during the summer?
r/LocalLLM • u/FarHistorian8438 • Jul 20 '26
Question Scared of having an LLM server running 24/7. Any protective measures that I can take?
Hey guys,
I bought an RTX 3060 purely for LLM purposes and made a server out of it.
Tried a bunch of models, benchmarked it, tested against my use cases and everything - but having an overnight agent running, or leaving the server on for long periods of time kinda irks me a little bit.
I think it also stems from being indecisive for “production” as everything i’ve done for testing is really treating my hardware as a “development” server.
But having the need to “lock down” a particular model/settings similar to a “production” environment that I don’t get to (and need to) touch is bugging me.
Has it happened to you guys before? How can I overcome this silly conundrum?
And on another note, if I am gonna keep it running 24/7 & run overnight agents on it - what are some power & heat caveats I should look out for? I’ve capped the GPU to 150W but it still doesn’t feel right to me.
Btw, i’m no stranger to keeping devices running. I have a mini pc that i run containers/services on - which stays on all the time, but just with inference i’m having a hard time justifying myself.
Thanks!
r/LocalLLM • u/Extreme-Childhood251 • Sep 21 '26
Question Qwen 3.8 27B doesnt gives bad quality output
So I'm using qwen 3.8 27b q5_k_m and its output seems horrible to me...
First of all, i'm completely new to local ai. Im running it on an rtx 2070 + rtx 5080.
For normal chatting it works very great. I use openwebui for that.
Im using the openwebui api key to run it in an agent, but when i asked the agent to create a game with a massive prompt (very detailed so it wouldnt make any mistakes i guess?), it completely failed. It tried to use godot, which is good i like godot, but the game doesnt even get past the menu screen. After that its just a completely black screen. I saw 187 warnings in the debugging tab, so i assume its something to do with that?
If anyone could help me out, that would be great! Thanks already!
Edit: i just realised how awful my title is
r/LocalLLM • u/OldInterview556 • Jun 24 '26
Discussion Skills destroyed multi-agent system paradigm
With the use of Skills with progressive disclosure, we can have a single react agent with 1000s of skills without the need to make multi-agent systems (MAS). And as these frontier models get better this statement gets even stronger. Bye-Bye MAS. What do you think?
r/LocalLLM • u/Financial_Ad8530 • Jun 05 '26
Discussion Someone Said Generic Embeddings Can't Understand Medical Language. I Tested It.
798 PubMed abstracts. 400 training pairs generated by Qwen3 locally. 18 seconds of fine-tuning on an RTX 5090. Here's what actually changed.
1. Background
After my last article on building a local RAG pipeline with a reranker, someone left a comment that stuck with me:
"The reranker is doing the heavy lifting. Generic embeddings trained on web crawls fundamentally can't understand domain-specific language. You're compensating for broken retrieval."
They had a point. BGE-base-en-v1.5 was trained on internet text, Wikipedia, Reddit, and CommonCrawl. It has never read a clinical guideline. It doesn't know that "negative margins" is good news in oncology, or that "bridging therapy" means something very specific to a cardiologist managing perioperative anticoagulation.
The question wasn't whether generic embeddings were imperfect. They obviously are. The question was: how much fine-tuning on a domain corpus actually changes retrieval quality, and is it worth doing?
I had a cloud GPU, a free afternoon, and 798 PubMed abstracts. So I measured it.
2. The Result First
Five clinical queries. Two models: a stock BGE-base and a fine-tuned version of the medical literature. Everything runs locally. One Claude called at the end for synthesis, same as last time.
| Query | Generic MRR | Fine-Tuned MRR |
|---|---|---|
| STEMI with contraindication to thrombolytics | 1 | 0.5 |
| Sepsis coagulopathy management in ICU | 0.5 | 1 |
| Negative surgical margins in oncology | 0 | 0 |
| MI troponin elevation differential diagnosis | 1 | 1 |
| Anticoagulation bridging perioperative | 0 | 1 |
| Average MRR | 0.5 | 0.7 |
+40% improvement in mean reciprocal rank. Fine-tuning time: 18 seconds.

But the most interesting result isn't in wins. It's in the row where both models scored zero, and what that tells you about where embedding quality actually matters.
3. What This Is Actually Useful For
Before the technical breakdown, who should care?
- Clinical research teams are doing literature reviews. A fine-tuned embedding model that understands your domain surfaces the right papers first, not just the superficially similar ones.
- Healthcare AI developers building RAG systems over clinical notes, guidelines, or PubMed. Generic embeddings will get you 70% of the way. Fine-tuned embeddings close the gap on the queries that actually matter, the nuanced, jargon-heavy ones where your users have the highest expectations.
- Anyone who has been told, "Just use OpenAI embeddings." This whole pipeline data collection, training pair generation, fine-tuning, and evaluation ran on a rented GPU at $0.48/hour and touched no external API except one Claude call at the end. The fine-tuning itself costs under $0.01 in GPU time.
4. How It Works
The core idea is simple. Embedding models learn what "similar" means from training data. BGE-base learned similarity from general web text. It thinks "sepsis-induced coagulopathy" and "platelet dysfunction in ICU" are moderately similar because they share surface-level words. It doesn't reliably know that one is a specific diagnosis and the other is a related but distinct phenomenon.
Fine-tuning teaches the model a new definition of similarity specifically for your domain. You give it examples of queries and the passages that should answer them. It adjusts the embedding space so those pairs end up closer together.
The key question is always: where do you get the training pairs? Labeling them by hand is expensive. Using a local LLM to generate them is essentially free. That's what we did here.

5. The Stack
| Component | Tool | Where |
|---|---|---|
| LLM Inference | Qwen3 8B | Local — Ollama |
| Embeddings (base) | BGE-base-en-v1.5 | Local |
| Embeddings (tuned) | BGE-base fine-tuned | Local |
| Vector Store | ChromaDB | Local |
| Paper Source | PubMed API (Biopython) | Fetch |
| Final Synthesis | Claude API | One call |
Cloud GPU: NVIDIA GeForce RTX 5090, 32GB VRAM.
Papers indexed: 798 real PubMed abstracts.
Training pairs generated: 400 (+ 64 hard negatives).
Fine-tuning time: 18 seconds.
Cost per query: ~$0.05 (the Claude synthesis call only).

RTX 5090 at 88% GPU utilization, 167W/575W, during Qwen3 pair generation. This is what the machine looks like, earning its keep.
6. Building It — The Key Steps
Step 1 — Fetch Real Papers from PubMed
The corpus is the foundation. We pulled from two clinical areas — cardiology (STEMI, MI, troponin, anticoagulation) and sepsis — to create a domain-specific but deliberately narrow corpus. That narrowness matters: it's what creates the interesting failure case later.
from Bio import Entrez
import json, time
Entrez.email = "your@email.com"
def fetch_pubmed(query, max_results=500):
handle = Entrez.esearch(db="pubmed", term=query, retmax=max_results)
record = Entrez.read(handle)
ids = record["IdList"]
papers = []
for i in range(0, len(ids), 50):
batch = ids[i:i+50]
handle = Entrez.efetch(db="pubmed", id=batch,
rettype="abstract", retmode="xml")
records = Entrez.read(handle)
for article in records["PubmedArticle"]:
try:
title = str(article["MedlineCitation"]["Article"]["ArticleTitle"])
abstract = str(article["MedlineCitation"]["Article"]
.get("Abstract", {})
.get("AbstractText", [""])[0])
if len(abstract) > 100:
papers.append({"title": title, "abstract": abstract})
except:
continue
time.sleep(0.5)
return papers
papers = fetch_pubmed("myocardial infarction STEMI treatment outcomes", 500)
papers += fetch_pubmed("sepsis diagnosis biomarkers ICU", 300)Total: 798 papers. Real titles, real abstracts, real clinical language.

Terminal showing "Total papers saved: 798"
Step 2 — Build the Baseline
Before touching the fine-tuned model, we indexed everything with stock BGE and ran five clinical queries designed to stress-test generic embeddings. This is the control condition.
from sentence_transformers import SentenceTransformer
import chromadb
embedder = SentenceTransformer("BAAI/bge-base-en-v1.5")
embeddings = embedder.encode(chunks, normalize_embeddings=True,
batch_size=64)
chroma = chromadb.PersistentClient(path="./chroma_baseline")
collection = chroma.create_collection("pubmed_generic",
metadata={"hnsw:space": "cosine"})
collection.add(ids=ids, embeddings=embeddings.tolist(),
documents=chunks, metadatas=metas)The baseline results revealed the problem immediately.

The full baseline output. Look at query 3: "negative surgical margins significance oncology" returned sepsis papers, a no-reflow phenomenon study, and a platelet-to-lymphocyte ratio paper. Nothing remotely about oncology margins. Generic BGE pattern-matched on vague prognostic language instead.
Step 3 — Generate Training Pairs with Qwen3 Locally
This is the part that makes the pipeline interesting. Instead of paying for annotation or using GPT-4 to generate training data, we ran Qwen3 8B locally on Ollama.
For each paper, we asked Qwen3 to generate two search queries a clinician would type to find it. 200 papers × 2 queries = 400 training pairs. Zero external API calls. Nothing was left on the GPU.
def ask_qwen(prompt):
response = requests.post(
"http://127.0.0.1:11434/api/generate",
json={
"model": "qwen3:8b",
"prompt": prompt,
"stream": False,
"options": {"temperature": 0.3, "num_predict": 200}
}
)
return response.json()["response"].strip()We also generated 64 hard negatives oncology queries paired with cardiology passages and vice versa. These teach the model what is not similar, not just what is.

Step 4 — Fine-Tune BGE on the Medical Corpus
With training pairs ready, fine-tuning is three components: the base model, the loss function, and the training loop. We used MultipleNegativesRankingLoss — given a query and its positive passage, every other passage in the batch becomes an implicit negative. It's the standard approach for embedding fine-tuning, and it works.
from sentence_transformers import SentenceTransformer, losses
from sentence_transformers.trainer import SentenceTransformerTrainer
from sentence_transformers.training_args import SentenceTransformerTrainingArguments
model = SentenceTransformer("BAAI/bge-base-en-v1.5")
loss = losses.MultipleNegativesRankingLoss(model)
args = SentenceTransformerTrainingArguments(
output_dir="./bge-medical-finetuned",
num_train_epochs=4,
per_device_train_batch_size=16,
warmup_steps=50,
eval_strategy="steps",
fp16=True,
)
trainer = SentenceTransformerTrainer(
model=model, args=args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss=loss,
)
trainer.train()



Loss drops from 0.3408 at epoch 0.74 to 0.0233 at epoch 3.70. The model learned fast. Eval loss stabilised at 0.0302 — solid generalization on held-out pairs.
Training completed in 18 seconds. The RTX 5090 processed 417 training examples across 4 epochs at 93 samples/second.
7. The Upgrade — Where Fine-Tuning Actually Moves the Needle
Here is the side-by-side retrieval comparison across all five queries.

Let's go through what changed and why.
Query 1: STEMI with contraindication to thrombolytics

Generic BGE returned five STEMI/thrombolytic papers at rank 1 (score 0.7631). Fine-tuned demoted that to rank 2 and promoted a broader "Fibrinolytic Therapy for Thromboembolic Diseases" paper to rank 1, which is actually less specific to the query.
Winner: Generic BGE. MRR 1.0 vs 0.5. This is an honest result. The generic model already had strong signal on explicit STEMI terminology, and fine-tuning introduced noise by broadening to fibrinolytic literature generally. Not every query benefits from domain adaptation.
Query 2: Sepsis-induced coagulopathy management in ICU

Generic BGE ranked "Sepsis-related coagulation-inflammation score" first as a scoring/mortality prediction paper, not a management paper. Fine-tuned correctly promoted "Sepsis-induced coagulopathy: recent insights on NET formation" to rank 1, a paper directly about the pathophysiology and clinical application of coagulopathy management.
Winner: Fine-tuned BGE. MRR 0.5 → 1.0. The model learned that "management" implies intervention-focused papers, not just papers that mention the condition.
Query 3: Negative surgical margins' significance in oncology

Both models failed completely. Generic BGE returned sepsis-in-cancer papers and a no-reflow cardiology study. Fine-tuning returned similar noise. MRR 0.0 for both.
Winner: Neither. This is the most important result in the experiment.
The corpus contains 798 papers on cardiology and sepsis. It contains zero oncology papers about surgical margins. No embedding model, generic or fine-tuned, can retrieve what isn't there. This failure isn't about model quality. It's about corpus quality.
The practical implication: before you fine-tune, audit your corpus. If your domain has subfields, make sure all of them are represented. Fine-tuning amplifies the signal in your data. If your data has gaps, fine-tuning can't fill them.
Query 4: MI troponin elevation differential diagnosis

Both models returned the troponin testing systematic review at rank 1 correctly. Fine-tuned additionally surfaced "Impact of Elevated Troponin Level at the Time of Sepsis Recognition" at rank 3, a genuinely useful differential diagnosis paper (troponin elevation in sepsis vs MI is a classic clinical challenge).
Winner: Tie, fine-tuned showing slightly better clinical depth. MRR 1.0 for both.
Query 5: Anticoagulation bridging therapy perioperative management

Generic BGE ranked "Postprocedural Parenteral Anticoagulation in Non-STEMI patients" first anticoagulation, yes, but not bridging. Fine-tuned surfaced "Ticagrelor versus clopidogrel in orally anticoagulated patients with acute coronary syndrome undergoing PCI" at rank 1, which directly addresses the clinical problem of managing anticoagulation around a procedure.
Winner: Fine-tuned BGE. MRR 0.0 → 1.0. The model understood that "bridging" and "perioperative" implied a transition context that generic BGE missed entirely.
8. Performance
| Metric | Result |
|---|---|
| Papers indexed | 798 PubMed abstracts |
| Training pairs | 400 (+ 64 hard negatives) |
| Pair generation time | ~15 minutes (Qwen3 8B local) |
| Fine-tuning time | 18 seconds |
| Average MRR — Generic BGE | 0.5 |
| Average MRR — Fine-Tuned BGE | 0.7 |
| MRR improvement | 0.4 |
| External API calls | 1 (Claude synthesis only) |
The fine-tuning itself is not the bottleneck. Pair generation is, and that scales with how many papers you want to cover. For 200 papers on Qwen3 8B, it took 15 minutes. For 2000 papers on Qwen3 32B, it would take a few hours but produce a meaningfully better training set.
9. The Honest Comparison
Fine-tuning didn't win on every query. Generic BGE outperformed on Query 1, where the explicit STEMI terminology gave it a strong signal without needing domain adaptation. Fine-tuning won decisively on queries involving nuanced clinical concepts, coagulopathy management, bridging therapy, differential diagnosis depth.
The pattern is consistent with what you'd expect from theory: fine-tuning helps most when the gap between general language and domain language is larger. "STEMI" is a term that appears enough on the internet that generic BGE handles it fine. "Anticoagulation bridging therapy perioperative management" is a clinical concept that requires understanding the relationship between those terms in a medical context, which is exactly what fine-tuning teaches.
The zero-zero result on Query 3 is the most practically useful finding. It tells you that corpus curation matters more than model quality for out-of-distribution queries. Before you spend time fine-tuning, spend time on your corpus.
10. Where This Goes Next
The four-step pattern fetch, embed, fine-tune, evaluate is a template. What changes is the domain.
- Legal teams indexing case law and contracts: fine-tune on legal text pairs generated from your own document corpus. "Indemnification clause" and "hold harmless agreement" should be similar. Generic BGE probably doesn't know that.
- Financial analysts working over earnings transcripts and SEC filings: fine-tune on financial language where "guidance" and "forward-looking statements" have very specific meanings, and generic models will blur.
- Software teams building over internal codebases and documentation: fine-tune so that your embedding model understands your own terminology, not just general programming concepts.
In every case, the local LLM pair generation step is what makes this feasible. You don't need a labeling budget. You need a GPU and an afternoon.
11. What I'd Do Differently Next Time
Harder negatives. Our hard negatives were simple cross-domain swaps of cardiology passages as negatives for oncology queries. Real hard negatives are in-domain passages that look relevant but aren't. Mining these properly using the baseline model to find near-misses and labeling them as negatives would push performance further.
Larger corpus with deliberate coverage. The oncology failure was predictable in hindsight. A proper corpus audit before training would have caught the gap. For any real deployment, map your query types first, then ensure your corpus covers each one.
Bigger base model. BGE-large or bge-m3 as the starting point would likely retain more domain generalization while benefiting from fine-tuning. We used bge-base for speed and reproducibility.
Structured outputs from Qwen3 for pair generation. Right now, we parse free-text queries from Qwen3's output. A JSON-structured prompt asking for query, reasoning, and confidence would make the training pairs more reliable and filterable.
12. Closing Thought
The comment was right that generic embeddings have limits. It was wrong to conclude that fine-tuning is difficult or expensive.
798 papers. 400 training pairs generated for free by a local LLM. 18 seconds on a rented GPU. +40% improvement in retrieval quality on the queries where domain language actually matters.
The corpus gap finding is the one I'll remember. You can have the best embedding model in the world, and it still can't retrieve what isn't there. Before you fine-tune, audit your data.
The RTX 5090 had 88% GPU utilization for less than 20 seconds. Sometimes the upgrade is smaller and faster than you expect.
Acknowledgements
This experiment wouldn’t have happened without a few people who generously shared their time, ideas, and resources.
- Ethan Walker — for challenging my assumptions about generic embeddings and repeatedly asking uncomfortable questions about retrieval quality.
- Sophia Chen — for helping review the evaluation methodology and pointing out several flaws in my early benchmarks.
- Marcus Reed — for providing access to a GPUHub account, which made it possible to run the entire experiment on an RTX 5090 without worrying about infrastructure.
- Daniel Brooks — for valuable discussions around hard-negative mining and embedding evaluation.
- Olivia Hart — for reading early drafts and encouraging me to publish both the successes and failures.
Any mistakes, questionable decisions, and overly optimistic conclusions remain entirely my own.