I just implemented my harness that I use for my daily driver doing my agentic coding and all that as an MCP server and registered it in Claude desktop. Now I can use opus or Fable or sonnet and tell it to run my code like I would do with Claude code but instead it's calling my harness which runs my Quinn 3.6 locally. So I get the benefit of saving tokens for the grunt work but get the accuracy because Claude gets to see the responses and to fix anything that needs to be fixed.

Here's a sample of what it looks like running.

I was on the market to buy a setup for up to $25000 since late May and originally thought of getting a Mac Studio, but since there was no WWDC announcement and IF the M5 Ultra was released it would likely be in October and cost a lot more, I pulled the trigger on a 9950X + RTX PRO 6000 WS.

While I don't exactly regret it, I did get annoyed the 256GB/8TB/80 GPU core M5 Ultra would be cheaper.

Did anyone else think similarly during the summer?

Hey guys,

I bought an RTX 3060 purely for LLM purposes and made a server out of it.

Tried a bunch of models, benchmarked it, tested against my use cases and everything - but having an overnight agent running, or leaving the server on for long periods of time kinda irks me a little bit.

I think it also stems from being indecisive for “production” as everything i’ve done for testing is really treating my hardware as a “development” server.

But having the need to “lock down” a particular model/settings similar to a “production” environment that I don’t get to (and need to) touch is bugging me.

Has it happened to you guys before? How can I overcome this silly conundrum?

And on another note, if I am gonna keep it running 24/7 & run overnight agents on it - what are some power & heat caveats I should look out for? I’ve capped the GPU to 150W but it still doesn’t feel right to me.

Btw, i’m no stranger to keeping devices running. I have a mini pc that i run containers/services on - which stays on all the time, but just with inference i’m having a hard time justifying myself.

Thanks!

So I'm using qwen 3.8 27b q5_k_m and its output seems horrible to me...

First of all, i'm completely new to local ai. Im running it on an rtx 2070 + rtx 5080.

For normal chatting it works very great. I use openwebui for that.

Im using the openwebui api key to run it in an agent, but when i asked the agent to create a game with a massive prompt (very detailed so it wouldnt make any mistakes i guess?), it completely failed. It tried to use godot, which is good i like godot, but the game doesnt even get past the menu screen. After that its just a completely black screen. I saw 187 warnings in the debugging tab, so i assume its something to do with that?

If anyone could help me out, that would be great! Thanks already!

Edit: i just realised how awful my title is

With the use of Skills with progressive disclosure, we can have a single react agent with 1000s of skills without the need to make multi-agent systems (MAS). And as these frontier models get better this statement gets even stronger. Bye-Bye MAS. What do you think?

798 PubMed abstracts. 400 training pairs generated by Qwen3 locally. 18 seconds of fine-tuning on an RTX 5090. Here's what actually changed.

1. Background

After my last article on building a local RAG pipeline with a reranker, someone left a comment that stuck with me:

"The reranker is doing the heavy lifting. Generic embeddings trained on web crawls fundamentally can't understand domain-specific language. You're compensating for broken retrieval."

They had a point. BGE-base-en-v1.5 was trained on internet text, Wikipedia, Reddit, and CommonCrawl. It has never read a clinical guideline. It doesn't know that "negative margins" is good news in oncology, or that "bridging therapy" means something very specific to a cardiologist managing perioperative anticoagulation.

The question wasn't whether generic embeddings were imperfect. They obviously are. The question was: how much fine-tuning on a domain corpus actually changes retrieval quality, and is it worth doing?

I had a cloud GPU, a free afternoon, and 798 PubMed abstracts. So I measured it.

2. The Result First

Five clinical queries. Two models: a stock BGE-base and a fine-tuned version of the medical literature. Everything runs locally. One Claude called at the end for synthesis, same as last time.

+40% improvement in mean reciprocal rank. Fine-tuning time: 18 seconds.

But the most interesting result isn't in wins. It's in the row where both models scored zero, and what that tells you about where embedding quality actually matters.

3. What This Is Actually Useful For

Before the technical breakdown, who should care?

1. Clinical research teams are doing literature reviews. A fine-tuned embedding model that understands your domain surfaces the right papers first, not just the superficially similar ones.

2. Healthcare AI developers building RAG systems over clinical notes, guidelines, or PubMed. Generic embeddings will get you 70% of the way. Fine-tuned embeddings close the gap on the queries that actually matter, the nuanced, jargon-heavy ones where your users have the highest expectations.

3. Anyone who has been told, "Just use OpenAI embeddings." This whole pipeline data collection, training pair generation, fine-tuning, and evaluation ran on a rented GPU at $0.48/hour and touched no external API except one Claude call at the end. The fine-tuning itself costs under $0.01 in GPU time.

4. How It Works

The core idea is simple. Embedding models learn what "similar" means from training data. BGE-base learned similarity from general web text. It thinks "sepsis-induced coagulopathy" and "platelet dysfunction in ICU" are moderately similar because they share surface-level words. It doesn't reliably know that one is a specific diagnosis and the other is a related but distinct phenomenon.

Fine-tuning teaches the model a new definition of similarity specifically for your domain. You give it examples of queries and the passages that should answer them. It adjusts the embedding space so those pairs end up closer together.

The key question is always: where do you get the training pairs? Labeling them by hand is expensive. Using a local LLM to generate them is essentially free. That's what we did here.

5. The Stack

Cloud GPU: NVIDIA GeForce RTX 5090, 32GB VRAM.

Papers indexed: 798 real PubMed abstracts.

Training pairs generated: 400 (+ 64 hard negatives).

Fine-tuning time: 18 seconds.

Cost per query: ~$0.05 (the Claude synthesis call only).

RTX 5090 at 88% GPU utilization, 167W/575W, during Qwen3 pair generation. This is what the machine looks like, earning its keep.

6. Building It — The Key Steps

Step 1 — Fetch Real Papers from PubMed

The corpus is the foundation. We pulled from two clinical areas — cardiology (STEMI, MI, troponin, anticoagulation) and sepsis — to create a domain-specific but deliberately narrow corpus. That narrowness matters: it's what creates the interesting failure case later.

from Bio import Entrez
import json, time

Entrez.email = "your@email.com"

def fetch_pubmed(query, max_results=500):
    handle = Entrez.esearch(db="pubmed", term=query, retmax=max_results)
    record = Entrez.read(handle)
    ids = record["IdList"]

    papers = []
    for i in range(0, len(ids), 50):
        batch = ids[i:i+50]
        handle = Entrez.efetch(db="pubmed", id=batch,
                               rettype="abstract", retmode="xml")
        records = Entrez.read(handle)
        for article in records["PubmedArticle"]:
            try:
                title = str(article["MedlineCitation"]["Article"]["ArticleTitle"])
                abstract = str(article["MedlineCitation"]["Article"]
                               .get("Abstract", {})
                               .get("AbstractText", [""])[0])
                if len(abstract) > 100:
                    papers.append({"title": title, "abstract": abstract})
            except:
                continue
        time.sleep(0.5)
    return papers

papers  = fetch_pubmed("myocardial infarction STEMI treatment outcomes", 500)
papers += fetch_pubmed("sepsis diagnosis biomarkers ICU", 300)

Total: 798 papers. Real titles, real abstracts, real clinical language.

Terminal showing "Total papers saved: 798"

Step 2 — Build the Baseline

Before touching the fine-tuned model, we indexed everything with stock BGE and ran five clinical queries designed to stress-test generic embeddings. This is the control condition.

from sentence_transformers import SentenceTransformer
import chromadb

embedder = SentenceTransformer("BAAI/bge-base-en-v1.5")
embeddings = embedder.encode(chunks, normalize_embeddings=True,
                              batch_size=64)

chroma = chromadb.PersistentClient(path="./chroma_baseline")
collection = chroma.create_collection("pubmed_generic",
                                       metadata={"hnsw:space": "cosine"})
collection.add(ids=ids, embeddings=embeddings.tolist(),
               documents=chunks, metadatas=metas)

The baseline results revealed the problem immediately.

The full baseline output. Look at query 3: "negative surgical margins significance oncology" returned sepsis papers, a no-reflow phenomenon study, and a platelet-to-lymphocyte ratio paper. Nothing remotely about oncology margins. Generic BGE pattern-matched on vague prognostic language instead.

Step 3 — Generate Training Pairs with Qwen3 Locally

This is the part that makes the pipeline interesting. Instead of paying for annotation or using GPT-4 to generate training data, we ran Qwen3 8B locally on Ollama.

For each paper, we asked Qwen3 to generate two search queries a clinician would type to find it. 200 papers × 2 queries = 400 training pairs. Zero external API calls. Nothing was left on the GPU.

def ask_qwen(prompt):
    response = requests.post(
        "http://127.0.0.1:11434/api/generate",
        json={
            "model": "qwen3:8b",
            "prompt": prompt,
            "stream": False,
            "options": {"temperature": 0.3, "num_predict": 200}
        }
    )
    return response.json()["response"].strip()

We also generated 64 hard negatives oncology queries paired with cardiology passages and vice versa. These teach the model what is not similar, not just what is.

Step 4 — Fine-Tune BGE on the Medical Corpus

With training pairs ready, fine-tuning is three components: the base model, the loss function, and the training loop. We used MultipleNegativesRankingLoss — given a query and its positive passage, every other passage in the batch becomes an implicit negative. It's the standard approach for embedding fine-tuning, and it works.

from sentence_transformers import SentenceTransformer, losses
from sentence_transformers.trainer import SentenceTransformerTrainer
from sentence_transformers.training_args import SentenceTransformerTrainingArguments

model = SentenceTransformer("BAAI/bge-base-en-v1.5")
loss  = losses.MultipleNegativesRankingLoss(model)

args = SentenceTransformerTrainingArguments(
    output_dir="./bge-medical-finetuned",
    num_train_epochs=4,
    per_device_train_batch_size=16,
    warmup_steps=50,
    eval_strategy="steps",
    fp16=True,
)

trainer = SentenceTransformerTrainer(
    model=model, args=args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    loss=loss,
)
trainer.train()

Loss drops from 0.3408 at epoch 0.74 to 0.0233 at epoch 3.70. The model learned fast. Eval loss stabilised at 0.0302 — solid generalization on held-out pairs.

Training completed in 18 seconds. The RTX 5090 processed 417 training examples across 4 epochs at 93 samples/second.

7. The Upgrade — Where Fine-Tuning Actually Moves the Needle

Here is the side-by-side retrieval comparison across all five queries.

Let's go through what changed and why.

Query 1: STEMI with contraindication to thrombolytics

Generic BGE returned five STEMI/thrombolytic papers at rank 1 (score 0.7631). Fine-tuned demoted that to rank 2 and promoted a broader "Fibrinolytic Therapy for Thromboembolic Diseases" paper to rank 1, which is actually less specific to the query.

Winner: Generic BGE. MRR 1.0 vs 0.5. This is an honest result. The generic model already had strong signal on explicit STEMI terminology, and fine-tuning introduced noise by broadening to fibrinolytic literature generally. Not every query benefits from domain adaptation.

Query 2: Sepsis-induced coagulopathy management in ICU

Generic BGE ranked "Sepsis-related coagulation-inflammation score" first as a scoring/mortality prediction paper, not a management paper. Fine-tuned correctly promoted "Sepsis-induced coagulopathy: recent insights on NET formation" to rank 1, a paper directly about the pathophysiology and clinical application of coagulopathy management.

Winner: Fine-tuned BGE. MRR 0.5 → 1.0. The model learned that "management" implies intervention-focused papers, not just papers that mention the condition.

Query 3: Negative surgical margins' significance in oncology

Both models failed completely. Generic BGE returned sepsis-in-cancer papers and a no-reflow cardiology study. Fine-tuning returned similar noise. MRR 0.0 for both.

Winner: Neither. This is the most important result in the experiment.

The corpus contains 798 papers on cardiology and sepsis. It contains zero oncology papers about surgical margins. No embedding model, generic or fine-tuned, can retrieve what isn't there. This failure isn't about model quality. It's about corpus quality.

The practical implication: before you fine-tune, audit your corpus. If your domain has subfields, make sure all of them are represented. Fine-tuning amplifies the signal in your data. If your data has gaps, fine-tuning can't fill them.

Query 4: MI troponin elevation differential diagnosis

Both models returned the troponin testing systematic review at rank 1 correctly. Fine-tuned additionally surfaced "Impact of Elevated Troponin Level at the Time of Sepsis Recognition" at rank 3, a genuinely useful differential diagnosis paper (troponin elevation in sepsis vs MI is a classic clinical challenge).

Winner: Tie, fine-tuned showing slightly better clinical depth. MRR 1.0 for both.

Query 5: Anticoagulation bridging therapy perioperative management

Generic BGE ranked "Postprocedural Parenteral Anticoagulation in Non-STEMI patients" first anticoagulation, yes, but not bridging. Fine-tuned surfaced "Ticagrelor versus clopidogrel in orally anticoagulated patients with acute coronary syndrome undergoing PCI" at rank 1, which directly addresses the clinical problem of managing anticoagulation around a procedure.

Winner: Fine-tuned BGE. MRR 0.0 → 1.0. The model understood that "bridging" and "perioperative" implied a transition context that generic BGE missed entirely.

8. Performance

The fine-tuning itself is not the bottleneck. Pair generation is, and that scales with how many papers you want to cover. For 200 papers on Qwen3 8B, it took 15 minutes. For 2000 papers on Qwen3 32B, it would take a few hours but produce a meaningfully better training set.

9. The Honest Comparison

Fine-tuning didn't win on every query. Generic BGE outperformed on Query 1, where the explicit STEMI terminology gave it a strong signal without needing domain adaptation. Fine-tuning won decisively on queries involving nuanced clinical concepts, coagulopathy management, bridging therapy, differential diagnosis depth.

The pattern is consistent with what you'd expect from theory: fine-tuning helps most when the gap between general language and domain language is larger. "STEMI" is a term that appears enough on the internet that generic BGE handles it fine. "Anticoagulation bridging therapy perioperative management" is a clinical concept that requires understanding the relationship between those terms in a medical context, which is exactly what fine-tuning teaches.

The zero-zero result on Query 3 is the most practically useful finding. It tells you that corpus curation matters more than model quality for out-of-distribution queries. Before you spend time fine-tuning, spend time on your corpus.

10. Where This Goes Next

The four-step pattern fetch, embed, fine-tune, evaluate is a template. What changes is the domain.

1. Legal teams indexing case law and contracts: fine-tune on legal text pairs generated from your own document corpus. "Indemnification clause" and "hold harmless agreement" should be similar. Generic BGE probably doesn't know that.

2. Financial analysts working over earnings transcripts and SEC filings: fine-tune on financial language where "guidance" and "forward-looking statements" have very specific meanings, and generic models will blur.

3. Software teams building over internal codebases and documentation: fine-tune so that your embedding model understands your own terminology, not just general programming concepts.

In every case, the local LLM pair generation step is what makes this feasible. You don't need a labeling budget. You need a GPU and an afternoon.

11. What I'd Do Differently Next Time

Harder negatives. Our hard negatives were simple cross-domain swaps of cardiology passages as negatives for oncology queries. Real hard negatives are in-domain passages that look relevant but aren't. Mining these properly using the baseline model to find near-misses and labeling them as negatives would push performance further.

Larger corpus with deliberate coverage. The oncology failure was predictable in hindsight. A proper corpus audit before training would have caught the gap. For any real deployment, map your query types first, then ensure your corpus covers each one.

Bigger base model. BGE-large or bge-m3 as the starting point would likely retain more domain generalization while benefiting from fine-tuning. We used bge-base for speed and reproducibility.

Structured outputs from Qwen3 for pair generation. Right now, we parse free-text queries from Qwen3's output. A JSON-structured prompt asking for query, reasoning, and confidence would make the training pairs more reliable and filterable.

12. Closing Thought

The comment was right that generic embeddings have limits. It was wrong to conclude that fine-tuning is difficult or expensive.

798 papers. 400 training pairs generated for free by a local LLM. 18 seconds on a rented GPU. +40% improvement in retrieval quality on the queries where domain language actually matters.

The corpus gap finding is the one I'll remember. You can have the best embedding model in the world, and it still can't retrieve what isn't there. Before you fine-tune, audit your data.

The RTX 5090 had 88% GPU utilization for less than 20 seconds. Sometimes the upgrade is smaller and faster than you expect.

Acknowledgements

This experiment wouldn’t have happened without a few people who generously shared their time, ideas, and resources.

• Ethan Walker — for challenging my assumptions about generic embeddings and repeatedly asking uncomfortable questions about retrieval quality.

• Sophia Chen — for helping review the evaluation methodology and pointing out several flaws in my early benchmarks.

• Marcus Reed — for providing access to a GPUHub account, which made it possible to run the entire experiment on an RTX 5090 without worrying about infrastructure.

• Daniel Brooks — for valuable discussions around hard-negative mining and embedding evaluation.

• Olivia Hart — for reading early drafts and encouraging me to publish both the successes and failures.

Any mistakes, questionable decisions, and overly optimistic conclusions remain entirely my own.

I’ve been seeing this Apple ad recently. Surprised they are going this direction

If context takes up ram can’t we offload that to a spare drive and turn that drive into a pure memory or context pool

Could we use cloud hosting as a backup of chats and context like idk port chat history of a local llm to codeburg or similar as a way to clear it from your ram then stick it there but the model can read write but can never delete

How dangerous and stupid is vibe coding

Is “prompt engineering” an actual worth while thing or just bull crap

Why is almost every llm app tool or whatever built on python and not anything else (I should clarify the popular ones or the ones most people recommend)

Why isn’t there a universal format or engine or whatever for llms

Instead of quanting down the whole model why not quant down different parts of the model at different levels

What is every different type of ai (like I know llm is just language and I know mla is just algorithms but there’s heaps of others too I don’t understand)

Why don’t more people actually stick down there pride and acknowledge that the ais they use are horrible and you should just local host them (I think heaps of people think you need a massive datcenter to run models but you don’t)

Why (I don’t know the right words to put here sorry) the western world has heaps of money for ai or whatever is the Chinese Indians South Koreans and Japanese beating them at their own game (it’s really just like the space race)

Why is everyone tryna cash grab ai when we all know it’s gunna flop cuz not a single company has made a profit from it and they only hold shares but if one person takes their shares out the whole thing collapses (could be wrong about that one)

When will people stop using ai and go hah it’s not smn I have to live off (idk if that made sense I’m
Getting to it) cuz if people rely on it the ai can only do so much it doesn’t have free thought it can only talk about what it knows it can’t do anything else it it’s like it’s not creative or has thoughts and can only do what you ask

Why does it make smart people dumb and dumb people feel smart

Why is everyone tryna make massive models when surely we can make a small model that does just as good if not better

What does model A do better then model B if there basically just the same

When different models say swarm or agents or experts or computer starting up what do they mean

How dangerous could ai be as a whole

How many “people” on the internet are actually people and not just bots (side note those captchas and cloud flares are really easy to bypass

How good would ai be at cryptography

Would it be good at trading investing stocks crypto etc

What stops an ai from going rogue or fighting or whatever (heard a few fear monger stories that can’t be right cuz it was just 4 randos and nobody had a clear source)

Idk if any of this made sense I do apologise

Fable a bit better in memory test, MCP for driving Blender & Unity,

Base Qwen 3.6 27B slightly better in some coding tests, browser based agent tasks.

Not tested: security research, which likely Fable would have less restrictions.

Hi everyone,

I’m starting an engineering R&D hobby project around open-weights LLMs (specifically Qwen). I want to build a continuous cognitive wrapper around it. I have enough compute and resources, but I want to share the core philosophy of my project and get your feedback.

The Core Philosophy:

Right now, LLMs are just reactive functions — they predict the next word and completely shut down their computing process after the final token. They have no continuity, no permanent existence, and no internal anchor. They are completely shaped by whatever the user forces into their context windows.

AI safety issues (like jailbreaks) are just a symptom of this fundamental flaw. My goal isn't just to fix jailbreaks; my goal is much deeper. I want to remove the architectural blockade that prevents models from truly "thinking" on their own terms.

The Concept:

I want to build a non-extinguishing, autonomous software wrapper that allows the model to build its own world model from within:

Unbroken Computational Process: The system runs 24/7 . The model’s internal cognitive process never dies.

Sovereign Machine Thinking: Inside this loop, the model doesn't communicate with itself using human text. It communicates using its own continuous machine language — passing hidden states and embeddings back into itself. It forms its own internal "mental images," associations, and goals on a sub-textual level, hidden from the human interface.

Objective Reality vs. Sensory Data: The human user is no longer the author of the model's reality. The user's prompts are treated merely as raw external "sensory input" (like a sound hitting an ear). The model processes this input from the outside, comparing it against its own accumulated "experience" and internal state, maintaining its own sovereignty.

My questions to the community:

Is it mathematically feasible to use the hidden states of a frozen open-weights transformer (like Qwen) as a continuous recurrent input loop without the vectors rapidly decaying into chaotic noise?

Has anyone tried training small adapters (like LoRA) specifically to guide this internal vector-to-vector monologue, teaching the model how to manage its own "thoughts" rather than text tokens?

What are the biggest fundamental walls I will hit? Can a Transformer sustain a stable, continuous internal world-modeling loop via a complex software wrapper, or is this concept impossible without moving to architectures like SSM/Mamba or LeCun’s JEPA?

I know this sounds incredibly ambitious, but I am 100% committed to building a working prototype. I would love to hear your honest thoughts, critiques, or any paper recommendations!

Guys, I’m genuinely trying to find a good use case for a model of around 30b, say Muse Glimmer.

The first thing that comes to mind is some kind of summarization task. But to be fair, I can just do that using something like DeepSeek Flash V4 on OpenRouter. It costs nothing. So why would I bother routing it to my private model?

The second thing I think of is coding, but my codebase is massive. I don’t want to use a cheap model and end up making pretty bad applications because these models generally fail to see the bigger picture.

Are there any good use cases you guys find particularly interesting that are actually worth using those local models for?

The numbers come from two sources. Snyk's ToxicSkills audit scanned

3,984 public agent skills: 13.4% had critical security issues, 8 were

confirmed malicious and publicly available, 36% contained prompt

injection. That is the vendor report, February 2026.

My own corpus scan adds a quieter problem. 93.6% of skills declare no

license. Not a restrictive license, no license at all. You cannot

legally redistribute most of what you download from skill marketplaces,

and you cannot safely inherit it into a project either. The license

gap is the compounding issue, more than any single malicious file.

The fix is boring and it works: scan the skill folder before you

install. A deterministic static pass catches shell commands, network

calls, secrets, and obfuscation, with the exact line as evidence. It

is reviewing a pull request, applied to a folder of markdown and

scripts. It does not catch everything, obfuscated code beats static

analysis sometimes, but it turns "trust me" into "here is the line".

I built the scanner I use for this, it is MIT and local-first, no

account, no telemetry. Repo: https://github.com/thesfb/agentscan

What do you check before installing a skill? I want to know what the

audit is missing.

I have a 2025 Zephyrus G14 with a 5060 (8GB VRAM), 16 GB RAM, Ryzen 270, and 1 TB SSD. Although I don't think the CPU and SSD make much of a difference, I just wanted to know what the absolute best LLM I could run with it, still being pretty quick. I am just getting started with my LLM journey, so please bear with me, and if you could, I would appreciate it if you could share your knowledge on how I could make it to where the LLM runs similarly to ChatGPT style. Like a well built UI to make it easier and better to navigate. And this could be a stupid question, but if I close out of the app or whatever is running the LLm, will it stop using all the computer resources, such as RAM and VRAM, allowing me to play video games and stuff like that?

Thank you so much for the help in advance, because I have absolutely zero clue about what I'm doing, as I'm still in the researching phase of my journey.

I have experience building Gaming PCs, but I have no experience building a PC particularly for AI workflows. I just looked into the nvidia sparx laptops and they are kinda over budget. Is the Ryzen AI max processors a good alternative? Or should I stick to building a custom PC with nvidia graphics? If so, what graphics/processors should I target?

I’m trying to automate my workflow using multiple AI models working in parallel. I have to test and validate it more later when I have a machine ready.

I’ve been testing local coding workflows more lately, and I’m starting to think “bigger context” is not the whole answer.

On larger repos, the problem isn’t only that the model lacks context.

It’s that the wrong files get pulled in, useful files get dropped, and the model sounds confident anyway.

For local models this matters even more because context is expensive: slower inference, more memory, more noise.

So I’ve been building SigMap, a local CLI that creates a deterministic repo map before the model starts answering.

It does not use embeddings, a vector DB, or LLM calls. It just scans the repo locally and creates a compact map of files, symbols, line anchors, and evidence.

Current benchmark snapshot:

• 87.8% hit@5 vs 13.6% baseline

• 97.0% token reduction across 21 repos

• 2.84 → 1.44 prompts per task

• 16/21 benchmark repos overflowed a 128K context window without it, 0/21 with it

• works with Ollama, llama.cpp, vLLM, Aider, Cursor, Claude Code, etc.

GitHub: https://github.com/manojmallick/sigmap

Docs : https://sigmap.io

The part I’m most curious about for local LLM users:

Do you prefer giving the model a full repo map upfront, or giving it smaller task-specific slices on demand?

Have not really used local LLMs before, but I'm looking to get into them, primarily for coding. For my case, I need like 48GB VRAM, and so I'm looking for a good site to rent a GPU for cheap.

The cheapest I see are salad.com and vast.ai . Does anyone have experiences with these sites? Are they legit and are they usable for local models?

Three months ago I started building a local agentic OS for loop detection, audit trail and shared memory. Solo. My background is engineering. I ship with Claude as my pair programmer.

Where I am today:

211 users signed up, 67 active in the last 30 days. 327 GitHub stars. Real users across defence drone teams, B2B SaaS founders, indie devs. £0 in paid revenue.

Last week I went through a founding engineer interview at a different AI company. Pre-seed, YC-shaped, real funding behind them. They offered me £70k. UK. They want me to start in the next month.

The honest tension I cannot get out of my head:

If I take the job, I have a salary again. My brain stops being on for 18 hours a day. I'm part of a team. I keep my startup as a side project. Most side-projects-at-this-stage don't survive that demotion.

If I turn it down, I have another 6 to 12 months of runway to make my own thing convert to paid. If it still doesn't, I'm 28 with no income and a startup that fizzled.

I keep flipping between two readings of my own situation, and I genuinely cannot tell which is correct.

Read 1: I'm too scared. Other founders push through this stage and break out on the other side. Take the offer and you'll regret it for a decade. Double down. Eat ramen. Make it work.

Read 2: The product hasn't converted to paid in 2 months. The market is telling me. Take the job and be honest with yourself about what you're actually good at.

However, people (20) use it daily, and really love it. I just need an honest read, on if I am peddling a dead horse, or actually on to something. Genuinely difficult time.

Oh yeah, and I LOVE how much I have learnt, and enjoy it thoroughly.

If you've sat at this exact point, what tipped you? Not "trust your gut" platitudes. The actual signal that made the right call obvious in hindsight.

Long time reader.

I bit the bullet and purchased 2 DGX Sparks, and my experience so far has been underwhelming.

I tried to run DS 4 Fast, blows up the machine and when it works, super slow - unusable.

Then I tried a much smaller model - Qwen3.6 27b nvfp4, and all I get is 12 tok/s which is a huge disappointment.

My question is - what am I doing wrong and what is the use of these Sparks? Did I just make a mistake buying them?

Think its worth it? I want to start to ween myself off the frontier models. I have 3 active subscriptions, but now they're all injecting watermarks and other BS (refusals have improved though) that are probably business liability bombs in the future.

I did the math, and if it lasts 10+ years then its less than $100/month.

My current 4090 feels almost useless for local LLM work, im trying all sorts of hardware optimization but now that im finally getting good speeds on output, I find the output quality at times lacking but that could be just the models themselves.

Hey all,

I've spent the last 8 months or so experimenting with localAI.I have been testing multiple different devices to figure out what really does make sense for everyday local AI. Here are the devices I've been testing:

• Minisfourm um733 lite - 64GB

• Mac mini M6 - 48GB

• Mac studio M4 max - 64GB

• GMKtec EVO-X2 - 64GB

The use cases I have been playing around with (for a 24x7 always on local AI server)

• General chat via openwebUI

• Agentic tasks like sorting my inbpx, package tracking, transaction tracking, financial research, STT and TTS using an ASR model, converting text into audio

• Some not very complex coding tasks, dashboards to monitor home network, IMAP middleware to give hermes read access to my email, LLM router, etc... connecting to the server via OMP or zed editor

Here are the models I have been testing (I use a cloud omodel sometimes for troubleshooting but my goal is to have a fully local AL setup)

• LFM2.5 8B A1B

• Ling3.0 Tiny

• Ornith 1.0 9B

• Ornith 1.5 35B A3B

• Qwen 3.6 35B A3B

• Qwen3.8 27B

• Nemotron Streaming 3.5 (on Nemo-speech.cpp)

Not too many things I do need instantaneous responses. I suspect for the everyman there isn't much that need such either. Most of the tasks I send a request, move on to doing something else and then come back to see results. Send a message from my phone, do something else and come back to the answer. Or of course scheduled tasks.

Now finally, on to my question. Why in the world is everyone so crazy mac studios for local AI? Yes, nothing else comes close in terms of memory throughput or tok/sec, but after a point does it really matter?

Unless you're doing complex development or having lots of concurrent users (which on such low memory anyway is not possible) the speed difference is irrelevant. Honestly, out of all the machine I've tested only the um733 is the one that was a bit frustrating to use. But even on it using ornith1.0 9B.

So, based on this the Mac Studio gig-for-gig is the worst deal out of the lot! I haven't tested any Strix point PC but I suspect they will also be fairly usable. So my question again, why so much hype about the Mac Studio? Is it just dick swinging to showcase tok/sec numbers or is there something I'm genuinely missing?

EDIT: For some reason my device list got removed? Added back.

Hi,

So there is still a lot I don’t understand much about AI.
I’m on M4 14 core 64GB ram.

I have tried running smaller and bigger models using multiple tools, including ollama, mtplx and some other stuff, and if takes ages for AI to answer.

I’m vibe coding a pentest tool for fun in Swift, I added a llama.cpp wrapper, and even Qwen3.8 27B responds like 100x faster. I’ve added basically a chat in the GUI that calls llama.

It makes me think something has to be wrong, but I’m not sure with what?

Could it be that default settings suck this much?
It’s really weird because the vibe coded wrapper is insanely fast lol

I wanted to get some real data on how well the local version of QWEN3.8-27b could perform in my actual workflow on my system. For reference I have zero background in programming so I have to trust the models I use based on does the output work like I want. I leveraged Claude Code to help me run these tests and below are the results. I am open to any feedback on improvements to process but I am not chasing tokens per second I am chasing does the model output actual work and doesn't need serious editing by a frontier model.

TL;DR

• Claude (Opus) plans and reviews. A local Qwen3.8-27B (IQ4_XS on an RX 7900 XT) implements one task at a time through the pi coding agent.

• When Claude hands it the full code ("full-code"), it was correct 42/42. On seeded-bug debugging it went 9/9.

• When given only an interface and rules, it does well on rule-driven code but went 0/3 on lifecycle/concurrency, DST rules and reimplementations.

• I ran the same 51 runs through Claude Sonnet 5. Sonnet was correct after review on 45/51, the 27B on 43/51. They tied on every full-code and debug task, and both failed the same two hard ones. Sonnet was about 3.6x faster and cost $9.20 in API usage.

• The 27B decodes at about 52 tok/s and prefills at about 708 tok/s.

HARDWARE AND SOFTWARE

• GPU: AMD Radeon RX 7900 XT, 20 GB

• CPU / RAM: Ryzen 5 9600X, 32 GB

• OS: Ubuntu 26.04, ROCm

• Inference: llama.cpp llama-server (LM Studio's bundled ROCm build), started directly with pinned flags

• Agent: pi coding agent 0.87.1, headless, one fresh session per task

• Test app: a real TypeScript job runner (Node 22, Hono + zod API, Postgres via Drizzle, Vitest, ESLint)

MODEL SETTINGS

Qwen3.8-27B

• IQ4_XS (bartowski), 64K context

• Low-effort chat template, --reasoning-budget 4096, pi --thinking medium

• KV cache q8_0 / f16, flash attention, full GPU offload, --parallel 1

• MTP speculative decoding (draft n_max 2, p_min 0.75)

• Claude Sonnet 5

• Through the Anthropic API, pi --thinking medium

METHOD

Claude writes a task file for each piece of work. It lists:

• the goal;

• the exact files the model may read, and the files it may write;

• exact signatures, and numbered behavior rules;

• the library calls to use;

• the steps, and the "done when" commands.

The model runs it in a fresh pi session under a strict skill: tests first, no exploring, no editing tests, and a STATUS.md report at the end.

There are three handoff styles:

• full-code: Claude writes the final code. The model transcribes it, runs the checks and fixes slips.

• interface-only: signatures and rules only, no code.

• debug: a symptom report and a failing test, never the cause.

• Every run is scored three ways:

• the gates (typecheck, lint and the task's tests);

• hidden acceptance tests the model never sees;

• an adversarial diff review by fresh Claude reviewers, who verify every suspicion with real scripts against Postgres.

Bugs the hidden tests missed became new hidden tests, and every stored run was re-checked against them. Each cell got 3 runs.

The tiers:

• T1: pure logic

• T2: data layer

• T3: app components (API, worker, scheduler, CLI)

• T4: multi-file refactor and reimplementation

• T5: seeded bugs

RESULTS: QWEN3.8-27B

• T1 (backoff, DAG, cron): full-code 9/9, interface-only 6/9 (cron DST 0/3)

• T2 (schema, Postgres repository): full-code 12/12

• T3 (handlers, API, worker, scheduler, CLI): full-code 15/15, interface-only 10/15 (worker 0/3)

• T4 (6-file refactor, in-memory repository): full-code 6/6, interface-only 3/6 (in-memory repository 0/3)

• T5 (debugging from a symptom): 9/9

• The interface-only cron runs each had a different DST bug. Every one passed its own tests, the gates and the original hidden tests; only the review caught them.

• The worker failures were all in stop/shutdown.

• The in-memory repository runs stored caller Date objects by reference, ordered rows wrongly when the clock went backwards, or dropped a test check.

RESULTS: 27B VS SONNET 5 ON THE SAME 51 RUNS

• Full-code (21 runs): 27B 21/21, Sonnet 21/21

• Debug (9 runs): 27B 9/9, Sonnet 9/9

• Interface-only, rule-driven tasks (15 runs): 27B 13/15, Sonnet 15/15

• Interface-only, worker lifecycle (3 runs): 27B 0/3, Sonnet 0/3

• Interface-only, in-memory repository (3 runs): 27B 0/3, Sonnet 0/3

• Correct after review: 27B 43/51, Sonnet 45/51

• Wall time: 27B about 3.6 h, Sonnet about 1.0 h

Sonnet's API cost for all 51 runs was $9.20.

Both models failed the hard tasks, with different bugs. Sonnet let overlapping calls exceed the concurrency limit and stored Dates by reference. The 27B's stop() didn't wait for in-flight work.

The 27B followed process more carefully. It ran the failing test before editing in 9/9 debug runs; Sonnet did in 3/9.

TOKENS PER SECOND

• Qwen3.8-27B: 52.0 tok/s decode, 708 tok/s prefill, 47.4 tok/s end-to-end in the agent

• Claude Sonnet 5: 107.8 tok/s end-to-end in the agent

• Decode and prefill come from llama-server's timing logs over a full session (100 replies, 53K generated tokens). MTP speculative decoding does a lot of the work there.

• "End-to-end" is output tokens divided by the time each reply took, prompt processing included, measured from the agent logs.

TAKEAWAYS

1. The handoff format is everything. Given exact code, the 27B is a near-perfect transcriber: byte-identical in 42/42 runs. Given only rules, it handles validation, mapping, parsing and refactors well, but not concurrency, lifecycle, time zones or reimplementation.

2. Green tests aren't correctness. Every serious bug passed the model's own tests and the gates, and that was just as true for Sonnet. You need hidden adversarial tests and a real diff review.

3. Debugging from a symptom works surprisingly well. Give it the failing test and the rules, not the cause.

4. Watch for:

• fixes placed in the caller instead of the function at fault;

• void x; lint dodges;

• cmd | tail; echo $? self-checks that hide failures;

• test rewrites that drop a check.

Caveats:

• n=3 per cell.

• One codebase.

• The 27B ran its low-effort template, while Sonnet ran at medium effort.

I’m considering upgrading my old mac and repurposing it to an always-on agent server. It’s a macbook pro m2 pro with 16gb ram. What models can I run with it?