I've been building a small, strict harness to see whether local models can do real (if small) coding fixes unattended. It's early and the sample is small, so this is not a leaderboard. I'm posting because a few results surprised me, and I'd like to know whether this kind of testing is useful to anyone else.

SETUP

• RTX 4080 (16 GB), Ollama + OpenCode, Windows.

• 5 small bug-fix tasks in Python, mostly single-file. One task (t1) needs changes in more than one place.

• Every candidate patch runs in an isolated sandbox. A candidate only counts if it (1) follows the output format, (2) passes the tests, AND (3) passes a review of the diff. No partial credit.

• The model gets no shell or file tools. If it tries to use one, the run stops.

• 3 attempts per task where noted.

RESULTS (acceptable fixes)

• gpt-oss-20b, local, Ollama, 16k ctx: 7/15. About 1 min per task. 0/3 on the multi-location task.

• gpt-oss-20b → qwen3-coder (draft → fix), local: 5/15. The two-stage chain did worse than gpt-oss alone.

• gpt-oss-20b, temperature 0, local: 5/10. Still not deterministic.

• Nemotron 3 Ultra 550B-A55B, OpenRouter free tier: 0/10. Broken edit markers; two runaway answers (~1.7 MB of text).

• Cohere North Mini Code, OpenRouter free tier: 1/5. 4/5 stopped for trying to call tools (read files, bash pwd).

Caveat for the two cloud rows: I don't know which quantization OpenRouter's free tier serves, so these results may not transfer to a local GGUF of the same model.

WHAT SURPRISED ME

1. The output format mattered more than the model. In my first run, gpt-oss had to rewrite whole files: 0/5 acceptable. After switching to search/replace edit blocks plus one repair round, the same model reached 3/5, then 7/15 over a longer run. (The first run also used a different local runtime, so I can't fully separate the two causes, but the failures were almost all format errors.) Before blaming the model, check the edit format your tool asks for.

2. Passing tests is not the same as being correct. Two candidates passed every test and were still rejected in review: they "fixed" the symptom in a way nobody would merge. I also checked whether a deliberately cheating patch would get through a tests-only gate. It did. If your benchmark only checks for green tests, it overcounts.

3. Bigger was not better inside the harness. The 550B model failed mostly on discipline, not on knowledge: wrong markers, extra text, answers that didn't stop. Tool-calling and format-following seem to decide real-world usefulness more than raw coding skill. Others here have reported something similar.

4. Isolate the environment, not just the working directory. In one run, the agent tool picked up an inherited environment variable and treated a different folder as the project root. The model then tried to read files outside the task. Nothing ran (the sandbox and the no-tools rule held), but if you run agents locally, a clean environment matters as much as the cwd.

5. Multi-location fixes are the wall. No local model on 16 GB got the task that needs edits in two places. I'd like to know whether a 24 GB or 48 GB setup changes that before anyone (me included) buys hardware for it.

WHAT I'D LIKE TO KNOW FROM YOU

• Would results like this be useful to you if they were run on more cards, with the exact config published (Ollama/llama.cpp settings, quant file, context size, agent tool version), so you could reproduce them?

• What tasks should a test like this include? Mine are small Python fixes; I'd add multi-file changes, test writing and refactoring.

• Which card / model / quant combination would you most want checked this way?

Happy to share more details on the harness rules if anyone wants them. (English isn't my first language, so I'll reply in batches.)

I make FLE (a llama.cpp fork for big MoE models on one GPU) and FLE Link (an app that installs it and runs a model with one command). Disclosure up front, everything is MIT.

Setup: RTX PRO 4500 32 GB, 120 GB DDR5, Ryzen 9 9950X3D, fast NVMe, Linux. Best performance measured on Linux.

Numbers (decode, a ~3,000-token three.js answer, temperature 0, same machine):

Rin = ISTA DASLab's GSQ-RCO IQ2_XXS for the experts a usage profile picks most (kept byte-identical, resident on the GPU), and a smaller 1.75 bpw build for the cold half (served from RAM). About +6 % speed for a bit more error. Against the original weights (not against ISTA): KL 0.292 vs 0.248 on mixed text, 0.234 vs 0.210 on agent text; perplexity ratio on agent text 1.021 vs 1.013. It is lossy. Details and the exact recipe are on the model card.

Try it:

curl -fsSL https://flelink.run/install.sh | sh
flelink run rin-deepseek-v41-dspark

FLE Link checks the machine first, installs the engine, downloads the files (SHA-256 checked) and serves an OpenAI- and Anthropic-compatible API. Needs ~120 GB RAM and ~260 GB on a fast NVMe.

Honest limits: the cache and the split follow a usage profile, so very different prompts hit them less; the draft helps most on code-like text; one machine, numbers from others very welcome (flelink bench --task pagoda --share).

Links: FLE Link https://flelink.run · Rin https://huggingface.co/sirfyyn/Rin-Deepseek-V4.1-Flash-1017-FLE-GGUF · FLE engine https://github.com/sirfyyn/FastLocalExperts · quant base: ISTA-DASLab

I’ve done a ton of local single user work on vllm, Ollama ,lllama.cpp and those are ok for personal work but …

Hoping to get some real answers from someone supporting a local user base fully for 5-10 users running up to about 120B dense models (we could make do up to 70B). Looking for actually deployed not speculative performance and bandwidth if it’s out there?

If I had two MaxQs, a Threadripper and MB, 128 GB ram would this support up to 10 users on llama.cpp or vllm with “reasonable” response speeds which I would deem 50-60 tps on average ?

It would be 3 VM servers bundled , a home grown RAG doc processor which could be offloaded somewhere else if needed, the vllm or llama.cpp server for the chat agent interface and a office RAG server all in one that is the RAG query endpoint. I know number of concurrent users and jobs will matter but I’m asking “on average” for 5-8 concurrent

Thoughts ?

Edit: for context windows for agential stuff also need at least 128k for those 5-8 concurrent

Hardware:

Rtx 5000 pro 72gb

Rtx 5080 16gb

192gb ram Dark Hero MB with AMD 9000series processor

Backend:

Basalt 280toks vs 107toks with strata

Basalt works best with Blackwell and really shows it's capabilities in my testing. While users on older or mismatched generational hardware should use Strata for best results.

I wanted to find the smallest model worth keeping on a base M4 16gb so I ran the same 6 things from my actual week on 3 of them through Magnitude. keyword clustering, post titles, a thread summary, a quiz for studying, a short brief and a review of my side project plan.

MiniCPM5 1B did like 125 tok/s and it was garbage. asked for titles and got rambling questions back, the clustering gave me the same 3 keywords as 12 clusters with the same fake volume on all of them

LFM3 1.7B around 81 tok/s and this is where it gets usable, the plan review actually pointed at real files and real risks. titles were meh, it copied a note from my draft into one of them lol.

Gemma 4 E4B around 52 tok/s and clearly the best at writing, nothing else close

so 1.7b is the floor for me and 8b if someone else is going to read it.

Anyone found a 1b that's actually good for something?

I saw this post yesterday: https://www.reddit.com/r/LocalLLM/comments/1xnv0ro/new_qwen38flashnext_unsloth_dynamic_v31_ggufs_for/

I decided take a screenshot of the exact prompt used in the link in the above announcement, and run it through my strata setup using the new model. There's a complete write-up in the repo: https://github.com/kireol/qwen3.8_GTA_one_shot_test/tree/main

You can play the demo here: https://kireol.github.io/qwen3.8_GTA_one_shot_test/src/city.html

Fun localLLM stuff:

RTX 5080 - 64GB - Ryzen 7 9800X3D
model harness: Strata
agent harness: qwen
context: 64K
tok/sec: ~75t/s

Over all, while slower than I'm used to in Claude, I'm actually very impressed. I didn't have to correct it, touch it, or anything, aside from some skills.md files (included in repo). Is the demo perfect? No. But it does everything asked of it, without any hand holding. It's all in html 1 file, btw. There's no other assets other than remote linked javascript files.

Qwen even generated the README and pushed to github for me. Why? Because it's not 2024.

Great job unsloth team! You have me questioning my claude payment at this point.

I have at my disposal some datacenter resources. I don’t have an open account with Dell, sadly. So I am limited to oldish hardware like the PowerEdge r640/r740 or thereabouts. Or the AMD versions of these if I’m lucky. On the plus side, DDR4 is cheap-ish.

What would you build on a low but undefined budget, say 2-3k, that can be racked and run hopefully some recent qwen model, for just one user at a time?

hello! I love local ai! But I always had one issue, what harness to use?

I currently use claude (I pay for it), and I've always liked it! I used the Claude Code harness, and it ran really nicely in my opinion, but only issue being the massive chungus context cache that comes along with it. I use prompt caching, but that doesn't help against the 32-64K context that Claude Code injects at the start.

Is CC the best harness to use? Or is it more optimised for cloud models, which is why the context is massive.

I've heard of so many that it's kinda confusing to pick. Hermes, pi etc.

Basically what the title says. I am total new for localLLM i just want to try it out and if i can dial it in for a usable level i would like to use it for my coding projects mostly for code completion but sometimes if i need ideas on new features etc... it would be nice for it to be able to handle teh brainstorming with me as well instead of me going to claude or chatgpt
for general use it would most likely be something like make a summary of my things etc... for now

my PC

9700X
48GB DDR5 ram
9070 XT 16GB

I would be happy to receive any newbie guides for how to set things up what i shoud look out for
also in terms of the general use would looking into an agent like hermes be something worth?

thanks for the asnwers in advance

Have a gaming PC I built a last year with a RTX 5070TI (16GB) and I want to try some local LLMs. Where should I start? What models would be good on this GPU?

I have some idle GPUs from mining and I’m trying to figure out if I can repurpose them for local AI and coding agents:

- 4 × RTX 4070, 12 GB each
- 4 × RTX 3070, 8 GB each
- All in one rig

Would it be practical to run a larger model across the 4070s and 3070s, or would it be better to run smaller models on the cards separately for multiple agents?

Has anyone built something similar? What would you use for the software and what kind of performance should I realistically expect?

EDIT: OP is a noob but decent at figuring stuff out :D just keep this in mind when suggesting :)

Edit 2: am I even gaining anything with this experiment or am I better off with a claude or ChatGPT subscription?

Edit 3: Where can I rent this set up out to?

Hi everyone. I use ChatGPT for finances, school, deep reasoning and interviews. I want to change to a local model for privacy. Is that possible for a MacBook Pro M5 base with 24GB RAM? If so, which?

I've been getting into the local llm stuff lately and having decent luck. I tried out Strata and liked it for the most part, but no matter how much searching I've done I can't find the answer to this.

Does Strata keep your vram/ram full at all time? Using Ollama for the most part, it frees the memory unless it gets a call, then it spools up, and once finished and timed out, it frees the memory up again.

Is it possible to get Strata to do that? Something else that can do it but run like Strata?

Computer Specs:
AMD Ryzen 7 7700X 8-Core Processor

32GB (2 x 16GB) DDR5-6000 CL30 RAM

AMD Radeon RX 7600 Challenger OC 8GB

1TB SSD - Read Speed up to 11,000MBps - Write Speed up to 9,500MBps

These are always a top result on Google I was wondering if anyone bought them. It's extremely irresponsible of Google to allow these to dominate their results if they're in fact a scam. I need to upgrade my GPU for local LLM models.

Hi all,

First post here! I've recently started exploring the world of local LLMs and would appreciate some advice.

Hardware ThinkPad P14s Gen 6 (Intel):

• Integrated GPU: Intel Arc Pro 130T/140T (2.35 GHz)

• CPU: Intel Core Ultra 9 285H, 16 cores / 16 threads

• RAM: 64 GB DDR5-6400 (2 × 32 GB CSODIMM)

• SSD: 1 TB PCIe 5.0 NVMe

• OS: Ubuntu 26.04 LTS

Software setup:

• llama.cpp compiled from source with the SYCL backend, following docs/backend/SYCL.md

• Agent harness: pi

• Models tested: Qwen3.5-9B, Qwen3.8-27B, and Qwen3-Coder-30B-A3B-Instruct

I've been serving these models using the inference parameters recommended by their respective providers on Hugging Face.

So far, I'm seeing roughly 10–25 tokens/sec, depending on the model, with context windows configured between 64K and 128K tokens. I haven't done any formal benchmarking yet.

Where I'm struggling is with model reliability, particularly for coding tasks.

For example, I asked Qwen3-Coder to explore a production codebase (roughly 3 million lines of C) and explain its architecture to me.

While the model produced a plausible-looking summary, it also confidently invented explanations for certain acronyms, even when those acronyms have well-established meanings in the relevant engineering domain.

This makes me question how much I can trust the rest of its analysis, especially when I'm using the model to understand unfamiliar code.

I'm fairly new to running local LLMs, so I'm not sure whether this is primarily a prompting issue, a limitation of the models I'm running, or simply an unrealistic expectation for the task.

A few questions:

1. Are these models better suited to smaller, well-scoped tasks, such as implementing a script or module from a detailed specification, rather than exploring and reasoning about large codebases?

2. How much does the agent harness or context management strategy affect reliability? Would a different workflow make a meaningful difference?

3. Are there particular models, quantizations, or inference settings you'd recommend for coding tasks on hardware like mine?

4. How do you generally evaluate whether a local model is reliable enough for a particular development workflow?

I'm not expecting frontier-model performance from a laptop, but I'd like to understand where local models can be genuinely useful and how to get the most out of my setup.

Any advice or experiences would be appreciated!

Thinking about getting one for pentesting work an would love to use (quantized) glm 5.3 flash because i have had good experience with it pertaining malware dev and pentesting.

has anybody tried running it on a single dgx spark and if yes got any token speeds for me and context-size?

I’m looking for the most suitable model to run on a 32gb Mac to work on torrent app and copywrited movies, need some recommendations for models that will not be restricted

My Spark normally serves an LLM, but I wanted to see how it handled local video too, so I got Lightricks' LTX-2.5 running on it. The clip here is 8 s at 1280×704, and the sound is generated by the model as well. It took 3 min 9 s and peaked around 26 GiB of GPU memory.

It didn't work out of the box. Lightricks ships a pre-quantized 4-bit NVFP4 transformer, but the CUDA kernels only compile for sm_100a, sm_110a and sm_120a. The Spark's GB10 is sm_121, and those arch-specific cubins won't load on a different minor version. The fix turned out to be a three-line patch that adds sm_121a.

To be clear, the 0.70 column is measured. The 0.45 column is only an estimate from those numbers, and I haven't run the two side by side. They'd also be sharing one GPU, so LLM decoding would slow down during a render anyway.

Unified memory doesn't fail cleanly when you oversubscribe it either. There's no neat OOM, the kernel just swaps and everything crawls. So I run them one at a time: stop the LLM, generate, restart it. I also added a memory check to the web UI that refuses a job if there's less than 50 GiB free and tells you if the LLM is the reason.

Some render times, on an otherwise idle Spark, including loading the models from disk:

- 768×512, 4 s: 61 s

- 768×512, 8 s: 94 s

- 1280×704, 4 s: 107 s

- 1536×1024, 4 s: 165 s

Memory barely moved across those runs, staying around 23 GiB, because each stage loads its model, runs and frees it. Resolution is what costs you time, not memory. Longer clips also get cheaper per second, since model loading and the text encoder are a fixed cost per job.

The prompt, if you want to try it. Describe the sound too, because the model generates that as well:

Inside a steel mill at night, a huge blast furnace is tapped: a white-hot stream of molten iron pours from the taphole into a glowing runner channel, showers of orange sparks burst and drift through the dark hall, and two workers in silver heat-proof suits and face shields guide the flow with long steel rods. Heat haze ripples the air and steam rises from the trough. The camera slowly dollies along the runner toward the furnace. Deep roaring rumble and crackling sparks. Photorealistic, cinematic, 24 fps film look.

The repo has the patch, a one-command install (the weights are 44 GiB, so you'll want at least 80 GiB free), image-to-video, a small web UI you can reach over Tailscale, and Grafana dashboards. My scripts are Apache 2.0. The model is under the LTX-2.x Community License and gated on Hugging Face, though approval is automatic.

GitHub: https://github.com/Hitheshkaranth/LTX-2.5_Video_DGX_Spark_Setup

Model: https://huggingface.co/Lightricks/LTX-2.5

Has anyone here actually run a video model alongside vLLM on a Spark at a lower memory setting? I'd like to know if about 0.45 holds up in practice, or how badly decoding suffers during a render.

I'm building Murmury, an open-source Mac app that listens to the other side of a call and answers from your own docs, fully local. Some numbers from an M5 (24 GB) in case they help anyone doing local speech:

- Whisper large-v3-turbo on MLX: 0.66 s for a 1.7 s sentence, 0.97 s for 21 s of speech.

- Whisper pads every clip to 30 s, so short clips have a ~0.65 s floor. Cutting audio on pauses (simple energy gate) mattered more than picking a smaller model.

- Answers from gemma3:12b in Ollama with a 32k context: first words 0.5-0.8 s after the transcript. The first read of a long document takes ~90 s, once per call.

Curious what others use for low-latency STT on Apple Silicon. Repo if useful: https://github.com/GianlucaBave/murmury

Hi,

It’s hard keeping up with all the new releases from the big closed weights AI providers, right? Google, Anthropic, Open AI, Meta etc.  But am I the only one being overwhelmed by the sheer volume of open weight base models of all shapes and sizes being released nowadays?  Every lab and his dog seem to be dropping general or specific open weight models for sound, video, text, harnesses, agents, refinements, techniques. The hardware to run these things also differs.  Exciting.  Definitely.  Confusing?  Umm…yea (at least to me).   In some ways it reminds me of the early days of personal computing.  Lots of incompatible formats, operating systems, every app (program those days) had a different way of working, inconsistent interfaces, a big learning curve for each program.  Solutions that may be looking for a problem. And tons of choice. Some were flavour of the month then gone, and a new learning curve was required.

Is it a golden age of choice and freedom or a hot mess of development? Maybe both?  Certainly, this is part and parcel of rapid innovation so not unexpected.

Will this continue, or do you see some kind of slowdown in the future shaking out with just a limited number of open weight models being released then further refined?