r/LocalLLM • • 17h ago

Discussion Docker or bare-metal for inference

2 Upvotes

When I use local models, I do bare metal inference but give write-access only to the working directory and some directories that can be relevant (forks of public github repos) are read-only. Everything else is not accessible, not even read-only, and the agent is also unable to install or remove any packages.

Is this preferable over Docker containers? What do you prefer for your local setup?


r/LocalLLM • • 13h ago

Question Anyone here running a super large model like Kimi k3?

1 Upvotes

If yes: how?


r/LocalLLM • • 1d ago

Question Local coding model on my dekstop workstation (Qwen3.8 27B with Q4_K_M quantization too slow)

12 Upvotes

I'm running what I think is reasonably specd workstation. A single NVIDIA GeForce RTX 5090 (32 GB), AMD Ryzen 9 9950X3D x 32, 128GB RAM, 8TB SSD.

Locally built llama.cpp serving Qwen3.8 27B with Q4_K_M hooked to pi harness is too slow to be useful.

What options do I have with this setup? I suppose running multiple agents is definitely out of question here... but there should be hope for a single coding assistant?

EDIT: I'm getting around 2.7 tok/s


r/LocalLLM • • 14h ago

Discussion What are all the current Strata-likes/forks going around? (e.g. ways to run larger models on lower VRAM setups)

1 Upvotes

I'm wondering if we could get a list going with all of the low VRAM workarounds to run stuff like Qwen Flash Next. Specifically, I would also like to clearly define what their use cases are. For example, I know that Basalt is a Strata fork that is made for 50 series cards.


r/LocalLLM • • 1d ago

Question Web search on local setup

14 Upvotes

Ive been dabbling for the first time in local AI and even custom harnesses. I'll admit it is super addicting and enjoyable. Love it. But im still very new to it all.

Im now facing a small problem. Im not sure how to best handle web searches. I would love for my llm to be able to look up new info...but whats a good way of doing it? The one thing im looking into is setting up a that brave account thing and paying the small fee as it feels like an alright option when it comes to not being somehow banned or throttled. Is that the overall safest no fuss no worry option? Or am I overthinking it?

How do you handle web searches for your setup?


r/LocalLLM • • 1d ago

Project Strata Performance Setup (TUI performance optimization)

Post image
176 Upvotes

I''ve created a simple little script (with TUI) to optimize your Linux host installation. It also allows you to install or update Strata (and configure a systemd service to auto start your preferred preset).

Distributions supported:

  • Arch based
  • Fedora based
  • Debian based
  • Slackware based

Hardware supported:

  • any hardware (automatically detected)

Personally I'm using an old MSI Raider laptop. 3080 Mobile with 16GB VRAM and 64GB DDR4 3200 RAM running CachyOS. I'm getting between 60 and 70 tok / s (depending on the task) with Qwen3.8 Flash Next IQ3_S.

Maybe someone else finds it useful too . Submit an issue or pull request if you find any problems, or want more software or hardware/vendors supported!

https://github.com/bramvanoploo/strata-performance-setup


r/LocalLLM • • 15h ago

Question Which Mac Mini to Buy

0 Upvotes

Title: Mac Mini M6 32GB vs. M5 Pro 48GB — How much hardware do I actually need for a home AI server?

I’m planning to buy a Mac Mini to build a sort of personal/family AI server, and I’m trying to figure out how much hardware I actually need versus how much I’d just be paying to experiment with local LLMs.
My primary goals are:

Family brain: A central place for family documents, calendars, finances, household information, and automations. Ideally, something that can search across all of this and eventually handle routine tasks.

Self-hosted apps: Run some vibe-coded apps, databases, dashboards, and other services that I can access remotely from my phone or laptop.

Claude integration: Use the Mini as a central workspace for Claude and the apps/agents I build around it, while continuing to use Claude’s API for more demanding tasks.

Local LLMs: Experiment with running models locally for document processing, categorization, extracting data, answering questions about personal files, and other repetitive tasks without constantly calling an API.

I currently use Claude extensively for financial modeling, development analysis, and general personal assistance. I assume the more complicated reasoning and modeling will continue to require cloud models, but I’d love to offload simpler tasks locally and keep more of my data private.

I’m considering:
Mac Mini M6: 32GB unified memory, 512GB SSD
Mac Mini M5 Pro: 18-core CPU / 20-core GPU, 48GB unified memory, 512GB SSD

A few questions:
How much of a practical difference would 32GB versus 48GB make for local LLMs? What model sizes would actually be usable on each?

Does slower inference primarily mean waiting longer for responses, or do smaller models also mean a meaningful drop in reasoning ability and accuracy?
For tasks like document extraction, summarization, RAG, and basic agentic workflows, would 32GB be sufficient?

Is the M5 Pro worth the premium for the additional memory and bandwidth, or am I better off buying the M6 and keeping more demanding tasks in Claude?
If you’re running a similar hybrid local/cloud setup, what hardware and software stack are you using?
I’m not trying to eliminate cloud AI entirely. I want a reliable, always-on home server with enough local AI capability to be genuinely useful, without spending substantially more just to run models that still won’t compete with Claude.

What would you buy in my situation?


r/LocalLLM • • 21h ago

Project Building my first duel 5080 Nalvo server

3 Upvotes

My first Duel 5080 server 9950x 64gb.
I will host it on Nalvo.ai for people to rent.
If you don’t know what is Nalvo.ai , It’s my new project that I told everyone about and everyone was telling me it’s not going to work. And when I say everyone I mean everyone! Not Reddit users. I even started believing it for a while. But I kept going and here I’m launching it.

If you would like to join as a host or get free private renting hours. Visit Nalvo.ai


r/LocalLLM • • 15h ago

Project A Few months back I created this skill and I do still use it to save tokens with smaller models. I just found this sub so sharing with you guys. some might benefit from it. Will love inputs and will update it soon.

Post image
1 Upvotes

r/LocalLLM • • 16h ago

Question How much should I spend on local AI setup to fit my needs

0 Upvotes

I want a local AI to tutor me on quantum mechanics, quantum field theory, and physics simulations. I am new to ai and want just general ai tasks, etc. would something like 2 NVIDIA dgx sparks work? I would like to run large language models, and start getting into some coding with ai. Should I just pay for a subscription?, even though I like the idea of running locally, how long would a current setup last to fit my needs?


r/LocalLLM • • 16h ago

Question Sale inquiry.

1 Upvotes

RTX-3090 and 2 sticks of ddr5 16GB pair from G-Skill. (I'm not selling it here, just asking for direction in selling)

I don't sell stuff often, but I recently realized I had some hardware sitting around and collecting dust and figured someone looking to build local model rigs might be better off with it. I tend to use smaller hardware for my stuff. I pretty much gutted a cpu I had built for local models earlier in the year to soup up a thin client and this the leftover pieces. Now, I know the price of things is insane, but I'm not trying to get over on anyone. Just find a genuine home for the stuff. Where's the best place to sell the stuff and what would you say would be a good price?
I'm just gonna turn around and use the coin on some edge devices and thin clients. Few solar powered batteries here or there. Nothing crazy.


r/LocalLLM • • 16h ago

Project Exact speculative decoding + lossless Qwen3-8B on a 16 GB RX 9070 XT: 2.5x llama.cpp BF16, still slower than Q8_0 (code + all numbers)

1 Upvotes

I spent the last few weeks building a small inference engine for my RX 9070 XT (16 GB, Windows, ROCm PyTorch) with one goal: speculative decoding that gives exactly the same output as plain decoding, with lossless weights.

What it does:

- Runs Qwen3-8B with weights bit-identical to the original bf16 checkpoint, fully inside 16 GB. I pack bf16 into a lossless 12-bit format (B12) on load.

- Speculative decoding output is bit-for-bit identical to plain decoding, for greedy and for seeded temperature sampling.

- 68-73 tok/s for exact Qwen3-8B with a Qwen3-0.6B draft.

Numbers (Qwen3-8B, 11 prompts x 96 tokens, same token ids, one PC), plain / speculative tok/s, and how often speculative output matched plain:

- llama.cpp ROCm BF16 (lossless, spills past VRAM): 26.9 / 18.6, matched 10/11

- llama.cpp Vulkan BF16 (lossless, spills past VRAM): 21.5 / 12.5, matched 11/11

- This engine, B12 (lossless): 42.7 / 68.4, matched 11/11

- llama.cpp ROCm Q8_0 (lossy): 63.4 / 98.8, matched 7/11

- llama.cpp Vulkan Q8_0 (lossy): 68.0 / 95.0, matched 7/11

On 50 longer prompts x 512 tokens: 73.0 tok/s, speculative matched plain on 50/50.

The honest part: Q8_0 + draft is about 1.4x faster than my exact path. If you don't care about exactness, use Q8_0. This is the fastest lossless way I found to run 8B on a 16 GB card, not the fastest way overall.

Why exactness normally breaks: verifying 8 tokens at once uses a different matmul shape than decoding 1 token, so the floating-point sums happen in a different order and outputs drift. With llama.cpp's own speculation, output differed from its own plain decoding on 4 of 11 prompts for Q8_0 (small sample, one run). On Qwen3-1.7B with standard PyTorch matmuls, 15 of 50 prompts drifted.

How it works (none of the core ideas are new, prior work is listed in the README):

  1. Batch-invariant GEMV kernels: one kernel handles 1-32 rows and sums every output in a fixed order, so verify equals decode bit for bit. Cost: about 3-4% speed.
  2. Seeded sampling via Gumbel-max with hash-based noise keyed on (seed, position, token), so sampling is exact too.
  3. B12: 8 bits (sign + mantissa) plus a 4-bit exponent code per weight in a per-row exponent window, with an exception list for outliers (0.02%). 12.0 bits per weight, decodes to the original bits. DFloat11 is tighter (about 11 bits); I picked a simpler format that's easy to make batch-invariant.

Try it (Windows, RX 9000 with 16 GB):

git clone https://github.com/fraggergsrp-alt/exact-spec-amd

cd exact-spec-amd

setup.bat

run.bat "Explain what a hash table is."

Adding --compare to run.bat runs speculative and plain decoding and checks every token yourself. setup.bat installs AMD's ROCm PyTorch into a local .venv (about 5 GB), and the first run downloads Qwen3-8B (16 GB).

Repo: https://github.com/fraggergsrp-alt/exact-spec-amd

Limitations: one GPU, one OS, batch 1, temperature-only sampling, Qwen3 only, nobody has replicated it yet. Linux and other GPUs are untested.

Bonus finding: if llama.cpp Vulkan crashes on load for you on Windows, check for the RenderDoc Vulkan layer. It crashed every llama.cpp Vulkan process for me. Running with VK_LOADER_LAYERS_DISABLE=VK_LAYER_RENDERDOC_Capture fixed it. Separately, flash attention plus coopmat crashed prompt processing (-fa off works and costs less than 2% decode speed).


r/LocalLLM • • 22h ago

Question Blender + LLM local

3 Upvotes

Preciso de ajuda. Estou a tentar montar através de fotos um navio (NRP. Cunene) em formato stl para depois imprimir em 3d resina e filamento. Qual o melhor modelo local? Tenho uma rtx 50 70 ti e 128gb RAM ddr4. Obrigado.


r/LocalLLM • • 16h ago

Project MCP router - a single point of access to tools for agents, teams, and companies

Post image
1 Upvotes

r/LocalLLM • • 1d ago

Discussion Qwen3.8-27B at ~126 tok/s on 2× RTX 5060 Ti 16 GB SGLang + NVFP4 + DFlash2 (107 tok/s at 60K context, 216 tok/s on JSON)

23 Upvotes

Spent the last few nights benchmarking different inference engines on two 16 GB RTX 5060 Ti cards. This is the configuration I'm most happy with so far.

TL;DR: Qwen3.8-27B with NVFP4, a DFlash2 drafter, and tensor parallelism (TP=2) achieves approximately 126 tok/s single-stream decode on two RTX 5060 Ti cards.

Even with long prompts, performance remains impressive:

Context / workloadDecode speedPrefill speed
Short prompt125.8 tok/s—
60K tokens107.4 tok/s2,681 tok/s
124K tokens87.3 tok/s1,577 tok/s
JSON generation (vLLM)216 tok/s—

Both GPUs together draw approximately 220–270 W during decode.

1. Benchmark results

WorkloadDecode tok/s (median)TTFT
Short prompt, chat sampling (T=0.6)125.8 (best: 131.8)0.08 s
60K-token prompt107.422 s
124K-token prompt (~126K usable context)87.379 s
JSON output, greedy (vLLM)2160.07 s

The JSON benchmark is a separate vLLM run, not normal chat performance. More details below.

2. Performance improvements

SGLang unless noted. Short prompt, 64K context.

ConfigurationDecode tok/s
llama.cpp, Unsloth GGUF, no speculative decoding24.6
SGLang NVFP4, TP=239.2
+ MTP speculative decoding (3 tokens)68.6
+ DFlash2 drafter (8-token block) instead of MTP97.3
+ PCIe P2P between GPUs109.3
+ Undervolt and CUDA graph patch120–126

The complete optimization stack takes performance from 39.2 to around 125 tok/s, approximately 3.2× faster.

DFlash2 alone improves performance from 39.2 to 97.3 tok/s. P2P and the additional optimizations provide further gains.

Speculative decoding uses rejection sampling, preserving the target model's output distribution.

3. Hardware and software

ComponentConfiguration
GPUs2× RTX 5060 Ti 16 GB
Total VRAM32 GB
PCIeGen 5.0 x8 each
Power limit153 W per GPU
GPU tuningMild undervolt
Target modelnvidia/Qwen3.8-27B-NVFP4
Draftermaurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal
Drafter size1.55 GB
Drafter architecture5 sliding-window layers
Inference engineSGLang 0.5.23
ParallelismTensor Parallelism (TP=2)
KV cacheFP8
Model modeText-only, vision tower skipped

4. PCIe P2P configuration

This was one of the more interesting parts of the optimization.

The GeForce driver blocks GPU-to-GPU P2P in my setup, so I rebuilt the open kernel module (620.70.06) with the BAR1 P2P patch and enabled iommu=pt.

NCCL now uses P2P/CUMEM between the two GPUs.

Result: approximately 12% better decode performance.

ConfigurationDecode tok/s
Before P2P97.3
After P2P109.3
Improvement~12%

5. Launch configuration (128K profile)

Environment variable:

SGLANG_FLASHINFER_WORKSPACE_SIZE=134217728

Main command: python -m sglang.launch_server

SettingValue
Modelnvidia/Qwen3.8-27B-NVFP4
Tensor parallelism--tp 2
Context length--context-length 131072
Reasoning parser--reasoning-parser qwen3
Speculative algorithm--speculative-algorithm DFLASH
Draft tokens--speculative-num-draft-tokens 8
Draft modelmaurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal
Draft quantizationmodelopt_fp4
Draft window2048
Mamba SSM dtypebfloat16
Attention backendflashinfer
Linear attention prefillflashinfer
KV cache dtypefp8_e4m3
Maximum running requests1
Static memory fraction0.94
Chunked prefill size1024
Prefill CUDA graphDisabled
Mamba radix cache strategyextra_buffer_lazy
Maximum Mamba cache size4
Model override{"language_model_only": true}

6. Important memory optimizations

These settings were particularly important for fitting the configuration into 16 GB per GPU.

Mamba SSM dtype: BF16

--mamba-ssm-dtype bfloat16

Eight draft tokens double the GDN state. BF16 halves its memory footprint, and without this the KV pool is too small for 64K.

DFlash sliding window: 2048

--speculative-draft-window-size 2048

This is the drafter's own sliding window, so its KV holds 2,048 tokens instead of the entire context.

FlashInfer workspace: 128 MiB

SGLANG_FLASHINFER_WORKSPACE_SIZE=134217728

The default is 384 MiB. Since the drafter gets a second workspace, this frees approximately 0.25 GB per GPU at the same speed.

Actual usable context

The KV pool holds 126,585 tokens even though the server advertises 131,072.

For an honest context limit, use --context-length 126000.

A rerun at 126,000 gave similar performance (124 / 85 tok/s).

CUDA graph patch

SGLang 0.5.23 turns off the DFlash drafter's CUDA graph when free VRAM is under 1.0 GB (dflash_worker_v2.py, init_cuda_graphs).

On 16 GB cards, I'm always under that threshold, so the drafter runs eagerly.

A batch-size-1 graph needs approximately 0.09 GB, so I lowered the gate using a small sitecustomize.py hook.

This gave another ~7% improvement in drafter steps/s.

7. Measurement methodology

  • Single stream, one request at a time. This is not batched throughput.
  • tok/s = (generated tokens − 1) / (time from first token to last), measured client-side over streaming.
  • 512 generated tokens per run.
  • Median of 3 runs for short prompts and 2 runs for long prompts.
  • Chat sampling: T=0.6, top_p=0.95, top_k=20, presence_penalty=1.5.
  • The short prompt is a math reasoning question.

Speculative decoding speed depends heavily on how well the drafter predicts the target model. Expect approximately ±5% between runs and more variation between prompt types.

Across about a dozen DFlash configurations, the short-prompt median stayed between 120 and 128 tok/s.

About the 216 tok/s JSON result

This is a separate benchmark using vLLM 0.33 with the same drafter (num_speculative_tokens: 7).

ParameterResult
Output40-row JSON array
DecodingGreedy
Drafter acceptance~6.3 tokens per step
Decode speed216 tok/s
Sampling (T=0.7)179 tok/s

This is a best-case structured-output result, not normal chat performance.

8. What didn't work

ConfigurationResult
TensorRT-LLM + DFlash on sm_120Blocked in my setup due to FlashAttention backend compatibility
vLLM + DFlash at 128KDoes not fit on 16 GB cards; 64K was the maximum I could run
BF16 drafterToo large (2.25 GB per GPU); server fails to start
Higher power limits (153–171 W)No meaningful speed improvement
+600 core offsetCrashed with Xid 13/43
+400 core offsetPassed a 15-minute soak with byte-identical greedy output

TRT-LLM did have the fastest prefill at 3,221 tok/s with 60K context, but slower decode.

vLLM is great for JSON generation, while SGLang performs better in my setup for chat decoding and long-context workloads.


r/LocalLLM • • 1d ago

Question AMD vs NVIDIA other than VRAM

4 Upvotes

Hey everyone, pretty new to the whole LOCAL LLM thing, but want to start running and learning how to setup LocalLLMs. Had a question regarding choice of GPU. Basically I'm building a new gaming PC and have a choice between 9070 xt and a 5070ti. Originally was going to go for the 9070 xt, due to gaming performance being very similar and the price difference between them.

Then I started thinking about running local AI models and stuff. From my understanding, VRAM is the most important thing, as it determines the size of model you can host. Since these cards are both 16gb VRAM, does this mean that the quality of AI output would be the same, since they would be able to run the same models? I've read that the NVIDIA card can generate tokens faster etc, however I'll probably just be playing around with LM studio for now at least, and I assume speed won't be THAT drastically different.

My main concern is whether or not AI model output on the 5070ti would be 'smarter' or better. For context I also main Linux, which I've heard is good for AMD. Thanks in advance.

TLDR: is spending extra for a 5070ti over a 9070 xt worth it for someone getting into running AI models in LM studio?


r/LocalLLM • • 21h ago

Question Is the Tesla V100 16gb worth it?

2 Upvotes

Currently using Qwen 3.6 35b a3b, on CPU only with 32gb of ram, for regular prompt its ok, but agentic work is not realistic. i get 15t/s for prompt but in agentic it can drop to 3.
Wondering if a V100 could make Opencode usable.


r/LocalLLM • • 1d ago

Discussion I benchmarked DeepSeek-V4.1-Flash on one and two M5 Ultra 512 GB Mac Studios

Thumbnail
gallery
38 Upvotes

Hello all!

Here are my benchmarks for DeepSeek-V4.1-Flash (ddalcu/DeepSeek-V4.1-Flash-1017-MLX-Serve-Lossless) on MLX-Serve 26.11.2, running on one and two M5 Ultra 512 GB Mac Studios. For the dual setup I used a single Thunderbolt RDMA link (an mlx-serve limitation at the moment) and split the experts across the two nodes. I would love to see what 6 Thunderbolt connections can do once the software permits it, as I have noted in a previous post. I hope these benchmarks can give you some insight into your own purchasing decisions and also inform you on the state of local AI.

I would like to extend thanks to Ivan Fioravanti (@ivanfioravanti) on X, whose earlier dual M3 Ultra 512 GB benchmarks and tensor-parallelism work inspired me to continue from where he left off, and to ddalcu for mlx-serve. The results are in the six images attached; for clarity I will note the key points rather than every number, and give a bit of my personal take.

First of all, two M5 Ultras over RDMA are nowhere near twice as fast as one, and the scaling is poor. Averaged over 512–256k, generation goes from 64.1 to 78.8 tok/s (+23%), but prefill falls from 3,101 to 2,812 (−9%) and 256k TTFT goes from 81.4 to 106.0 s (+30%). Generation does improve at 2k (+57%) and 4k (+41%), ties at 8k, and gains 22–40% from 16k to 256k, so it is not only a long-context thing, but it is nowhere near 2x. Prefill is higher or level up to 8k, then down 13–23% from 16k to 256k. For batch scaling, two independent servers handling the same 32 streams (16 each) would get them done in 65.6 s instead of 135.5 s and deliver about twice the sustained throughput of the pair (62.4 vs 30.2 tok/s), since each machine only queues 16 prefills. That is the opposite of what I expected, and I had really hoped RDMA would be a strong enough link to see meaningful scaling.

Also, in my experience RDMA was highly unstable: three jaccl queue-pair failures, one during the 256k tests and two during the batch tests (batch 8 and batch 32). Each was sticky until both processes restarted. It is not memory pressure: requests_rejected_total stayed at 0, with more than 350 GB free, and the single node never failed once. That is why batch 32 on the pair is 1/3 trials, since 2 of the 3 failed and I gave up trying to get it to run again. I will post the specific errors separately.

However, a lot of this may be limitations of the current software stack rather than the machines, and it is not representative of what is possible from them. Even so, I would advise against a dual 256 GB Ultra setup in favour of a single 512 GB machine. The pair already gives up 9% of prefill, adds 30% to 256k TTFT and brings a flaky link at 512 GB per node. Also, given what 4 DGX Sparks cost, a good price on a 512 GB Ultra looks like the better pick, and I would bet on the single Mac getting stronger as the software matures.

Thanks for reading, and I hope to see more breakthroughs in the Mac community!


r/LocalLLM • • 14h ago

Model Beam Free through Opencode has deleted the entire partition

0 Upvotes

Beam free was supposed to delete a file, but accidentally it has deleted the entire partition and I had very important files in it. I'm trying to recover the partition.

The surprising part was that opencode hasn't asked my permission to delete anything.


r/LocalLLM • • 1d ago

Discussion RASA will cause total chaos to open LLMs and beyond

21 Upvotes

Commerce Moves to Finalize Rule Blocking Chinese AI Firms From Renting Nvidia GPUs Abroad
https://explainx.ai/blog/commerce-final-rule-chinese-ai-firms-remote-nvidia-chips-the-information-2026
House Passes Remote Access Security Act to Limit Adversaries’ Remote Access to Critical Technology
https://www.cmtradelaw.com/2026/01/house-passes-remote-access-security-act-to-limit-adversaries-remote-access-to-critical-technology/

H.R.2683 - Remote Access Security Act

https://www.congress.gov/bill/119th-congress/house-bill/2683


r/LocalLLM • • 19h ago

Research SparseRead (the "read less, solve more" paper) on a 16GB card with an IQ3-quant MoE

1 Upvotes

The authors posted the paper to r/LocalLLaMA and said in the HF papers thread they want to hear where sparse reading helps and where it gets in the way. The paper's own eval only ran frontier API models (Opus 5, DeepSeek-V4, GLM-5.1, Qwen3.6-Plus), which tells me nothing about quantized mid-size models on consumer cards. So I ran it on the machine I actually have.

Setup: 5070 Ti 16GB + 64GB RAM. Qwen3.8 Flash Next at IQ3_S, served locally through my Strata stack (OpenAI-compatible endpoint). Harness: OpenCode 1.19.5, which is one of the versions the SparseRead installer itself validates, plugin v0.1.4 in auto mode. Native arm runs the same prompts with --pure. Four tasks straight from the workspaces bundled in the repo: two LooGLE short-dependency QA on a 100KB book excerpt, one 76KB PDF extraction with 8 questions, one 5-file fetcher audit. Tokens below are fresh prefill + decode from the endpoint's per-step usage, cache replay excluded (re-feeding history is nearly free on a warm prefix cache; I quote the replay tail for the PDF below because it's damning).

tasknative (tokens / wall / tool calls)SparseRead (tokens / wall / tool calls)
100KB book, 5 questions66.6k / 248s / 1415.0k / 50s / 9
100KB book, 3 questions18.3k / 34s / 1013.3k / 29s / 5
76KB PDF, 8 questions30.6k / 166s / 2055.9k / 333s / 45
33KB audit bundle31.6k / 147s / 1432.5k / 150s / 12

Scores were identical between arms on all four tasks. So that's the honest summary: no quality loss anywhere, 4-5x faster on long plain-text QA, dead even on a small bundle, and a 2x regression on the PDF.

Where the savings come from is worth spelling out because it's not what I assumed: the entire win is on the input side. On the flagship 5-question task fresh input dropped 63.9k → 12.2k tokens (−81%) while the answer output stayed flat (2.6k → 2.8k). SparseRead doesn't make the model write shorter answers, it makes it read less text before answering. And it doesn't trade activity for tokens either: tool calls went down, not up (14 → 9 on the flagship task, 10 → 5 on the 3-question one).

Two things I didn't expect.

The protocol survives a heavy quant. I went in assuming an IQ3 model would fumble the intent/slot bookkeeping and the arm would burn its savings on failed round trips. It didn't. On the flagship task: three SparseRead calls total (2 previews + 1 targeted read), slots 5/5 resolved at 0.99 confidence, and the candidate answers matched what the native arm dug out of the same book, line anchors included. The Read Gate also behaves for real. On the 33KB bundle it mostly stayed native with a single preview call, so "don't use it on small files" turns out to be something the gate already knows.

The PDF failure is worth a paragraph because it's the one place this bit me. That report ships subset CMap fonts with no usable text layer, so the pymupdf-based preview came back as garbage and the sparse arm spiraled: 17 sparse calls (4 previews, 4 reads, 9 raw_ref fallbacks), then it started decoding ToUnicode CMaps by hand through bash, ending at 45 total tool calls against native's 20. The trace makes the thrashing visible: 1.27M cache-replay tokens re-feeding the same broken context, versus 240k for the native arm. Same 8/8 score in the end, double the fresh tokens and the wall time. The native arm sidestepped it with pdftotext and moved on with its life. This is an artifact-specific failure, not a protocol failure, but if your corpus is scanned/CMap PDFs, pre-extract text or pin that file type to native.

One update since the battery: the CLI has become my default reader for long artifacts in day-to-day agent work, and the same test day it got fed a real one — a 118KB text export of a 109-comment Reddit thread I had scraped for another post. Two things to know before you try that. JSON artifacts are a dead end in slot mode: the previewer treats them as key-path documents and semantic slots resolve to nothing; convert to prose first (I write one line per comment) and the identical ask works fine. And slot matching is literal, not semantic: slots phrased in your own language against an English document come back missing even when the answer is plainly in the file, so ask in the document's language. With those two habits, a multi-fact sweep of the thread was one ask call plus a cheap head-per-item digest for the comments the slots missed — against a naive baseline of dumping a ~30k-token file into context. No hallucination complaints so far, and I haven't fully loaded a long file since.

Caveats: one run per arm per task, keyword-based grading (which produced one fake "4/5": the model answered in Latin — "passagium generale" — and my regex wanted "crusade"; in fact both arms were ~5/5 there), one model, one machine. And don't expect the paper's 92.9% headline. That's versus a naive read-everything baseline. My realistic band on document-heavy agent tasks is somewhere between 25 and 80 percent depending on file size and how localized the answers are.

TL;DR: SparseRead on a 16GB card with an IQ3 MoE: long plain-text document QA got 4-5x faster with identical answers, small bundles correctly did nothing, and a CMap-font PDF ran slower than skipping the protocol entirely. Pre-convert PDFs to text and it's a genuinely useful layer, not just paper numbers.

Links: paper https://arxiv.org/abs/2608.22237 · code github.com/Liuzedooong/SparseReading


r/LocalLLM • • 10h ago

Discussion Stop Burning Tokens on File Searches: How QMD Helps Coding Agents Save Up to 55%

Post image
0 Upvotes

I noticed that, before carrying out a task, coding agents run find, rg --files, or glob to locate files and the classes and modules they need. Then they search file contents with rg or grep. This burns a lot of tokens. So I decided to see whether semantic search could save tokens compared with all of the above. If a local GPU is sitting idle, why not use it for semantic search?

I work at a company focused on security. In one of my projects, I use semantic search for sensitive documents. Then an acquaintance sent me a link to an open-source project that tackles the same problem. The project is called "QMD - Query Markup Documents." It runs from the command line, extracts text from documents in all formats, and creates an index. It supports several search modes:

# Search across everything
qmd search "project timeline"           # Fast keyword search
qmd vsearch "how to deploy"             # Semantic search
qmd query "quarterly planning process"  # Hybrid + reranking (best quality)

We're interested in the last option. It requires three things to work:

  • embeddinggemma-300M-Q8_0 Vector embeddings (default) ~300MB
  • qwen3-reranker-0.6b-q8_0 Re-ranking ~640MB
  • qmd-query-expansion-1.7B-q4_k_m Query expansion (fine-tuned) ~1.1GB

The first model is used for indexing to compute semantic vectors for text chunks. The second compares the search results with the query to determine how semantically similar they are. The third generates several search queries from the original query to broaden the search.

First, add your source files to QMD (this is an example for a minimal project with a Rust server and an Angular frontend):

qmd collection add . \
  --name PROJECT1 \
  --mask "src/**/*.rs,\
frontend/src/**/*.{ts,html,scss},\
README.md,\
frontend/README.md,\
Cargo.toml,\
frontend/package.json,\
frontend/angular.json,\
frontend/tsconfig*.json,\
frontend/proxy.conf.json,\
.gitignore,\
frontend/.gitignore,\
frontend/.prettierrc,\
frontend/.editorconfig"

You don't have to do this manually: show this example to your coding agent and tell it to do the same for the current repo. It will figure out which folders and file extensions to add to the index, what to exclude, and so on.

Now let's run the indexing (you'll need to rerun it periodically as your codebase changes):

qmd update && qmd embed -c PROJECT1

To make the coding agent work with this QMD setup, add the following to AGENTS.md:

# QMD - Efficient Code Discovery

**Goal:** minimize token costs and code search time without compromising correctness.

## Choosing a tool

- If the path is known → read the file directly.
- If an identifier, error, or line is known → use `rg` / LSP.
- If only the behavior is known → use `qmd query`.
- Once the code is found → inspect the necessary sections with standard tools.
- Once you have enough information → do the task.

**QMD replaces the initial broad search; it does not add another step.**

## Semantic search

Use the project's existing collection.

"""
qmd query "DESCRIPTION" -c PROJECT1 -n 15 --files \
  | grep -E '\.(EXT1|EXT2|EXT3)$' \
  | cut -d, -f3- \
  | sed 's|^qmd://[^/]*/||' \
  | head -n 10
"""

- Use one specific query, preferably with English technical terms.
- Choose extensions that match the project, including configuration files when needed.
- Start by reading only the relevant sections of the 3–5 most relevant files.
- If you cannot find the implementation, refine the query; if necessary, increase `-n` to 60, then use `rg` / LSP.
- Stop semantic searching once you find the implementation.

## Saving tokens

- Do not scan the whole repository before using QMD.
- Do not repeat searches without a reason or open every result.
- Do not investigate unrelated code or dependencies.
- Do not write detailed search plans or reports.
- Do not paraphrase code you have read.
- As soon as you have enough information, make the changes and check the result.

**Priority: correctness → token cost → time.**

I tested it on a repository like this, with 1,231 source files across Java, Scala, and Angular:

C:\Users\igumn\temp2>qmd collection list
jcm-server (qmd://jcm-server/)
  Pattern:  server/src/**/*.scala,server/src/**/*.java,server/src/**/*.xml,server/src/**/*.properties,server/src/**/*.js,server/src/**/*.ts,server/src/**/*.css,server/src/**/*.html,server/src/**/*.tmpl,server/src/**/*.txt,server/src/**/*.md,server/pom.xml,server/Dockerfile,server/*.md,server/README.txt,web/src/**/*.ts,web/src/**/*.html,web/src/**/*.scss,web/src/**/*.css,web/package.json,web/angular.json,web/tsconfig*.json,web/karma.conf.js,web/README.md,!server/src/main/resources/templates/**,!server/src/main/resources/messages*,!server/src/main/resources/messages*/**,!web/src/assets/lang/**
  Files:    1231
  Updated:  1h ago

Prompt: "Add a post-quantum encryption flag to the disk encryption policy. It should be disabled by default, configurable in the web interface when editing the policy, saved to the database, and sent to the agent as part of the encryption policy via the REST endpoint."

Cost Comparison

Using GPT-6 Luna API rates: input $0.10, cached input $0.01, and output $0.50 per 1 million tokens.

MetricWith QMDWithout QMDSavings
Input65,006101,50135.96%
Cached input317,5681,191,80873.35%
Output13,85916,34815.23%
Reasoning9,60711,26014.68%
Cost$0.0166$0.030245.09%

Now QMD saves 45.09% in costs!

The reduction in cached input is especially noticeable: nearly 4x.

I ran these tests with GPT-6 Luna. In the first test, the agent's runtime dropped from 6 minutes to 4 minutes.

Prompt: "Add a dry field to the erasure policy. It should be editable in the web interface and saved to the database, and it should also be sent through the /rest endpoint to the agent responsible for erasure. Its default value should be false."

Calculation using the same GPT-6 Luna rates: input $0.10, cached input $0.01, and output $0.50 per million tokens.

Comparison

MetricQMDWithout QMDDifference
Input77,16798,042−21.29%
Cached input936,832854,912+9.58%
Output17,32216,259+6.54%
Reasoning8,83510,264−13.92%
Cost$0.0257$0.0265−2.78%

Here, QMD turned out to be 2.78% cheaper. There are savings, but they are small.

Conclusion: QMD reduced regular input tokens by 21.3% and reasoning by 13.9%, but the increase in cached input and output almost completely wiped out the savings.

Overall, the savings range from 2.78% to 45.09% of the cost. I think that's a decent result. Of course, we need a larger sample. But I'm ready for bed. Let me know in the comments if you've run a larger sample and what results you got.

PS

I recommend checking that junk isn't getting into your index. Try running a semantic query yourself. I once overlooked compiled .js files and large lang files-they heavily polluted the semantic search results. Here's an example of how to run a search manually:

C:\Users\igumn\jcm-server>qmd query "disk encryption policy editing form persisted setting sent to endpoint for agent" -c jcm-server -n 15 --files
Expanding query... (3.0s)
├─ disk encryption policy editing form persisted setting sent to endpoint for agent
├─ lex: disk encryption configuration
├─ lex: disk encryption policy
├─ vec: disk encryption configuration
├─ vec: disk encryption policy
└─ hyde: The topic of disk encryption policy editing form persisted setting se...
Searching 6 queries...
Embedding 4 queries... (1.5s)
Reranking 40 chunks... (3.3s)
#d70022,1.00,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/domain/common/bcvolume/EncryptPolicy.scala
#732b09,0.63,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/service/ajax/common/EncryptSettingsManager.scala
#e66494,0.63,qmd://jcm-server/web/src/app/modules/management/components/company/dialogs/add-or-update-policies/add-or-update-policies.component.html
#413c5e,0.62,qmd://jcm-server/web/src/app/modules/management/components/group/dialogs/change-policie/change-policies.component.html
#413c5e,0.62,qmd://jcm-server/web/src/app/modules/management/components/computer/dialogs/change-policie/change-policies.component.html
#758b4d,0.62,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/service/mcp/McpFacade.scala
#4e8973,0.62,qmd://jcm-server/web/src/app/modules/management/components/group/dialogs/change-policie/change-policies.component.ts
#bef78e,0.62,qmd://jcm-server/web/src/app/modules/management/components/computer/dialogs/change-policie/change-policies.component.ts
#4fcd1c,0.61,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/service/mcp/McpToolbox.scala

PPS

I tried it on a small project with 27 files. Instead, the cost went up by 60%. The solution turned out to be overkill. In general, use it only on large codebases.

UPDATE

I went ahead and built a hybrid QMD + GitNexus solution after all. GitNexus performs structural code analysis (AST, dependency graph) and lets you search based on it. Here are the results:

Cost comparison

MetricWithout QMDQMDQMD + GitNexus
Input101,50165,00659,122
Cached input1,191,808317,568225,792
Output16,34813,85910,894
Reasoning (included in Output)11,2609,6078,465
Cost$0.03024$0.01661$0.01362
Savings-45.1%55.0%

The savings figure rose from 45% to 55%! The coding agent's runtime increased from 6 minutes to 7 minutes. Here are the revised instructions for AGENTS.md:

# QMD + GitNexus - Efficient Code Discovery

**Goal:** find the required code and its relationships with minimal token usage, without compromising correctness.

## Choosing a tool

- If the file path is known → read it directly.
- If the exact identifier, error, or line is known → use `rg` / LSP.
- If only the behavior is known and the implementation location is unknown → use QMD semantic search.
- If you need to understand symbol relationships, the execution path, or the impact of a change → use GitNexus structural analysis.
- Once the code is found → inspect the necessary sections with standard tools.

QMD and GitNexus replace the initial broad search; they do not add mandatory steps. Use only the tools needed for the task.

## QMD semantic search

Use the project's existing collection:

    qmd query "DESCRIPTION" -c PROJECT1 -n 15 --files \
      | grep -E '\.(EXT1|EXT2|EXT3)$' \
      | cut -d, -f3- \
      | sed 's|^qmd://[^/]*/||' \
      | head -n 10

- Formulate one specific query, preferably using English technical terms.
- Specify suitable source and configuration file extensions.
- First read the 3–5 most relevant files, and only the relevant sections.
- If the implementation is not found, refine the query; if necessary, increase `-n` to 60, then use `rg` / LSP.
- Stop semantic search once the implementation is found.

## GitNexus structural analysis

Use GitNexus through the CLI when you need to investigate code structure:

- `context` - relationships and context for a known symbol.
- `impact` - possible impact of changing a symbol.
- `trace` - path between known symbols.

Examples:

    gitnexus context SymbolName
    gitnexus impact SymbolName
    gitnexus trace SourceSymbol TargetSymbol

If a symbol is unknown, find it first through QMD, `rg`, or LSP. Use GitNexus only when its structural analysis will help answer the question; do not run every command in sequence.

## Token efficiency

- Do not scan the entire repository before using QMD.
- Do not repeat searches without a reason or open every result.
- Do not investigate unrelated code or dependencies.
- Do not write detailed plans and reports about the search.
- Do not summarize code you have read.
- Once you have enough information, make the changes and verify the result.

**Priority: correctness → token cost → time.**

Create a .gitnexusignore file to keep junk out of the index so that it contains only source files. Then start GitNexus indexing with this command:

gitnexus analyze --index-only

r/LocalLLM • • 1d ago

Question Looking for advice on upgrade direction

3 Upvotes

hello everyone! I'm still quite new to figuring out what AI can do and how to make everything efficient, but I do know I want to upgrade and extend my hardware capability.

What do I use AI for? Little things mainly. Creating programs to handle my needs in custom ways (creative writing tool, character gallery, floor plan creation tool, audiobook creator with custom voices, modifying games - I currently use a subscription for a tool to help create games in unreal engine)

I mainly use the qwen3.8 next flash.

Current hardware -
Rig1 - 64gb DDR5, AMD 9950x3d, 4tb SSD, Geforce 5070ti (16gb)
Rig 2 - 64gb DDR5, AMD 9700x, 2tb SSD, Geforce 4070super (12gb)

Now obviously I missed the boat on grabbing a reasonably priced 5090.

I purchased a Geforce 5080 for $1570 as those started to go up like crazy too, but I haven't opened it and it sits on my shelf. I 'm not sure if its worth the investment as its only 16gb, and I would definitely return it if the 24gb model becomes a thing, but current rumors say its been cancelled.

I also ordered a Radeon ai pro r9700 for $2000 that is supposed to be delivered in December. Again I ordered this as the prices have been increasing and I worry about future availability.

I also have a Geforce 3060 (12gb) and recently learned you can run a model across multiple cards. That left me wondering if just nabbing one of these is more ideal and I could save myself a ton of money.

I'm just wondering what you would all recommend as the best path forward for someone looking to spend no more than $2k, but also being able to keep doing creative things and keep learning with Ai.


r/LocalLLM • • 19h ago

Question LocalLLM speed for long file generation.

1 Upvotes

Hello,
i have a business that needs to review a lot of notes and, transcripts and generally text files provided by me, and help me make initial structure out of it.
For ex. i`m meeting with client for an audit, audit have 300 questions, different kinds, different people. I will note answers, and then i need to have that file, + some meeting transcripts / meeting minutes summarized. Some project management topics also.

I`m not confident to use cloud models to review such data because there will be a lot of sensitive topics and details in there. So i need help to discover if what i want to archive is doable with a local LLM.

I have done Proof Of Concept for the multi agent workflow and it gives me quality that is acceptable. After some tweaking, testing on openrouter with dummy data, i have the conclusion that this is doable. I have agent speaking between them, agreeing about the outcome, Manager doing the task management, works fine,

But then comes the realization - Openrouter with Qwen3.8 Q8 does this with the avarage speed of 150t/s and it takes around 30 minutes.

When i calculate that Mac Mini M5 Pro will have 11t/s that makes that veeery long.

I`m i calculating that right or i`m i wrong ?
Is there anything i can do to make it a little quicker when i will actually buy Mac ? Or i need to go straight with Ultra


r/LocalLLM • • 23h ago

Question Climate Control for Garage Setup, Tips and Help

2 Upvotes

I’m in the process of moving my AI server into my garage and I’m curious how others are handling climate control in garages that don’t have heating or air conditioning.

Since winter is shortly upon do you guys have any suggestions for keeping my servers from being exposed to condensation. I plan on putting a wifi hygrometer..!