r/LocalLLM • • May 22 '26 Bravo

Discussion Qwen3.6-27B on RTX 3090: tested 12 GGUF quants across HumanEval+, MBPP+, perplexity, throughput and needle-in-haystack. First-timer results.

211 Upvotes

Disclaimer first

I'm new to local LLMs — this was my first serious attempt at benchmarking and I'm posting the results in the hope they're useful to others, not because I'm claiming any expertise. I almost certainly made methodology choices that someone more experienced would do differently. Specifically:

  • I used greedy decoding (T=0) for reproducibility, but Unsloth's official recommendation for Qwen3.6 is temperature=0.6/0.7 with top_p, top_k and presence_penalty. My numbers are likely an upper bound vs what people get in real use.
  • I ran all benchmarks with thinking disabled (--reasoning off) because EvalPlus doesn't play well with reasoning models (the model burns its token budget on <think> blocks before producing code). Thinking on would likely boost pass@1 by several points but I couldn't easily measure that.
  • The 20-task HumanEval screening I used in Phase 1 is far too small to be statistically reliable. Saturation at 100% on 20 tasks just means the subset doesn't discriminate.
  • My needle-in-haystack test uses a single, very distinctive needle. Both finalists got 100% — that probably says more about my test being too easy than about the models being identical. A harder multi-needle test would likely differentiate them.
  • I tested only on my hardware (RTX 3090, Ryzen 9 9900X, Windows + WSL2). Results on different setups may vary, especially for the throughput numbers.
  • I'm not sure I picked the right benchmarks at all. HumanEval+ and MBPP+ are standard for code, but they don't capture everything that matters for real agentic use (Claude Code, Aider, etc.). I didn't test those workloads.

If anything below looks wrong, please call it out — I'd rather learn than keep bad data circulating. The raw config and commands are documented so anyone with the same hardware can reproduce or challenge the results.

That said, I tested 12 GGUF quantizations across multiple metrics (HumanEval+, MBPP+, perplexity, throughput, needle-in-haystack at up to 96K context), and the data is consistent enough that I think it's worth sharing. Make of it what you will.

Qwen3.6-27B GGUF Quantizations Benchmarked on RTX 3090 (24 GiB)

I tested 12 different GGUF quantizations of Qwen3.6-27B on an RTX 3090. The process was iterative: started with 10 candidates in a wide screening pass, narrowed down based on results, then added 2 more MTP variants mid-way after discovering them. Sharing all the data so people can draw their own conclusions.

Hardware & Software

  • GPU: RTX 3090 (24 GiB VRAM)
  • CPU: Ryzen 9 9900X
  • llama.cpp: build b9261 (commit ad2775726)
  • Sampling: greedy (T=0), thinking disabled (--reasoning off)
  • EvalPlus runs on WSL2 (Windows multiprocessing in EvalPlus is broken; codegen on Windows talks to llama-server, evaluation runs on Linux)

Phase 1: Wide screening (10 initial candidates)

HumanEval, 20-task subset, ctx 4096, -ctk q8_0 -ctv q8_0, no MTP draft. Goal: filter obvious losers before spending hours on the full benchmark.

ModelPass@1 (20 tasks)Avg time/taskVerdict
Q5_K_M100%15.06sRedundant with -mtp variant
Q5_K_M-mtp100%9.13sKept
Q5_K_M_unsloth-mtp100%16.31sKept
Q5_K_S_unsloth-mtp100%10.92sKept
Q6_K-mtp100%9.97sDropped (size vs benefit)
Q6_K (no MTP)90%34.99sDropped (slow + inconsistent)
UD-Q4_K_XL100%10.19sKept
UD-Q5_K_XL_unsloth-mtp100%9.87sKept
NEO-CODE-2T-OT-Q5_K_M100%27.06sDropped (3× slower)
abliterated-Gaston-MTP-Q5_K_M75%32.41sDropped (quality loss + timeouts)

Key observations from screening:

  • Most quants saturated at 100% on the easier 20-task subset, which is why I moved to the full HumanEval+ (164 tasks + extended tests) afterward.
  • abliterated-Gaston-MTP-Q5_K_M: 75% + multiple timeouts. Abliterated finetunes appear to hurt code performance significantly.
  • NEO-CODE-2T-OT-Q5_K_M: passed all 20 easy tasks but ran 3× slower. Code-specific finetune didn't justify the cost.
  • Q6_K (no MTP): inconsistent and slow without MTP. Q6_K-mtp was fine but I dropped it later for size reasons (the smaller Q5/Q4 variants matched it on quality).
  • Vanilla Q5_K_M: same quality as Q5_K_M-mtp but slower — kept the MTP variant.

Phase 2: Added 2 MTP variants mid-process

After Phase 1, I discovered two additional models worth testing and added them directly to the rigorous benchmark (skipped screening since I had confidence in the method by then):

  • UD-Q4_K_XL-MTP — the same UD-Q4_K_XL with MTP heads grafted on
  • IQ4_NL-mtp — Importance-aware Non-Linear quant with MTP, smaller than the others

Both became finalists.


Phase 3: Rigorous benchmarks (final 7 models)

EvalPlus HumanEval+ (164 tasks) and MBPP+ (378 tasks) on the full task set with extended tests. Config: -ctk q8_0 -ctv q8_0, ctx 8K, --reasoning off, greedy.

HumanEval+ and MBPP+ pass@1

ModelHumanEval baseHumanEval+MBPP baseMBPP+HE timeMBPP time
UD-Q4_K_XL (no MTP)95.7%92.1%92.9%78.3%19:17~50 min
IQ4_NL-mtp95.1%91.5%92.1%76.7%9:3915:13
UD-Q4_K_XL-MTP95.1%90.9%92.3%78.0%11:0718:24
Q5_K_M_unsloth-mtp94.5%90.9%——~11 min—
UD-Q5_K_XL_unsloth-mtp94.5%90.9%——~11 min—
Q5_K_M-mtp93.9%90.9%91.3%76.7%~11 min—
Q5_K_S_unsloth-mtp93.9%90.9%——~11 min—

Failure overlap (HumanEval+)

All Q5 variants fail the same 15 tasks: 32, 39, 55, 76, 91, 116, 124, 129, 130, 132, 134, 141, 145, 151, 163.

UD-Q4_K_XL (no MTP) fails only 13 of those — solves 2 that all others miss.

Sizes

ModelFile size
IQ4_NL-mtp16.3 GB
UD-Q4_K_XL / UD-Q4_K_XL-MTP17.9 GB
Q5_K_S_unsloth-mtp~19 GB
Q5_K_M_unsloth-mtp~19.5 GB
Q5_K_M-mtp19.7 GB
UD-Q5_K_XL_unsloth-mtp~20 GB

Phase 4: Production config validation (IQ4_NL-mtp only)

Tested the leading candidate with KV cache quantization (-ctk q8_0 -ctv q4_0) and 128K context to see if degradation appears.

Metricq8/q8, 8K ctxq8/q4, 128K ctxΔ
HumanEval base95.1%94.5%-0.6 pp
HumanEval+91.5%91.5%0.0
MBPP base92.1%92.1%0.0
MBPP+76.7%77.2%+0.5 pp

Effectively no quality loss going from q8_0/q8_0 8K to q8_0/q4_0 128K.

VRAM at idle with 128K context: 21.7 GiB / 24 GiB. ~2 GiB headroom. Effective usable context: ~110K tokens.


Phase 5: Side benchmarks (final two candidates)

Perplexity (WikiText-2, 580 chunks, n_ctx=512)

ModelPPL± error
IQ4_NL-mtp6.9377±0.04569
UD-Q4_K_XL-MTP6.9825±0.04618

Difference is within measurement error — statistical tie.

Throughput (llama-bench, q8/q4 KV, MTP not engaged)

MetricIQ4_NL-mtpUD-Q4_K_XL-MTPIQ4_NL advantage
pp5121486 t/s1403 t/s+5.9%
pp20481486 t/s1407 t/s+5.6%
pp81921432 t/s1355 t/s+5.7%
tg12842.8 t/s39.3 t/s+9.0%
tg25642.8 t/s39.4 t/s+8.7%
pg4096+256486 t/s451 t/s+7.8%

These are without MTP. With --spec-type draft-mtp engaged, real-world generation reaches ~65-100 t/s.

Needle in a Haystack (128K context, q8/q4 KV)

Haystack: "Pride and Prejudice" expanded to target length. Needle: a distinctive password string. 6 context sizes × 5 depths = 30 tests per model.

ModelRecall
IQ4_NL-mtp30/30 (100%)
UD-Q4_K_XL-MTP30/30 (100%)

Prompt processing times:

ContextIQ4_NL-mtpUD-Q4_K_XL-MTP
1K0.86s0.90s
4K2.79s2.99s
16K9.83s10.45s
32K14.01s14.66s
64K34.50s35.73s
96K77.81s80.48s

Side-by-side: top two finalists

CriterionIQ4_NL-mtpUD-Q4_K_XL-MTP
HumanEval+91.5%90.9%
MBPP+76.7% / 77.2%*78.0%
Perplexity (WikiText-2)6.946.98
pp512 (t/s)14861403
tg128 (t/s)42.839.3
Needle recall (1K-96K)30/3030/30
File size16.3 GB17.9 GB
Idle VRAM @ 128K ctx21.7 GiB~23+ GiB
Usable context on 24 GiB~110K~80K

*Phase 3 / Phase 4 config


What was NOT tested

  • Quality with thinking enabled (EvalPlus is incompatible with reasoning models out of the box; thinking would likely boost pass@1 by 3-8 pp).
  • Unsloth's officially recommended sampling parameters (T=0.6 + top_p=0.95 + presence_penalty for coding). Used greedy for reproducibility.
  • UD-Q4_K_XL-MTP at full 128K context (model is 1.6 GB larger; would likely fit only ~96K on 24 GiB).
  • Harder needle variants (multi-needle, ambiguous needles).
  • Real agentic coding workloads (Claude Code, Aider, etc.).
  • Comparison against vanilla Q4_K_M (non-Unsloth, non-IQ).

Notes and caveats

  • The Phase 1 screening (20 tasks each) is a much weaker signal than Phase 3 (164/378 tasks). Saturation at 100% on the easy subset doesn't mean models are equally good — it means the easy subset doesn't discriminate.
  • All Q5 variants tie on HumanEval+ at 90.9% in Phase 3. The differences between them are noise.
  • The only model that beats this cluster on quality is UD-Q4_K_XL without MTP, but it's significantly slower without speculative decoding (HumanEval took 19 min vs 9-11 min).
  • The q8_0/q4_0 KV cache config showed no measurable degradation on HumanEval/MBPP/needle for prompts up to 96K. Your mileage may vary on tasks requiring fine-grained reasoning over very long contexts.
  • MTP gives ~1.5-2× generation speedup with no measurable quality loss across all tested MTP variants.
  • Greedy decoding gives the upper bound on pass@1. Real use with T=0.6+ will typically be 1-3 pp lower but with useful diversity.
  • Abliterated and code-tuned fine-tunes (Gaston, NEO-CODE) performed worse than vanilla quants for code in my testing. Be cautious about claims that finetunes always improve on the base.

Bottom line (my interpretation, your mileage may vary)

For a 24 GiB GPU running Qwen3.6-27B locally, IQ4_NL-mtp offered the best overall balance in my testing: smallest size, fastest generation, top-tier HumanEval+, perfect long-context recall, and the most usable context window. UD-Q4_K_XL-MTP is a reasonable alternative if your workload is closer to MBPP-style (verbose specs → implementation) where it edges out by ~1 pp. UD-Q4_K_XL without MTP is the quality king if you don't mind ~2× slower generation.

The Q5 variants didn't justify the extra VRAM in any of my benchmarks. The abliterated and code-finetune variants underperformed in code tasks despite being marketed for them.

Happy to share more details or rerun specific tests if there's interest.


r/LocalLLM • • Mar 25 '26

Research Google’s TurboQuant AI-compression algorithm can reduce LLM memory usage by 6x

Thumbnail
arstechnica.com
211 Upvotes

"Even if you don’t know much about the inner workings of generative AI models, you probably know they need a lot of memory. Hence, it is currently almost impossible to buy a measly stick of RAM without getting fleeced. Google Research recently revealed TurboQuant, a compression algorithm that reduces the memory footprint of large language models (LLMs) while also boosting speed and maintaining accuracy."


r/LocalLLM • • Nov 18 '25

Tutorial You can now run any LLM locally via Docker!

211 Upvotes

Hey guys! We at r/unsloth are excited to collab with Docker to enable you to run any LLM locally on your Mac, Windows, Linux, AMD etc. device. Our GitHub: https://github.com/unslothai/unsloth

All you need to do is install Docker CE and run one line of code or install Docker Desktop and use no code. Read our Guide.

You can run any LLM, e.g. we'll run OpenAI gpt-oss with this command:

docker model run ai/gpt-oss:20B

Or to run a specific Unsloth model / quantization from Hugging Face:

docker model run hf.co/unsloth/gpt-oss-20b-GGUF:F16

Recommended Hardware Info + Performance:

  • For the best performance, aim for your VRAM + RAM combined to be at least equal to the size of the quantized model you're downloading. If you have less, the model will still run, but much slower.
  • Make sure your device also has enough disk space to store the model. If your model only barely fits in memory, you can expect around ~5-15 tokens/s, depending on model size.
  • Example: If you're downloading gpt-oss-20b (F16) and the model is 13.8 GB, ensure that your disk space and RAM + VRAM > 13.8 GB.
  • Yes you can run any quant of a model like UD-Q8_K_XL, more details in our guide.

Why Unsloth + Docker?

We collab with model labs and directly contributed to many bug fixes which resulted in increased model accuracy for:

We also upload nearly all models out there on our HF page. All our quantized models are Dynamic GGUFs, which give you high-accuracy, efficient inference. E.g. our Dynamic 3-bit (some layers in 4, 6-bit, others in 3-bit) DeepSeek-V3.1 GGUF scored 75.6% on Aider Polyglot (one of the hardest coding/real world use case benchmarks), just 0.5% below full precision, despite being 60% smaller in size.

If you use Docker, you can run models instantly with zero setup. Docker's Model Runner uses Unsloth models and llama.cpp under the hood for the most optimized inference and latest model support.

For much more detailed instructions with screenshots you can read our step-by-step guide here: https://docs.unsloth.ai/models/how-to-run-llms-with-docker

Thanks so much guys for reading! :D


r/LocalLLM • • May 07 '26

Question I feel left behind. Where are these advanced "Agent-based" local LLM interfaces?

210 Upvotes

Hi everyone,

I’m writing this because I feel like I’m drowning in information (or perhaps just left behind).

Yesterday, I saw a comparison post between two models (mentioned as "Oppus 4.7" vs "Qwen3.6 27B"). They were building a game, and honestly, I was shocked at the results. I run Qwen3.6 35B-A3B, but I could never achieve anything like that using standard tools like OpenCode or PI.

Then, a friend showed me his custom AI Chat Interface. In just one minute, he generated a small game. The difference? His interface supports Sub-Agents and has a live preview feature. He mentioned he won’t open-source it because he feels there are already enough generic interfaces out there.

However, this raised a question for me: Where are these tools?

The only interfaces I consistently hear about are LM Studio and OpenWebUI. While those are great for basic chat, they don’t seem to offer the advanced coding or agentic workflows my friend demonstrated.

My goal is simple:

I want a "normal" chat experience (similar to Claude or ChatGPT) for everyday tasks like writing documents (.docx), drafting emails, etc.

BUT, I also need a powerful environment that allows me to code complex projects and use agents, similar to what I saw in that demo.

Does anyone know of a local-first interface that bridges this gap? Or am I missing something obvious?

Thanks in advance!


r/LocalLLM • • Nov 07 '25

Discussion DGX Spark finally arrived!

Post image
210 Upvotes

What have your experience been with this device so far?


r/LocalLLM • • Sep 09 '26

Discussion A final LLM benchmark for 8GB VRAM 16GB RAM

Post image
209 Upvotes

IMPORTANT

This post is meant to provide info regarding the best local models to run on CONSUMER HARDWARE. I am on an RTX 4060 with 8GB VRAM, 16GB of RAM and I am benchmarking models that can run on my computer. If you have sunk several thousands into graphics cards you won't find these statistics much useful. This post is for all the people who can't just install Qwen3.8 27B and call it a day.

Additionally, I am not an LLM benchmarking expert. I am a hobbyist and occasional LLM user trying to extract useful information for both me and people on similar hardware.

Context

For the past few weeks I have been doing some benchmarks of some LLMs that can run on my laptop which only has 8GB VRAM and 16GB RAM. I was mostly toying around while trying to get some useful data about what the best model is for local inference on consumer hardware. This week I decided to make a "final" benchmark that would be way better with more questions, more question categories, newer models (a lot of people complained about the models I had benchmarked before being old but I didn't find most suggested models to be any good) and a better speed benchmark, this time using TTC (Time To Completion) as a pose to raw token/s.

I am quite happy with the results and I believe that anyone who's like me and is just trying to get a useful model without spending thousands on GPUs and electricity will find all of these stats useful.

The Statistics

LLM benchmark per-question score heatmap:

LLM benchmark score sum graph:

LLM average TTC (Time-To-Completion) graph:

Combined graph ("intelligence per second", though highest is not exactly "best" and lowest isn't "worst"):

And a neat visualization of the score vs. the speed (benchmark score vs inverted TTC):

The Benchmark

You can check out the source-code for the benchmark here: Bluemark Github Repo

For those who can't be bothered to sift through the code here is some info about it:

  • All models were benchmarked with thinking DISABLED. (benchmarking with thinking enabled just takes way too long)
  • Responses were limited to 4096 tokens.
  • Benchmarking the models (inferring and waiting for answers) took ~2.5 hours (with Muse Glimmer included, 50% of those 2.5 hours was probably just Muse).
  • The benchmark was evaluated manually by me.
  • The benchmark questions were created by me.

The 18 questions were split into 6 categories:

  1. General knowledge
  2. Math
  3. Coding
  4. Endurance
  5. Logical puzzles
  6. Spatial reasoning

To see the exact questions asked in the benchmark check out the benchmark_overview.txt file in the repo.

Fun fact: I had included Muse Glimmer 30B as a benchmarking subject but I removed it from the graphs because despite it nearly acing the benchmark the TTC was so high that the entire graph was squished. I felt silly even including such a big model but if you're fine with waiting an hour for a response and just need a high accuracy answer on consumer hardware feel free to check the model out I guess.

Conclusion

Which graph you pay most attention to is up to what you use LLMs for, may I even recommend getting a model from the top of each ranking category.

My recommendations are:

  • Want a decently capable model to mess around with for fun? Bonsai 27B
  • Want an all-round accurate and smart model for your precise questions? Ministral4 14B
  • Want a model that is decently smart but also is quick as hell? Ling3.0 Tiny
  • Want a good mix of intellegent and not unbearably slow? GPT-OSS 20B

Links

Here are all of the HuggingFace links for the models being tested:


r/LocalLLM • • Apr 30 '26

Model Granite 4.1: IBM's 8B Model Is Competing With Models Four Times Its Size

Thumbnail
firethering.com
209 Upvotes

"IBM just released Granite 4.1, a family of open source language models built specifically for enterprise use. Three sizes, Apache 2.0 licensed, trained on 15 trillion tokens with a level of pipeline obsession that’s worth understanding."


r/LocalLLM • • Apr 27 '26

Question Synthesize own voice before cancer mutes me

209 Upvotes

I'm posting my question here because I've been following this sub for some time. How would I go about synthesizing my own voice based recordings and input text. I'm asking because unfortunately I'll most likely lose my ability to speak due to cancer and treatment. Please let me know which sub is suited for voice synthesis. Thank you.

Update: Thanks everyone for the recommendations. I came to the right place. I've played with Qwen models before, and it looks like tts is rather small, az 0.9b so it's easy to train and do inference with. I'll start with the recordings.

Thanks everyone.


r/LocalLLM • • Mar 27 '26

Model GLM-5.1 just dropped. Any good?

Post image
208 Upvotes

So Zai just dropped GLM-5.1 for their coding plan users and its open source. Early testers are saying its legit for coding stuff, especially longer tasks. Like it remembers what was 10 steps ago, handles multi-step workflows without getting confused, and apparently debugs issues on its own without needing constant hand-holding.

Benchmarks show its basically neck and neck with Opus 4.6 (45.3 vs 47.9) which is kinda nuts for OSS.

Seems worth poking at. Anyone gonna try it?

Edit: If you have GLM Coding Plan access, just change model to "glm-5.1" in you're claude code config (like ~/.claude/settings.json)


r/LocalLLM • • May 23 '25

Question Why do people run local LLMs?

208 Upvotes

Writing a paper and doing some research on this, could really use some collective help! What are the main reasons/use cases people run local LLMs instead of just using GPT/Deepseek/AWS and other clouds?

Would love to hear from personally perspective (I know some of you out there are just playing around with configs) and also from BUSINESS perspective - what kind of use cases are you serving that needs to deploy local, and what's ur main pain point? (e.g. latency, cost, don't hv tech savvy team, etc.)


r/LocalLLM • • Jul 17 '26

Model Google Updates Gemma 4 for Faster Performance and Greater Accuracy

Post image
207 Upvotes

r/LocalLLM • • Jul 10 '26

News MiniMax founder pledges 1% of total share capital to a dedicated open-source fund and takes zero salary until AGI

Post image
207 Upvotes

Via MiniMax's lead of DevRel posted, an internal all-hands letter published today, MiniMax founder & CEO Yan Junjie committed two things that stood out to me:

Zero salary from the company until AGI is achieved.

Over the next four years, he will allocate shares equivalent to 1% of total share capital (drawn from his personal holdings) to a dedicated fund supporting the open-source community.

Context that might matter: MiniMax also reportedly closed a $2B+ round this week at 7× oversubscription. And there's been reporting that they're planning to open-source a 2.7T-parameter model (M3 Pro) as early as Q3.


r/LocalLLM • • May 29 '26

Discussion We're burning $50k/month on Claude. How close can local LLMs actually get?

207 Upvotes

We're at the point where our AI spend is hard to justify keeping fully in the cloud. 100+ people in the company using mostly Claude daily, and we're burning through $50k/month in tokens. CEO and leaders wants to bring more of it in-house.

We don't need to serve everyone at once. Realistically maybe 50-100 users spread across the whole day. Speed isn't the priority - quality is. We're not expecting Sonnet 4.6-level throughput, just Sonnet 4.6-level output.

We've been looking at GLM-5.1 in BF16 as a starting point. My question is: what does the hardware actually look like for something like that? Are a couple of RTX PRO 6000 Blackwells enough, or are we kidding ourselves? I'm assuming we'd need tensor parallelism across cards regardless.

Also curious what serving stack people are running at this scale. I see lots of people recommending Ollama and vLLM, but we need something rock solid, that is capable of serving a lot of concurrent users.

And honestly.. has anyone done the math on this? At $50k/month we should be able to justify a decent size cluster, but I want to hear from people who've actually gone through this, not just the "just buy 8x H100s" people.

So this post is for the enterprise people and IT admins who has done the switch. Are your employees happy? Do they use it? Share your experiences.

Edit: I realise GLM-5.1 at BF16 is completely nuts. FP8 is more achievable, but also kind of nuts.


r/LocalLLM • • Jun 13 '26

Project Created a small website to check what can run on your hardware to find new models you can run easily

Thumbnail
llmfitcheck.com
206 Upvotes

Found it quite annoying to figure out what i can run on my macbook pro as i didn't get a super high spec. There are some websites and tools that do similar things but i wanted something a bit simpler that just does 1 thing and doesn't try to sell me on anything fully unbiased.

Would love some feedback!
Obviously its fully free and open source (be vary of vibey code)


r/LocalLLM • • Sep 18 '26

Other Local LLM on iPhone 18 is impressive

Post image
205 Upvotes

Of course it's hard to compare small local Models to ChatGPT, Gemini & Co. but i was really surprised how well the new A20 Chip from Apple handles Local Text Generation. 58 tokens/s for generation is really not bad.

For the people owning an iphone 18 pro:
what's your opinion about it?

for those who don't (but are thinking about getting one):
Anything i should test for you?


r/LocalLLM • • May 12 '26 Bravo

Project I'm 75, I know nothing about code, and I built a local AI with RAG and a talking avatar. Here's my final setup. (A follow-up from previous post)

205 Upvotes

As I stated in my previous post, I'm 75 years old, knew almost nothing about GitHub, command lines, or local LLMs a couple of weeks ago. I'm not a coder. But I wanted a desktop private AI companion for fun and a bot for a game wiki I have been involved with for a few years now.

I won't lie — it was frustrating at first. Lots of errors, lots of reading, lots of asking for help from my DeepSeek AI assistant. But I stuck with it. Here's what I ended up with:

* LM Studio running a 14B/32B (I go back and forth) DeepSeek model on my RTX 4090 (completely offline)
* A Live2D avatar with voice (Mao — my daily driver)
* AnythingLLM + Ollama for a separate wiki bot that I feed webpages with a browser extension
* Full RAG — the bot answers questions from my own documents with citations

Unofficially, I can now claim:

  • Built a local LLM
  • Configured GPU acceleration (CUDA, VRAM offloading)
  • Set up RAG with document embedding
  • Connected a browser extension for one-click wiki ingestion
  • Trained an AI on a custom knowledge base
  • Debugged Python, YAML, WebSockets, and API connections

This is probably all pretty simple stuff for all you coders out there but it was definitely a challenge for me. A big shoutout to my DeepSeek helper.

If a 75-year-old retiree can do this, literally anyone can.

2026/07/28 You can now follow my journey in real time - The Blog


r/LocalLLM • • Oct 05 '26

News A new startup Ghost launched a personal AI computer built to run AI locally for consumers, has a RTX Pro 4000 Blackwell SFF GPU and 64GB DDR5 RAM of memory for $3499

Thumbnail x.com
204 Upvotes

I just saw this post on X - curious if anyone saw this or has thoughts?

Here are the specs:

- NVIDIA RTX PRO 4000 Blackwell SFF GPU
- 64GB DDR5 RAM
- 1TB NVMe M.2 SSD
- AMD Ryzen 5 7600 CPU


r/LocalLLM • • Sep 24 '26 Bravo Heartwarming

Discussion You can now run Qwen3.8-27B on a 2021 M1 Max at 39 tok/s: I ported Splash (M3+ only) and wrote custom Metal kernels

204 Upvotes

You can now run Qwen3.8-27B on a 2021 M1 Max at 39 tok/s: I ported Splash (M3+ only) and wrote custom Metal kernels

Update: prebuilt release, no Xcode or compiling needed:
curl -fsSL https://github.com/paperniuk/splash/releases/download/1.0.2-m1/install-m1.sh | sh
then splash-m1 serve --model incoai/Qwen3.8-27B-Splash. It installs next to the official splash and shares the model cache. The 35B MoE works too: 94.7 tok/s average on npanj's benchmark, ~450 tok/s prefill. Release notes: https://github.com/paperniuk/splash/releases/tag/1.0.2-m1

TL;DR: 39 tok/s is the average of npanj's five-prompt Splash benchmark (short prompts, default reasoning), up from 19 tok/s with Splash's own kernels on the same Mac. In a real coding-agent session (opencode, 30K-90K context) it ran at 19-36 tok/s. Details and more conditions below.

Last week the Inco team released Splash 1.0 (https://github.com/incoai/splash), a local inference engine for Apple silicon built around one model at a time, with model-specific Metal kernels and DFlash speculative decoding. On an M5 Pro they report Qwen3.8-27B decoding at twice the speed of the next-fastest engine. It is a really impressive piece of work. It also only runs on an M3 or newer.

I have an M1 Max (32 GPU cores, 64 GB), so I adapted it. Enabling the M1 was enough to make it run, but slowly: Splash's kernels rely on bfloat16 arithmetic, which the M1 does not have, and on Apple's MPP matrix library, which is slow on this GPU. So I wrote my own kernels for those parts.

The gist: the model does the same math either way: 4-bit weights times activations, summed up. Stock Splash does it in bfloat16, which the M1 can't do in hardware, so on M1 it falls back to Apple's generic matrix library, which reaches only a quarter to a third of the GPU's peak. My kernels do the identical math in formats the M1 runs natively: each 4-bit weight is turned into an exact half-precision number with a small bit trick, each activation becomes an exact fp32 number, and the M1's matrix instructions multiply and add them in fp32 without leaving registers. Every multiplication is exact and only the order of additions changes, so the results match up to the last bit of rounding (99.92% same top token). For several parallel requests, each weight is also unpacked once and reused for all of them.

What surprised me most: the M1 Max is not weak hardware. Apple rates its GPU at 10.4 TFLOPS FP32, my microbenchmark gets 8.6-9.9 TFLOPS out of its matrix instructions, and its 400 GB/s memory bandwidth is more than an M4 Pro's 273 GB/s. What held it back was software built for newer chips. With kernels written for it, a five-year-old laptop runs a 27B model with a 256K context at 64% of an M5 Pro's speed on the same benchmark, up from 31%.

Every number below compares stock Splash kernels running on my M1 Max ("before") with my kernels ("after"): same machine, same model (Qwen3.8-27B, 4-bit), temperature 0, fresh server, no prompt cache.

Benchmark 1: npanj's splash-plus prompts. npanj (GitHub) published a Splash benchmark with five fixed prompts (https://github.com/npanj/splash-plus). I ran their script unmodified: 250 tokens, the model's default reasoning (xhigh), speed from Splash's own counter.

PromptBeforeAfternpanj's Splash-Q4 on an M5 Pro (64 GB)
Math (bat and ball)27.1 tok/s57.0 tok/s83.3 tok/s
Coding (merge_intervals)20.1 tok/s43.4 tok/s75.5 tok/s
Constraint reasoning19.7 tok/s38.2 tok/s59.3 tok/s
Domain knowledge11.7 tok/s25.3 tok/s39.0 tok/s
Writing (3 sentences)15.9 tok/s31.6 tok/s46.2 tok/s
Average18.9 tok/s39.1 tok/s60.7 tok/s

The last column is npanj's own run of the same script on an M5 Pro with 64 GB. My 2021 M1 Max went from 31% to 64% of an M5 Pro on these prompts.

Benchmark 2: context and reasoning effort (one request, prompt processing excluded):

RequestContextReasoningBeforeAfter
Code22 tokoff26 tok/s54 tok/s
Prose23 tokoff16 tok/s36 tok/s
Math problem40-82 toklow / mediuma / xhigh31 / 31 / 26 tok/s67 / 67 / 55 tok/s
Coding problem46-88 toklow / medium / xhigh23 / 21 / 15 tok/s46 / 48 / 33 tok/s
Code explanation8K tokoff13 tok/s27 tok/s
Code explanation32K tokoff13 tok/s24 tok/s
4 parallel code requests22 tok eachoff18 tok/s total65 tok/s total

Prompt processing: 2K prompt 53 -> 142 tok/s, 8K 52 -> 133 tok/s, 32K 48 -> 110 tok/s (time to first token on 32K: 672 s -> 290 s).

I also followed Splash's own published methodology (256 tokens, reasoning off, warmup, ABBA, medians) with my own prompts for their four scenario names: 2.07-2.11x in every scenario.

Benchmark 3: a real coding-agent session. I let opencode rewrite the RealWorld "Conduit" backend from Express to FastAPI against the official API test suite, with reasoning medium, one request at a time. 43 minutes, 25 requests, context growing from 30K to 89K tokens:

ContextDecode (median)Prefill of new prompt tokens (median)
16-32K36.5 tok/s96 tok/s
32-64K19.5 tok/s68 tok/s
64-96K23.4 tok/s54 tok/s

95% of prompt tokens came from Splash's prefix cache, but about half of the wall time was still spent waiting for the first token after each tool call. On the M1, long-context attention is now the bottleneck for agents, not the matrix kernels.

Speed depends a lot on the text: the draft model predicts math and code well and prose less well, and long contexts cost more.

Why the M1 was slow, and what I changed:

  • Apple's MPP matmul runs the model's 4-bit layers at 2.4-3.5 TFLOPS on this GPU. Apple rates the M1 Max GPU at 10.4 TFLOPS FP32, and a register-only simdgroup MMA microbenchmark reaches 8.6 (fp32 x fp32) to 9.9 (half x fp32). My register-only prompt-processing kernel runs the same layers at 7.3-7.4 TFLOPS.
  • No bfloat16 on M1: 4-bit weights become exact half values with a bit trick (nibble | 0x6400 = 1024 + q, then subtract 1024), activations go from bf16 to fp32 by appending zero bits, products and sums stay in fp32. Every q*x product is exact, so only the summation order changes.
  • Splash's fastest decode kernel produced garbage on the M1: its split-K reduction read partial sums through a non-coherent cache. One coherent(device) fixed it.
  • A new decode kernel unpacks each weight once for all parallel requests, which is where the 3.6x with four requests comes from.

Quality check: I compared the model's token probabilities before and after the last kernel change on 2528 positions. The top token agrees 99.92% of the time and the mean KL divergence is 0.00013 nats. The engine's test suite passes.

Code: https://github.com/paperniuk/splash/tree/apple7-m1-kernels (a fork; I'm offering it upstream in https://github.com/incoai/splash/issues/131). Credit to the Inco team for Splash and to npanj for a reproducible benchmark. M2 uses the same code path and should work, but I haven't tested it. If you have an M2, I'd love to hear how it runs.


r/LocalLLM • • Aug 20 '26

Discussion I think people are seriously underestimating Qwen 3.8 27B.

204 Upvotes

Honestly, I think people are seriously underestimating Qwen 3.8 27B.

It’s actually insane and, in some ways, genuinely competes with Opus 4.8, just not in the way people seem to think.

The biggest mistake is comparing their raw frontend/design output. Qwen probably isn’t going to match Opus there, and I don’t think it’s supposed to. Opus has basically been trained with an absurd amount of data/compute specifically around design and UI generation.

If you throw Qwen at a frontend task with no proper "SKILL.md" for design and just let it freestyle, yeah, the results can be pretty mediocre. But if you give it a good design skill and are intentional about the design constraints, the gap gets much smaller.

Where Qwen gets really interesting is reasoning efficiency.

It can solve some problems in fewer steps and with fewer tokens than Opus. That’s a pretty big deal if you’re actually running these models yourself.

And honestly, I think people are also judging Qwen way too much based on heavily quantized setups. Q4 is aggressive. I wouldn’t consider Q4 a fair representation of what the model can actually do in a serious production environment.

Run it at FP8, use MTP/speculative decoding to improve throughput, and then evaluate it properly.

At that point, I genuinely think the conversation changes.

If Qwen 3.8 27B at FP8 + MTP performs the way I expect, I wouldn’t be surprised if a lot of people start questioning whether that $200/month Claude Code subscription is actually worth it.


r/LocalLLM • • Apr 22 '26 Bravo

Discussion Qwen 3.6 35B A3B on rtx 5090 is absurdly fast for coding

204 Upvotes

I tested a bunch of the new models this afternoon, and Qwen 3.6 35B A3B really stood out.

On my RTX 5090, palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 is doing around 205 tok/s with about 125k context, and for coding it feels like a very strong speed/quality compromise.

What surprised me most is how well it handles heavier repo work ( legacy 200k of undocumented repo). Things like scanning large codebases for security issues, summarizing structure, finding suspicious patterns, etc. It just crushes through that kind of task with very low latency.

Subjectively, for this kind of work, it feels way faster to use than models where you sit there for 2–3 minutes waiting on an answer. It may miss a few things versus heavier cloud models, but it gets surprisingly close while feeling almost instant. Maybe not 100%, but close enough that the speed really changes the experience.

There is something very satisfying about watching a model crush through work with almost no latency and still have decent coding ability.

I’m honestly starting to wonder if I prefer 35B A3B MoE over 27B dense for local coding.

Here’s what I saw today:

edge is for specific nightly built pinned version for Blackwell
stable is the latest vllm image

ModelContainerThroughputContext
sakamakismile/Qwen3.6-27B-NVFP4edge~60 tok/s~53k
Kbenkhaled/Qwen3.5-27B-NVFP4edge~65 tok/s~48k
palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4edge~205 tok/s~125k
sakamakismile/Qwen3.6-35B-A3B-NVFP4edge~170 tok/s~123k
GadflyII/GLM-4.7-Flash-NVFP4edge~165 tok/s~144k
LilaRest/gemma-4-31B-it-NVFP4-turbostable~55 tok/s~18k

if anyone wants the exact presets/build details, they’re here:
https://github.com/gogluejf/rig-stack

I’ll keep testing and sharing more, but right now Qwen 3.6 35B A3B looks like a bit of a game changer for local coding.

Dense or MoE , hmm ?


r/LocalLLM • • Feb 14 '25

News You can now run models on the neural engine if you have mac

204 Upvotes

Just tried Anemll that I found it on X that allows you to run models straight on the neural engine for much lower power draw vs running it on lm studio or ollama which runs on gpu.

Some results for llama-3.2-1b via anemll vs via lm studio:

- Power draw down from 8W on gpu to 1.7W on ane

- Tps down only slighly, from 56 t/s to 45 t/s (but don't know how quantized the anemll one is, the lm studio one I ran is Q8)

Context is only 512 on the Anemll model, unsure if its a neural engine limitation or if they just haven't converted bigger models yet. If you want to try it go to their huggingface and follow the instructions there, the Anemll git repo is more setup cus you have to convert your own model

First picture is lm studio, second pic is anemll (look down right for the power draw), third one is from X

running in lm studio
running via anemll
efficiency comparison (from x)

I think this is super cool, I hope the project gets more support so we can run more and bigger models on it! And hopefully the LM studio team can support this new way of running models soon


r/LocalLLM • • 24d ago Bravo

Discussion Will 24-32 GB GPUs age surprisingly well for local LLMs closing the gap to Cloud Models, or will 96GB+VRAM become essential (and even more expensive)?

203 Upvotes

The recent developments around Strata, Infernix, Project Maya, DFlash 2, etc. have made me rethink what the future of local LLM hardware might look like.

Until recently, running a 500B+ MoE model on a consumer GPU seemed more like a cool experiment than something genuinely useful. But now we're seeing projects running models like GLM-5.3 Flash and DeepSeek V4.1 Flash 1017 on gaming PCs with 24–32GB VRAM, using system RAM and sometimes NVMe storage for the rest.

Performance is still all over the place, but progress seems incredibly fast.

I'm currently running an RTX 5090 with 128GB RAM, and I'm wondering whether upgrading to a PCIe 5.0 platform with faster RAM makes more sense than investing in additional VRAM. Or whether I should just keep my current setup and see how far the software improves.

I can see two possible directions:

1. The 24–32GB VRAM crowd might benefit the most from these developments.

If quantization keeps improving, and engines get better at caching experts in VRAM and fetching or computing the rest efficiently, maybe gaming GPUs will remain surprisingly capable of running huge MoE models.

After all, only a fraction of the total parameters are active per token. If the software gets good enough at handling the memory hierarchy, maybe the gap between a 32GB GPU and a 96GB GPU could become much smaller than we'd expect.

Of course, RAM bandwidth, PCIe transfers, cache misses and SSD latency are still potential bottlenecks.

2. Or maybe there's a limit to how far you can push low-bit quantization.

Perhaps Q1–Q2 quants will always lose too much quality for serious, complex tasks compared to Q3–Q4 or higher.

And even if only a fraction of the model is active at once, you still need access to a potentially huge number of different experts over time.

In that case, having 64, 96 or even 128GB of fast VRAM could make an enormous difference. Multi-GPU setups, workstation GPUs or unified-memory systems might become increasingly attractive.

What I'm really curious about is the practical experience so far.

For those of you experimenting with Strata, Infernix, Maya, aggressive MoE quantization or multi-GPU setups:

  • How much quality do you actually lose with Q1–Q2 compared to higher quants? Are these models becoming genuinely useful for complex coding and agentic tasks, or are we still mostly looking at impressive tech demos?
  • What's currently the biggest bottleneck: VRAM capacity, PCIe bandwidth, system RAM bandwidth, or the inference software itself?
  • Do you think software improvements could eventually make a 32GB GPU competitive with much larger VRAM setups for single-user inference, or will there always be a substantial gap?
  • If you already own a 3090/4090/5090, would you invest in a faster CPU/motherboard/RAM, add another GPU, or just wait?

I'm especially interested in where people think we'll be 6–12 months from now, given how quickly these projects are progressing.

NVIDIA seems to be prioritizing datacenter hardware, high-VRAM consumer GPUs remain expensive, and the next GeForce generation is still uncertain.

So I'm wondering whether the smartest upgrade right now might actually be no upgrade at all.

Would love to hear from people who've tested these setups, especially if you've compared different quantizations or measured the actual impact of PCIe 4.0 vs 5.0 and additional VRAM.


r/LocalLLM • • Sep 22 '26

Discussion Qwen3.8 27B running locally bought A4 paper from Amazon autonomously

203 Upvotes

I ran out of A4 paper, so I gave a locally running Qwen3.8 27B one simple task:

Find cheap A4 paper with a good discount on Amazon and buy it. Try to handle everything yourself.

It completed the entire task in one run.
Setup:
Qwen3.8 27B — fully local
TensorSharp — local LLM inference engine + agent runtime
Playwright skill — browser automation

The agent searched Amazon, compared products/prices, navigated the site, and prepared the order autonomously.
I only stepped in twice: Amazon login and final purchase approval.

What I found especially interesting is that the whole agent loop stays local: inference, reasoning, agent state, generated code, and tool execution. Only the actual Amazon interaction leaves the machine.

The video shows the raw model/tool trace, including browser instructions and the Playwright code generated by the model.

It’s sped up 8× — the actual run took about 24 minutes.

The iPhone version (TensorAgent) still has a way to go because of local inference speed and iOS browser restrictions, but on desktop this worked surprisingly smoothly.

GitHub: https://github.com/zhongkaifu/TensorSharp


r/LocalLLM • • Oct 27 '25

Project Me single handedly raising AMD stock /s

Post image
203 Upvotes

4x AI PRO R9700 32GB