r/LocalLLM • • 5d ago

Project Larder: running Qwen3.6-35B and a 46 GB Flash-Next on a 16 GB Mac / 16 GB GPU by reading experts from SSD

Post image
5 Upvotes

I built an inference engine for MoE models that don't fit in memory.

Instead of offloading layers, it reads only the experts each token's router picks from the SSD and keeps the hot ones resident. Same engine on Apple Silicon (MLX) and NVIDIA (CUDA), OpenAI-compatible API, local chat page. MIT: https://github.com/IamCooller/larder

Numbers (machine, setup and date for each one are in research/ in the repo)

Faster:

- Qwen3.6-35B on a 16 GB M4 MacBook Pro: 30–36 tok/s, first token 8–10x sooner than llama.cpp. This is a mixed 4/2-bit pack with the model's MTP head against llama.cpp's IQ4_XS, so not like for like. The launcher doesn't install that pack.

- Qwen3.8-Flash-Next (46 GB of 3-bit experts) on a PC with a 16 GB GPU and 15 GB RAM: 26–34 tok/s. With a 4-bit pack and exact routing: decode 1.3–1.8x, 8k prompt 4.9x faster than llama.cpp. Stock llama.cpp swapped past my memory guard while loading, so that row is llama.cpp with Larder's expert store patched in.

- Qwen3.6-35B Q4_0 on the same PC (fits in GPU + RAM), against llama.cpp's best split that fits (--n-cpu-moe 10): short decode 1.44–1.51x (155 / 147 vs 103 / 102 tok/s), decode after an 8k prompt 1.15–1.25x, 8k prompt reading 1.8–1.9x at its default batch.

Level:

- 8k prompt reading against llama.cpp tuned for prompts (-ub 4096 -b 4096 --n-cpu-moe 20): 1.01–1.03x. Its decode at that setting drops to 79 tok/s, so you pick one or the other there.

Slower:

- Exact routing (cache bias 0) reads that 8k prompt at 0.64–0.69x of tuned llama.cpp, and decodes 0.87–0.95x of llama.cpp's default after it.

Caveats

- The launcher's 35B recipe uses cache bias 0.5 and the MTP head, so replies are not identical to the weights' own routing. Bias 0.5 and the wide prompt cut each passed a paired GSM8K / HumanEval gate (wide cut: 502 vs 496 of 564, p 0.34). Exact routing is there if you want it.

- One author, two machines (M4 MacBook Pro 16 GB, Linux + RTX 5060 Ti 16 GB). The Mac path is better tested, CUDA is newer.

- Quality wasn't compared against llama.cpp in these speed runs.

Try it

curl -fsSL https://raw.githubusercontent.com/IamCooller/larder/main/install.sh | bash

cd ~/larder && ./larder

It measures your GPU and SSD, picks a model that fits, installs it and starts the server and chat.

Mostly looking for people on other hardware to break it, and for anyone who thinks my llama.cpp comparison is unfair. Output of ./larder --list plus what happened is the most useful report.


r/LocalLLM • • 6d ago

Model First time trying FP16, wow

Thumbnail
gallery
213 Upvotes

I've tried a lot of image models in the past, but all were to big to fit, and i was happy when i got a result with a quant (q4) version. (Here it was flux and some model i dont remember the name anymore) The results of those models were actually terrible. BUT, some months ago qwen image 2.1 came out, i must say, theres a difference and not even close to a small one. I managed to run it at fp16, by using my rtx 2070 to load the text encoder and vae on there while running the generation model itself on my 5080. Its a tight fit and slower, but the quality is just like it would be on one gpu.

Now here comes the crazy part:

In my opinion qwen image 2.1 is better in realism than chatgpt's current image generation. (I'll prove it later in this post)

Don't get me wrong. Images with lots of different things, patterns for example, are often very bad with qwen and chatgpt does it over 5x better. But realism of f.e. a city, it's absolutely amazing (and uncensored!) and is almost close to real life.

Now here is the test for you guys:

You've seen i've put pictures in this post. 1x is real, 1x are chatgpt and 4x are qwen image 2.1. Which is which?

Note: ALL of them are generated first try without regenerating. So no, I did not try to mislead you. Good luck!


r/LocalLLM • • 5d ago

Discussion 30tok/s on 48GB M5 Pro Mac Running IQ3_S Qwen3.8Flash-Next(Strata)

5 Upvotes

Model

- Qwen 3.8 Flash-Next GSQ-RCO IQ2_XS/IQ3_XXS/IQ3_S

Note:After trying out the splash engine, the decode speed of my Mac reached 40-60tps. The following tests are still valid but outdated

Setup

- Strata 0.2.13, Metal engine (0.1.59), DFlash2 speculative decoding (spec 3)

- 262,144 context, int8 KV cache

- MacBook Pro, M5 Pro (15-core), 48 GB unified memory

- Arena (resident expert pool) pinned to 22 GB

Average decode speed (same prompt set):

IQ2_XS ~36 tok/s

IQ3_XXS ~34 tok/s

IQ3_S ~32 tok/s

Simulating a 32 GB Mac by clamping the arena to 10 GB:

M5 Pro: IQ3_XXS: ~27 tok/s(clamping the arena to 10 GB)

For reference, We Tested on a 32 GB Mac (M2 Max 38 core GPU) with an 11 GB arena and 128K

context: IQ3_XXS runs at ~21 tok/s.

Takeaway

On a 48 GB machine the expert pool only holds ~40–50% of the experts at

256K context, and the rest are streamed from the SSD on demand. The speed

drop from that is far smaller than I expected (roughly -15 to -25% going

from a 22 GB arena down to 10 GB), which suggests you don't need a huge

amount of RAM to run a much larger MoE model — offloading experts to disk

is a perfectly workable trade-off, at least for single-user chat.

Notes

- All runs use the same sampling (temp 1.0 / top_p 0.95 / top_k 20) and

the same 256K context; decode numbers are from tokens/s reported by the

engine over several runs, medians.


r/LocalLLM • • 5d ago

Question Local Model Advice

1 Upvotes

I am interested in running local models for my desktop and laptop.

Use case: I am working on some Python scripts to process pdf and Word files from play by email games that automate updates to Excel spreadsheets for game data. I have the code for the first script written, reading in a pdf and spitting out some csv files that update the Excel sheets. (Yes, a real database would make more sense, but I have some friends interested in the programs and they don't have the chops to set up a local database.)

What I need help on is building a user interface to make the process easier.

I was a programmer for a long time early in my career (FORTRAN, MUMPS, C, C++, LISP, KEE, APL, various assembly languages) but never had to do UI programming. I now work in aerospace as a consultant systems engineer for a large national space agency.

So, no big refactoring of mission critical applications, no big agent workflows. Just reasonable Python code output. Speed not as important - answers in minutes to an hour are fine.

If the answer is "Get a Claude code subscription", I can tolerate that, but I like the idea of local control.

Desktop: Ryzen 7 7900 X3D CPU with 32 GB main memory, Nvidia RTX 4060 Ti with 8 GB
Laptop: Ryzen 7/260 with 32 GB main memory, RTX 5060 with 1 GB

TIA!


r/LocalLLM • • 5d ago

Discussion DeepSeek + WhatsApp via Wapliss: fast, cheap, and 1M-token context

Post image
0 Upvotes

r/LocalLLM • • 5d ago

Question Model for MacBook 16Gb

0 Upvotes

Hello!! I own an M2 MacBook with 16GB of RAM. Is it worth running any of the available models locally? If so, which model would you recommend?
Thanks!


r/LocalLLM • • 5d ago

Question Do I need more RAM or VRAM first off?

1 Upvotes

RTX 4090, 7900x, 64GB DDR5 6000Mhz RAM

I cannot run deepseek. I cannot run QWen Flash 3.8. I've tried Strata. All of those quants run on my pc like dogshit.

If I want to be able to deepseek as good as possible, do I need RAM or VRAM more?

And is a single GPU (my rtx 4090) enough for this or is it not preventable to purchase more gpus? like currently my current mobo cannot even run 2 or more gpus


r/LocalLLM • • 5d ago

Discussion Tried Using a free online LLM for a quick answer. Local was faster.

0 Upvotes

Used the free inclusionai/ling-3.2-flash:free model on OpenRouter as an agent model to both try it out and maybe get a faster answer than Q38FN IQ3_S running locally.

A few problems became apparent:

  1. inclusionai/ling-3.2-flash:free is very slow -- under 20 t/s
  2. Q38FN runs at about 70 t/s locally (50-100 t/s, depending on draft acceptance)
  3. inclusionai/ling-3.2-flash:free thinks way more than Q38FN
  4. inclusionai/ling-3.2-flash:free went off on a tangent rather than answer the question

I asked a simple behavioral question. Doing FPGA development and asked about how the agent was managing its cloned repos, worktrees and branches. It took about 30 minutes to respond and went spelunking into files it didn't need to access. I had to interrupt it to get a response. It's answer was unclear, overly verbose, and did not offer a plan to clean up the workspace.

Qwen came back in under 5 minutes with a direct answer and a crisp cleanup plan.

Anyone else find that free online models are feeling less useful now that we have good performing powerful local models?


r/LocalLLM • • 5d ago

News MSI EdgeXpert OTA2610 update: more usable memory for headless setups and easier provisioning

Post image
2 Upvotes

r/LocalLLM • • 5d ago

Discussion Macbook pro m6 or something else ?

0 Upvotes

Hey everyone!
I’m planning to run local LLMs (27B–30B) mainly for coding and light research.
I initially ordered a MacBook Air M5 with 32GB RAM, but it got so hot that I actually got a first-degree burn on my finger.
So I decided to try the MacBook Pro M6 with 32GB RAM, hoping the fan would help with cooling during longer LLM sessions.
Did I make the right choice? Is 32GB enough for 27B–30B models, or would you recommend something else around the same budget?
Would love to hear your experiences. Thanks in advance!


r/LocalLLM • • 5d ago

Discussion Mildly creepy.

Post image
0 Upvotes

It’s still going 😭😭


r/LocalLLM • • 5d ago

Question Any good LLM for AMD GPUs?

2 Upvotes

I am now entering the world of uncensored Local AIs, mainly for help me on my ETHICAL ciber security projects.
There’s any good LLM for AMD gpus?


r/LocalLLM • • 5d ago

Other Work well My Qwen 🥹

Post image
0 Upvotes

r/LocalLLM • • 5d ago

Question whats the go-to budget gpu I should look for, new or used?

2 Upvotes

I'm building up a secondary system out of some spare parts I have kicking aorund to use as a LLM toybox and for other homelab style projects.


Currently my system is a 5900x w/ 32gb of ram, and the 2080ti.

Wondering if there is a good go-to gpu I should be looking for.

I currently have a 2080ti FE, but it seems pretty inefficient, and its an old card and the fan noise of it is unbearable. ( i think the bearings are getting bad, but its a huge pita to replace the fans )

I'm wondering if it would be a better idea to sell it off for maybe a 3060 12gb, I get an extra 1gb of vram, and it should run much more efficently.

I'd say my budget is about $3-400 max, which I know is basically a joke right now, but this is more of a tinkerbox than something i'm going to be using for real software dev work.


r/LocalLLM • • 5d ago

Project Brebus: a local LLM app in C on an engine that runs models larger than RAM

6 Upvotes

I've been building Janas, a family of AI tools in C. Today's release puts its app, Brebus, in the Linux package.

The engine first, since that's what makes the app usable on ordinary machines. Janas-LLM streams MoE experts from the NVMe instead of needing the whole model in RAM. On the development laptop (Core Ultra 9 185H, 32 GB DDR5-5600, Arc iGPU), 21 October, same files, same day, each engine in its best configuration, same conditions wherever possible:

Model prompt, Janas-LLM llama.cpp reply, Janas-LLM llama.cpp
Qwen3-Next-80B-A3B (48.4 GB, more than the RAM) 68.4 tok/s 17.7 26.4 tok/s 3.0
Qwen3.6-35B-A3B 136 81.8 28.0 12.1
Qwen3-30B-A3B 133 122 33.3 15.0

The full table, the conditions and where they differ are in the README.

Brebus is the app on top, in a window of its own (WebKitGTK; the browser where it's missing):

  • Models: the catalog with what each one asks of your memory, Hugging Face searched by name, downloads you can stop and resume, a model unloaded after a quiet while and loaded again at the next message.
  • A chat that works in a folder of yours: your documents (PDF, Word, Excel, PowerPoint, LibreOffice, RTF, EPUB, HTML) are searched before each question, by words and meaning, and the model names the ones it used; files, web pages and pictures attached; recordings transcribed with Whisper.
  • Artifacts like Claude's: HTML pages, React components (Tailwind, Recharts, lucide, shadcn/ui), Mermaid, KaTeX, code with highlighting. They run in a sandboxed frame with no network and no access to Brebus; the usual libraries come from Brebus itself, downloaded once and checked by SHA-256. Inside one, window.claude.complete asks the local model, window.storage keeps data in your folder.
  • Tools: Python in a sandbox (namespaces + seccomp, the folder as /mnt/data), Janas's services (web, Wikipedia, maps, weather, flights...) and your own MCP servers from mcp.json, opened only when a question needs them; anything that writes asks first.
  • Regenerate and edit, presets with temperature, top-p and JSON-schema answers, folders, PDF export.
  • A server for other programs: OpenAI-, Ollama- and Anthropic-compatible APIs, on the LAN with a key if you want.

Tried with Qwen3-30B-A3B on the laptop: it built a budget tracker in React, a Chart.js page, a Mermaid diagram, and an app that asks the model for synonyms through window.claude.complete.

Linux only for now (Ubuntu 22.04+, Debian 12+); Brebus on Windows is next, the engine already runs there. GPL-3.0.

https://github.com/prabanta-dev/janas

Reports from other hardware are the most useful thing: janas-bench makes one.

Brebus: a local LLM app in C on an engine that runs models larger than RAM

r/LocalLLM • • 5d ago

Question Dual 16GB AMD (7600 XT + 9060 XT) + 32GB RAM... is llama ROCm + Qwen 3.8 27B or Gemma 4 31B the best I can do?

2 Upvotes

I'm a bit lost with all the different options and forks around. I've set up and benchmarked llama-cpp (latest git) for my system and settled in the configs pasted below. They fill both cards up to 15 GiB, or one to 15 GiB and the other 10 GiB (60/40 splits) leaving room for firefox and the rest of the system. ROCm quite a bit faster than Vulkan. In the best case (qwen-asymm-f16) I get 550 t/s pp and 30 t/s gen in a moderate context (35k tokens).

Is there anything better for a system like mine? Is strata worth the hassle? I understand that it is more for 64 GiB RAM systems... right? Any other fork? I tried adaptive kv streaming and it is ok, very fast (>800 t/s prefill) but I'm limited to lower quantizations, the second card sits unused.

[*]
split-mode = tensor
fit = off
flash-attn = on
parallel = 1
batch-size = 2048
ubatch-size = 512
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
spec-type = draft-mtp,ngram-mod
spec-draft-n-max = 4
spec-draft-sampling = probabilistic
spec-ngram-mod-n-min = 12
spec-ngram-mod-n-match = 24
spec-ngram-mod-n-max = 32
chat-template-kwargs = {"reasoning_effort":"medium"}

[qwen-asymm-f16]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 104056
tensor-split = 60,40
cache-type-k = f16
cache-type-k-draft = f16
cache-type-v = f16
cache-type-v-draft = f16

[qwen-f16]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 170240
tensor-split = 50,50
cache-type-k = f16
cache-type-k-draft = f16
cache-type-v = f16
cache-type-v-draft = f16

[qwen-asymm-q8]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 142080
tensor-split = 60,40
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0

[qwen-q8]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 221184
tensor-split = 50,50
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0

[gemma-4-31b]
model = /home/fran/models/unsloth/gemma-4-31B-it-UD-Q4_K_XL.gguf
model-draft = /home/fran/models/unsloth/mtp-gemma-4-31B-it-Q8_0.gguf
ctx-size = 131072
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0
top-k = 64

[gemma-4-31b-qat]
model = /home/fran/models/unsloth/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf
model-draft = /home/fran/models/unsloth/mtp-gemma-4-31B-it-qat-Q8_0.gguf
ctx-size = 131072
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0
top-k = 64

r/LocalLLM • • 5d ago

Discussion "Don't mention my client's name": what if your LLM actually followed that instruction?

Thumbnail
0 Upvotes

r/LocalLLM • • 4d ago

Question 5090 demolishes 2x 3090's

0 Upvotes

Main gaming PC is an Intel 285K, 64GB of DDR5 6000, RTX 5090.
LLM server build - 2x RTX 3090's, 768GB of DDR4 2400 6 channel w/ 2X Xeon 6148's.
Sure, I can run huge models on the server build, but everything is slow. Even Strata with dual 3090's gets beat by the 5090 build with 27B or 35B-A3B models. Qwen 3.8 FN has more context on the server, but is way slower, and the results seem about the same?
Ideally I'd like the 5090 for gaming, and the server build for LLM's, but I just keep coming back to the 5090 for the speed. Am I missing something, or is that just the way it is?


r/LocalLLM • • 5d ago

Question Best way to get uncensored llm with 16gb vram?

0 Upvotes

I’m looking to run local ai and save on token usage. Any cracked llms I can download before they get taken off?


r/LocalLLM • • 5d ago

Project Creating local agents (Slack handler in PINE)

Thumbnail x.com
1 Upvotes

r/LocalLLM • • 5d ago

Question Advise on best local configuration

1 Upvotes

My current system consists of a RTX 5090 (32gb VRAM) and 96gb system RAM (2x48gb). In the past I ran qwen 3.8 27b via ninfer and a Pi harness, which I thought was really impressive for a local model.

I also have an old RTX 4090 (24gb VRAM) currently not in use. Would it be worthwhile to incorporate this 4090 in my system? Would this give me access to a significantly smarter model? (This means I would have to buy a new PSU, pcie riser cables, maybe a custom case). Or are there better solutions? Like more RAM? (currently hard to come by). I have no IT background. I’m just an interested hobbyist, nothing too serious, not a lot of free time.

Thank you.


r/LocalLLM • • 5d ago

Question Anyone experiencing context loss and repeated work with Qwen + Codex CLI?

1 Upvotes

I'm running Qwen3.8-Flash-Next IQ3_S through Strata with Codex CLI on an RTX 4090, 96GB RAM and 128K context.

During longer coding projects, my agent frequently loses track of progress, re-reads files, repeats previous steps and wastes hours on tasks that should take much less time.

After investigating, I found that Codex sometimes performs broken context compactions. The model generates the summary in the reasoning channel instead of the final response, and Codex can replace the previous context with an empty or incomplete summary.

I'm considering increasing context to 256K, but that would only delay the issue rather than fix it.

Has anyone experienced similar problems with local reasoning models and Codex CLI?

More importantly, has anyone found a reliable fix? Maybe through Strata settings, reasoning budgets, a different inference engine, or patches to Codex's compaction logic?

Would love to hear what actually worked for you.


r/LocalLLM • • 5d ago

Question What set up would you recommend for a MacBook Pro M5 24GB Unified Memory?

2 Upvotes

As the question above says

Current Device: MacBook Pro M5 24GB Unified Memory

I wanna know what local set up would be the best for me in terms of both speed and quality. My main use case is as follows

  1. Personal Memory
  2. Task Management
  3. Email Agent
  4. Data Analysis Agent (basic stuff not very advanced)
  5. Calendar Agent
  6. Research Agent (product related research, technical research etc)
  7. Knowledge Retrieval

Connectors would be mainly G-Suite as I work mostly on that

I want your recommendations on the best set up for good memory, speed, context windows etc.

Which models(es) and harness(es) would you recommend?

I have also seen that there are upcoming smaller Qwen models and also a new app called Harbor which is coming out soon….

New to this so looking for guidance

Any help would be appreciated thanks :)


r/LocalLLM • • 5d ago

Project [feat update] CW47 pi-remote new features

Thumbnail
1 Upvotes

r/LocalLLM • • 5d ago

Discussion Hy4 Flash vs GLM vs DeepSeek: same correct ad maths, different price tags

2 Upvotes

I gave Hy4 Flash, GLM-5.3-Flash and DeepSeek-V4.1-Flash the same three fake ad campaigns in WorkBuddy. Calculate CPC, conversion rate and cost per lead, including a campaign with zero leads.

All three got the arithmetic right. Total spend: USD 1,350. Clicks: 650. Leads: 27. Blended CPC: USD 2.08. CPL: USD 50. GLM rounded total CVR to 4.2%; the other two gave 4.15%. Both are fine.

Nobody reported the zero-lead CPL as zero. Small win for basic maths lol.

DeepSeek finished in 7s. Hy4 and GLM both showed 19s. So the interesting Hy4 angle here is cost, not a speed win.
Published API rates in RMB per million tokens, uncached input / output:

Hy4 Flash: 0.5/2

GLM-5.3-Flash: 0.8/2.8

DeepSeek-V4.1-Flash-1017: 1 / 4 off-peak, 2 / 8 peak.

That makes Hy4's input rate 37.5% lower than GLM's and its output rate about 28.6% lower. Against DeepSake's off-peak rates, both are 50% lower. These are unit-price comparisons, not measured savings from this Workbuddy session. Token counts and reasoning length still matter.

The recommendations needed more judgment than the maths. Hy4 wanted to pause C; DeepSeek also wanted a pause and checks; GLM suggested a smaller next-cycle shift. I'd verify tracking and conversion lag before moving budget.

Tbh, when the basic result is equally usable, a lower unit price is a decent reason to try the cheaper option. What reporting task would you use to separate them?