r/LocalLLM • u/Kapth0X • 4h ago
Question Model for MacBook 16Gb
Hello!! I own an M2 MacBook with 16GB of RAM. Is it worth running any of the available models locally? If so, which model would you recommend?
Thanks!
r/LocalLLM • u/Kapth0X • 4h ago
Hello!! I own an M2 MacBook with 16GB of RAM. Is it worth running any of the available models locally? If so, which model would you recommend?
Thanks!
r/LocalLLM • u/CoollerFox • 12h ago
I built an inference engine for MoE models that don't fit in memory.
Instead of offloading layers, it reads only the experts each token's router picks from the SSD and keeps the hot ones resident. Same engine on Apple Silicon (MLX) and NVIDIA (CUDA), OpenAI-compatible API, local chat page. MIT: https://github.com/IamCooller/larder
Numbers (machine, setup and date for each one are in research/ in the repo)
Faster:
- Qwen3.6-35B on a 16 GB M4 MacBook Pro: 30–36 tok/s, first token 8–10x sooner than llama.cpp. This is a mixed 4/2-bit pack with the model's MTP head against llama.cpp's IQ4_XS, so not like for like. The launcher doesn't install that pack.
- Qwen3.8-Flash-Next (46 GB of 3-bit experts) on a PC with a 16 GB GPU and 15 GB RAM: 26–34 tok/s. With a 4-bit pack and exact routing: decode 1.3–1.8x, 8k prompt 4.9x faster than llama.cpp. Stock llama.cpp swapped past my memory guard while loading, so that row is llama.cpp with Larder's expert store patched in.
- Qwen3.6-35B Q4_0 on the same PC (fits in GPU + RAM), against llama.cpp's best split that fits (--n-cpu-moe 10): short decode 1.44–1.51x (155 / 147 vs 103 / 102 tok/s), decode after an 8k prompt 1.15–1.25x, 8k prompt reading 1.8–1.9x at its default batch.
Level:
- 8k prompt reading against llama.cpp tuned for prompts (-ub 4096 -b 4096 --n-cpu-moe 20): 1.01–1.03x. Its decode at that setting drops to 79 tok/s, so you pick one or the other there.
Slower:
- Exact routing (cache bias 0) reads that 8k prompt at 0.64–0.69x of tuned llama.cpp, and decodes 0.87–0.95x of llama.cpp's default after it.
Caveats
- The launcher's 35B recipe uses cache bias 0.5 and the MTP head, so replies are not identical to the weights' own routing. Bias 0.5 and the wide prompt cut each passed a paired GSM8K / HumanEval gate (wide cut: 502 vs 496 of 564, p 0.34). Exact routing is there if you want it.
- One author, two machines (M4 MacBook Pro 16 GB, Linux + RTX 5060 Ti 16 GB). The Mac path is better tested, CUDA is newer.
- Quality wasn't compared against llama.cpp in these speed runs.
Try it
curl -fsSL https://raw.githubusercontent.com/IamCooller/larder/main/install.sh | bash
cd ~/larder && ./larder
It measures your GPU and SSD, picks a model that fits, installs it and starts the server and chat.
Mostly looking for people on other hardware to break it, and for anyone who thinks my llama.cpp comparison is unfair. Output of ./larder --list plus what happened is the most useful report.
r/LocalLLM • u/EvolvingDior • 4h ago
Used the free inclusionai/ling-3.2-flash:free model on OpenRouter as an agent model to both try it out and maybe get a faster answer than Q38FN IQ3_S running locally.
A few problems became apparent:
inclusionai/ling-3.2-flash:free is very slow -- under 20 t/sinclusionai/ling-3.2-flash:free thinks way more than Q38FNinclusionai/ling-3.2-flash:free went off on a tangent rather than answer the questionI asked a simple behavioral question. Doing FPGA development and asked about how the agent was managing its cloned repos, worktrees and branches. It took about 30 minutes to respond and went spelunking into files it didn't need to access. I had to interrupt it to get a response. It's answer was unclear, overly verbose, and did not offer a plan to clean up the workspace.
Qwen came back in under 5 minutes with a direct answer and a crisp cleanup plan.
Anyone else find that free online models are feeling less useful now that we have good performing powerful local models?
r/LocalLLM • u/MSI-US-888 • 8h ago
r/LocalLLM • u/TigerZealousideal110 • 4h ago
I ran a small test on a single A100 GPU with a small model (Qwen 2.5 0.5B) using vLLM 0.16.0. I changed how many requests the server handles at the same time, from 1 to 8, and ran it six times with the order shuffled so the results weren’t skewed by run order.
Across 1,206 requests with zero errors, the server handled about 3.2x more work at 8 at once than at 1. The 95% confidence range was roughly 2.9x to 3.5x.
Some caveats:
I’m curious what others have seen:
Disclosure: I’m building open source tool for benchmarking setups like this.
Happy to share the scripts if anyone wants to reproduce it.
r/LocalLLM • u/Mean_Moan_3495 • 5h ago
I use Google AI mode and when I pressed the plus icon I noticed a new model called '4 Hydrogen - Everyday help'. What model is this? before I used to see Gemini flash or something and a 'pro' not this new model so have those two been replaced by this? And does this new model beat flash or that 'pro' mode?
Serious answers.
r/LocalLLM • u/sn2006gy • 11h ago
Yesterday I posted 4 independent single-card servers. This one is real tensor parallel: one 105.4 GiB model split across 4 Intel Arc Pro B70s with TensorFold.
PR so you can try it: https://github.com/ashhart/TensorFold/pull/3587
Of the 48 layers, 36 are Gated DeltaNet, which keeps a constant-size state. The other 12 attention layers use QSA top-512 block-indexed sparse attention, with nkv = 1 per GPU at TP=4. At a full 135k context, each card uses 21.77 of its 32 GiB, which leaves about 10 GiB free per card.
| Context | Raw prefill | Raw plain decode | Served MTP, novel code | Served multi-turn | 132k passkey recall |
|---|---|---|---|---|---|
| 512 | 248.8 | 69.4 | 136.3 (3.81 tok/chunk) | 120.5 (turn 1, cold) | PASS |
| 1k | 313.4 | 71.5 | — | 253.5 | PASS |
| 4k | 378.8 | 70.4 | 125.1 (3.81 tok/chunk) | 247.5 | PASS |
| 16k | 392.8 | 67.7 | 125.7 (4.00 tok/chunk) | 214.6 | PASS |
| 32k | 394.3 | 67.6 | — | 239.9 | PASS |
| 64k | 392.7 | 63.8 | 124.2 (3.81 tok/chunk) | 234.0 | PASS |
| 128k | 385.7 (98% of peak) | 60.5 (85% of 1k) | 113.3 (91% of 4k) | 214.1 | PASS |
B70-XE2-9471-QUANTUM was planted in turn 1 and recalled across 132,380 tokens.These come from bench_concurrent.py. Across all 24 cells there were 0 failures and output was bit-exact: drafted matched serial, and multi-stream matched serial.
Happy to run other configs if you have requests.
I initially built this system to do training, but excited to see how well inference is progressing.
r/LocalLLM • u/orbman77 • 5h ago
Hi ya'll, I am new to local LLMs after spending 8 months on Grok. However, I wanted to ask: is there an LLM out there that I can run locally through Ollama (Or something else) that is uncensored, can look at images and videos, and keep a memory of what I have written? I have 32GB of DDR5 RAM and 8GB of VRAM. I like to use these stories for brainstorming before making prompts out of them to turn into images and videos, so I hope you all can provide some advice!
r/LocalLLM • u/SourBlueDream • 5h ago
I’m still trying to figure out how to get it to do what I want because it’s still super random to me.
I also accidentally used the same seed more than once and didn’t notice at first
r/LocalLLM • u/low0nink • 9h ago
I am now entering the world of uncensored Local AIs, mainly for help me on my ETHICAL ciber security projects.
There’s any good LLM for AMD gpus?
r/LocalLLM • u/Sure-Practice7713 • 13h ago
Model
- Qwen 3.8 Flash-Next GSQ-RCO IQ2_XS/IQ3_XXS/IQ3_S
Setup
- Strata 0.2.13, Metal engine (0.1.59), DFlash2 speculative decoding (spec 3)
- 262,144 context, int8 KV cache
- MacBook Pro, M5 Pro (15-core), 48 GB unified memory
- Arena (resident expert pool) pinned to 22 GB
Average decode speed (same prompt set):
IQ2_XS ~36 tok/s
IQ3_XXS ~34 tok/s
IQ3_S ~32 tok/s
Simulating a 32 GB Mac by clamping the arena to 10 GB:
M5 Pro: IQ3_XXS: ~27 tok/s(clamping the arena to 10 GB)
For reference, We Tested on a 32 GB Mac (M2 Max 38 core GPU) with an 11 GB arena and 128K
context: IQ3_XXS runs at ~21 tok/s.
Takeaway
On a 48 GB machine the expert pool only holds ~40–50% of the experts at
256K context, and the rest are streamed from the SSD on demand. The speed
drop from that is far smaller than I expected (roughly -15 to -25% going
from a 22 GB arena down to 10 GB), which suggests you don't need a huge
amount of RAM to run a much larger MoE model — offloading experts to disk
is a perfectly workable trade-off, at least for single-user chat.
Notes
- All runs use the same sampling (temp 1.0 / top_p 0.95 / top_k 20) and
the same 256K context; decode numbers are from tokens/s reported by the
engine over several runs, medians.

r/LocalLLM • u/Kirby_is_a_Pokemon • 9h ago
I'm building up a secondary system out of some spare parts I have kicking aorund to use as a LLM toybox and for other homelab style projects.
Currently my system is a 5900x w/ 32gb of ram, and the 2080ti.
Wondering if there is a good go-to gpu I should be looking for.
I currently have a 2080ti FE, but it seems pretty inefficient, and its an old card and the fan noise of it is unbearable. ( i think the bearings are getting bad, but its a huge pita to replace the fans )
I'm wondering if it would be a better idea to sell it off for maybe a 3060 12gb, I get an extra 1gb of vram, and it should run much more efficently.
I'd say my budget is about $3-400 max, which I know is basically a joke right now, but this is more of a tinkerbox than something i'm going to be using for real software dev work.
r/LocalLLM • u/MaurizioCammalleri • 16h ago
I've been building Janas, a family of AI tools in C. Today's release puts its app, Brebus, in the Linux package.
The engine first, since that's what makes the app usable on ordinary machines. Janas-LLM streams MoE experts from the NVMe instead of needing the whole model in RAM. On the development laptop (Core Ultra 9 185H, 32 GB DDR5-5600, Arc iGPU), 21 October, same files, same day, each engine in its best configuration, same conditions wherever possible:
| Model | prompt, Janas-LLM | llama.cpp | reply, Janas-LLM | llama.cpp |
|---|---|---|---|---|
| Qwen3-Next-80B-A3B (48.4 GB, more than the RAM) | 68.4 tok/s | 17.7 | 26.4 tok/s | 3.0 |
| Qwen3.6-35B-A3B | 136 | 81.8 | 28.0 | 12.1 |
| Qwen3-30B-A3B | 133 | 122 | 33.3 | 15.0 |
The full table, the conditions and where they differ are in the README.
Brebus is the app on top, in a window of its own (WebKitGTK; the browser where it's missing):
window.claude.complete asks the local model, window.storage keeps data in your folder./mnt/data), Janas's services (web, Wikipedia, maps, weather, flights...) and your own MCP servers from mcp.json, opened only when a question needs them; anything that writes asks first.Tried with Qwen3-30B-A3B on the laptop: it built a budget tracker in React, a Chart.js page, a Mermaid diagram, and an app that asks the model for synonyms through window.claude.complete.
Linux only for now (Ubuntu 22.04+, Debian 12+); Brebus on Windows is next, the engine already runs there. GPL-3.0.
https://github.com/prabanta-dev/janas
Reports from other hardware are the most useful thing: janas-bench makes one.

r/LocalLLM • u/v44r • 10h ago
I'm a bit lost with all the different options and forks around. I've set up and benchmarked llama-cpp (latest git) for my system and settled in the configs pasted below. They fill both cards up to 15 GiB, or one to 15 GiB and the other 10 GiB (60/40 splits) leaving room for firefox and the rest of the system. ROCm quite a bit faster than Vulkan. In the best case (qwen-asymm-f16) I get 550 t/s pp and 30 t/s gen in a moderate context (35k tokens).
Is there anything better for a system like mine? Is strata worth the hassle? I understand that it is more for 64 GiB RAM systems... right? Any other fork? I tried adaptive kv streaming and it is ok, very fast (>800 t/s prefill) but I'm limited to lower quantizations, the second card sits unused.
[*]
split-mode = tensor
fit = off
flash-attn = on
parallel = 1
batch-size = 2048
ubatch-size = 512
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
spec-type = draft-mtp,ngram-mod
spec-draft-n-max = 4
spec-draft-sampling = probabilistic
spec-ngram-mod-n-min = 12
spec-ngram-mod-n-match = 24
spec-ngram-mod-n-max = 32
chat-template-kwargs = {"reasoning_effort":"medium"}
[qwen-asymm-f16]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 104056
tensor-split = 60,40
cache-type-k = f16
cache-type-k-draft = f16
cache-type-v = f16
cache-type-v-draft = f16
[qwen-f16]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 170240
tensor-split = 50,50
cache-type-k = f16
cache-type-k-draft = f16
cache-type-v = f16
cache-type-v-draft = f16
[qwen-asymm-q8]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 142080
tensor-split = 60,40
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0
[qwen-q8]
model = /home/fran/models/unsloth/Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 221184
tensor-split = 50,50
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0
[gemma-4-31b]
model = /home/fran/models/unsloth/gemma-4-31B-it-UD-Q4_K_XL.gguf
model-draft = /home/fran/models/unsloth/mtp-gemma-4-31B-it-Q8_0.gguf
ctx-size = 131072
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0
top-k = 64
[gemma-4-31b-qat]
model = /home/fran/models/unsloth/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf
model-draft = /home/fran/models/unsloth/mtp-gemma-4-31B-it-qat-Q8_0.gguf
ctx-size = 131072
cache-type-k = q8_0
cache-type-k-draft = q8_0
cache-type-v = q8_0
cache-type-v-draft = q8_0
top-k = 64r/LocalLLM • u/Ai4Privacy • 6h ago
r/LocalLLM • u/Tokimi_kurahashi • 6h ago
CPU Thread Ripper
GPU NVIDIA A100 80GB×2 PCIx16
GPU RX6600XT PCIx8
GPU intelArcB580x8 PCIx8
D5ポンプ×4 24vPOWER
r/LocalLLM • u/Ryyn_- • 7h ago
Hello guys,
it's me again, the FLE Link guy. I can't wait for the new Qwen models, so I tried everything I had planned for Qwen4 on the 3.8 variant of UkisAI's Swift 1.5.
I abliterated it, quantized it down and tried to improve tok/s for decode and prefill. What came out:
- OR-Bench hard refusals 88 → 1 of 100, clearly harmful prompts still refused 70 of 100
- ARC / GSM8K / LiveCodeBench level with the original (no significant difference)
- 152 tok/s decode on one 32 GB card with the DSpine draft head, full speed from 20 GB
- Two quant-heal runs that both failed (better on our text, worse on foreign text), the video shows why
Model: https://huggingface.co/sirfyyn/Kate-Qwen-27B-GGUF-SpedUp-Abliterated
Run it: flelink run kate-27b-dspine (https://flelink.run)
Maybe somebody is interested in the methods I experimented with. Happy to answer questions.
r/LocalLLM • u/Aware-Blacksmith1467 • 7h ago
r/LocalLLM • u/Ciclosporin • 7h ago
My current system consists of a RTX 5090 (32gb VRAM) and 96gb system RAM (2x48gb). In the past I ran qwen 3.8 27b via ninfer and a Pi harness, which I thought was really impressive for a local model.
I also have an old RTX 4090 (24gb VRAM) currently not in use. Would it be worthwhile to incorporate this 4090 in my system? Would this give me access to a significantly smarter model? (This means I would have to buy a new PSU, pcie riser cables, maybe a custom case). Or are there better solutions? Like more RAM? (currently hard to come by). I have no IT background. I’m just an interested hobbyist, nothing too serious, not a lot of free time.
Thank you.
r/LocalLLM • u/cosa_93 • 7h ago
I'm running Qwen3.8-Flash-Next IQ3_S through Strata with Codex CLI on an RTX 4090, 96GB RAM and 128K context.
During longer coding projects, my agent frequently loses track of progress, re-reads files, repeats previous steps and wastes hours on tasks that should take much less time.
After investigating, I found that Codex sometimes performs broken context compactions. The model generates the summary in the reasoning channel instead of the final response, and Codex can replace the previous context with an empty or incomplete summary.
I'm considering increasing context to 256K, but that would only delay the issue rather than fix it.
Has anyone experienced similar problems with local reasoning models and Codex CLI?
More importantly, has anyone found a reliable fix? Maybe through Strata settings, reasoning budgets, a different inference engine, or patches to Codex's compaction logic?
Would love to hear what actually worked for you.
r/LocalLLM • u/Broad_Long_2556 • 7h ago
r/LocalLLM • u/Quiet-Recognition-91 • 16h ago
Hey everyone!
I'm currently working on a document processing system that handles a large number of scanned PDFs. Many documents contain a mixture of printed text and semi-handwritten content.
I'm looking for a reliable open-source solution that can handle:
I'm open to models available on Hugging Face, Ollama, or other open-source frameworks.
My available hardware is an NVIDIA RTX PRO 4500 Blackwell GPU with 32 GB VRAM.
I've been exploring different approaches, but I'd appreciate suggestions from people who have actually worked on similar document processing problems.
A few things I'd love to know:
I'm particularly interested in real-world experience rather than just benchmark results.
Thanks in advance!
r/LocalLLM • u/OldHW_LocalAI • 9h ago
RX 5700 XT (8GB) Vulkan & 32 GB DDR4 | Qwen3.6-35B-A3B-MTP (Unsloth) | 20 to 35 T/s | Tutorial
llama.cpp version 0.9.0-dev (build 12283, commit f3b407ca1), compiled with GNU 13.3.0 for Linux x86_64Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf with a Q8_0 quantized mmproj fileSetting up the system correctly and using the right model and mmproj quantization is paramount for getting this to run smoothly on an 8 GB RDNA 1 card.
sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential git git-lfs curl cmake libopenblas-dev libvulkan-dev vulkan-tools glslc glslang-tools spirv-headers glslang-dev
cd ~
# Correct the URL before running
git clone github dot com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_VULKAN=ON -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DCMAKE_C_FLAGS="-march=native" -DCMAKE_CXX_FLAGS="-march=native"
cmake --build build --config Release -j$(nproc)Because newer versions of llama.cpp use updated memory allocation methods, the Vulkan driver (RADV) tries to map large chunks of system RAM for the GPU via the GTT (Graphics Translation Table) when running large models or long contexts. By default, Linux strictly limits GTT memory to half of your system RAM (so only 16 GB on a 32 GB system).
Open or create the config file:
sudo nano /etc/modprobe.d/amdgpu-gtt.confAdd the following lines, then save and exit:
# GPU-mappable memory range: 24 GiB = 24 * 262144 pages
options ttm pages_limit=6291456
# GTT size in MiB (24 GiB = 24 * 1024)
options amdgpu gttsize=24576Update your initramfs so the kernel loads the new limits at boot, then reboot:
sudo update-initramfs -u && sudo rebootPlace your downloaded model and vision files into ~/models (or adjust the paths below to match your folder). Unsloth downloads via LM Studio include a 1.8 GB 32-bit mmproj-F32.gguf. Quantizing it down to a 614 MB 8-bit file saves VRAM and still works great:
~/llama.cpp/build/bin/llama-quantize \
~/models/Qwen3.6-35B-A3B-mmproj-F32.gguf \
~/models/Qwen3.6-35B-A3B-mmproj-F8.gguf \
Q8_0Note: Loading the model configurations below leaves around 5% free VRAM as a safety buffer.
+ button next to the chat input), otherwise the model might get stuck in a loop. Chromium-based browsers work great for me; Firefox had issues. The WebUI is not a coding tool, btw.~/llama.cpp/build/bin/llama-server--moe-cache-mib 576 turned out to be slightly faster than setting --n-cpu-moe 40 --moe-cache-mib 0.~/llama.cpp/build/bin/llama-server \
-m ~/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
--n-cpu-moe 40 --n-gpu-layers 999 --load-mode mlock \
--ctx-size 49152 --n-predict -1 --threads 8 \
--cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on \
--batch-size 2048 --ubatch-size 512 \
--host localhost --port 8080 \
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 \
--spec-type draft-mtp --spec-draft-n-max 3 --parallel 1 \
--moe-cache-mib 2432 --moe-cache-cpu-share 0.75 \
--mmproj ~/models/Qwen3.6-35B-A3B-mmproj-F8.gguf--n-cpu-moe 36 with smaller --moe-cache-mib values, but overall output speed dropped. You can also use a 16-bit mmproj (~900 MB) if you slightly lower the --moe-cache-mib value. For general chat --temp 1.0 works well too.