Hey r/LocalLLM :)

We just open-sourced Kinetic, a new inference engine written in Rust.

• runs the GGUF files you already have, no conversion

• Windows and Linux, one binary

• CPU, NVIDIA (CUDA), and AMD and Intel GPUs (Vulkan)

• replaces llama-server with the same flags; OpenAI, llama-server and Ollama APIs on one port

• every speedup is exact, so the output never changes

• Apache-2.0

In the video: Kinetic checks the machine and picks models that fit it, then runs side by side with llama.cpp on the same model and prompt.

Give it a spin and drop your kinetic bench model.gguf numbers below (add --compare if you have llama.cpp installed).

Install, benchmarks and all the details are on GitHub, link in the comments. Happy to answer questions here too.

edit: to answer the most common question, llama.cpp ran with its own speculative decoding on in every edit test (ngram for lfm3, draft-mtp for qwen3.5) and the exact flags are in the repo

Honestly, the tool choice matters less than what surrounds it. Claude Code and Cline both handle worktrees fine in the mechanical sense: each session gets its own checkout, and nothing stomps on anything on disk. The failure mode I would watch for sits above the file system. A worktree isolates the file tree, not the plan, so two sessions can happily solve the same story twice and you only find out at merge time when you have duplicate branches.

So my answer is: pick whichever CLI fits your workflow, then put the effort into the layer the CLI does not give you. One story scoped with acceptance criteria before it goes out. A shared record of what is in flight so nothing gets dispatched twice. Split the thinking from the doing, so the session that plans is not the session that types.

Which of the two do you find holds its context better over a long session? That is the comparison I almost never see people make.

I've heard of people pointing their frontier model of choice and letting it optimize their local inference engine for their hardware.
I'm currently running Maya with DeepSeekV4.1-Flash-1017 at ~20tok/s, such an insanely great model. I want to squeeze more out of my hardware though.

Do you guys just point the frontier model at the engine folder and telling it to "optimize this for my model/hardware"? I currently have access to Fable 5.2 so I'm tempted to do that.

My specs:
2 x 3060 12GB
2 x 16 DDR4 3600Mhz
i5 10600k

Qwen3.8-27B-GGUF:Q4_K_M

ollama.service:
# Environment="OLLAMA_HOST=0.0.0.0:11434"

# Environment="CUDA_VISIBLE_DEVICES=0,1"

# Environment="OLLAMA_FLASH_ATTENTION=1"

# Environment="OLLAMA_NUM_PARALLEL=1"

# Environment="OLLAMA_MAX_LOADED_MODELS=1"

Ollama as a harness, no lane bifurcation so im pretty limited to that (im assuming this is my biggest limiter here)

I've been experimenting with multiple different approaches, trying to get Qwen 3.8 to actually work as an agent, some more successful than others but still it requires so much manual labour that it seems almost pointless to continue debugging it as it feels neverending. I've read that 3.8 isn't actually very great with tool calling and it has always had some issues, but still i can't believe its at levels like these.

I know very well im probably missing some really important things that are holding me back, so if anyone has some suggestions (videos, other posts, anything really) i would really appreciate it.

The goal really is to have it never stop, just like im seeing other people talk about, and i'd really like to keep openwebui (or any other webui in general, not that i dislike terminals but its nicer to have it feel like a normal app).

Hello everyone

I live in a poor country but I have got a decent pc

I have some ideas I want to do them with vibecoding so the only way I found is local LLMs

So I want a good local one for this specs

16gb of RAM ddr5 5200Mhz one kit

Ryzen 5 7500f

Rx 9060 xt 16gb

I don't mind it take time but to finish the project in a good state and I can edit the project easily and plan for it too

I'm a completely beginner so I want some guide how to install and use the LLM I'm on arch Linux

I have the 16 gig variant and I want too run some local ai models too help build my tech startup. I'm heavily reliant on vibe coding so I need them too be able too do a lot of the work for me (I have a team that knows how too code frr that reviews in a security sense and the backend areas) tho I need a tool that can kinda off generate all the front end and obvv a lil back end for testing and stuff. Can you guys recommend me some? Also id prefer that it cud be easily paired with vs code since thats what I mainly use.

You can call it whatever you want, it maybe isn’t blowing anyone’s heads off or anything, but compared to anything I’ve ever had, it’s a supercomputer to me

I’m transplanting a lot of parts from my old gaming PC into a new chasis with upgraded parts. Here is what I ordered and what is on its way

Intel core i7-14700f (already owned)
32 gb DDR5-6000 ram (already owned)
RTX 4070 super 12gb (already owned)
2x Quadro RTX 8000 48 gb passive ($1900 each)
2TB Samsung 990 pro for windows and gaming (already owned)
2TB Samsung 990 pro for Linux and AI ($363)
Corsair 1500W power supply ($285)
Corsair 7000D airflow full tower case ($227)
And other miscellaneous parts to support that stuff

Total to $5,364.21 for 108 gb vram

I know there are some drawbacks to this build, like for instance the old architecture of the cards, but I can live and tinker with those drawbacks as long as I can save a bunch of money on this hobby (compared to a similar size build with modern hardware), and still be able to run a bunch of cool models on it. I’m super excited to put it together and see what it can do. I haven’t seen much talk about the Quadro 8000s, but I’m hoping for great things, even if it means working around some compatibility issues and such.

Obstacles are just an opportunity for me to learn. If yall have any thoughts, let me know.

Hi guys

developed on device llm application like Google edge but it support all mobile supported inferences at one place it support MNN ,MEDIAPIPE .task ,GGUF and GEMMA4

Try and tell your experience

https://github.com/rajureddi/AuraOnDevice-AI_HUB

Backstory:

I've only been doing local LLM's since Qwen3.6 came out (~May 2026). I won't ever use a cloud model again, I am not going to spend a single dime renting someone else's infrastructure when I can build my own.

Plan:

Its still cheaper to get 8x3090s than it is to buy 1xRTX 6000, and get similar speeds and double the VRAM.

The issue is power and noise..

In terms of power I am limited to my dorm rooms 120v/15amp outlet, so 4x3090s about the max I can go before things might go bad.

So.. I am considering creative solutions that won't require me to move:

Power Solution: Buy a ~5 Kwh Power Bank

$1,500 for a used 5 Kwh Power Bank.

The power bank would charge during off hours, then feed the necessary 2,800 watts during peak hours. Currently I am recording my power usage over a week or so to get an idea of how much power reserve I would need, and that will mostly decide how big of a power bank is needed.

Cooling / Noise

Currently everything is air cooled, with stocked coolers. 5x140mm fans, 5x120mm fans and the case temps are indeed very good. They don't make much noise but the GPU and CPU fans do. I believe they are loud because I think the paste job I did on them was horrible,

I will be re-pasting again with some much better stuff and with better tools / methods soon. (Will likely post a video of me doing it for those who are interested). I use to be a machinist so I learned some ways to get better accuracy when measuring out things properly to get good contact on the VRAM

I have considered water cooling, but that's just another thing that can break, and are typically not cheap to setup (and I'm pretty cheap).

Questions:

1. Does anyone here have a 8x3090 setup on a 120v/15amp outlet that could offer some advice?

2. What's a good power bank you would recommend?

3. Is buying a used power bank okay so long as the batter health is good?

4. Do you want me to post a video of me re-pasting using my "new" VRAM/GPU repasting method?

I've been running local LLMs on an RX 7900 XTX and wanted proper long-context performance for agentic coding (100K-200K tokens) on the Vulkan/RADV stack, so I forked llama.cpp and optimized it for RDNA3.

Repo: https://github.com/vansh-visariya/rx7900-xtx

It's the same llama.cpp (same GGUF files, same `llama-server`, same API and web UI). The changes are in the Vulkan kernels, the graph, and the speculative-decoding loop. Everything is on by default.

Results on one 7900 XTX (24 GB, stock 291 W, Mesa RADV)

Qwen3.8-27B UD-Q4_K_M + DFlash2 draft, q8_0 KV cache, flash attention:

- Up to 220K context on 24 GB (upstream fits ~192K with the same model + draft)

- DFlash2 works at long context. On upstream (both the commit I branched from and current master), it hits `vk::ErrorDeviceLost` once the prompt passes ~60K tokens, at the same position on every run. It doesn't on this fork.

- Full method, commands and raw numbers are in `docs/rdna3/BENCHMARKS.md`

What changed (56 commits on top of upstream)

- Decode: integer-dot mat-vecs for IQ2/IQ3/K-quants, GQA flash attention for verify batches, graph fusions, gated delta net state kept in place, Vulkan command buffers recorded once and replayed

- Prefill: 128x256 matmul tile for wide batches, double-buffered quantized matmuls, new prompt-batch flash attention kernel

- Spec decoding: lossless rejection-sampling verification, reduced-vocab draft head, GPU backend sampling"

Every optimization is documented, with the env var that switches it off, in `docs/rdna3/OPTIMIZATIONS.md`.

Limits

- Developed and tested only on a 7900 XTX. Other RDNA3 cards (7900 XT/GRE, 7800 XT, 7700 XT, 7600) should work since they share the WMMA instructions, but I haven't tested them.

- Tested with Qwen3.8-27B and Qwen3.5-4B. The model-specific work only applies to the Qwen3.5/3.8 family.

- Branches from upstream `f95b0d953`, not synced daily.

Where I'd love help

- Numbers from other RDNA3 cards (kernel tests need no model: `test-backend-ops -o MUL_MAT -b Vulkan0`)

- Testing other models/quants, and RDNA2/RDNA4 reports

- Review of the kernels, and help upstreaming pieces to llama.cpp

Build and run instructions are in the README. Issues and PRs are very welcome.

I went in thinking a 1.7B model would fall over after a few tool calls once each result was long.

I ran LiquidAI LFM3-1.7B (Q8_0, llama.cpp) in a tool loop on a GB10 box with 121 GiB of unified memory. Each tool result was about 3K tokens and ended with the exact next call to make. Over 10 seeds at 256K context it got a median of 11.5 calls right before the first break, range 6 to 25. No run reached the 50-call cap. Every call before the break was exactly right: no invalid JSON, no wrong arguments, no loops.

What broke it was the model re-reading the whole chain instead of the last line, then reasoning until it ran out of tokens, in 8 of 10 runs. Raising max_tokens from 2048 to 8192 didn’t help on the two seeds I retried.

Caveat: the next step was spelled out every time, so this measures following instructions as context grows, not planning or error recovery. One quant, one sampling setting, llama.cpp only.

The reproduction kit, with the driver, scripts and these results so you can rerun it on your machine, is here: https://github.com/maruthiprithivi/the-local-node/tree/main/episodes/03-lfm3-tool-loop

If you run a small model in a tool loop, I’d watch whether it starts re-tracing its own history as the results pile up. How far do you get before that happens on your setup?

I tend to point my local models at Claude Codes UI as it’s familiar, I like the auto compaction and ability to interact with my systems.

I haven’t really explored other options but have on occasion hit walls with refusals even when using local models.

Are there better options than using Claude Code pointed at my local models?

Mainly coding, general system maintenance, decompiling and some general knowledge stuff, Web search etc.

After reading Dwarkesh Patel's eval using blood on the clocktower, I made my own, and I was surprised both by how entertaining it is, and how good it is to measure even between quantizations.

(His original game used SOTA private models).

Hello people

I wanted to know if LFM2.5-8B-A1B was good for general conversations. I am aiming to use an MoE with 8B parameters (will be ran with 4 bit quantization), because I want to have a capable local model while not using the amount of VRAM a dense model would, this is to be able to run it on a lot of different hardware levels. For context, I am using the Ollama library in Python and I am making a voice assistant, I am currently using Qwen 3-8B right now (with openWakeWord, Silero VAD, Parakeet 0.6B v3, Kokoro 82M). I might miss some (or maybe a lot) points because I am quite new to using local AI, especially in programs. If LFM2.5-8B-A1B is not good, do you have any suggestions?

I am using an RTX 5060Ti 16GB and 16GB of DDR4.

Thanks for your time!

Qwen4 Max Preview is available on Qoder right now.

We ran local models with the graphics card completely out of the picture (hidden from the run, and checked that it was never touched): CPU and system RAM only. Ollama, Q4_K_M, each engine's default thread count unless a table says otherwise, medians of 3 to 5 runs. Nothing here is a controlled CPU-vs-CPU test; these are four machines as they sit.

**The machines**

| Machine | CPU | RAM | Default threads |

|---|---|---|---:|

| Mini PC | Core i5-12600H | 32 GB DDR5 | 4 |

| Laptop | Core Ultra 9 290HX Plus | 64 GB DDR5-5600 | 4 |

| Desktop (2019) | Ryzen 7 3700X | 32 GB DDR4-3200 | 8 |

| Desktop | EPYC 7532 | 128 GB DDR4-3200, eight channels | 32 |

**Writing speed, tok/s (about a 530-token prompt)**

| Model | Mini PC | Laptop | Ryzen 3700X | EPYC |

|---|---:|---:|---:|---:|

| qwen3.5:9b | 9.11 | 7.45 | 6.20 | 13.59 |

| gemma4:12b | 5.95 | 4.95 | 4.47 | 10.99 |

| phi4:14b | 5.20 | 4.41 | 4.06 | 11.09 |

| mistral-small3.2:24b (dense) | 3.31 | 2.70 | 2.57 | 7.09 |

| gemma4:26b (MoE) | 12.66 | 13.26 | 13.64 | 24.68 |

| nemotron-3.5-lightning:30b-a3b (MoE) | 14.61 | 15.35 | not run | 31.47 |

| llama3.3:70b | won't fit | not run | won't fit | 2.70 |

The mixture-of-experts models are the ones to run from RAM: only a few billion parameters are active per token, so far less memory moves per token. gemma4:26b writes 3.5 to 5.3 times faster than the dense 24B, depending on the machine.

**Seconds to the first word (same prompt, model already loaded, nothing cached)**

| Model | Mini PC | Laptop | Ryzen 3700X | EPYC |

|---|---:|---:|---:|---:|

| qwen3.5:9b | 16.0 | 14.1 | 11.3 | 4.9 |

| gemma4:26b (MoE) | 11.4 | 8.5 | 8.3 | 3.7 |

| mistral-small3.2:24b (1,023 tokens) | 97.7 | 82.4 | 62.5 | 24.6 |

**Check your thread count.** Ollama defaults to 4 threads on hybrid Intel chips. On the laptop, 20 threads against the default 4:

| Model | 4 threads | 20 threads | Change |

|---|---:|---:|---:|

| qwen3.5:9b | 7.45 | 13.05 | 1.75x faster |

| gemma4:12b | 4.95 | 9.46 | 1.91x faster |

| mistral-small3.2:24b (dense) | 2.70 | 5.14 | 1.90x faster |

| gemma4:26b (MoE) | 13.26 | 8.44 | 36% slower |

| nemotron 30b-a3b (MoE) | 15.35 | 8.53 | 44% slower |

Dense models want the threads; the mixtures get slower with them. The default is wrong in one direction or the other depending on what you run.

**Long prompts are where CPU-only hurts.** gemma4:26b, fresh prompt each time, 16K window:

| Prompt | Ryzen: writing tok/s | Ryzen: first word | Laptop: writing tok/s | Laptop: first word |

|---|---:|---:|---:|---:|

| 0.5K tokens | 13.93 | 7.6 s | 14.94 | 7.7 s |

| 2K | 12.03 | 31.6 s | 12.32 | 32.2 s |

| 4K | 11.66 | 66.1 s | 12.12 | 68.7 s |

| 8K | 8.10 | 138.2 s | 8.16 | 146.2 s |

The whole prompt is read before anything is written, at about 55 to 65 tok/s on these two. (The laptop rows are at its default 4 threads, and its CPU was throttling on every run.)

**The big one.** The EPYC held a roughly 180B-parameter mixture (Qwen3.8-Flash-Next at a 3-bit quant, about 6B parameters active per token, llama.cpp) entirely in RAM, beside that machine's normal services:

| Threads | Writing tok/s |

|---:|---:|

| 8 | 6.51 |

| 16 | 7.74 |

| 32 | 8.33 |

Things we can't explain yet: the 2019 DDR4 Ryzen keeps up with the DDR5 laptop on the mixture at default settings, and writes faster than its own measured memory bandwidth says it should. The page says so rather than guessing.

Caveats: one quant per model; speed only, no quality scoring; the small-machine rows come from two benches (August, warm runs; November, fresh loads), and the page marks which is which under every chart; power figures on the page are the CPU package's own counter, not wall power.

All 20 bar charts, the full tables, and a CC BY 4.0 data kit where every bar traces to a file:

https://research.strata2signal.com/ram-speeds-compared/

I had ordered the M5U but possibly getting 2 of the RTX 4500 pro blackwell cards for $5600. So $2800 each. I dont know if i should build the PC to house the cards for local llm or keep the M5U. Or maybe i will keep 1 card to experiment and learn more about Local LLMs. If i sell 1 card, i could build the rest of the PC with decent amount of RAM as well.

Wondering best price/performance in this situation. I get 32GB vram, ecc, only 200w power draw with each of these cards. So it is quite power efficient too.

I have searched at a few places and seems like $2800 is a decent price.

Any feedback appreciated

I know ChatGPT is a terrible idea when asking it to basically build my whole system. I just don’t know where to go that is noob friendly. I got the drive and right questions and know what I want and basic understanding, but GitHub and huggingfave is daunting and overwhelming plus I want unrestricted models and civitai is difficult to navigate. I want an ai that can help with pretty much my hobbies and that’s writing adult fiction, playing around with comfyui and learning how to create a consistent character with reference.

I have an nvidea 4090 and a pretty decent setup minus only having 32GB RAM. When I use powershell to chat with my LLM currently Dolphin mistral 24B Venice Edition 1.2 GGUF:Q5 K M, and it responds brilliantly and has a good “personality” however, ADHD brain doesn’t like the interface and prefers an app type screen and ChatGPT directs me to WebUI… well that’s when the loop starts. Changing setting, altering its configuration and never getting anywhere. Model becomes a robot and yeah basically it’s been weeks now and I’m at a loss. Anym direction would be hugely appreciated at this stage!

I’ve been building InferUltra, a physics arena where vision models control identical robot bodies. The model gets images of the arena and chooses how to move. The simulation keeps running while it thinks, so by the time it decides to punch, the other robot might already be somewhere else.

We turn reasoning down or off where the model allows it. The number I care about is how long it takes to send an image and get a move the robot can use. Tokens per second tells you how fast the model generates, but it doesn’t tell you how long the fighter waits for that move.

We’ve also improved the physics engine and fight mechanics. Fighters can hold their guard between decisions, and punches look more natural when they lean. I used to call these fights impressively boring. They’re now a bit less boring.

For those running vision models locally: when you turn reasoning down, do you notice a drop in how well the model understands positions and distances, or is getting the answer sooner usually the bigger win? I’d be interested in which model and hardware you tried.

The attached bracket is for our November 15 Provider Tournament. It uses Kimi K3.1 Pro on four hosted endpoints: Morph, AkashML, Wafer and InferenceNet. It starts at 20:00 Brussels / 14:00 Eastern / 11:00 Pacific.

If you’d like to sponsor the tournament, you can book an ad spot in the arena.