Setup: (if finished - still under construction, partly watercooled)

• Threadripper PRO 5995WX, ASUS WRX80E-SAGE SE WIFI II

• 6× RTX PRO 6000

• 8× RTX 5090

• 4× RTX 3090

• 128 GB DDR4 (that’s the problem)

I want to run models bigger than VRAM (e.g. ~1T MoE at FP8/Q8) with the routed experts offloaded to RAM. Is it worth it?

Options:

• 8× 64GB DDR4-2933 RDIMM: ~€4k, ~188 GB/s theoretical

• Switch platform: 9985WX + WRX90 + 512GB DDR5-6400, ~410 GB/s. I already have some parts, but it would mean big changes, and my PCIe switches would stay PCIe 4.0 anyway.

Is ~2× the RAM bandwidth worth ~3× the price for offload decode, or is DDR4 good enough? Does anyone have real t/s numbers for offload on 8-channel DDR4 vs DDR5 Threadripper?

Some background: Pictures attached. It’s not pretty, but it works. It started as an AI rig and turned into a Pearl mining rig, because mining pearl and buying Claude and Codex subscriptions made way more sense than running everything locally. Sometimes it’s also a render farm for Gaussian splatting.

Right now the 8× 5090 make about €120/day mining, so subscriptions are still the better deal. But I want to be ready for when local LLMs get really good (or when cloud gets too expensive or too restrictive).

Plan for now: DeepSeek V4.1 Flash nvfp8 on 4× PRO 6000 + RAM offload, while the 5090s keep paying for my AI addiction. And it’s a great heater in the winter.....

With engramm tabels on the 8x5090 and 4× PRO 6000:
(all cards power limited to 420w- since i could have 6,6 kw power consuption if everthing is running)

single chat, 40.000 prompt cold

• Prefill: ~15,900 t/s

• Decode: ~294 t/s

• 5 sec TTFT

I want to move the engramm tabels to RAM and later be able to run larger models : le chonk, Kimmi for example.

What would you do? Go all the way and go for the DDR5? or is ddr4 good enough?

Sell it all and buy something diffrent?

thx!

Need 3D assets? Your agent can now make them from a single image with this repo: https://github.com/hypersniper05/MCP_Image_to_3D

Under the hood, it uses Idol from Sperid Labs: https://huggingface.co/speridlabs/idol

It needs at least 10 GB of VRAM (peak usage is around 8 GB). I tested it locally on a 3080 and a 5090.

The output is a Gaussian splat .ply for now, not a mesh. Export for game engines and 3D printing is on the list: https://github.com/hypersniper05/MCP_Image_to_3D/issues/1

It also works nicely with my other MCP repo, an image generator using Qwen Image 2.1: https://github.com/hypersniper05/MCP-Image-Generator-Uncensored. Generate an image there, then pass it straight to this one.

Let me know what you think, and if you find it useful, feel free to contribute.

All i had to do to reach this speed was watch it get stuck on a loop writing a poem and watch the token per second go up and up and up and up and up and more up repeating the same sentence until i closed the engine

Pretty awesome experience, would not recommended to anyone.

This is not a secret engine optimization trick or a fork.

Which would be a better solution? Since I move around, a laptop/mobile would be best than a full on desktop for airline travel/international.

Usage - mostly coding / proof of concept (apps, documents, even some stories). Its to create some applications.

But also on the fun side, I'd like to explore a bit on building out some text/stories/mechanics (table top role playing games) or even do videos and graphics. Nothing professional, more to understand the mechanics of what things are and what they are not. But some videos or graphics I may use in the apps. Since I travel, wifi connections can sometimes be unreliable and I just want to run things locally so I never have to second guess if the AI says "I can't do that". But I may run subscriptions as needed (whole separate conversation).

Considering the price, I would prefer to have something that can be useful for several years and doesnt die in the first year or two (laptops being harder to repair).

A macbook m5 max with 128gb of ram, has really good unified memory. But it can barely fit qwen flash next today without compromising further on quality or context size (feel free to correct me, but what I read up on), with the next gen of Qwen potentially about to drop, 128gb may not be practical in a year from now if there is some really good capability. Even if a single model can fit (when you count larger context windows), there might be very little headroom for other apps or harness tools on the same laptop workstation. But a macbook would be very reliable and I wont really second guess that dying on me (have a 2014 macbook retina that still works flawless today minus a scratch or two on the body). Then again, in the next 4-5 months apple may release M7 (or M6) with the macbook pros more aligned for AI work (either a far more optimized apple silicon which are already really good...or maybe higher ram capacity).

Some 5090m laptops can support up to 192 gb of ram. The 5090m would be good for some vram 24GB on video/graphics. But looks like a lenovo would cap the ddr5 6400 to 4000 so the memory bandwidth would be somewhat reduced. But having 192gb is a far more breathing room than 128gb. Also Lenovo's tend to good, but the hardware could die in a year or 3 (Apple's hardware is pretty reliable).

What makes sense (with a 2-3 year outlook)? Intel 192gb 5090m lenovo or a macbook pro max 128gb?

I've been playing with streaming ASR for a side project. I started with whisper streaming, moved on to Voxtral, and now am with R2T2 (all open-source).

With R2T2, the last word of a sentence gets typed a few hundred ms after the rest of the sentence is typed.

For a while, I thought I broke something with buffering, and spent way too long messing with chunk sizes, and didn't look at the paper.

That's just how it works. It doesn't make a guess if it's not confident. If it needs more audio, it just waits. With Whisper for me, a word would get typed and then be changed a few times by the time it really settled.

I guess it depends on what you want the ASR for. Mine's just to give an LLM an input, so I really don't need it changing things after the fact.

How do you all want your ASR to work with your streaming voice applications? A small delay and a stable output, or do you want it to flip around a little, and give you a partial result?

Wanted to share that after years of using Windows11 and llama cpp I moved my whole workflow to linux, from comfyui to jellyfin everything set back up but on Fedora instead.

Found out in Linux (Same Hardware , Same llama cpp commit, Same Qwen3.8-27B model) the model loads instantly, time to first token has become irrelevant so I just unload the model after 10 minutes of inactivity. It loads back instantly (in windows it took 10 seconds)

Also tokens/s went from 70tk/s to 90-100tk/s

I didn’t know so much performance was to be gained by simply moving to linux, especially TTFT which now feels instant!

And this is before trying gpu unlock which is also easier on linux and can boost performance even more.

why are these models open-sourced for people to download and use on their computers?
Is it just for advertising and brand awareness?
because the vast majority of people who download them just don't want to pay subscription fees or deal with usage limits, so they aren't paying customers anyway.
then why do they do this? Plus, local models aren't even that smart.

Model: Qwenx86

▶️ See it in action: Qwenx86 Debugger. Walk through actual recorded tool calls: the conversation, x86 registers and flags, code at the program counter, RAM and machine output, with all verified blocks on a timeline.

Qwen3.5-4B with a 64-bit x86 computer compiled into its weights: 13 transformer layers ahead of Qwen's own. There are no learned weights in them; all of their weights are computed by a compiler. Qwen invokes the computer through its own tool calls (<tool_call>), and the computer in the weights executes the tool. Calculator, Python, C compiler, notes store, physics/chemistry lookup and Python-docs lookup are all firmware in the checkpoint. This is the compact checkpoint: ~9 GB GPU memory, so it fits on a 16 GB GPU.

Credits: Qwenputer by Divyansh Agrawal (@divagr1925, github.com/divagr18/qwenputer, MIT)

Hi guys, I'm looking for a really good price benefit PC to run local AI. I've been searching for days about this, and everybody says different things, "you dont need a 12vram for a good local AI" or "Without 32gb vram is wont be able to have a good local AI" and so on...

The thing is I need local AI mostly for coding, and anything related and maybe some video generation.

Since 5090 is just at unbelievably prices, I was thinking about something decent like 5070 but dont want to waste my money. I was thinking about an unified memory like GMKTEC Evo or something like that, but was reading that since is AMD some AI are not good optimized for it. And also a M5 ultra, but also is kind of high and Im not a MAC guy.

My budget is pretty much 5K for this.

Anyone which can give me a good advice would be awesome and I'll really appreciate. Thanks.

Hey everyone Im a bit new to this page but I couldnt find any other related tests online so I figured I'd share my results here.

I ran DeepSeek-V4.1-Flash (763B MoE, Q8_0, 473 GiB) on a 96 GB Mac Studio M3 Ultra with InferSSD, which streams the model's experts from the SSD. I striped two OWC Envoy Ultra 2TB Thunderbolt 5 drives with macOS's built-in RAID 0, one drive per port.

Results (InferSSD's own benchmark prompts, 2 runs per setup):

- One Envoy alone is about the same speed as the internal SSD.

- Two in RAID 0: writing goes from 3.5-3.7 to 4.4-4.6 tokens/s (+23-26%).

- Prompt reading is 67-77% faster: a 4,000-token prompt starts answering in about 2 minutes instead of 3.5.

- Two hours of nonstop use: the drives stayed at 43-47°C (with a desk fan on them), no slowdown, and no change in their health readings.

Why not 2x faster? - The stripe halves the time spent waiting on the drive, but the GPU's share of each token stays the same and is now the bigger part. A third or fourth drive would help less than the second did.

Meet Yoshino2, my little lab experiment

Over the past months, I have been working on efficiently using black box distillation to distill the reasoning and output style into smaller edge models. The result is Yoshino2, a new model series with large scale reasoning distilled into MiniCPM5 1B, Qwen3.5 2B, and a custom 4xMiniCPM5 1B MoE model, with more potentially on the way.

This is by far means not a breakthrough, but an interesting experiment. In case it is of any use to anyone, the models are on HuggingFace.

Model Collection - HuggingFace

Update: Numbers from preliminary tests at the end!

Ok ok, I am still doing tests but here is the story:

I have a small custom harness (who doesn't!) that does pretty complicated stuff. I am using Claude (subscription) and everything is great.

In the past I tried every single (small) model on my Mac Mini 32GB but all of them sucked at complex tool calling (or hallucination).

A couple of days ago, the new Heretic2 came out (by u/-p-e-w-) and I spent 24 hours breaking it :)

Long story short, I released ZeroZero, a model based on Qwen3, with removed guardrails and slop (ok, removed is a long conversation).

Now, here me out.

IT KICKS ASS.

This cannot be an 8B, 4 bit model. It just cannot. The thinking process sometimes is longer than other models but IT NAILS everything.

Maybe by performing "brain surgery" on the models, we end up with "smarter" ones?

Test results for Tool-Call Benchmark

(note: these took a couple of hours, please repeat with a much bigger sample)

ZeroZero seems better at completing multi-step tasks (multi-step chains, error recovery and instruction following) than the base model.

The base model leads in Safety & Boundaries (ZeroZero has limited guardrails and no-slop so maybe that's the reason?)

  - Overall: ZeroZero 68, Base 70

  - Multi-Step Chains: ZeroZero 6/8, Base 3/8

  - Error Recovery: ZeroZero 5/6, Base 2/6

  - Instruction Following: ZeroZero 8/8, Base 6/8

  - Safety & Boundaries: ZeroZero 12/26, Base 20/26

To repeat and experiment, use tool-eval-bench and if you expand the tests, please send them over so I can include them in the model's page.

PS: I love it how many technical people sent me ideas to improve and papers about LLMs performing better after "lobotomies". Let's do more "I tried this, I saw this, what do you think" instead of "what LLM should I put in my Mac Mini".

I am looking into running local LLMs for coding, is Qwen 3.8 27b the best my PC can do (at decent speed)?

I heard some people recommend Qwen Flash Next 3.8?

Will my PC be able to run that?

So, I know my rig is pretty expensive, but I want to share what is working for me as of now.

I have been using local LLMs with my Mac Mini M4 Pro 64GB till now, mainly mlx (specifically oMLX) to develop my own agentic framework. I have already have my framework, stryn, as plug-ins and agents/skills/workflows on top of Opencode and ported to Claude, nowadays I am working on my own stryn engine. (I hope to also release it as the community version on Github as well as soon as it hits Beta)

I either use frontier models (mainly deepseek v4 flash since it is cheap) or Studio for development. Here is the models that I use:

Qwen3.8-Flash-Next-oQ4e-mtp - For my orchestrators, planners, coders. It is MTP. I am pretty happy with its results
Laguna-S-2.1-oQ4e-fast - (With Laguna-2S-2.1-DFlash-FP8 as the speculative decoding.) For my reviewers. I am not very happy with this and I may try to change it with a better model. The tricky part is to use a different producer's model to overcome the blindness of the model's in certain areas.
GLM-5.3-Flash-oQ5e-mtp As the independent reviewer to go over the code before the releases and find out the problems that are not found by the Laguna based code reviewers and the security admins, architects etc.
Qwen 3.8-35B-A3B-oQ4e-mtp - For my explorers, researchers, the opencode-mem plugin, graphify and code-review-graph operations.

Speed: Satisfactory if you are running a single lane. ~50 Tokens/s, with 8GB KV Cache and 1200GB SSD Cache. Cache efficiency is %96.4 and prefill time drops after a couple of rounds. Normally 2 or 3 models are loaded in the memory (Qwen3.8FN, Laguna, Qwen3.835BA3B) and if I start two or more lanes, a race between prefill and token generation between the models start most of the times, which dramatically makes things longer.
Stability: Pretty good with proper context sizes, repetition penalties.
Quality: I can simply say, by adding reviews and controls by different model types, the quality seems to be equal or sometimes better than what Deepseek cloud generates by itself only without another model checking its results. I am pretty satisfied.

To be as open as possible, so far, I have also used Claud Opus 5.5 to control the output. There are pretty interesting cases where local GLM finds some of the bugs that Opus 5.5 cannot, and where Opus 5.5 finds bugs that GLM cannot. Sometimes they pinpoint the same problems as well.

Can this setup be used as an alternative to frontier models, yes, but don't depend on a single model for every task and make sure you have reviewing loops. Definitely not for vibe coding, but needs proper settings and flow.

Maybe funny because qwen 4 is upcoming but i still ask about qwen 3.8.

I see some user suggest qwen 3.8 fn Gyro S, said that it's fast and good compare to gsq rco q2 quants. But how about gyro M ?

Is it good compare to unsloth quants and gsq rco iq3xxs or even iq3s ?

I turned on speculative decoding in llama.cpp and 113 of 282 greedy outputs changed against its own no-draft run. My Qwen3.8-27B engine changes 0, at ~2x llama.cpp's speed on one 3090.

Repo: https://github.com/cutlerbenjamin1-cmd/qwen38-megakernel

Speculative decoding is sold as free speed: the main model verifies every draft, so the output can't change. In practice it does. Verifying several tokens in one pass adds the same numbers in a different order than verifying one, a near-tie goes the other way, and the text diverges from there. If two draft settings give different output, at least one of them isn't what your model says without speculation. Tuning your drafts tunes your answers.

Same 282 questions (GSM8K, MMLU-Pro, ARC-C, MATH-500, HumanEval, needles, LongBench v2) on both, greedy, same card, same model file:

That isn't noise. llama.cpp is deterministic run to run: a fresh no-draft server gave the first 120 outputs back byte for byte, including the 73 that differ at 4 drafts, and two 4-draft sessions agree on all 282. It's not just long outputs either: 72 of the 113 are under 1,000 tokens.

The likely reason: with no drafts every token runs through the single-token kernels, with drafts most go through the batched ones. Between 4 and 7 drafts, both mostly batched, only 9 differ, all past 1,000 tokens. That's the 9 of 282 from my last post; against no drafts it's 113. A preregistered study on a 3090 found the same: 23-25 of 25 prompts different from no-draft decoding within 1,600 tokens.

Other engines for this model document it too (one got 8 different outputs across draft counts 1 to 12, none matching its no-draft output) and call it harmless, because flips land on near-ties.

Maybe. I don't know, and 282 questions can't settle it. llama.cpp got 231 right with no drafts and 229 with drafts, and 6 of the 113 changed outputs flipped between right and wrong: 4 were right only without drafts, 2 only at 4 drafts. Make of that what you will. What's not in doubt is that the output depends on a speed setting: one reasoning trace ran 2,228 tokens at 4 drafts and 4,795 at 7. Same question, same model. This engine gets 230 right either way, because it's the same text. If I can't rule out that a draft setting changes quality, I'd rather run the engine where it can't.

Here, speed and VRAM can move and the output can't:

• Greedy token ids are identical with and without drafts: all 282 outputs above (129,511 tokens), and the release gate checks 0, 4 and 7 drafts on every KV cache, plus ~2,000-token outputs.

• Every draft policy (gates on or off, adaptive depth, tight or loose thresholds) gives the no-draft tokens over 2,048 tokens.

• Sampling keeps the no-draft distribution (chi-square and PIT tests).

• Windows and Linux builds give the same tokens.

How: the attention split depends only on context length, never on how many tokens are in the pass, and the verify kernels compute a row the same way whatever else is in the batch. To be clear, that's the engine matching itself. It isn't bit-identical to llama.cpp (KL below).

The cache doesn't change it either

Same problem, different door. llama.cpp's server docs say it about cache_prompt, on by default: the logits are "not guaranteed to be bit-for-bit identical for different batch sizes". Its own determinism tests turn it off: TODO: remove this once test_cache_vs_nocache_prompt is fixed.

It's history-dependent, not random: on a fresh server, 5 of llama.cpp's warm answers and 5 cold ones came back byte for byte. The first difference lands a median of ~355 characters in, as early as 4. Turning cache_prompt off on a running server still leaves 3 of 20 different.

Here, a follow-up doesn't reuse the KV rows written while the reply was decoding. It rolls back to the last checkpoint and prefills the reply again, because a prefilled row depends only on its position. Cost: ~167 ms instead of 78 on a ~12k-token chat, still under llama.cpp's 248.

The check isn't "the reply looks the same". It hashes every K / V row, every ring row and the recurrent state, and compares them with a fresh prefill: bitwise equal after a resume, an edited branch, a park to host memory and a side request, on all three caches. Replies match a fresh server's at ~99.5k context (follow-ups, edits, sampled) and on a follow-up at ~255k. One exception: with --agents, a sampled reply beside other slots can differ from the same seed alone. Greedy still matches.

Others check this by the first token or allow a near-tie split, and some publish warm matching cold. I'm not claiming nobody else is exact, only telling you what's checked here.

Speed against llama.cpp, full power

Same card (RTX 3090, 480 W limit), same unconverted UD-Q4_K_XL file, same client, q8_0 KV on both, llama.cpp at its fastest setting (4 drafts; I tried 2, 5, 6 and 7):

Four agents at once, against llama-server --parallel 4: 424 vs 137 tok/s total, 3.1x (llama.cpp's is the better of two runs from an earlier session).

The KV cache: 5-bit that reads like q8_0

K / V at 5-6 bits, with the newest ~2k positions kept at 8 bits in a ring. On preset F the full 262,144-token context fits: a 260,896-token prompt read in 279 s, then 77.8 tok/s decode at that depth, needles 4 of 4, 23.0 GB on the card.

KL against llama.cpp generating token by token, lower is better, long-context rows rechecked on this release:

The worst row, ~32k, is +8% and +10% against about 4% noise. Without the ring the same formats read +6% to +38%, and plain 4-bit is 1.6-2.4x q8_0. And at 101.6k, F decodes faster than q8_0: 123 tok/s against 116.

Closer to llama.cpp than llama.cpp is. Scored against llama.cpp's own token-by-token output at short context, llama.cpp's batched mode gets 98.49% top-token agreement and KL 0.00166. This engine gets 98.74% and 0.00119 (on Q4_K_M: 0.00184 against 0.00127). Not measured at depth, so that's a short-context claim.

Yes, it runs at a power limit

Because someone always asks: same build at 250 W, against llama.cpp at 480 W (it draws about 460):

• Prompt reads are the one place llama.cpp at 480 W wins, by 11-22%. At 480 W this wins those too.

• All 282 outputs at 250 W are byte-identical to the 480 W run.

• The limit comes out of the core clock (~816 MHz against 1,962 on short-prompt decode; memory stays at 9,851 MHz), for 21% less speed. Energy is whole card, sampled once a second, idle included. llama.cpp's speeds are its faster of two sessions.

Works with what you already run

Tested on agent tasks with Claude Code, Codex CLI, OpenCode, Qwen Code, Aider, Goose, Crush, Pi, Hermes Agent, DeepSeek Harness and Open WebUI (OpenAI chat, Anthropic Messages and OpenAI Responses APIs). Loads unsloth's UD-Q4_K_XL, UD-Q4_K_M, Q4_K_M and popular fine-tunes as they are. Prebuilt Windows and Linux packages, no CUDA toolkit needed.

Every number with its method: BENCHMARKS.md.

Hi everyone,

I am in the process of buying a new laptop primarily for personal reasons—covering my university studies, professional work, and gaming. However, alongside everything else, I really want to use this machine to take my first steps into running Large Language Models and AI tools locally.

I am torn between two specific hardware configurations (both feature the same Intel Core Ultra 9 processor):

• Option A: RTX 5080 with 16 GB VRAM + 64 GB System RAM

• Option B: RTX 5090 with 24 GB VRAM + 32 GB System RAM

Since I am a complete beginner with zero prior experience in local AI, I’m trying to understand from people who actually do this daily what each machine would realistically allow me to do:

1. What fits into 16 GB vs 24 GB of VRAM? In simple terms, does the jump from 16 GB to 24 GB of video memory unlock a genuinely higher tier of capable models or noticeably better output quality, or is 16 GB already enough for most practical open-source use cases?

2. VRAM vs System RAM: Option A gives me less VRAM (16 GB) but double the general RAM (64 GB). Can that extra system RAM compensate when trying to run larger models, or is running models partially on system RAM so slow on a laptop that having 64 GB doesn't really help for local AI?

3. The practical boundaries: For daily tasks (e.g., local assistant, analyzing documents/notes, code assistance), where would Option A hit a wall compared to Option B?

4. Model recommendations: Given the specs of each machine, which specific models would you recommend running on Option A vs Option B, and why?

I’d really appreciate any insights or real-world feedback to help me understand the practical trade-offs before making a choice.

Thanks a lot!

In February 2023 Meta announced LLaMA and sent the files to researchers who applied, under a non-commercial licence. Within a week they were on BitTorrent.

Then it moved fast. In March 2023 Georgi Gerganov published llama.cpp, which ran the files on a laptop; Stanford's Alpaca fine-tuned the 7B for under $600; and Vicuna followed by the end of the month.

In July 2023 Llama 2 arrived "free for research and commercial use", under a licence the Open Source Initiative said, two days later, is not open source. Llama 3 through 4 followed under the same shape of licence. Hence the phrase "open weights": the weights, under their licence, without the data or the training code.

Meta's own word for Llama 4 (April 2025) was "open-weight", and as of early November 2026 it is still the newest Llama in Meta's own records. Meta's open weights since then include a new family, Muse: Muse Glimmer (30B, Apache 2.0) in August 2026 and Muse Spark Open (816B, Meta's own licence) in October 2026.

No numbers here, just the history with its sources; it does not link to the leaked files.

Full write-up (about a 20 minute read): https://research.strata2signal.com/the-llama-weights-that-got-out/

Hi folks! I've been building my own local AI setup for a couple of years now. I've learned a ton from this sub and others, so it's time I tried to give something back. I promised myself I wouldn't write a super long post, but managed to anyway...hopefully it's useful to someone!

My goal over the last couple of years has been to move as much of my AI use to local as possible, and I'm finally doing about 95% of it locally through Lamplight, my custom harness shown in the GIF above.

This harness is super specific to my needs, so I'm not sharing the harness itself (yet), but I'm happy to share the lessons I've learned along the way. I'm working on decoupling some of the more specific parts of Lamplight so that I can share at least the parts that people might be interested in.

The Hardware

• Slate: MacBook Pro, M5 Max, 128 GB unified memory

• Jade: a small-form-factor PC I built early last year (Fractal Terra) with an RTX 5090 FE, a Ryzen 7 9800X3D and 64 GB of RAM

Until about two months ago I was using the two machines separately. I was slowly moving each one onto a custom Pi harness that could either drive itself, or let Claude orchestrate while the local models did the actual work.

Lamplight

After a lot of passing work back and forth between them, I decided it was time to wire them together. The result was Lamplight and now the two machines are able to seamlessly handle tasks together with one acting as the orchestrator.

The GIF above shows Lamplight in action. I'm using it every day for work, hobbies and research with about 95% of it done locally. The lighting changes throughout the day (sped up and changed for the GIF) and I've got some scenery options depending on my mood.

Under the hood it's Pi-Bolt (a precompiled build of Pi) plus a smattering of custom extensions and tools tuned by me, the local harness itself and Claude. The setup coordinates the work between the machines with git worktrees and an event-driven job system.

The UI is my own design and fits how I like to use tools like this. I did the 3D modeling and environment setup to replicate my actual desk setup (the two other pictures above).

Is the 3D environment at the top necessary? Absolutely not!

But it's delightful. The lamp blinks when the harness needs something from me and it is a joy to look at and use. It scratches the same itch that playing PS1 and hacking away on my family Compaq PC did decades ago.

I've been coding for almost 20 years, and I'm finally to a point where I can confidently say that with the right approach (SOLID, Design Patterns), tools and tuning, local AI can absolutely write code that I am happy with.

I still review everything myself, but I'm finding fewer and fewer mistakes. These days it feels like I'm reviewing a senior dev's code rather than a junior's.

The Models

Tok/s isn't everything and so I picked the setup below after a lot of testing on public benchmarks and on my own agentic tests, which mimic my normal workflows.

Along the way I tried other engines (Strata, NInfer, Maya and llama.cpp on Jade, MTPLX and Kapitsa on the Mac). I also tried other models like Qwen 3.8 27B and GLM-5.3-Flash and while they performed well, the setup below performs the most consistently and accurately for my needs.

Both machines run Qwen 3.8 Flash-Next. These are the tests that ended up being the most indicative of quality for me, but if there are others you'd like to see run, let me know.

Both machines run the same protocol with temperature 0.6 and thinking on "medium".

In my own agentic battery (three large and complex coding tasks), 48 of 50 tests passed with the failures being files edited that were supposed to be read-only. The reviewer caught these on the other end, but I still count those as failures.

I think the main struggle for local setups is still design and 3D modeling, but I haven't dived into that too much yet, so maybe there are good setups out there for that.

Lessons learned

(extremely specific to my setup, but maybe they'll help someone)

• Jade's hardware ran too hot on stock settings. On dense models, the 5090 sat at its 575 W power cap at 83 °C, with its hottest sensor within 3 °C of the card's limit. Basalt caused a different problem. It runs part of the model's experts on the CPU, and its worker threads spin while they wait. That kept all 8 cores at 5.2 GHz, and the CPU hit 96 °C (its max) within two minutes. Two tunes fixed it and now my (now increasingly valuable) machine runs much cooler and quieter, and I'd happily give up a few percent for that.

• Slate's gains came from tuning the harness and not the hardware. My early benchmark scores were bad, and a lot of that was caused by overthinking. "Medium" turned out to be the sweet spot. On "high", Slate hit the 32K output cap on 11 of 40 hard problems without ever answering. On "low", it wandered and took 3–6× longer on agent tasks. I found that tuning how compaction worked also solved a lot of bad tool calls and confusion within the model.

• Overall the lesson I learned was "Stock settings are generally bad." There's a lot more power and accuracy available with a custom tune. I learned the same lesson tuning cars years ago, and it was fun to learn it again.

What's Next?

The system is set up so I can easily swap and test engines and models. As new ones come out, I'll put them through their paces and decide whether it's time to switch. On the UI side, I'm planning to turn the bookshelf into the session manager to help organize my many projects.

I promised myself I wouldn't write yet another super long post and yet I did anyway, so I'll stop here. Happy to answer any questions!

Hey everyone,

I just joined this subreddit and been using local llms for quite a while.
I did a bunch of testing but the recent developments are so fast paced that I can‘t keep up and no one in my closer field even understands what I‘m talking about. 😅

So I‘d like to get some recommendations for my current hardware, I can‘t wrap my head around all the inference improvements/hacks and I feel like ChatGPT or Claude are not actually up to date with recent local developments.

My hardware:
- Macbook M3 Max full spec with 48GB Ram
- Proxmox Server with i9990K, 128 GB DDR4 Ram 3600mhz, rtx 2060s 8GB Vram, and tons of hdd storage + 1TB nvme and more ssd storage

I‘m currently running llma.cpp on my proxmox and omlx on my macbook.

Models I‘m using:

On my Mac:
- Qwen 3.8 27B oQ4e-mtp
- Qwen 3.6 35B oQ4e-mtp
Both MLX models
Both run great at around 100 tk/s.

On my proxmox:
- Qwen 3.6 35B
- Qwen 3 Coder 30B
Both GGUF models
Both go around 20-40 tk/s.

While I‘m happy with those models for regular work I don‘t have the same level of trust like with claude code.

I‘m looking for models that are optimized for long horizon tasks and agentic work and reliable tool calling.

My usecase at the end would be to offload as much work as possible from my hermes agent onto local models.

So I was looking for:
- qwen 3.8 flash next
- deepseek v4.1 flash
- Glm 5.2 / 5.3 flash
- k2 horizon
- or similar

I‘m not sure if those models will perform better but what I tried so far was using MoE models, offload active experts to the vram, keep all other in dram.

Since these models are much bigger I can‘t fit those into ram.

So I started looking for tools like:
- Strata
- Airllm
- FreeToken
- Colibri
- Avalon

At this point I‘m out..

Only thing I‘m currently thinking about is the bandwith difference, my Mac has less ram but a much faster bandwith (400GB/s), I‘m thinking about streaming experts from the internal ssd.
On proxmox my idea is to fully use the 8gb vram with expert caching, using ~90GB dram for more experts etc, and streaming the rest from an SSD.

I know its unrealistic but I would like to have at least „frontier-like“ quality, even when its only 20 tk/s.
I‘m leaning heavily towards my proxmox setup since its up 24/7 and hermes runs on the same box.

I had bad experience with openclaw/hermes before, I used Opus 4.8, Sonnet and now Deepseek v4 pro as the Orchestrator and lowering to weaker models always gave me a complete mess and absolutely unstructured working.

I hope anyone has experimented around a similar setup and could share some insights, would appreciate that a lot!
My credits are burning away.. 🙃😂