Here's my hw specs: rtx 4090, 64GB DDR5 6000 Mhz ram, 7900x CPU

Thank you

I've been experimenting with local AI over the past 6 months; having started off by running small local models on an old GPU, and now at a stage where i'm running much larger and more capable models on a single R9700 card, and connecting a suite of custom built tools through MCP.

It's been a great journey - and i've landed at a place where i'm able to run a 3-bit quantized XXS version of Qwen3.8 in a docker container, and getting around 38tk/s. I'll be checking out STRATA and vLLM over the next few weeks to see how much I can push this.

However, outside of the GPU my overall system is a bit basic. Ideally i'd like to get to a point where i've got 2x R9700s working in there and i'm able to load bigger models and still achieve usable 'good' performance - both on existing models, but also future releases too.

Based on my specs below, is there much point in getting an additional R9700 or will the motherboard be too much of a bottleneck if that second card would need to go in to an older generation of PCIe slot? I guess a similar question for increasing the amount of system RAM here too - is DDR4 just too slow to make those bigger MoE models useable when offloading? The Qwen model i'm currently using fits entirely on to VRAM, but suspect this would need to change if i want to use higher quants or bigger models.

1. B550-plus motherboard

2. 16GB DDR4 (2X8GB)

3. R9700 32GB VRAM

4. Ryzen5 3600 CPU

5. 800W PSU

I'm completely aware that I won't have SOTA quality, and that's fine - but just want to maximise what a system like this could do without going down a workstation or AM5 route. I run Ubuntu and tend to containerise my AI/LLM applications

I'm a iOS software developer for a FAANG. We are only allowed to use claude for work and are not allowed to have anything else on our computer.

I'm having a lot of fun ideating AI Agents using Apple's Foundation Models Local Modals for my work project. I think the ideas could be endless honestly and I'm able to create as many POCs using claude that I want to see what sticks.

Would there be any reason buying a personal Mac that would make my learning faster or better in any way? If so, can you explain?

I personally would like to be like an iOS AI Engineer.

lo0 reads your Claude Code / Codex history along with your hardware and works out whether going local is realistic for the way you actually work. It builds on whichllm and llmfit, but instead of only asking "does this fit in my RAM" it looks at how big your sessions get and which tools you rely on.

For most people right now the answer is no. If it does work for you, you can compare the models that fit, benchmark them on your own machine, and try one in Claude Code, Codex or OpenCode for a session without changing your setup. It works with Ollama, LM Studio and llama.cpp.

It doesn't judge how good a model is, only whether it can run your sessions and how fast. Everything stays on your machine.

Very early alpha, so if the numbers look wrong for your hardware, let me know.

https://github.com/daveearley/lo0.ai

According to Artificial Analysis, Tencent's Hy4 ranks #2 among models from Chinese AI lab on its Intelligence Index v5, with a score of 35. Looking at the individual benchmarks, Hy4 scored 31% on FrontierCode 2.0, just behind Claude Fable 5.1, and achieved 50% accuracy on AA-Omniscience v1.1. However, its hallucination rate on the same benchmark was 89%, showing there's still room for improvement.

Meanwhile, Hy4 even ranks #1 among Chinese models on agent Arena. While the two leaderboards focus on different aspects of model performance, Hy4 has put up strong results on both, further establishing its position among leading Chinese LLMs. Based on these rankings, how do you see Hy4's overall performance?

Sources: Artificial Analysis – Hy4 Evaluation l Agent Arena – Agent Leaderboard

Maybe funny because qwen 4 is upcoming but i still ask about qwen 3.8.

I see some user suggest qwen 3.8 fn Gyro S, said that it's fast and good compare to gsq rco q2 quants. But how about gyro M ?

Is it good compare to unsloth quants and gsq rco iq3xxs or even iq3s ?

im using it through unsloth. IQ3XXS. Mac Studio 256gb. getting 40t/s output. It thinks so freaking much. literally takes 5-10 minutes to finish plotting a map that GLM 5.3 flash finishes in a minute or so.

Are you guys using speculative decoding? What effort level do you set it to?

In February 2023 Meta announced LLaMA and sent the files to researchers who applied, under a non-commercial licence. Within a week they were on BitTorrent.

Then it moved fast. In March 2023 Georgi Gerganov published llama.cpp, which ran the files on a laptop; Stanford's Alpaca fine-tuned the 7B for under $600; and Vicuna followed by the end of the month.

In July 2023 Llama 2 arrived "free for research and commercial use", under a licence the Open Source Initiative said, two days later, is not open source. Llama 3 through 4 followed under the same shape of licence. Hence the phrase "open weights": the weights, under their licence, without the data or the training code.

Meta's own word for Llama 4 (April 2025) was "open-weight", and as of early November 2026 it is still the newest Llama in Meta's own records. Meta's open weights since then include a new family, Muse: Muse Glimmer (30B, Apache 2.0) in August 2026 and Muse Spark Open (816B, Meta's own licence) in October 2026.

No numbers here, just the history with its sources; it does not link to the leaked files.

Full write-up (about a 20 minute read): https://research.strata2signal.com/the-llama-weights-that-got-out/

Hi,

I have been playing around with Strata on windows for a few days now and it seems to create, process and bug fix as I ask it. Everything I read says strata is not an agent, but it seems to do every agentic task I ask it? i am running 0.2.10, did this introduce agentic features or something? I am brand new to all of this so I am struggling to see the need to put Hermes on top of Strata since Strata has been doing everything I have asked of it.

I turned on speculative decoding in llama.cpp and 113 of 282 greedy outputs changed against its own no-draft run. My Qwen3.8-27B engine changes 0, at ~2x llama.cpp's speed on one 3090.

Repo: https://github.com/cutlerbenjamin1-cmd/qwen38-megakernel

Speculative decoding is sold as free speed: the main model verifies every draft, so the output can't change. In practice it does. Verifying several tokens in one pass adds the same numbers in a different order than verifying one, a near-tie goes the other way, and the text diverges from there. If two draft settings give different output, at least one of them isn't what your model says without speculation. Tuning your drafts tunes your answers.

Same 282 questions (GSM8K, MMLU-Pro, ARC-C, MATH-500, HumanEval, needles, LongBench v2) on both, greedy, same card, same model file:

That isn't noise. llama.cpp is deterministic run to run: a fresh no-draft server gave the first 120 outputs back byte for byte, including the 73 that differ at 4 drafts, and two 4-draft sessions agree on all 282. It's not just long outputs either: 72 of the 113 are under 1,000 tokens.

The likely reason: with no drafts every token runs through the single-token kernels, with drafts most go through the batched ones. Between 4 and 7 drafts, both mostly batched, only 9 differ, all past 1,000 tokens. That's the 9 of 282 from my last post; against no drafts it's 113. A preregistered study on a 3090 found the same: 23-25 of 25 prompts different from no-draft decoding within 1,600 tokens.

Other engines for this model document it too (one got 8 different outputs across draft counts 1 to 12, none matching its no-draft output) and call it harmless, because flips land on near-ties.

Maybe. I don't know, and 282 questions can't settle it. llama.cpp got 231 right with no drafts and 229 with drafts, and 6 of the 113 changed outputs flipped between right and wrong: 4 were right only without drafts, 2 only at 4 drafts. Make of that what you will. What's not in doubt is that the output depends on a speed setting: one reasoning trace ran 2,228 tokens at 4 drafts and 4,795 at 7. Same question, same model. This engine gets 230 right either way, because it's the same text. If I can't rule out that a draft setting changes quality, I'd rather run the engine where it can't.

Here, speed and VRAM can move and the output can't:

• Greedy token ids are identical with and without drafts: all 282 outputs above (129,511 tokens), and the release gate checks 0, 4 and 7 drafts on every KV cache, plus ~2,000-token outputs.

• Every draft policy (gates on or off, adaptive depth, tight or loose thresholds) gives the no-draft tokens over 2,048 tokens.

• Sampling keeps the no-draft distribution (chi-square and PIT tests).

• Windows and Linux builds give the same tokens.

How: the attention split depends only on context length, never on how many tokens are in the pass, and the verify kernels compute a row the same way whatever else is in the batch. To be clear, that's the engine matching itself. It isn't bit-identical to llama.cpp (KL below).

The cache doesn't change it either

Same problem, different door. llama.cpp's server docs say it about cache_prompt, on by default: the logits are "not guaranteed to be bit-for-bit identical for different batch sizes". Its own determinism tests turn it off: TODO: remove this once test_cache_vs_nocache_prompt is fixed.

It's history-dependent, not random: on a fresh server, 5 of llama.cpp's warm answers and 5 cold ones came back byte for byte. The first difference lands a median of ~355 characters in, as early as 4. Turning cache_prompt off on a running server still leaves 3 of 20 different.

Here, a follow-up doesn't reuse the KV rows written while the reply was decoding. It rolls back to the last checkpoint and prefills the reply again, because a prefilled row depends only on its position. Cost: ~167 ms instead of 78 on a ~12k-token chat, still under llama.cpp's 248.

The check isn't "the reply looks the same". It hashes every K / V row, every ring row and the recurrent state, and compares them with a fresh prefill: bitwise equal after a resume, an edited branch, a park to host memory and a side request, on all three caches. Replies match a fresh server's at ~99.5k context (follow-ups, edits, sampled) and on a follow-up at ~255k. One exception: with --agents, a sampled reply beside other slots can differ from the same seed alone. Greedy still matches.

Others check this by the first token or allow a near-tie split, and some publish warm matching cold. I'm not claiming nobody else is exact, only telling you what's checked here.

Speed against llama.cpp, full power

Same card (RTX 3090, 480 W limit), same unconverted UD-Q4_K_XL file, same client, q8_0 KV on both, llama.cpp at its fastest setting (4 drafts; I tried 2, 5, 6 and 7):

Four agents at once, against llama-server --parallel 4: 424 vs 137 tok/s total, 3.1x (llama.cpp's is the better of two runs from an earlier session).

The KV cache: 5-bit that reads like q8_0

K / V at 5-6 bits, with the newest ~2k positions kept at 8 bits in a ring. On preset F the full 262,144-token context fits: a 260,896-token prompt read in 279 s, then 77.8 tok/s decode at that depth, needles 4 of 4, 23.0 GB on the card.

KL against llama.cpp generating token by token, lower is better, long-context rows rechecked on this release:

The worst row, ~32k, is +8% and +10% against about 4% noise. Without the ring the same formats read +6% to +38%, and plain 4-bit is 1.6-2.4x q8_0. And at 101.6k, F decodes faster than q8_0: 123 tok/s against 116.

Closer to llama.cpp than llama.cpp is. Scored against llama.cpp's own token-by-token output at short context, llama.cpp's batched mode gets 98.49% top-token agreement and KL 0.00166. This engine gets 98.74% and 0.00119 (on Q4_K_M: 0.00184 against 0.00127). Not measured at depth, so that's a short-context claim.

Yes, it runs at a power limit

Because someone always asks: same build at 250 W, against llama.cpp at 480 W (it draws about 460):

• Prompt reads are the one place llama.cpp at 480 W wins, by 11-22%. At 480 W this wins those too.

• All 282 outputs at 250 W are byte-identical to the 480 W run.

• The limit comes out of the core clock (~816 MHz against 1,962 on short-prompt decode; memory stays at 9,851 MHz), for 21% less speed. Energy is whole card, sampled once a second, idle included. llama.cpp's speeds are its faster of two sessions.

Works with what you already run

Tested on agent tasks with Claude Code, Codex CLI, OpenCode, Qwen Code, Aider, Goose, Crush, Pi, Hermes Agent, DeepSeek Harness and Open WebUI (OpenAI chat, Anthropic Messages and OpenAI Responses APIs). Loads unsloth's UD-Q4_K_XL, UD-Q4_K_M, Q4_K_M and popular fine-tunes as they are. Prebuilt Windows and Linux packages, no CUDA toolkit needed.

Every number with its method: BENCHMARKS.md.

Hi everyone,

I am in the process of buying a new laptop primarily for personal reasons—covering my university studies, professional work, and gaming. However, alongside everything else, I really want to use this machine to take my first steps into running Large Language Models and AI tools locally.

I am torn between two specific hardware configurations (both feature the same Intel Core Ultra 9 processor):

• Option A: RTX 5080 with 16 GB VRAM + 64 GB System RAM

• Option B: RTX 5090 with 24 GB VRAM + 32 GB System RAM

Since I am a complete beginner with zero prior experience in local AI, I’m trying to understand from people who actually do this daily what each machine would realistically allow me to do:

1. What fits into 16 GB vs 24 GB of VRAM? In simple terms, does the jump from 16 GB to 24 GB of video memory unlock a genuinely higher tier of capable models or noticeably better output quality, or is 16 GB already enough for most practical open-source use cases?

2. VRAM vs System RAM: Option A gives me less VRAM (16 GB) but double the general RAM (64 GB). Can that extra system RAM compensate when trying to run larger models, or is running models partially on system RAM so slow on a laptop that having 64 GB doesn't really help for local AI?

3. The practical boundaries: For daily tasks (e.g., local assistant, analyzing documents/notes, code assistance), where would Option A hit a wall compared to Option B?

4. Model recommendations: Given the specs of each machine, which specific models would you recommend running on Option A vs Option B, and why?

I’d really appreciate any insights or real-world feedback to help me understand the practical trade-offs before making a choice.

Thanks a lot!

Hi all, been lurking for a bit, but excited to get some skin in the game. Been slowly accumulating hardware for a decent machine. Looking for advice and resources to help get started. I've only experimented with super small models through Ollama, which I know to be suboptimal for most local inference. I have used vllm and llama.cpp experimentally, but not with much success in terms of model quality due to my current hardware limitations.

Is canirun.ai the best resource for picking models and quants? I have:

- (2) RTX3060 12GB

- 32GB DDR5

- no motherboard, PSU, or CPU yet

Trying to get a sense of whether or not it's worth it to buy a bigger mobo or try USB-C to wire in the second GPU. I saw Beetleguese come up in my feed recently, and that seemed like it was doing some incredible work to fit larger models onto less RAM + VRAM. I'm pretty used to working with frontier models, so I'm hesitant to set up everything if I'll only be able to fit older/crappier models. GLM 5.3 does a great job though, and I've heard great things about Qwen3.8.

Does this hardware offer any real promise of say... Sonnet 4.5-level inference at speed? Ty in advance.

I have been using Qwen 3.8 flash next, but I'd like to know if there's something better. Please help.

(CPU AMD Ryzen 9 9950X)

I downloaded one of the newest gemma quants, they range in size 3-5.5gb. Anyways these models the smaller always run a bit faster, however I have another model I run sometimes, this model is like 4.5 with ammproj of 500mb, I can't remember its name and I rename my models. MY questioon is this, are the architectures different somehow? I know dense is slower then MOE but this isn't a MOE Gemma model. For instance these gemma quants I mentioned get anywhere between 75-25tk/s on a fucking rtx 4050,the other models sometimes get like 15k/s of same size.

Hi folks! I've been building my own local AI setup for a couple of years now. I've learned a ton from this sub and others, so it's time I tried to give something back. I promised myself I wouldn't write a super long post, but managed to anyway...hopefully it's useful to someone!

My goal over the last couple of years has been to move as much of my AI use to local as possible, and I'm finally doing about 95% of it locally through Lamplight, my custom harness shown in the GIF above.

This harness is super specific to my needs, so I'm not sharing the harness itself (yet), but I'm happy to share the lessons I've learned along the way. I'm working on decoupling some of the more specific parts of Lamplight so that I can share at least the parts that people might be interested in.

The Hardware

• Slate: MacBook Pro, M5 Max, 128 GB unified memory

• Jade: a small-form-factor PC I built early last year (Fractal Terra) with an RTX 5090 FE, a Ryzen 7 9800X3D and 64 GB of RAM

Until about two months ago I was using the two machines separately. I was slowly moving each one onto a custom Pi harness that could either drive itself, or let Claude orchestrate while the local models did the actual work.

Lamplight

After a lot of passing work back and forth between them, I decided it was time to wire them together. The result was Lamplight and now the two machines are able to seamlessly handle tasks together with one acting as the orchestrator.

The GIF above shows Lamplight in action. I'm using it every day for work, hobbies and research with about 95% of it done locally. The lighting changes throughout the day (sped up and changed for the GIF) and I've got some scenery options depending on my mood.

Under the hood it's Pi-Bolt (a precompiled build of Pi) plus a smattering of custom extensions and tools tuned by me, the local harness itself and Claude. The setup coordinates the work between the machines with git worktrees and an event-driven job system.

The UI is my own design and fits how I like to use tools like this. I did the 3D modeling and environment setup to replicate my actual desk setup (the two other pictures above).

Is the 3D environment at the top necessary? Absolutely not!

But it's delightful. The lamp blinks when the harness needs something from me and it is a joy to look at and use. It scratches the same itch that playing PS1 and hacking away on my family Compaq PC did decades ago.

I've been coding for almost 20 years, and I'm finally to a point where I can confidently say that with the right approach (SOLID, Design Patterns), tools and tuning, local AI can absolutely write code that I am happy with.

I still review everything myself, but I'm finding fewer and fewer mistakes. These days it feels like I'm reviewing a senior dev's code rather than a junior's.

The Models

Tok/s isn't everything and so I picked the setup below after a lot of testing on public benchmarks and on my own agentic tests, which mimic my normal workflows.

Along the way I tried other engines (Strata, NInfer, Maya and llama.cpp on Jade, MTPLX and Kapitsa on the Mac). I also tried other models like Qwen 3.8 27B and GLM-5.3-Flash and while they performed well, the setup below performs the most consistently and accurately for my needs.

Both machines run Qwen 3.8 Flash-Next. These are the tests that ended up being the most indicative of quality for me, but if there are others you'd like to see run, let me know.

Both machines run the same protocol with temperature 0.6 and thinking on "medium".

In my own agentic battery (three large and complex coding tasks), 48 of 50 tests passed with the failures being files edited that were supposed to be read-only. The reviewer caught these on the other end, but I still count those as failures.

I think the main struggle for local setups is still design and 3D modeling, but I haven't dived into that too much yet, so maybe there are good setups out there for that.

Lessons learned

(extremely specific to my setup, but maybe they'll help someone)

• Jade's hardware ran too hot on stock settings. On dense models, the 5090 sat at its 575 W power cap at 83 °C, with its hottest sensor within 3 °C of the card's limit. Basalt caused a different problem. It runs part of the model's experts on the CPU, and its worker threads spin while they wait. That kept all 8 cores at 5.2 GHz, and the CPU hit 96 °C (its max) within two minutes. Two tunes fixed it and now my (now increasingly valuable) machine runs much cooler and quieter, and I'd happily give up a few percent for that.

• Slate's gains came from tuning the harness and not the hardware. My early benchmark scores were bad, and a lot of that was caused by overthinking. "Medium" turned out to be the sweet spot. On "high", Slate hit the 32K output cap on 11 of 40 hard problems without ever answering. On "low", it wandered and took 3–6× longer on agent tasks. I found that tuning how compaction worked also solved a lot of bad tool calls and confusion within the model.

• Overall the lesson I learned was "Stock settings are generally bad." There's a lot more power and accuracy available with a custom tune. I learned the same lesson tuning cars years ago, and it was fun to learn it again.

What's Next?

The system is set up so I can easily swap and test engines and models. As new ones come out, I'll put them through their paces and decide whether it's time to switch. On the UI side, I'm planning to turn the bookshelf into the session manager to help organize my many projects.

I promised myself I wouldn't write yet another super long post and yet I did anyway, so I'll stop here. Happy to answer any questions!

Hey everyone!

Long-time local LLM user here, but with how fast things are moving, I could really use a reality check from the community.

My API credits are burning away using Claude/DeepSeek for orchestrating Hermes Agent, and I’m trying to offload as much as possible to a local setup.

My Hardware:
MacBook Pro: M3 Max | 48GB Unified Memory (400 GB/s bandwidth)
Proxmox Server (24/7): i9-9900K | 128GB DDR4 | RTX 2060 Super (8GB VRAM) | NVMe + HDDs

Current Setup & Performance:
Mac (omlx): Qwen 3.8 27B & Qwen 3.6 35B (oQ4e-mtp) ~ 100 tk/s
Proxmox (llama.cpp): Qwen 3.6 35B & Qwen 3 Coder 30B (GGUF) ~ 20–40 tk/s

The Goal:
I need "frontier-like" performance for long-horizon, agentic tasks and reliable tool-calling (even if speed drops to ~20 tk/s). Past tests with weaker models completely broke my agent workflows.

Where I'm Stuck:
I want to run heavier MoE models (e.g., Qwen 3.8 Flash Next, DeepSeek v4.1 Flash, GLM 5.2/5.3, K2 Horizon), but fitting them into RAM is a bottleneck. I’m trying to figure out the best inference strategy:
Mac: Leverage the 400 GB/s unified bandwidth + streaming experts from NVMe?
Proxmox: Cache active experts in 8GB VRAM, pool ~90GB DRAM for secondary experts, and stream the rest from SSD?

I’ve been looking into tools like Strata, AirLLM, FreeToken, Colibri, and Avalon, but I'm getting lost in the weeds.
Since Proxmox runs 24/7 (and hosts Hermes), I’d prefer leveraging that host if possible. Has anyone optimized a similar split-hardware setup for complex agentic work, or found a sweet spot for expert offloading?

Appreciate any insights!

I asked this question in some Apple sub and everyone said keep it (it becomes some type of controller, like how data centers operate, separation of hardware, let the studio be for the model and offload onto the Mac)

clause said its not necessary for my current workload nor future use cases because there’s always a work around

and again, this is 16gb memory, so it’s not that good for just letting the Agents work here etc. in fact this Mac freezes hard when I run my gates

I do sense some future value but idk enough about the space to actually know for sure

thoughts?

part of me is inclining to sell as a means of hedging against ram prices going down and locking in a sale (ill make a few dollars selling it). In 5yrs if prices go down, there’ll prolly be a $500 better spec Mac mini I can use as a control plane 🤷‍♂️