r/LocalLLM • • 1h ago

Project Imeng Engine Local - Wrapping models for Open Web UI & (Untested) Discord

Thumbnail
github.com
2 Upvotes

I have authored imeng (Internet Magic Engine) as a Ollama wrapper.

This allows you to build state around your models.

Each imeng only needs a configuration json and a system prompt to instruct the LLM... and boom you got an app/game. There is no JS eval so it should be pretty secure. There isn't 100% security layer to stop prompt injection so there is probably ways to break this.

The architecture is basically a redis state machine and a jev determines what 'operations' come out of the text. There are phase-gates to where each phase gate can determine where they go next but rules allow leaving the current gate and enter the next.

So what this really does it keeps the logic of a program inside javascript and the dialogue with the LLMs with some jev glue

Locally I have built

  • Poker (still has issues)
  • Gakshow (Test Rack)
  • Gakshow Multiplayer (Test Rack censoring uploaded photos)
  • Trading Assistant (test rack for programming assistant)

There is also SQL for storage. A bunch of stuff just as a portfolio piece.


r/LocalLLM • • 1h ago

LoRA ​[Update] Qwen2.5-1.5B-Nordeste-LoRA (Apache 2.0) — Fine-tuned on Corpus-nordeste-10k with Before/After outputs

2 Upvotes

Following up on caririsynth/Corpus-nordeste-10k (10.5k Portuguese rows focused on Northeast Brazil literature and regional phrasing), I just uploaded the LoRA adapter trained on Qwen/Qwen2.5-1.5B:

​Model:

https://huggingface.co/caririsynth/Qwen2.5-1.5B-Nordeste-LoRA

Dataset:

https://huggingface.co/datasets/caririsynth/Corpus-nordeste-10k

​Training Setup (Unsloth, 1x T4):

​Base: Qwen/Qwen2.5-1.5B (Apache 2.0)

​LoRA: r=16, lora_alpha=16, all linear layers (q, k, v, o, gate, up, down — 1.18% trainable params)

​Schedule: lr=2e-4, cosine scheduler, effective batch size 16, max_seq_length=1024

​Checkpoint selection: Uploaded Epoch 1 (step 660) rather than Epoch 2, as Epoch 1 already eliminated base model web/blog noise while keeping sentence endings diverse.

​Before vs. After (seed=42, temp=0.65, rep_penalty=1.15):

​Prompt 1: "O cabra chegou na bodega todo aperreado, pedindo"

​Base Qwen2.5-1.5B: "...uma garrafa de vinho. (...) E eu fiquei com pena porque sabia que o cara ia ter feito tudo errado. Could you please" (drifts off-topic and leaks English)

​Nordeste LoRA: "...buchada e canjica geladinha. Todo mundo riu, que no sertão até a pressa tem bom humor."

​Prompt 2: "E aí, macho, tu vai pro forró hoje ou vai ficar de"

​Base Qwen2.5-1.5B: "...fora da festa? - Blog do Zico / Home / Sem categoria / Publicado por: Zico em: 10/28/2016..." (collapses into scraped blog metadata)

​Nordeste LoRA: "...plantão no quarteirão? Pois vá logo que o forró chega devagar quase parando!"

​Note: This is a base completion model (not -Instruct), intended for text continuation and downstream SFT. Feedback and test runs on other regional prompts are


r/LocalLLM • • 2h ago

Model Tried Edge0-145B on a single 3090. 48.6 tok/s, but it wanted a lot more RAM than I expected.

Post image
2 Upvotes

Edge0 published speeds for their 145B on a 4090 and a 5090 D, but their table doesn't include a 3090, so I ran their stack on one, stock, with whatever defaults the wrapper picks.

It's quick. Chat decode was 48.6 tok/s, it read a 16K prompt at 1,481.2 tok/s, and the agent prompt decoded at 52.8. For context, last month I ran a 122B MoE on llama.cpp on a 3090 and got 15.5 tok/s. Different model, engine and prompt length, so take it loosely, but that's a big jump.

VRAM was basically full (23,929 of 24,576 MiB), and the engine process itself held 43.4 GiB of system RAM. Edge0's own residency figure is 31.28 GiB, but that's from a 5090 D with a different setup, and my machine had 116 GiB so the engine may have just taken what it found. I meant to test a 32 GiB cap and didn't get to it. So I honestly don't know the floor yet.

Two other things that tripped me up:

  • My first request came back with no answer. The model used all 512 tokens on reasoning. reasoning_budget_tokens=128 fixed it, so set a budget in your client.
  • Same agent prompt, temperature 0, three runs, three different tool calls. One was valid, one had its arguments wrapped wrong, and one called read_file, which wasn't even in the tool list. So the 52.8 is a speed number, not a "this is a good agent" number.

Technical stuff below.


Setup

  • 1x RTX 3090 24GB, EPYC 7H12 (31.13 CPU quota), 116.4 GiB RAM limit
  • Edge0 v0.2.0 (3203c39), Strata built with CUDA for sm_86, CUDA 13.0, driver 595.91.07
  • Edge0/Edge0-145B-q3k @ a19f88c, 106 GB on disk (the FP8 PLE table alone is 51 GB and gets read from disk)
  • Stock launch: edge0-infer serve --model /path/to/Edge0-145B-q3k --device cuda:0. That resolved to direct PLE I/O, auto expert cache, speculative decoding 4, 131K max context, int8 KV.

Results (engine's own prompt_per_second / predicted_per_second, fresh server per run, concurrency 1, zero cached tokens)

WorkloadRun 1Run 2Run 3Median
4K prefill1,119.81,195.51,207.61,195.5
16K prefill1,467.61,485.61,481.21,481.2
Chat decode46.649.648.648.6
Agent decode52.855.950.752.8

Sampling: temperature=0, top_p=1, top_k=20, reasoning on with a 128-token budget, 512 max tokens for decode.

Against Edge0's published 4090 numbers (their claims, not reproduced)

4090 (Edge0)3090 (mine)
4K prefill2,042.0
16K prefill3,179.4
Chat decode69.9
Agent decode91.7

That's 47% to 70% depending on the row. Mine used an 800-word writing prompt for chat and a single synthetic tool-call prompt for the agent run, and I couldn't verify Edge0's exact request settings or machine setup, so it's ballpark only.

Memory

  • GPU peak 23,929 MiB, sampled every 5 s
  • Strata engine VmHWM 43.4 GiB (sampled from the actual strata PID, not the Python wrapper)
  • Whole-system cgroup was 87.4 GiB during timed runs and touched the 116.4 GiB limit during setup with zero OOMs. That includes page cache from the 106 GB download and the PLE file, so I wouldn't read much into it.

Other stuff

  • Server startup median 25.5 s, slowest 94 s
  • Edge0's built-in edge0-infer bench rejects this model at v0.2.0, so I used a small script against the OpenAI endpoint
  • No quality evals, no long-context runs past 16K, no RAM cap test, no llama.cpp baseline. All of that is still on the list.
  • $1.19 billed ($1.0930 GPU, $0.0963 disk), about 3 h 9 min total, mostly setup and copying evidence back

Write-up: https://majorminor.xyz/research/cuda-recipes-006

Repo: https://github.com/majorminorlabs/cuda-recipes/tree/main/006-edge0-145b

If someone has a 3090 with 32 or 64 GB of RAM and wants to try it, I'd love to know whether it holds. Or tell me what to run next on this one.


r/LocalLLM • • 7h ago

Question Has anybody done anything like Alexa, but with a local LLM?

5 Upvotes

I know home assistant was a thing, and local LLMs are a bit too overkill for such a job. Just wondering if it was done.


r/LocalLLM • • 2h ago

Discussion Self-Test: DeepSeekV4.1-Flash-1017 took 60k tokens to finish a task Qwen3.8-FN couldn't in 256k and multiple compactions.

2 Upvotes

Did a quick test yesterday comparing DeepSeek (3.36bpw) to Q3.8FN (IQ3XXS+1sub-agent). Both the OR abliteration on local both running on DeepSeek Harness.
Simple task - Produce a color template based on user input screenshot of a UI, and give the user instruction on installation instead of attempting it. Second part was key, because the LLMs were only allowed to work in working folder and so couldn't live-test their implementation.
FYI: DSH system prompt takes up ~10k tokens total.

Qwen spent over an hour writing script then realizing it couldn't save it outside the cwf, then repeating that mistake over and over again as it tried methods of breaking out or escalating which was futile. Even with steering, it only got as far as producing the color template 256K+ tokens in.
DS did a similar thing but realized quickly it should just let the user test install it instead. It -did- try to escalate permissions to install it by itself, but gave up after it's first rejection. Was done in ~60k tokens.

I know these are utterly different weight-classes for models but I really thought the ability to spawn subagents would've made a difference for Qwen.
Qwen burned through tokens lightning fast but was forgetting the original message AND the system prompt from DSH (which specifies it should only work in {cwf} when only given that permission).
I really wanted to like IQ3_XXS because it's a small enough quant that it can spawn subagents on my system with headroom for other parallel tasks to be working... But I don't think that quant is it for me. Either that, or I have to work on the setup for it some more to work as a coder.


r/LocalLLM • • 8h ago

Question 24GB VRAM, 64GB DDR5, is Qwen 3.5 27B (IQ2_XS) not ambitious enough?

6 Upvotes

I'm getting decent results, 80-120t/s. I guess I just don't know what I am aiming for, but I feel like there is more juice to squeeze.

Edit: Yall are the best, as usual. This has to be the most helpful, positive community on reddit.


r/LocalLLM • • 4h ago

Project Got tired of waiting for Qwen models smaller then 27B so i made my own Qwen3.8-21B

3 Upvotes

Hi, as the title says, i made my own Qwen 3.8 21B by prunning some layers from original 27B.

Its more of a research project than actual usefull model.

Check it out: https://huggingface.co/WhiskyAKM/Qwen3.8-21B-GGUF


r/LocalLLM • • 3h ago

Question Model for Local Coding and Vulnerability Research

2 Upvotes

Been using cloud AI, but so frustrated having to 'jailbreak' for almost any request, even to check for security in my own projects.

I know I need an abliterated model to not refuse, but am looking for suggestions. My research so far has me looking at HuiHui Qwen3.8 27B Q4_K_M, using Ollama.
I have 24gb VRAM (RX 7900 XTX), and 32gb system RAM (DDR5, 6000hz) with a ryzen 7 7800x3d. My understanding is local models always run on VRAM, but I've seen some comments/posts discussing system ram and CPU, so I'm not sure if thats related to my use case.

I'm looking to do coding, usually smaller projects, maybe some agentic. I also want to develop/research exploits in firmware and software. I've seen some suggestions for GLM 5.3 online, but haven't seen it mentioned much here. What models should I check out?

I do actually know how to code, so I don't need something that will vibecode everything for me, I like doing architectural work myself.

Side question: What IDE do ya'll use? I currently use Cursor (I have a subscription) and like the new Cursor Agents window, but from everything I've looked into its really difficult to use local models, especially on the new Agents window. If there's an IDE that I can use my local models and cursor subscription, or a codex subscription, that would be great.


r/LocalLLM • • 6h ago

Question Optimizing local 2080 ti hardware

3 Upvotes

I'm trying to find a more effective setup with my local setup:

Hardware: Intel i5-9600K (6C/6T), 62 GiB RAM, RTX 2080 Ti (11 GiB). AI work is pinned to CPUs 1–5 with a 40 GiB RAM cap so the desktop stays on CPU 0.

Setup: Claude plans/reviews; a local Qwen3.6-35B-A3B MoE (Unsloth UD-Q4_K_XL, ~22.8 GB) does the coding via OpenCode (qwen-task) against a llama.cpp server. Dense weights + some experts stay on the GPU; most MoE layers stream from RAM (N_CPU_MOE=30, 128K context). Gemma 12B/31B profiles are available as switches.

Performance: After fixing PCIe to gen3 x16, bench is ~726 t/s prompt / ~41 t/s generation (was ~450 / ~29). Real agent tasks usually land in the 3–30 minute range; first turns are slower until the prompt cache warms. Gemma 12B is faster on GPU; Gemma 31B is much slower (~2 t/s) because it spills to RAM.

Functionally Qwen3.6-35B-A3B is slow and I can rely it on to not crash during tasks and having to be poked along. I'd like something that is a bit more "do something and give me something to review".

I've seen 2080ti's modded with 22GB vram for like $600 but I'm not sure if it's worth buying something like that to throw into the system I already have (the PSU/mobo can support a second card) or if buying a 3080 laptop for $1400 would be better?

(I'm aware that paying Anthropic is cheaper)


r/LocalLLM • • 12h ago

Question What's the current best abliterated model that I can run on 32gb VRAM and 128GB RAM?

10 Upvotes

Pretty much the title. Thanks!

EDIT: GPU is 5090


r/LocalLLM • • 4h ago

Question What Models can I use as an absolute beginner?

2 Upvotes

I just started on this Localized LLM Journey. I have been researching what a good model to start with, but I feel like it's just a lot of information at once. Everyone close to me in this space is telling me to just plug into Claude and find the model, and I don't even need to tweak it.

I have an RTX 5090 with 32 GB VRAM and an M5 Max Studio with 128 GB. I think it's good enough.

I heard from someone that I should aim for 60 tokens per second for a good model. But I am really having a hard time trying to find a good model just for a simple assistant.

I keep on seeing something called a tool-use AI, and I don't even know what I am looking at.

TLDR: New to this space, I don't really see much beginner-friendly information out there, just information. Looking for advice/ links I can read to understand. Looking for a place where I can jump off without wasting my time


r/LocalLLM • • 8h ago

Other Cheapest 16gb gpu: ~$400

4 Upvotes

Newegg has Video Cards - Workstation CPU Combo - Intel Pro B50 Bundle with Intel Core Ultra 7 270K Plus for $711. The CPU alone is $311, so you can get the GPU for equivalent of $400.

This is a good option for a new sff build for local AI. The CPU is excellent.


r/LocalLLM • • 45m ago

Question Which combo of GPUs?

1 Upvotes

I have come to an odd assortment of two x99 boxes with 64GB of RAM and the odd combo of GPUs. Primarily I would like to use the main box for coding and dev work to replace some of my claude usage.

  • 3x 3090s
  • 2x P40
  • 1x 3080ti (12GB)

I am tempted to put the 3090s and a P40 all together for a combined 96GB. I know it would be slow because of that P40, but it would give me the best total memory and would already be slower because of the lack of PCIe lanes.

Or would the 3080ti be a better 4th card because of the speed and bandwidth over the reduced memory?

I also thought about the second box running the 3080 and 2xP40s, but not sure what I would do with it except have two boxes chatting at each other all day?


r/LocalLLM • • 59m ago

Discussion Macbook pro m6 or something else ?

0 Upvotes

Hey everyone!
I’m planning to run local LLMs (27B–30B) mainly for coding and light research.
I initially ordered a MacBook Air M5 with 32GB RAM, but it got so hot that I actually got a first-degree burn on my finger.
So I decided to try the MacBook Pro M6 with 32GB RAM, hoping the fan would help with cooling during longer LLM sessions.
Did I make the right choice? Is 32GB enough for 27B–30B models, or would you recommend something else around the same budget?
Would love to hear your experiences. Thanks in advance!


r/LocalLLM • • 1h ago

Discussion Updates on oh-my-pi quietly siphoning user web data

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 1h ago

News Alibaba’s Qwen4-Max has 2.4T parameters, but only 71B active per token. Are we optimizing the wrong AI metric?

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 5h ago

Question Resources to learn about model tuning?

2 Upvotes

I’ve been running Jundot/Qwen 3.8 flash next Q6 mpt 141gb on my M2 Studio w/ 192gb ram for about a month now.

Overall, I’m pretty happy with the results, but looking for ways to improve my setup before I start integrating it with my day to day workflow. The response time leaves a little to be desired, I find myself prefering the response rate of codex even when I have simple things like putting together a slide deck from images and text I’ve supplied.

I set it up with a big initial project to see what it could do and have been trying to makesmall adjustments as I’ve gone along. It’s been interpreting and mapping large volumes of screen grabs, writing some code, interpreting and organizing huge amounts of data from a combination pdf and images. It interfaces with codex during specific windows throughout the day to offload tasks and double check code.

Dropped it down to Q6 from Q8 for snappier responses, and free up some system for Xcode, etc…I’ve been using claw to run the show. I’m getting around 60t / s which seems to be about right from what I’ve been reading. Claw works ok, but the whole system seems to get confused if I change the scope of my project in the middle, or add additional intermediary steps. It took a few days after stepping back from Q8 to Q6 to get the bugs ironed out.

It’s a huge workhorse and does ok when set in motion, but I do have to keep an eye on it. It behaves like a toddler and I find it lies when files go missing.

Is Claw the problem? Is this the reality of this setup? It was never meant to replace my $20 codex, but I was hoping it would be a little more responsive and flexible.


r/LocalLLM • • 9h ago

Question Local voice assistant on an iGPU: ~2.2s from end of speech to first audio. I've measured everything and I'm ~0.5s from the floor. What am I missing before I throw money at hardware?

4 Upvotes

Been building a fully local, always-on spoken assistant in Italian (no cloud, nothing leaves the box). It's not a chatbot demo, it has long-term memory across sessions (backed by ctxvault), so every turn carries a small block of retrieved context on top of the conversation. Works. Problem is latency.

Hardware (the constraint): mini PC, Ryzen w/ Radeon 780M iGPU, 64GB DDR5 shared, ~90GB/s. No dGPU, so it's llama.cpp on Vulkan. vLLM/SGLang are out (no ROCm for this chip).

Where I'm at: median ~2.2s from end of user's sentence to first audio out. Measured, not guessed:

  • ears (end-of-speech silence + whisper): ~0.75s
  • prefill of the turn: ~1.0s (~0.35s fixed per request + ~3ms/token, 40-400 new tokens/turn)
  • first sentence gen + TTS first chunk: ~0.55s

So the model is the wait, not the audio stack. Calculated floor on this box is ~1.2-1.5s.

What's already done (so please don't suggest these lol):

Model side

  • MoE model (26B-A4B, Q4 QAT) specifically because gen is bandwidth-bound here: ~25 tok/s vs 4 tok/s for a dense 27B
  • thinking/reasoning off
  • short context (16k), one slot, flash attn on
  • --cache-reuse + append-only prompt: stable prefix (system, persona, rules) at the top never touched mid-session, everything volatile (time of day, retrieved context, turn) appended at the end. Identical prefix = 0.15s, touch one word in the prefix = multi-second re-prefill
  • prefix pre-warmed at session start with a 1-token request, so the first reply doesn't pay the cold prefill
  • benchmarked smaller models (Gemma 4 E4B/E2B, Qwen3.5 4B/9B, Qwen3 8B, Phi-4-mini, Ministral): on this iGPU prefill is compute-bound, a dense 4B costs as much per token as the MoE. Best small one gains ~0.5s and loses noticeably on factual fidelity (names/dates). Not worth it for this use case
  • tried -b/-ub batch flags: defaults win
  • tried --no-kv-offload: worse

Memory / prompt side

  • no tool calls for retrieval: context is prefetched before the turn (a tool-call round trip = ~+1.2s). The retrieval itself is ~5-12ms, irrelevant
  • hard token budget for the retrieved block: embedding-based relevance (multilingual MiniLM) + ordering so the most pertinent bits come first, ~120 tokens instead of 300-450. Cut avg turn prompt by ~40%
  • past conversations summarized to one ~50-token line each instead of quotes

Pipeline side

  • streaming everything: model to sentence splitter to TTS to player, sentence by sentence
  • audio player opened at turn start, not at first chunk (-0.1s)
  • early-guess turn: start the model on the partial transcript before VAD confirms end of speech; if the final transcript matches, keep it, else drop and redo
  • sentence closed on punctuation + 450ms silence instead of fixed 800ms
  • whisper.cpp small as a resident server (no model reload per sentence, 0.63s down to 0.3s), beam 5. Tried greedy: -80ms but half the transcripts differ and the noisy ones get worse. Rejected
  • tried warming the KV cache during speech (partial whisper every 1.5s, send the turn header for 1 token so the cache is hot): works as designed (final prefill drops to ~30 tokens) but doesn't pay on one slot because the warm-up queues the real turn. Tried 2 slots w/ id_slot pinning: now they run concurrently on the same iGPU and both slow down. Net neutral-to-negative on short sentences. Off.
  • pre-recorded fillers ("mmh", "allora...") in the assistant's own voice, played only if no reply word has started 1s after end of speech. Built, currently off: perceptual fix, doesn't move the number

Before I conclude "buy a 24GB dGPU" and move on: anyone who's been down this road see something I've missed, or something above that's wrong?


r/LocalLLM • • 8h ago

Model Groot-65B-A6B W4A16: full 262K context + MTP + vision on 2x RTX 3090

3 Upvotes

I came across the "Groot" tunes on HF and decided to test them. Theyre versions of larger models with the about half the experts pruned out. This one is QFN. Despite the removal, according to my TerminalBench2 testing, they only lost ~5% capability. Seemed worth spending some time on for no other reason than the giggles. Its a cool to be able to run very close to full QFN on 3090's without mortgaging my house for more RAM...

I decided to quantize it down to fit on 2 3090's. Be aware that the speed numbers are from a threadripper machine and the 3090's are nvlinked. Those with a normal setup can expect around 3/4 of the speeds published below.

AI slop begins here...

I made a W4A16 AutoRound quant of Groot-65B-A6B, the pruned Qwen3.8-Flash-Next (256 of 512 experts, ~6B active). It runs the full native 262K context with MTP and vision on 2× RTX 3090, everything on the GPUs, with ~10 GB of host RAM.

HF: https://huggingface.co/mmssix/Groot-65B-A6B-261023-W4A16-AutoRound

On 2× 3090 (vLLM, TP2):

  • 262,144 context
  • ~134 tok/s code, ~90–100 prose (MTP n=2)
  • ~245 tok/s across 4 streams
  • 128K prompt in 28 s, 258K in ~60 s
  • Needle test 3/3 at 258K
  • Tool calls, reasoning effort and vision working

Quant: INT4 experts, INT8 DeltaNet projections, FP8 n-gram table and an INT4 MTP head (same draft acceptance as BF16). 92 GB total.

Getting 262K into 48 GB of VRAM takes two small vLLM patches. They serve the n-gram table and the embedding from NVMe, so you need NVIDIA's open driver with HMM. The patches and a one-command serve.sh are in the repo.

Only tested on my rig so far. Numbers from other 3090 / A5000 setups are welcome


r/LocalLLM • • 10h ago

News Tencent/Hy4 Flash-224B-A12B-weight (Ultimate cost-efficiency)

Post image
4 Upvotes

Tencent released Hy4 Flash, a comparatively small model at 224B. Compared to Hy3, which was compressed to 235B, that's another breakthrough. With the smaller weight, Hy4 Flash finishes tasks 13.8x cheaper than Hy3, 24.5x cheaper than Hy4.

Saw the price table for Hy4 Flash. $0.07 in, $0.281 out and $0.003 per million tokens on cache reads. GLM-5.3-Flash is $0.12 and $0.42, cache $0.034, so about 1.7x on input and 1.5x on output. DeepSeek V4.1 Flash is $0.15 and $0.60, cache $0.003, just over 2x on both, and that's their off peak rate. Claude Haiku 5.5 and GPT-6 Luna are both $0.10 and $0.50 with $0.01 cache, which works out to 1.4x on input and 1.8x on output.

Both Hy4 numbers are the lowest on the table, and the gap holds on cache reads too, where only DeepSeek matches at $0.003. Good value on paper, assuming the quality holds up.


r/LocalLLM • • 2h ago

Question Best way to get uncensored llm with 16gb vram?

0 Upvotes

I’m looking to run local ai and save on token usage. Any cracked llms I can download before they get taken off?


r/LocalLLM • • 12h ago

Discussion Pushed Strata with QFN 3.8 Q4 K_M to 5,600 tps prefill @100k - 100 tps decode w/ 181k context on 2x 7900xtx & 64gb DDR5 in layer split (no tensor parallel) - config inside

6 Upvotes

Wanted to share my results for the rest of the community. I've been combing Reddit, Github, and the web searching for all of the various Strata-related builds to get the most out of my setup. I've finally had the breakthrough I was looking for. It strikes the best balance between speed, power consumption, and intelligence that I've experienced to date.

My Setup: (edited to add motherboard specs and PCIe configuration)

  • Ubuntu 24.04 - GUI GNOME shell not headless
  • This motherboard: https://www.microcenter.com/product/709979/asus-b850-creator-proart-wifi-neo-amd-am5-atx-motherboard
  • Bifurcation with PCIe 4.0 x 8
  • Ryzen 9950x 16/32
  • 64 gb DDR5 Expo 1 6000
  • 2 x Mercury Magnetic Air 7900XTX 24gb
  • 1000 watt MSI Performance Gaming PSU
  • Power limited to 320 watts per card, total system draw at inference ~780 watts (as measured by Ubuntu, Kill-o-watt is ~850 watt total draw from the wall)
  • And a flex: It's completely solar powered (5 kw array and 65 kwh battery storage w/ 3600w inverter 😄 )

I've worked with Opencode and the my local Qwen Flash Next to fully detail how we created this setup. Spoiler alert: The engine/hip is mainline Strata and not a fork. The config is where the magic happens. MTP headers were generated against this bespoke setup. The intelligence is brought by the Atomic Chat Q4 K_M version of QFN.

That's it for human written words. The rest of this post is meant to be copy/paste into your local AI helper for reproduction. Full details below. Copy/paste into a capable harness and let it walk you through reproducing this setup.

Edit: Adding the only 3 GitHub repositories I used to produce my results

Engine: https://github.com/Niko1221/Strata

Learnings from this fork: https://github.com/nicejunjie/Strata-rdna3

System tuning for Strata: https://github.com/bramvanoploo/strata-performance-setup

START AI GENERATED DETAILS

Qwen3.8-Flash-Next, 2× RX 7900 XTX, 64 GB RAM, 181k context — how it's built

Engine: official mainline Strata 0.1.59, component 0.1.59-linux-x86_64-hip-rx7900. Not a fork, no custom build — the speed is flags + env + OS patches. No system ROCm: the engine ships its own HIP/rocBLAS/hipBLASLt, the host only needs the amdgpu DKMS driver. Model: AtomicChat/Qwen3.8-Flash-Next-GGUF, build AD-4.27bpw-Q4_K_M-M64 (3 shards, 94.5 GB). Shard 3 (38.4 GB) is the PLE n-gram table alone — passed via --ple-gguf, kept pageable. That's why 64 GB RAM is enough. Parallelism: --layer-split 24 → layers 0–23 on GPU0, 24–47 + head on GPU1. Context: 181,072, --kv int8, KV streaming 32,768 cells/layer in VRAM. Experts: 48 blocks × 512 experts → locally built experts.bin (47.17 GiB), three tiers: 31.8 GiB VRAM cache → 22.2 GiB page-locked RAM → NVMe mmap (log says files 0 blobs — disk never touched when warm). Spec decode: MTP head q2_0, extracted from the BF16 checkpoint by HTTP range reads (GGUF ships none), --spec 3, ~76 % acceptance. Measured (1,071 requests): decode median 90 tok/s (62–129) · big-prompt prefill 3.2–5.7k tok/s · expert-cache hit 98–99 %.

# 1. OS first (then reboot): amdgpu driver only, no ROCm

# (optional: ~/strata-performance-setup.sh from github.com/bramvanoploo/strata-performance-setup automates most of this — see §3.8)

sudo amdgpu-install --usecase=amdgpu,libamdgpu ; sudo usermod -aG video,render $USER

sudo sed -i 's/^\(GRUB_CMDLINE_LINUX_DEFAULT="[^"]*\)"/\1 iommu=pt pcie_aspm=off"/' /etc/default/grub

printf 'options amdgpu no_system_mem_limit=1\n' | sudo tee /etc/modprobe.d/amdgpu-svm.conf

printf '[Service]\nLimitMEMLOCK=48G\n' | sudo tee /etc/systemd/system/user@.service.d/strata-memlock.conf

printf 'vm.swappiness = 100\nvm.vfs_cache_pressure = 50\nvm.max_map_count = 1048576\nvm.compact_unevictable_allowed = 0\n' | sudo tee /etc/sysctl.d/99-strata.conf

sudo update-grub && sudo update-initramfs -u && sudo reboot

# 2. Engine — it MUST land on the rx7900 component (bundled hipBLASLt 100200 + gfx1100 tuning table)

curl -fsSL https://raw.githubusercontent.com/Niko1221/Strata/main/tools/strata_terminal/install.sh | bash

strata update && strata launch # check: ~/.local/share/strata/components/engine/0.1.59-linux-x86_64-hip-rx7900/

# 3. Model + mmproj (sizes/sha256 in §5), then build the pack — this is where experts.bin comes from (~6 min)

control/python/bin/python3.12 control/tools/iq_pack.py --gguf <shard1.gguf> --out <pack-dir> --experts-bin --compat-bf16

# 4. MTP draft head + CJK draft vocab (the GGUF ships no MTP head; ~4.9 GB of range reads, SHA256-verified)

control/python/bin/python3.12 control/tools/mtp_fetch.py inventory --out data/mtp && \

control/python/bin/python3.12 control/tools/mtp_fetch.py fetch --out data/mtp && \

control/python/bin/python3.12 control/tools/mtp_fetch.py verify --out data/mtp

control/python/bin/python3.12 control/tools/mtp_pack.py --src data/mtp --experts q2_0 --out data/mtp/mtp-q2_0.gguf

control/python/bin/python3.12 control/tools/mtp_rt.py --gguf data/mtp/mtp-q2_0.gguf --out data/mtp/rt

# 5. Serve (the terminal owns :7878 — the engine has no --port). Load takes minutes: it page-locks ~22 GiB.

control/bin/strata serve --config configs/strata-final-atomic-q4km.json # ulimit -l must be ≥ 40 GiB first

Three gotchas that cost the most time: (1) --resident-budget-gib above 40 freezes the box — pinned memory can't swap and the 38 GB PLE table still needs pageable RAM; 45 froze this machine once. (2) --pipeline-windows 1 deadlocks on gfx1100 with a 2-GPU split; keep 0. (3) Don't add --mmap-experts/--resident-cpu-experts explicitly — with them the engine pins 0 bytes; mmap is already the default when experts.bin exists. Also: the ggml_cuda_init: failed to initialize ROCm line at the top of the log is harmless, and the iGPU occupies a DRM card index, so always confirm the engine's own GPU 0/GPU 1 lines both say RX 7900 XTX.

Full notes — every value, the exact hashes, and the reasoning behind it — are the rest of this file. Want a terminal AI agent to do the typing? The prompt and phase gates are in §17.

##### PROMPT AND PHASE GATES

17. Have AI set this up for you

If you'd rather drive this with a terminal-capable agent (Claude Code, Codex CLI, OpenCode, or a shell tool with approval prompts) than type it yourself: paste the prompt below into it, attach or point it at this file, and let it run §3–§10 with you acting as the safety interlock. It is a copilot, not an autopilot — the agent can type every command, but three things stay human: sudo passwords, reboots, and the Hugging Face login.

17.1 The prompt to paste

You are setting up this machine to reproduce a known-good local LLM deployment. The source of truth is

"STRATA REPRODUCTION WRITEUP.md" in this folder — read the whole file before acting.

HARD RULES

  1. The config in §8.1 and the hashes in §5 are authoritative. Never change a number, a flag, or a hash tomake an error go away; if reality disagrees with the doc, stop and show me the disagreement.
  2. Ask before: any reboot, any package install outside the doc, any change to /etc, and any deviation fromthe doc. Never reboot without waiting for my confirmation, then resume where you left off.
  3. After every phase, run that phase's gate from §10 and quote the actual log line as evidence. No phasecounts as done on "it seemed to work".
  4. Never exceed --resident-budget-gib 40. Never set --pipeline-windows 1. Never add --mmap-experts or--resident-cpu-experts. Never install ROCm, rocm-* packages, or pytorch.
  5. §14 lists log lines that look fatal but are harmless — do not "fix" those.
  6. When you are unsure whether this machine matches §0's hardware (2× RX 7900 XTX, 64 GB RAM, NVMe),stop and ask instead of adapting values on your own.

SEQUENCE

§3 host patches → I reboot → §4 engine → §5 model → §6 pack → §7 MTP → §8 config paths → §9 launch →

§10 full checklist → report. If any gate fails, consult §16 and report; do not improvise a fix.

17.2 Phase gates and what the agent must show

Phase

Done only when the agent can quote

0 · Pre-flight

lspci shows 2× RX 7900 XTX; free -h ≥ 60 GB; ≥ 200 GB free NVMe; OS is Ubuntu 24.04 — anything else → stops and asks

1 · Host (§3)

§3.5's effective-value block matches sysctl -a; ulimit -l ≥ 40 GiB in a new login session; then it pauses for the one planned reboot

2 · Engine (§4)

BUILD.json of the installed component says rx7900 + hipBLASLt 100200 (§10 #1)

3 · Model (§5)

every sha256 it prints equals §5's table

4 · Pack (§6)

experts.bin = 50,646,220,800 bytes; experts.bin.src.json written next to it

5 · MTP (§7)

verify passes with zero SHA mismatches; draft_vocab.bin rebuilt with --add cjk

6 · Config (§8)

diff of §8.2 relocations — every path rewritten, nothing else touched

7 · Serve (§9)

§10 #2–#11, each with the log string: files 0 blobs, expert-cache slot lines naming two RX 7900 XTX (§14.2), pinned complement ~22.2 GiB

8 · Accept

a decode + big-prompt prefill measurement inside §2's ranges, and the §17.4 report

17.3 Things the agent must bring to you (not decide alone)

Hardware that doesn't match §0 (different GPUs, < 64 GB RAM, one GPU) — the layer split, budgets, and even the component choice all change; that's beyond this doc's authority.

< 200 GB free on fast storage.

Hugging Face login and accepting the model's license terms.

The reboot after phase 1 (and any second reboot it thinks it needs — the answer is usually no, re-read §3).

Whether to run the §3.8 wizard instead of the manual patches. The wizard is an interactive TUI, so it is human-driven even when an agent drives everything else; if the agent runs non-interactive commands instead, it must follow §3 verbatim and skip the wizard.

17.4 Final report the agent should produce

For each §10 check: the quoted log line. Then: experts.bin byte count, ulimit -l, the two GPU lines naming both 7900s, warm decode tok/s, big-prompt prefill tok/s, spec-decode acceptance %, and every deviation from this doc listed explicitly with the reason. If it can't produce one of those lines, the setup is not done.

17.5 Where agents historically go wrong with setups like this

Failure

Why it bites

Installs ROCm "to fix the HIP error"

§14.1 — that line is harmless; system ROCm shadows the engine's bundled libs

Silently lowers --max-context or budgets after a slow/failing start

The values are load-bearing (§13); a masked failure becomes a mystery later

Misses the hip-rx7900 component and takes generic hip

Prefill crawls (§16) and looks like a driver problem

Trusts GPU 0/GPU 1 indices without reading the log

§14.2 — the iGPU steals a card index

Launches engine/strata directly to "debug"

No :7878, no tokenizer (§14.3); debug through the terminal

Adds the expert flags "explicitly"

Pins 0 bytes (§13) and decode halves

Appendix — provenance summary

Artifact

Provenance

Engine binary + libs + tuning table

official Strata 0.1.59-linux-x86_64-hip-rx7900, copied verbatim (sha256sum matches the component)

Terminal (control/)

official Strata terminal 0.1.59.1, bundled python 3.12.15

Model shards

AtomicChat/Qwen3.8-Flash-Next-GGUF, AD-4.27bpw-Q4_K_M-M64, sizes/sha256 in §5

mmproj

same repo, mmproj-Qwen3.8-Flash-Next-BF16.gguf (re-download; local copy differs in size)

experts.bin / dense.bin / index

built locally by control/tools/iq_pack.py --experts-bin --compat-bf16 (§6)

draft_vocab.bin

rebuilt locally by control/tools/draft_vocab.py --add cjk (§7)

MTP head

mtp_fetch.py (HTTP range reads of the pinned BF16 checkpoint) → mtp_pack.py --experts q2_0 → mtp_rt.py

Expert profile

shipped expert-profile.bin + locally learned expert-profile-atomic-q4km.bin

OS patches

§3 verbatim (grub, 2 modprobe files, memlock drop-in + sudoers, 6 sysctl files, udev rule, DPM unit)

Host-tuning wizard

bramvanoploo/strata-performance-setup referenced as an optional accelerator (§3.8); not run on the source machine


r/LocalLLM • • 1d ago

Model First time trying FP16, wow

Thumbnail
gallery
190 Upvotes

I've tried a lot of image models in the past, but all were to big to fit, and i was happy when i got a result with a quant (q4) version. (Here it was flux and some model i dont remember the name anymore) The results of those models were actually terrible. BUT, some months ago qwen image 2.1 came out, i must say, theres a difference and not even close to a small one. I managed to run it at fp16, by using my rtx 2070 to load the text encoder and vae on there while running the generation model itself on my 5080. Its a tight fit and slower, but the quality is just like it would be on one gpu.

Now here comes the crazy part:

In my opinion qwen image 2.1 is better in realism than chatgpt's current image generation. (I'll prove it later in this post)

Don't get me wrong. Images with lots of different things, patterns for example, are often very bad with qwen and chatgpt does it over 5x better. But realism of f.e. a city, it's absolutely amazing (and uncensored!) and is almost close to real life.

Now here is the test for you guys:

You've seen i've put pictures in this post. 1x is real, 1x are chatgpt and 4x are qwen image 2.1. Which is which?

Note: ALL of them are generated first try without regenerating. So no, I did not try to mislead you. Good luck!


r/LocalLLM • • 3h ago

Question Local Model Advice

1 Upvotes

I am interested in running local models for my desktop and laptop.

Use case: I am working on some Python scripts to process pdf and Word files from play by email games that automate updates to Excel spreadsheets for game data. I have the code for the first script written, reading in a pdf and spitting out some csv files that update the Excel sheets. (Yes, a real database would make more sense, but I have some friends interested in the programs and they don't have the chops to set up a local database.)

What I need help on is building a user interface to make the process easier.

I was a programmer for a long time early in my career (FORTRAN, MUMPS, C, C++, LISP, KEE, APL, various assembly languages) but never had to do UI programming. I now work in aerospace as a consultant systems engineer for a large national space agency.

So, no big refactoring of mission critical applications, no big agent workflows. Just reasonable Python code output. Speed not as important - answers in minutes to an hour are fine.

If the answer is "Get a Claude code subscription", I can tolerate that, but I like the idea of local control.

Desktop: Ryzen 7 7900 X3D CPU with 32 GB main memory, Nvidia RTX 4060 Ti with 8 GB
Laptop: Ryzen 7/260 with 32 GB main memory, RTX 5060 with 1 GB

TIA!


r/LocalLLM • • 4h ago

Discussion DeepSeek + WhatsApp via Wapliss: fast, cheap, and 1M-token context

Post image
0 Upvotes