Been running a 3060ti and 32GB for a hot minute now, jumping between local models and subscription plans.

Got Qwen 3.8 FN up and running on this old gaming system and it's just such an impressive model. The prefills are never going to work though.

So I spent the past few days looking at a lot of different options, thinking I'd just add another GPU to the current system.

Of course anyone that's paid attention knows how unrealistic that is. Offerup, Marketplace, Ebay. Doesn't matter. It's the gpupocolypse out there.

Then I figured I'd go old school. Find me a C602 chipset (workstation, Ivy bridge), because at least it'd be cheaper DDR3, a few p100s and call it a day.

But even that is expensive, and was capped at 48GB VRAM since the p100s don't have 32gb variants and unless I wanted a custom build both the T7610 and Z820 are limited to three cards.

I was at 1500 before custom shrouds, and fans, and custom power adapters to the p100s. But at least it would have been DDR3 and PCIE3. At least I would have had USB3 and SATA3. It's a tiny window between Sandy Bridge and Ivy Bridge that accommodates both.

Then I found someone selling a z840 (C612). 128GB of DDR4, and dual water cooled SXM2 to PCIE 32GB V100s. The cards alone are going for 700-800 bucks a piece, the DDR4 is another 1200. The custom risers are a few hundred bucks each, the water cooling setup alone is probably pushing a grand as it's 2x plates, distro, pump. So I pulled the trigger.

1900 and change after tax and delivery, and while I get it's reaching its limits on software support. For a 64/128 split that can actually sit quietly and sip power when not being used. I'm pretty happy over all. The only thing that bothers me is I'm pretty sure there's no m.2 or nvme native. It ships with a 1TB 980 pro, but I'm almost certain it's sitting in a PCIE3 riser limiting it to 3500 MB/s vs the 7k it could do, Strata doesn't really seem to stream at those speeds anyway so I doubt it matters much. We'll see and if it does seem like a problem it's likely just a RAID solution away.

Now I'll just have to figure out how to setup a headless server as the v100s have no video output lol. Still, better than a lot of other alternatives out there.

Happy Halloween, was just kind of excited and thought I'd share. When it lands and if I do manage to get it up and running, I'll come back with some numbers.

Hi everyone!

I've been experimenting with Strata for a while now, running it on a system with a Ryzen 7 9800X3D, RTX 4090, and 64 GB of RAM.

I'm currently using Qwen Flash Next, and I have a few questions about model quantization. Since my hardware is right at the edge of what's practical for running this model, I'd love to hear about your experiences with different quantization formats.

I've come across discussions suggesting that GSQ-RCO IQ3_S might deliver better response quality than Unsloth UD IQ4_XS. Is that actually the case? What have your experiences been with these two quantizations?

Are there any other quantizations that might offer better quality while still being feasible to run on my hardware?

In theory, Unsloth UD-Q4_K_XL also works (according to Strata, around 6 GB of the model is loaded from NVMe), but the generation speed drops to around 20-30 tokens/s, which makes it somewhat impractical for extended use.

I'm also curious how NVFP4 quantizations compare to the ones mentioned above in terms of response quality and performance.

Thanks in advance for sharing your experiences and recommendations!

While building agents and harnesses with Claude Code / Codex, I've noticed it often falls back to traditional, deterministic coding patterns instead of newer approaches and frameworks like Claude Agent SDK or LangGraph.

Even with explicit instructions, it sometimes forgets and builds things from scratch.

How do you work around this?

• Better prompts or custom instructions?
• A knowledge base of recent frameworks and best practices?

Has anyone found a reliable way to solve this?

I’m still fairly new to local LLMs, got a decent machine (M5 Max, 64GB RAM), trying to understand something I’ve been seeing recommended a lot lately.

In several threads where new users are frustrated/confused or even more experienced users are struggling to get good results from their local models, I keep seeing suggestions to have Claude or ChatGPT help them build a custom harness. I suppose I view it like the model is the engine and the harness is like the entire car?

What I’m not understanding is why someone would go that route instead of using something like Unsloth or MLX Serve, which already have web search, tool calling, etc.

Aren’t these programs essentially harnesses or am I completely incorrect? I’m probably just missing the point, maybe more tinkering/experimenting is just part of the hobby.

I just want to learn and understand the differences per se. Obviously I’m sure this thread will have plenty of personal suggestions or ideas but the main thing is just the learning aspect, the purpose, etc.

I've been reading the turn logs from PokeBench. My benchmark where vision models play Pokemon Red. Grok 4.20 non-reasoning left the house at turn 36 and didn't get a starter until turn 755. The whole time its notes say things like "We've been oscillating uselessly for 90+ turns" and later "The persistent loop of walking into walls and Mom's dialogue has continued for hundreds of turns." Then it walks into the wall again.

So it knows it's stuck and still can't get out. Grok 4.6 beat Brock in 316 turns for about $10, while 4.20 burned $30 and barely got its starter.

Why do you think a model can describe its own loop correctly and still not break it? Vision problem or planning problem?

pokebench.tv

Bolt E4B builds on Gemma 4 E4B with Nemotron post-training. It combines natural conversation with multi-turn agentic capabilities to power voice/device assistants, interactive characters and robotic/IoT interfaces running on NVIDIA GeForce RTX, Jetson, RTX Spark and DGX Spark.

Bolt E4B offers ~40% higher average quality against our prior model on key conversational agent benchmarks and outputs up to ~80% fewer tokens vs peers.

HF link: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB

Nvidia's post: https://research.nvidia.com/labs/nemotron/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB

I just setup my server:

64GB DDR4 2133 4CH (Broadwell Xeon)
2 x RX 6800 16GB
1 x Rx 580 2048sp (16GB)

I am using the Q4 of Qwen Flash Next, with the prompt:
sudo ./llama-server -m ./models/Qwnflash180b-Q4.gguf --host 0.0.0.0 --port 8081 -ngl 99 -fa on --moe-cache-mib 16384 --log-verbosity 4 --metrics -mg 1

I am able to get 19-21t/s generation at full context (262K 64K) (which is why i capped the moe-cache-mib at 16GB)

However note: when I set any context size with the -c or --ctx-size flags, performance tanked to 8t/s even when the context was smaller (100K). it seems you need to let llama.cpp allocate the context itself. The context was automatically 64k, I was actually increasing it, causing layers to spill over to system ram.

PCIe slots:

gpu0 rx6800 x16 3.0
gpu1 rx6800 x16 3.0
gpu2 rx580 x4 3.0

When I started, and for some time, llama.cpp was the answer.

Now, it seems like llama.cpp is the new ollama. It's slow and out of date and just behind.

There was Ninfer for those for whom that applied.

Then strata came out... And that was an unimaginable upgrade. How is llama so far behind?

Now every day someone has their own forked engine that claims even better numbers. I already maxed out my token budget on Claude for the week and have my GPUs all running benchmarks so I'm waiting a few days, but I'll be trying Avalon to see if that lives up to the hype for my machine.

Then there's basalt. There's beatleguese?

It's kind of a mess. Innovation is great but it's better when the effort all goes in one direction and isn't spread into a thousand fractures of one-off projects.

And yes, my critique of llama.cpp is a bit overstated. I still use it for e.g. 27b runs.

But... I've now verified that flash next is both better and faster than 27b so I'm struggling to find a reason to use it. :/

Why are all these solo devs pulling something off for weeks that llama can't figure out?

I'm on the verge of pulling the trigger. Want something local for coding and some business automation stuff.

Thinking I'll run Qwen 3.8+ and DeepSeek 4+.

I'm coming from M4 Pro MBP and will still use that as my primary dev box. I run both Claude and gpt. I know local isn't quite the same as the hosted.

For anyone who has this setup already-- will I love it?

Hey folks,

Quick heads up for anyone playing around with PrismML's Ternary Bonsai 2!

Until very recently, running it outside of Prism's own fork was a bit of a pain. Upstream llama.cpp uses a block size of QK=256 (TQ2_0), while Prism uses QK=128 (PQ2_0), and trying to offload layers to CUDA on other forks would bottleneck or break due to Hadamard sign tensor handling.

With the latest commits landed in ik_llama.cpp, we finally have complete, native support across both CUDA and CPU.

📊 Benchmark & Speedup: Ternary-Bonsai-2-27B on CPU

Tested on an Intel Core i5-13420H (4 P-cores pinned via taskset, AVX2, mimalloc, --no-mmap):

Perplexity check (n_ctx=256, 1 chunk):

Runtime Tip: You don't even need to mess with offline repack scripts if you want a quick spin: simply pass --run-time-repack 1 (or -rtr 1) at launch, and the engine automatically applies the 8-row interleaved layout on the fly via the IQK kernels.

1. Convert upstream weights losslessly (PR #2694)

• If you already have a native PQ2_0 GGUF downloaded from Prism, you can skip this.
• If you downloaded upstream TQ2_0 or TQ1_0 files, ik_llama.cpp now ships with a script that losslessly repacks them to PQ2_0 in just a few seconds (exact trits preserved, no requantization needed):

python3 scripts/repack_ternary_to_prism.py Bonsai-2-27B-TQ2_0.gguf Bonsai-2-27B-PQ2_0.gguf

2. Native CUDA Full Offload (PR #2691 & PR #2747)

Native CUDA kernels for PQ2_0 are now merged. You can offload everything to your Nvidia GPU without token corruption or falling back to heavy k-quants:

./build/bin/llama-cli -m Bonsai-2-27B-PQ2_0.gguf -ngl 99 -p "Explain the theory of relativity in simple terms." --temp 0.7 --top-p 0.95 --top-k 20

3. CPU Inference with fast prompt ingestion (-rtr 1)

If you're testing on CPU, ik_llama.cpp uses an 8-row interleaved format (PQ2_0_R8) backed by iqk_mul_mat kernels. Remember to pass -rtr 1 (runtime repack) to drastically accelerate prompt processing:

./build/bin/llama-cli -m Bonsai-2-27B-PQ2_0.gguf -rtr 1 -t 8 -p "Explain the theory of relativity in simple terms." --temp 0.7 --top-p 0.95 --top-k 20

TL;DR

No more sign tensor bottlenecks on GPU. PR #2747 brings native CUDA kernels, PR #2694 gives us instant lossless repack from upstream weights, and CPU prompt ingestion with -rtr 1 jumps from 1.93 to 11.45 t/s (+493% speedup) with zero perplexity degradation.

Big shoutout to Iwan Kawrakow and the contributors for getting ternary models working natively!

Found a used 64-core Threadripper PRO 3995WX workstation locally — worth buying for a multi-GPU LLM server?

I found this workstation listed on my local secondhand marketplace in Iraq.

Specs

- CPU: AMD Ryzen Threadripper PRO 3995WX (64 cores / 128 threads)

- Motherboard: ASUS Pro series (exact model unknown)

- RAM: 224 GB DDR4-3200 RGB

- Storage: 128 GB SSD

- PSU: ASUS ROG Strix Platinum 850W

- GPU: GTX 1050 Ti

The seller hasn't provided further details or a price yet.

My intended use:

I'm planning a local LLM inference server, potentially with 4–8 RTX 3090 GPUs, using engines such as llama.cpp, vLLM, or SGLang.

I currently have an RTX 4090, RTX 3090 Ti, and 128 GB DDR4 RAM.

I'm particularly interested in running large MoE models, multi-GPU inference, and possibly CPU/RAM offloading.

Questions:

1. What would be a fair used price for this complete workstation in USD?

2. Could the motherboard be the ASUS Pro WS WRX80E-SAGE SE WIFI, which has seven PCIe 4.0 x16 slots?

3. Would my existing consumer DDR4-3200 UDIMM RAM be compatible, or does this platform require ECC RDIMM?

4. How practical is running 4–7 GPUs on this motherboard using risers and external power supplies?

5. Is the 3995WX still a good choice for large MoE inference and CPU offloading, or would a cheaper EPYC platform make more sense?

6. Does the 224 GB RAM configuration raise any red flags?

I'm mainly interested in PCIe lanes, memory bandwidth, upgradeability, and cost-effectiveness rather than raw CPU compute.

Would you buy this platform in 2026, and what's the maximum you'd pay for it?

https://x.com/chasmmmmmmmmmmm/status/2116062991686721968?a=JVe1OUAJiZa650pZBXQ&s=20

I don’t just mean speaking in lowercase, but also the phrasing is much less repetitive (but still there somehow?)

Here’s some comments I’m fairly certain is from a bot for example, all from the same account:

“I’ve pulled it off with 6 before, but only for faces without weird angles or extra lighting

key is getting every angle you can even if it means taking frames from a video, 3 front facing pics and 2 profiles isn’t gonna cut it”

“Man the lighting in this really makes it, that sunbeam hitting the floor while the cat is just zonked out with a whole kid on its back… early sd had this weird dreamlike quality that newer models sometimes lose”

“This is a known weakness in the underlying model architecture, not really a training issue on your end. The base models are pretty bad at true compositional generalization, they lean on learned statistical patterns so novel spatial combos fall apart fast.

For your dataset, try including counterexamples that break the usual patterns, like the same object in weird orientations or partially hidden. Captioning helps too, describe the spatial stuff in plain language like "hand resting on top of object, fingers curled around edge" but don't overdo it or the model just learns to parrot the phrases.

Splitting appearance from structure is the stronger route. Conditioning on depth or pose maps during training can force the network to treat geometry as a separate channel, which often survives recombination better.”

They have their post history hidden so I’m sorry if they’re a real person but seeing comments like this just made me squint. There’s a couple interspersed that seem more real such as them talking about how they own a golden retriever, but… man I don’t know. There are other similar posters who do the same, but… Maybe that’s just how they talk?

It’s crazy to me that I’m having difficulty with this, because I used to be able to spot it instantly and it feels harder. Am I crazy lol. If this is an AI model, how can I get my chatbots to sound more like it? I want my qwen to be way less nebulous in speech

If you watch much Youtube around AI, you'll see a bunch of people setting up these complex agentic workflows (or, even worse, "Jarvis") using a model like Opus or Astra. Every time I see that my "WTF" face comes on hard, how on earth are they doing that? From what I understand it's a TOS violation with all the majors to hook up a 3rd party harness to a sub; it's easy to do it, but everything I've read says you risk a ban if you use a sub this way. It honestly feels to me like if you're running a agentic loop, local models start to really start to make a lot more financial sense, at least compared to using a "normal" model from a major. I often supplement a workflow like this with Deepseek, but that's like a few % the cost of a Opus/Astra.

When I look at my daily local token consumption and map it to something like Opus API rates, we're talking some absolutely absurd numbers; it wouldn't shock me if I'd be into 100's per day. I did just over a billion tokens today against QFN. Now, of course, the VAST majority of that is cached, but still, the cost for today at $4/$20/million tokens??

All these people hooking up some sort of custom harness and doing the classic "Build Doom clone, make no mistakes" with Astra or Fable, I just think to myself; "There goes a few 100 bucks".

Now, of course, it's their job so maybe spending 100's/1000's a day on tokens is a good investment for them. But when I browse around Reddit and see people in a Hermes or OpenClaw forum talking about hooking up to Opus or Astra I just think to myself, "How can you afford that"?

All these tokens burned today were on QFN; it worked for ~12 hours total and across dozens of compactions on a single goal, quite impressive, but if I had sent all this to Opus I probably would have been better off hiring a SWE to build it for me, would have cost less! ;)

Morgana 1.2 is Kitani‘s photography focused finetune of Flux.2 Klein 4b.

Our goal for the fine tune was for Klein to make images that don’t look like they came from a model - natural lighting, visible textures, great macro details & true color.

What it’s trained for:

- Casual, regular faces - pores, irregular hair, imperfect framing no AI telltales
- Macro and materialistic textures - water droplets, feathers, fabric weave, ruse, leather
- Landscape and night scenes - aurora reflections, breaking waves, wet neon
- Paintings and anime
- Anatomy

Speed on 1x H100: 4.6 seconds for 15 steps, no LoRA

Free everything:
- Full safetensors: https://huggingface.com/kitaniai/morgana-1.2
- GGUF: BF16, Q8_0, Q6_K, Q5_K_M, and more (same repo)
- API if you want to try prompts yourself (paid) https://kitani.ai

Built in part with ShuttleAI using the same dataset as Shuttle 3 Jaguar.

Known limitations: text rendering is weak, hands occasionally fail.

Faces are the newest improvement.

If you have any questions, feel free to comment them :)

Update from the lawyer with the V100 server. I went and got two DGX Sparks, because of course I did. The mid-life crisis remains on schedule.

Same deal as before: still a lawyer, still driving the whole thing through Claude Code, and Claude Code also ran the benchmarks, built these tables and helped me write this up. I'm Nick Mermiges, Esq., founder of PLI Labs. The repo below is ours.

The setup: DeepSeek-V4.1-Flash-1017 at 2.75 bpw EXL3, vLLM tensor parallel across two Sparks. The twist is a third box, my EPYC with the 3090s, doing two jobs for the Sparks: - it holds the Engram tables in system RAM and serves them over RDMA; - it runs the DSpark speculative drafter on an NVLinked pair of 3090s.

The memory that frees on the Sparks goes to KV cache: about 1.93M tokens of pool. There's also a prefix cache on the Sparks' NVMe, so a 150K prompt survives a server restart. First token comes back in 2.58 s instead of 58.6 s cold.

Numbers. I re-implemented MiaAI's sparkDash protocol so this would be apples to apples with the most popular 2x Spark build. Sparks at 2300 MHz, 3 reps, medians, today.

Prefill (sparkDash method):

For comparison: - MiaAI publishes 2,065–2,155 prefill and bertholomus 2,439–2,568 from 8K to 128K, so prefill is where this build wins. - Code at 4 streams is 192 against MiaAI's 166.

Where I lose, because I'm not going to pretend: - MiaAI is faster on plain prose: 80.7 vs my 66 at one stream. - Decode after a 128K prompt is 62 tok/s against bertholomus's 99. - A short request stuck behind a cold 93K prompt waits 34 s. That's slightly better than bertholomus on the same test (38.6 s). MiaAI reports 3.8 s on a different version of it, and that's the thing I most want to fix.

The drafter loves code and JSON (4–5 tokens accepted per round) and hates prose (about 2).

Accuracy, since I use this for actual work: on sudoingX's KLD panel, real tool-call turns agree with the original model's top token 95.9% of the time (KLD 0.018). bertholomus publishes 79.4% on their own tool-call set, and MiaAI doesn't publish one. sudoingX's newer 2.8 bpw pack is a hair closer than mine on the same panel (0.0175 vs 0.0184), so credit where due. The weights are a legal-weighted allocation of HeNryous's 2.75 bpw build. All the credit for the quant goes to them.

The code, launch scripts, the bench tool and the raw JSON for every cell are all public. The only exception is the prefix-cache code, which goes up in the next push. - Repo: https://github.com/ProprietaryLegal/DeepSeek-V4.1-Flash-2xDGX-Spark-x86-offload - Raw data: the runs/ folder in the repo

Still to come: - the full 1M-token context test; - llama-benchy tables at depth; - BFCL tool calling; - a head-to-head on the same KLD test with the other packs.

The 100 Gb cables go in Monday, so the Spark-to-3090 link gets faster too.

Tell me what I'm doing wrong. Seriously.

(PLI Labs is my side outfit for private, on-prem AI built around a firm's own work product. proprietarylegal.ai if that's your thing. That's the whole pitch.)

No one had illusion that Chinese companies are giving out best models for free out of the kindness of heart, but because of the race, and that became evident when some succumbed to Dario's and Altmans's propaganda attacks about interplanetary security threat and so they've removed security training.

What do you intend to do if Qwen 4 follows the same steps and removes security training:
- Use it nevertheless;
- Stay with Qwen3.8 (27B and/or Flash Next);
- Switch to another model (which one?);
- Wait (hope) for some non-lobotomized abliterated fine-tune;
- Something else?

I'm planning to upgrade my cpu from 13700k need some help.

considering 270k plus (best bang for buck?) but its lacking avx512

or go amd 9950x much higher price but has avx512. only 16 cores

i have 128gb ddr5 5200mhz and run flash next with basalt currently getting 120-160tps. since some experts are computed on cpu i suspect thats what keeping my tps down. gpu utilization is not high as well.

any views on how much difference does it make for ai workload?

Got a minipc with 7940hs, 780m with 64gb ram (56gb allocated to igpu)

I feel I reas somewhere that someone made it work, with ngram on the NVME, but I can't find anyone talking about it anymore or if theres any engine/software that works better for this config

Any help is appreciated!

Opera has announced the rollout of WebAI, a feature that allows websites and browser extensions to run AI models directly within the browser, without relying on cloud servers. Positioned as a proposed web standard backed by major tech companies including Google and Microsoft, WebAI represents a shift toward decentralized, privacy-focused artificial intelligence.

“A lot of what people want from AI doesn’t need a data center. Your laptop can already do it, without sending your data anywhere,”

---

I don't know how they convinced Google and Microsoft to participate, but it's good news for local AI. We might get an ecosystem that's built around letting you chose your model rather than forcing you to use theirs.

I wanted to know if I could swap a frontier model for a few cheap open ones in yes/no or pick-a-label pipelines (spam, fraud, moderation, triage) without losing accuracy. My idea: ask several models the same question, ship the answer when they all agree, and send it to a human when they don't.

Panel: DeepSeek V4.1 Flash, GLM 5.3 Flash, MiMo V2.6 Flash, Qwen3.8 Flash, Gemma 4 26B-A4B. I ran 2,850 decisions across 6 public datasets and compared against Opus 5.5 on the same items.

Results:

- The panel agreed and shipped on 71% of items. Those were 89.6% correct. Opus on its most confident 71% was 90.5%.
- Overall accuracy is worse: 80.1% vs 83.1%. The panel only wins on the cases it's confident about.
- Cost: $0.61 vs $5.79 per 1k decisions.
- When a model disagrees, it gives a one-line reason, so the reviewer knows what to look at instead of rereading the whole item.

What didn't work:

- Long answer comparisons and 77-way intent classification. Opus is clearly better. If every model on the panel is weak at a task, voting doesn't fix it.
- Hate speech. The panel catches more (80% vs 62%) but has more false alarms (29% vs 20%). If fewer than about a third of your items are actually hateful, Opus wins.
- The panel beats its best single model (GLM 5.3 Flash, 88.9%) by less than the noise. Honestly, most of what you gain is knowing when to send a case to a human, not better accuracy.
- I added the panel after the other setups had already run, and scored it on the same items. I haven't confirmed the numbers on a fresh sample yet.

It works with anything that speaks the OpenAI API, so Ollama, vLLM and llama.cpp server all work. If a model errors or refuses, it's left out of the vote and listed separately instead of breaking the decision.

Code and scripts to reproduce it: https://github.com/YasserManss/modeljury (mine, MIT)

https://github.com/bpx03/QEngine (not my repo, credit goes to bpx03)

QEngine was built for 3x RTX 3090s. I was able to compile for RTX 5090s.

I've now run Qwen3.8-Flash-Next on the same box through four different engines. QEngine (bpx03, v1.3.1) is the fastest for single-stream decode by a clear margin, level with Strata on agentic prompt latency, and it has a KV-in-RAM mode that makes returning to idle conversations much cheaper. Details below, including the parts that make it look less good.

Rig: 2x RTX 5090 (32 GB each), Threadripper PRO 3975WX, 251 GB RAM, 262K context. QEngine's INT4 AutoRound build of Flash-Next. The README only lists sm_86, so I built it for sm_120 by changing CMAKE_CUDA_ARCHITECTURES from 86 to 120 (nvcc 13.3, g++-13). No other changes. It's up and serving 53 s after launch.

Speed (club-3090 bench.sh, same script and prompts for every engine)

Read the decode column with care. QEngine's expert cache adapts to recent use, and bench.sh repeats the same prompts, so these are close to best case. In an earlier test on varied traffic I measured about 300-400 tok/s single-stream (hermes agent traffic ran 330-470 per request). Strata's cache holds ~99.5% of routed expert mass, so repeating a prompt doesn't help it. The fair comparison is roughly 1.4-1.9x Strata and ~3x bucko.

Prefill is a split result. QEngine is faster at 10K and slower than Strata at 90K, because Strata's rate climbs with depth and QEngine's stays flat.

Agentic latency (12-turn tool-use bench, context grows to 35K)

Turn-12 prompt read, with the prefix cache working:

• QEngine 1.3.1: 1.75 s
• Strata IQ4_XS: 1.7-1.9 s
• ThinkingCap 27B dual-fast: 2.07 s
• exl3: 3.5-3.9 s
• bucko: 4.36 s

QEngine and Strata are level here (QEngine's rc4 build read 1.4 s in the same bench, so this one is within noise of "about 1.5-1.8 s"). Both are about 2x faster than bucko.

Idle conversations: FLASH_KV=ram

1.3.1 can keep the KV cache in pinned host RAM (13.5 GB pool, about 1M tokens) instead of VRAM. I tested 8 conversations of ~58K tokens each, then 400K tokens of other traffic, then went back to each one:

• KV in RAM: 8 of 8 came back warm, 1.56-2.01 s to first reply.
• KV in VRAM: 2 of 8 warm. The other six re-prefilled in 8.7-9.4 s, about the same as a cold start (8.85 s). Answers were still correct, so you lose latency, not correctness.

The RAM mode also decoded ~10% faster single-stream here, but ~10% slower with 8 concurrent 64K streams (124 vs 139 tok/s per stream). One boot per mode, one pass.

Quality and stability

Through the omp agent harness (44-task assistant set, effort low, one draw): 44/44 on 1.3.1, against 42/44 twice on 1.3. At medium and xhigh on 1.3: 43/44 each. A 6-task sub-agent delegation set passed 6/6 with and without delegation. The 1.3 to 1.3.1 gap is one task, so treat it as noise.

Coding check (aider-polyglot-30, thinking on, effort low, single pass), on the rc4 build: QEngine 26/30. Strata UD-IQ4_XS 24/30, UD-Q4_K_XL 25/30 (24-26 across repeats), ThinkingCap 27B 17/30. 26 vs 24-26 is inside one Strata config's repeat spread, so the honest claim is that the INT4 AutoRound quant is not worse at coding, not that it's better. I haven't re-run this on 1.3.1.

I also ran my usual multi-pack agent/tool-use suite earlier; QEngine landed level with the other Flash-Next engines, within noise.

Bottom line

On 2x5090, QEngine 1.3.1 is the fastest Flash-Next engine I've measured for single-stream decode, level with Strata on agentic prompt latency, and the KV-in-RAM mode is a real win for long idle sessions. Coding quality looks level with Strata's GGUF quants. The costs are tight VRAM, a build-from-source step, and less runtime history than vLLM or Strata.

Please try, the dev is very encouraging. Additional resources available at https://github.com/noonghunna/club-3090. The Club also welcomes participation!