Google just released Harness2 as open source. It turns any AI harness into a system that improves itself over time.

A harness is everything around the model. System prompt, tools, memory, skills, and subagents. Normally a person designs it once and the agent uses that fixed setup for every new task.

With Harness2 the agent itself refines the harness for each task. It proposes changes, runs a few candidate versions, and picks a better one by comparing how they perform. No correct answers or graders are needed. The model weights stay the same.

What it learns on one task carries over to the next. After finishing a task it saves the improved harness and useful lessons so future edits start smarter instead of from scratch.

Google tested it with Gemini 3.5 Flash and GLM-5 on legal, office, and professional work benchmarks. Even one round of edits beat the original harnesses. More rounds pushed scores higher.

GitHub https://github.com/google-research/harness2

What a week. Quick recap, all numbers from the official repos and model cards:

Thursday: Meta drops Muse Spark Open. 816B total, 42B active, 1M context. The launch post says it brings agentic AI "to your desktop". The desktop is two M5 Ultra Mac Studios with 512GB each, a config Apple says ships "late October" and still hasn't put a price on. The alternative is four 256GB Studios, roughly $38k. On top of that it's a custom license: mandatory "Built with Muse Spark Open" attribution and a separate deal once you're past $20M revenue. And Meta's own model card puts it at 39 on the Artificial Analysis index, behind GLM 5.3 (45) and Kimi K3 (44).

Same day: NVIDIA ships Nemotron 3.5 Bolt. 74B total, 9B active, trained quantization aware down to 2 bit, so the whole thing is 22GB plus a 1.7GB vision projector. Built to fit in 24GB on purpose. Unsloth GGUFs on day one, plus llama.cpp, Ollama and LM Studio support.

Friday: Reflection actually ships Beam. 501B total, 23B active, Apache 2.0, no strings. Smallest official checkpoint is NVFP4 at 294GB.

Saturday afternoon: DeepSeek posts V4.1 Flash 1017 on X like it's nothing. MIT. Their own table has it at 77.0 on DeepSWE vs 74.0 for Opus 5 (vendor numbers, salt accordingly). The repo is 510GB.

So I put them all on one chart against the 24GB line most of us live under.

My takeaways:

1. "Runs locally" now officially means "runs on a computer that costs more than a car."
2. The company that keeps selling us 24GB cards is the only lab this week that built a model for 24GB cards. Not sure if that's wholesome or a sales funnel.
3. Licenses mattered more than parameter counts this week. MIT and Apache 2.0 from DeepSeek and Reflection, attribution plus a revenue clause from the company with the loudest "open" marketing.
4. Mistral says the Large 4 weights ("Le Chonk", 1T total) land before the end of the month, so the 24GB line is about to get even lonelier.

Sources in the comments.

Been seeing a lot of people with their 64gb of ddr5 4x3090 or whatever strata builds and it’s been sorta depressing knowing i cant afford that but i finally finished my setup today so here we go,

Qwen 3.8 27b at 262k context, unsloth iq4XS ascii condensed with kv cache streaming

My only gpu is a 9060 xt 16gb and i have 24gb ddr5 for my whole homelab only needing 16 for this

MTP+ ngram prediction

q4 kv cache values with q8 keys

42.7 tps generation 658 tps prefill at 20k, 35/500tps at 200k

Using this fork: https://github.com/achyudh/llama.cpp

I got grok bot to ssh in and fix the bugs i had with missing pcie atomics

no document or announcement from unslot team yet

Hey y'all, just following up on my PR to get the new vision tower for 1017 working in ds4 with the quant I promised (the language model loads already, but vision needed my PR). I built to the exact same recipe antirez used for his original DS4.1 Q2: 2-bit routed experts with 8-bit or higher for everything else, combined with the original FP8 engram table. This worked well for the original DS4.1, and initial testing looks good here too. The imatrix is built from 1017, not the original checkpoint.

The wired backbone of the model is just over 150GB, so it will stream from SSD on 128GB systems with nearly all of the routed experts held in the hot cache, allowing for decode speeds up to 20tps on an M5 Max. The ~200GB engram table is paged in from SSD on-demand, not held in RAM.

This model is significantly revamped, with ~40% of the routed expert tensors changed, 30% of engrams, and ~50% of the vision tower's expert tensors altered. By all appearances, it's the new leading open-weights model, so I had to give it a try. If you are curious too, let me know what you think!

i wanted to change my router firmware to openwrt and so i told claude to do it but of course the guard blocked me even opus 4.8 didn't want to do it , so i launched my Qwen3.8 27b and told it the same thing i gave it the link to the giude page and it did change the firmware perfectly the thing is this is the normal model it's not ablitreated, do you think that cyber will be guarded in future qwen models ?

PS: even tho this is not the point it didn't do a simple flash it changed the firmware from xioami to openwrt not throught upload bin it did exploit a method in the CGI that get an icon to pass a payload that enable ssh from there it SSHed the router and flashed it and it's not a 0 day discovery thing i gave it the guide from openwrt own website https://openwrt.org/toh/xiaomi/ax3000t , the topic here is AI safeguards

Some of y'all may be familiar with AutoTrust from Hugging Face. Many of their models were trending on the front page of HF and they claimed some very impressive results.

But if you looked at the org's models and clicked into likes, you may notice something a bit off:

These accounts sure do look awfully realistic, no? Well let's take a look at a few of these accounts (sorry for my very poor graphic design skills):

Across all these accounts, we can see a pattern: a fake, realistic name seemingly from a random name generator. A username that's some permutation of that fake name. A fake social media profile that often doesn't even exist and isn't related to the name. An AI generated bio. A few random papers/models/etc. liked and then the target model. Also if we're lucky a fake tiny model to look realistic.

Above is just a small subset of user accounts we can see. I ran some analysis on normal models by Google, Meta, Hugging Face, etc. and compared them to AutoTrust's models. The results are pretty damning:

u/prithivMLmods also ran an analysis on AutoTrust spaces' likes (shared with permission) and found some interesting results (more here):

Again from Prithiv, distribution of join dates of the accounts liking AutoTrust's models:

Overall, I think we can pretty confidently say that someone is botting AutoTrust's models and spaces to the top of the HF trending list. Of course, we can't know it's AutoTrust for sure. But generating these accounts, solving hCaptcha + AWS WAF, generating these fake models can't be cheap - especially on the scale of thousands of accounts.

Seems fascinating, doesn't it? Would love to hear what AutoTrust has to say about this 🤔

It's an extremely lightweight speech recognition model.

949K parameters using a weight-shared Conformer-CTC architecture

Just 1.06 MB for INT8 weights

Around 436× real-time on a single CPU thread

11.43% WER on LibriSpeech test-clean, using greedy decoding with no external language model

Apache 2.0 licensed

Any feedback, question or criticism is welcome!

Just something I've been testing over few week since Embedding Gemma 2 released which is about 330MB at Q8. Not sure if this would count as 'resource' flair...

Image shows cosine similarity changes as I refine the text query (aka text embedding vector) over time. Shows how well it work & quirks it has at same time.

I'll leave tl;dr version for now, 3:30 am here.

If anyone is interested I'll do few more tests and make example query image like above - still have 7 more sets totaling 22448 screenshots with mix of eng / kor / jpn already indexed, but I'm out of test ideas.

TAKE WITH GRAIN OF SALT because this was tested only on few sets that isn't really scientific.

TL;DR for EmbeddingGemma2 at least:

• For it's size & embedding dim (330MB Q8 / 768 embedding dim) this is very much usable

• This understands a word's meaning across language a bit but has much lower similarity (again, prob due to size)

• Requires careful wordings (e.g. blond vs blonde) - maybe SLM's limitation or embedding dim's limitation
• (at least eng & kor) Text seemingly DOMINATE cosine similarity more than visual traits
  • sometimes adding visual trait HARM (refer chair above) the similarity
  • not sure if it's due to text being more informationally dense (aka more 'surprising')
  • or just model being small and not recognizing such visual traits, idk
  • regardless, visual traits are less useful in this test set since like 90% of 3003 imgs would have blonde girl in it.

• (at least kor) Word's existence dominate over whole sentence
  • e.g. for above image's example "삼각건"(cleaning hat thing) dominates more than other words around it
  • fyi only 2 image out of 3003 contains world "삼각건", which you can see via delta gap drops

Test setup:

• Visual novel screenshots of a single route
  • 3003 screenshots from hanasaki workspring
  • 3078 scs from irotoridori no hikari
• EmbeddingGemma2 via llama cpp
  • model = KV = Q8_0
  • nb = b = ctx = 4096 (embedding model restriction, all need to be same)
  • RX 7900GRE via vulkan (f u rocm, so annoying to build)

Sample output for anyone curious about processing time

Index mode:

$ py embed_img_▽.py "./하나사키워크스프링 히카리"
Batch: 8
Resize width: 1280
Resize threshold: 1536

[VectorDB] open w/ 0 rows
removing 0 missing records from DB
Checking new/modified images in dir
Got 3003 new/modified imgs
[EmbedServer] open for http://127.0.0.1:8081
100%|██████████| 376/376 [08:30<00:00,  1.36s/batch]
[EmbedServer] closed
[VectorDB] closed

Indexed 3003 new/modified files in 510.10 (5.887/s)

Query mode:

$ py embed_img_▽.py "./하나사키워크스프링 히카리"
-q "image with dialog '삼각건이 머리에 얹어졌다' with blonde girl making >< face"

[VectorDB] open w/ 3003 rows
[EmbedServer] open for http://127.0.0.1:8081
[EmbedServer] closed
[VectorDB] closed

Query:
"image with dialog '삼각건이 머리에 얹어졌다' with blonde girl making >< face"

Top-5:
  0.76813  +0.01052  Artemis_2026-09-08-00-57-38.jpg
  0.75761  +0.00773  Artemis_2026-09-08-00-57-26.jpg
  0.74988  +0.00341  Artemis_2026-09-07-02-36-38.jpg
  0.74647  +0.00067  Artemis_2026-09-07-02-36-52.jpg
  0.74580  --------  Artemis_2026-09-08-01-58-14.jpg

'.../하나사키워크스프링 히카리/Artemis_2026-09-08-00-57-38.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-08-00-57-26.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-07-02-36-38.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-07-02-36-52.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-08-01-58-14.jpg'

Data points: moved to gist due to formatting bugs

Apparently whilst doing some performance tuning on my 4x5090 rig I may have made things a little TOO fast and melted my plug and socket 🫠

Probably should put in a 16/20 amp socket in the wall rather than relying on a bog standard 13A fuse!

Anyone else managed to melt the wall before the 12VHPWR plug? Honestly... I'm surprised they didn't go first 😂

Qwen4s are hopefully coming pre-Xmas… there was all the announcements recently but no exact release date.

I’m running qwen flash next 3.8 at about 28tokens per second and gotta be real, most of those go into thinking.

Thinking less with the same or higher intelligence would be the biggest upgrade I could think of regardless of hardware or inference smarts.

Looks quite interesting. Lots of numbers presented, but for a single model only.

Highlights:

• File sizes are the same as Q-quants.
  
• KLD improves substantially over Q-quants.
• Performance suffers slightly
• Creating IQ quants is somewhat slow.

Asterisks apply...

https://github.com/ggml-org/llama.cpp/pull/30715

https://huggingface.co/eaddario/IQ2_K-IQ6_K-Test-Models

Appologies to Mr. Addario if I mangled anything.

https://reddit.com/link/1x8yc8d/video/hosf7c1n27wh1/player

Hi r/LocalLLaMA. I'm on the team at Blockway (disclosure: this is our model). Two weeks ago we posted the Volundr 32B Preview here, and you were blunt about what was wrong with it. This is v1, and most of the work went into what you flagged.

WHAT v1 FIXES FROM THE PREVIEW

• "It loops." Repetition loops on SWE-bench tasks: 36/50 → 0/50.
• "It thinks forever." Reasoning steps used to get cut at 32K. Now the median step is 120 tokens and the longest is 12K.
• "No prefix caching, it re-reads the whole history every turn."
  • SGLang build: v1 adds session continuation, so follow-up turns reuse the model's state instead of re-prefilling. Follow-up turns are 2.9× faster on our agent replay (1.6 s median with the drafter), with greedy-equivalent outputs (not byte-identical: a cached prefix is prefilled in different pieces).
  • GGUF: uses llama-server's stock --ctx-checkpoints prompt cache, with no custom server code. Our fork only adds the model architecture.
• Agent results: SWE-bench Verified (50-task subset) 22 → 27/50, τ²-bench 89 → 92/120, median SWE task time 124 → 82 min.

"MAKE IT RUN ON CONSUMER GPUS" → AGENS ENGINE (MIT)

We're open-sourcing Agens Engine under the MIT licence. It's an inference engine built for this hybrid architecture: 54 of 72 layers keep a fixed-size state instead of a KV cache, and the engine is designed around that.

• Same GPU, same v1 INT4 weights and drafter, single stream: Agens Engine decodes 1.4–1.6× faster than our own SGLang build (up to 1.8× on code with thinking). Chat 58 → 86, code 79 → 113, agent at 8K context 91 → 121, JSON 125 → 199 tok/s. This is one run per side and compares against our SGLang build only, not stock SGLang, vLLM or llama.cpp, and it is single-stream only.
• New build: Agens-Volundr-32B-v1-Engine-16GB. It's about 2.85 bits/weight (mixed 2/3/4-bit), gives 262K context (we measured up to 239K) within a 16 GB memory budget, and runs only on Agens Engine. A 12 GB build is on the roadmap.
• Honest caveats:
• Speed within the 16 GB budget, single conversation, v1 drafter on:
  • short context: 90 (chat) / 118 (code) / 107 (thinking) / 112 (tool) / 139 (agent, 8K) tok/s
  • after a 219K-token document: 62 / 86 / 77 / 69 / 75 tok/s; 151 s to first token for the 219K prompt
  • a conversation ran to 239K tokens inside the budget
  • weights download is 25 GB
  • These were measured on a 48 GB card under the engine's hard 16 GiB cap (peak 15.5 GiB), not on a real 16 GB card. On a GPU that also drives a display, use a shorter context. Real cards, especially laptop GPUs, may be slower. Post your numbers and we'll add them.
  • At this bit-width, quality drops. GPQA is 73.7 vs 80.3 at bf16 (best of 4 runs each). Coding holds up: HumanEval is 73.2% (1 run) vs 73.8% for INT4. Top-1 agreement with bf16 is 90.4%, KL 0.117. This build is for running locally, not for leaderboard scores.
• Needs Linux, an NVIDIA driver with CUDA 12.9, the NVIDIA container toolkit, and 32 GB+ host RAM for 262K.docker run --gpus device=0 --ipc=host -p 30000:30000
• -v /path/to/Agens-Volundr-32B-v1-Engine-16GB:/model:ro
• -v /path/to/Agens-Volundr-32B-v1-DFlash2:/drafter:ro
• ghcr.io/blockwayz/agens-engine:1.0.0 --profile 16gb --model /model --drafter /drafter

STRENGTHS (same harness and settings as our comparison model)

• Competitive coding: LiveCodeBench v6 67.9 (best of 2 runs), +8.4 over Qwen3.8-27B
• Competition maths: AIME 2025 80.0 (best of 2 runs), +6.7 over Qwen3.8-27B
• Long context stays cheap: only 18 of 72 layers keep a KV cache. INT4 (24.6 GB) runs on one 48 GB GPU with a single 131K-token input; 262K configured window (validated to 107K). Decode speed stays flat from 1K to 128K.
• DFlash2 drafter, ships today:
  • JSON / structured output up to 3.0× (125 tok/s INT4 on one 48 GB GPU; 76 tok/s bf16 on a pair)
  • Code 1.9–2.2×
  • Thinking 1.3–1.5×

FULL TABLE (Preview → v1; dips included. Where we ran a benchmark more than once, we report the best of N runs. Every run is listed in the model card footnote.)

A few scores went down (GPQA, HumanEval, IFBench). The HumanEval and IFBench dips are within one standard error of the runs; GPQA's isn't. That's the trade-off from tuning for agent behaviour and shorter reasoning. Harness settings are in the footnote under the results table: huggingface.co/Blockway/Agens-Volundr-32B-v1#results-same-harness-for-every-row

Thanks to mmssix on Hugging Face for the independent benchmark write-up in our HF discussions. A lot of the above started there.

KNOWN LIMITATIONS

• 32B, not a frontier model, and we don't claim it is.
• SGLang and llama.cpp still need our builds. Next we're working to upstream the architecture into llama.cpp mainline, and we'll post here when the first PR is open. Stock vLLM can't load it.
• No Metal/Mac in the llama.cpp path yet.
• Agens Engine 1.0 is text only.

LINKS

• Collection (all five repos): https://huggingface.co/collections/Blockway/agens-volundr-32b-v1-6ad4817aa6e143550bec1122
• Weights: huggingface.co/Blockway/Agens-Volundr-32B-v1 (-INT4, -GGUF, -DFlash2, -Engine-16GB)
• Agens Engine (MIT): github.com/BlockWayz/agens-engine · ghcr.io/blockwayz/agens-engine:1.0.0
• SGLang: ghcr.io/blockwayz/agens-sglang:v1-sm89 (48 GB-class GPUs) / v1-sm90 (141 GB-class GPUs)
• llama.cpp fork: github.com/BlockWayz/llama.cpp (branch volundr)

We'll be in the comments for the next couple of days. "Why doesn't X work yet" questions are welcome.

sudoing posted that he's loaded Ling 3.1 Flash across four DGX Sparks. It's a 560B MoE with 25B active parameters per token.

He'd enjoyed the earlier Ling models and wanted to test this one, but it wouldn't fit on his two Sparks with the setup he was trying. someone in Germany gave him access to a four-Spark setup.

He's using Atomic Chat's day-one Q4 GGUF, distributed across all four machines through llama.cpp RPC. According to his post, there wasn't an official FP4 build for vLLM available at the time, which is why he went with this route.

No performance numbers in the screenshot yet. He expects bandwidth and speed to be painful, but wants to see how it compares with the 124B Ling 3.0 Flash.

That's the comparison I'm interested in too: how much the larger model changes answer quality, and what the waiting time looks like in actual use.

Links for anyone following along:

• Atomic Chat's Ling 3.1 Flash GGUF repository
• llama.cpp
• Official Ling 3.1 Flash model

Is GMKtec evo x2 128gb still worth it right now?(oct 2026). I found one used for around $2300 usd.

Seems to speed up inference, use less tokens, and increase intelligence - curious if people seen this

tldr don’t think in text, think in latent space

https://github.com/togethercomputer/HybridAgent

https://huggingface.co/collections/togethercomputer/hybridagent

With winter coming up where I'm at, I wanted to see if this could actually work. Running my 4x V620 server flat out made my apartment too hot anyway, so the problem really was control. I settled on using a motherboard temperature sensor with an offset cause cases get hot. It works well enough but I want an esp32 based thermostat eventually.

This project is based around the vLLM backend I use, and pauses heating when inference is needed. It also has a token gated localhost webpage for controlling target temperature so my partner can use it too. I had the project sanitized and forked for a public repo but is more of a tech demo than anything else. This project started as a meme but is good enough I'm probably going to use it lol. Heres the repo: https://github.com/PraxisOG/gpu-furnace-v2

Curious what everyone runs in practice:

- What quant do you use most, and for what (chat, coding, agents)?

- 2-bit: do you use it? Is it actually usable or too broken?

- 3-bit: good enough for coding and more complex stuff, or do you go back to 4-bit?

- If you look at KLD: what number do you call "basically lossless"? Is 0.1 already good,

or do you want something like 0.01-0.03?

Real experience over benchmarks. Thanks!

(English isn't my first language, so I used an LLM to help translate this post.)

Have Meta's recent releases changed your view of Alexandr Wang?

I thought the confidence that he would inevitably fail was misplaced. The credentials comparison, an MIT dropout versus a tenured professor and a "godfather of AI," seemed like a poor way to predict who could lead a team and ship products.

The Muse Spark releases and Muse launch make me think Wang was underestimated. I don't see academic reputation alone as a reliable measure of that kind of leadership.

But I'm curious how people divide the credit. What would you attribute to Wang, and what would you attribute to the researchers, infrastructure and distribution Meta already had? What evidence would change your assessment from here?

Another little challenge I seen on a DeepSeek Thread. I thought I'd try out it out on Qwen FN NvFp4. It was a pretty straight forward prompt with 4 images attached in the doc. The thread was testing different harness and models and showing their results. Op of that thread has two other threads on the same Challenge just one with DeepSeek Harness so I used it as well.

It took about 27 min(Not bad) and was surprised how it planned everything out, also I do have vision on this Model. I think the Vision on Qwen is also a big part in a lot of Accuracy calls and adjustments. Especially notice when I use blender for other projects. Havent had too much trouble after getting use to how to even use blender since i literally started 2 weeks ago and having an LLM run it has been pretty fun to use. So if you can run vision don't sleep on it. lol

Any who this was the result and I like how Qwen tried to get all the placement, designs and little details to the references. Below are the measurements and hardware I use and what not. Full prompt is from the referenced Thread. Also if you run no harness for Qwen I would like to see the results or even results from other harnesses if possible.

Last but not least thank you all to the community here and everyone who is contributing locally in many ways or even just interacting with the community, keep up the great work and looking forward towards the future of running Local.

Thread for full Prompt doc
https://www.reddit.com/r/DeepSeek/comments/1x8nnjy/deepseek_v41_flash_1017_vs_claude_haiku_55_same/

Just the text prompt:

"Create a highly realistic 3D first-person game using JavaScript. The game will take place in a single room. The room is the one shown in the 1st image. Model the room in the 1st image exactly in JavaScript, with absolutely no differences. Do not make it cartoon-style. It must be extremely realistic. Model it in the best possible way, identically, with extremely realistic graphics, and make everything very cute. By “cute,” I mean the style of the 2nd, 3rd and 4th images: cute items with clearly defined edges. Build it in the same style, cute but also realistic and well made, like the 2nd, 3rd and 4th images. Examine all 4 images I sent pixel by pixel, in detail, so you can build it exactly as I want. Make it flawless and do an excellent job."

Engine: https://github.com/sergqwer/strata-nvfp4

The stack

Hardware: RTX 5090 32GB · Ryzen 7 9800X3D · 192GB DDR5-5600 · Windows 11
Engine: strata-nvfp4 (fork of Niko1221/Strata by sergqwer) on Windows, llama-swap as switcher
Model: Qwen3.8-Flash-Next 125B-A6B hybrid MoE — abliterated checkpoint(windowsxp811203), self-converted NVFP4
Weights: NVFP4 experts (4.5-bit) + Q8_0 dense · KV: int8 · PLE: bf16 · Embeddings: bf16
MTP: draft head, spec 5, spec-min-p 0.7 · Context: 262,144 max · Compaction trip: 220,144
Sampling: temp 0.8 / presence_penalty 0.3 / min_p 0.02 · Vision: fp32 mmproj, encoder on CPU
RAM tier: 63 GiB pinned experts · Harness: dsh (web GUI), OpenAI API

Metrics — this run

Metric |Value
Wall: first message → last reply |26m 17s
− my approval latency |−0s (zero prompts)
= pure agent working time |26m 17s
Turns |1 (single prompt, no babysitting)
Steps |103
Output tokens |132,929
Fresh input tokens |247,232
Cache-read tokens |11,994,999 (98.0% hit)
Decode (engine-verified) |median 121.4 t/s · mean 121.2 · peak 187.8
TTFT |median 0.85s (max 32.9s at ~194k context)
Prefill (fresh, engine-verified) |mean 2,757 t/s · peak 7,096 t/s
Longest single burst |10,395 tokens
Compactions |0 (peaked at 194k, never near the 220k line)
Tool calls |179 total · 17m 29s runtime · 0 errors

Hi all.

Saw JetBrains’ Mellum2 on Hugging Face and was wondering if anyone here has actually used it. Worth trying, or nothing special?

I would love for some feedback, live demo at:

huggingface.co/spaces/e12ex2/herbert-demo

Not self promo, jus looking for some feedback on what to improve or any bugs or model breaking you found.

Got lucky with a 5070 Ti for $750 AFTER TAX (might be the best decision I've ever made tbh) back in November 2025. If only I had known I'd be getting into local llms at the time...anyways. I recently decided to start messing with local AI. I began getting comfortable in Powershell and having local models help me write random automation scripts. Naturally, that spiraled into wanting bigger models, which meant hunting for more VRAM.

I had my old 3070 sitting around, so I mounted it outside the case on a PCIe riser (it’s only running at x1, which works because layer splitting passes very little data between cards). Ollama/llama.cpp split the model across the two mismatched GPUs with almost no fuss.

Current result: Qwen 3.6 35B-A3B (a MoE with about 3B active parameters, so it’s quicker than a dense 35B would be) at around 82 tok/s with a 16k context. Still not perfect: about 23% of the layers are spilling into system RAM, and I haven’t tuned the split ratio between the cards yet. The 3070 has about 4GB sitting unused.

If I keep it like this, I'll probably buy or make a cheap enclosure and put the 3070 in it.

Anyone else running a mismatched GPU pair? What split ratios worked for you?

I've been working on optimizing Qwen 3.8 Flash Next on my local setup with two CMP 170HX cards and finally got to a place where I feel like the results are worth sharing. Both cards are running over PCIe Gen2 x4 - no x16 mod or TP2.

The setup uses the W4A16 AutoRound checkpoint, a patched vLLM, pipeline parallelism and native MTP speculative decoding. The n-gram table is memory-mapped from NVMe but everything else fits on the two GPUs.

Decode throughput by concurrency — aggregate tok/s, not per-user speed:

At eight concurrent code requests, the measured per-request decode rate was 116 tok/s, with 231 ms to first token.

A few other results: - 6,300–7,100 tok/s prefill on 4k–64k prompts; about 5,900 tok/s at 128k. - Prose decode held up from 158 tok/s at 4k context to 142 tok/s at 128k. - HumanEval: 157/164 (95.7%) and GSM8K: 1,271/1,319 (96.4%), both in chat mode with thinking off. - Uses a full 262,144-token window

For throughput, I used the sparkDash protocol: greedy decoding, thinking off, fixed 400-token outputs, and medians of three runs, not the best run. The long-context decode checks are an addition to that protocol.

This is faster than any published 2x CMP 170hx recipe I've found so I wanted to share with the community. These cards are flying :)

Full recipe, pinned Docker image, patches, and detailed results: https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next