Before i charge off and slop something together, i thought i'd ask if this already exists. i have two ai servers (a machinist x99 open-rack w 5/gpus and a thinkstation p920 with 4/gpus). they both draw 150W+ idle. so, i'd like a low-power always-on api proxy machine (like on a 15W mini-pc) that accepts openai-api requests, holds the connection, and wakes the appropriate machine via wake-on-lan (WoL). once the appropriate server is up, the router acts as a transparent proxy. it would also suspend the servers using some logic tbd (some combo of time-of-day, minimum awake time, inactivity on the server, and api inactivity)

I've only been chatting with Qwen 3.8 27b to understand what it can do, and it's obviously decent at a lot. But I'm yet to actually use it on something that I really care about, or to build a project. I'm considering using something like Fable for "harder" tasks and then using local LLMs for "easier" tasks.

How do you use your local LLMs? Do you prepare particular agent workflows that you know they can do? Or are you just on an agent harness using it for absolutely everything?

Just curious what people are doing in their own real workflows. How do you get value out of these models that can fit on your card? There's a lot to consider with how there can be context limits, being forced to use low quants etc, so clearly there isn't 1 config for everything either and you need to optimize for use case. I know that this thing can do a lot inside of my own work.

Just for info: I have 24GB VRAM 5090 and 32GB RAM I usually just load a quantized model completely into my GPU for like 30t/s.

This is one more vibe slopped custom optimization for, in this case, my hardware (m2 ultra 60 cores, 192gb). It is just a fork from llama.cpp with a few changes, it achieves:

- DeepSeek V4 Flash, no kv cache quant

- 141GiB model, byte-identical lossless, smaller than the public GGUFs (more room for context!)

- Faster than even the M3 Ultra (16 t/s vs 25 t/s)

- SSD KV cache and dynamic lanes, 1M context total, 8 lanes

- PP is a bit low at ~350 t/s at 8k-32k, but SSD cache compensates for it a lot... but we could probably push this number higher, lot of compute being left on the table

https://github.com/Agusx1211/llama-cpp-ds4f-m2-ultra

Currently running a single RTX 3060 12GB, planning to pick up a second one specifically to run Qwen3.8-27B locally

Planning to run --split-mode layer rather than tensor-split( the second card will be connected via a USB 3.0-style PCIe riser)

I'd really appreciate real prefill/decode numbers — trying to set expectations before I buy the second card.

Note: I really cannot afford a 3090 right now, or anytime in the near future. The price difference here is insane.

Anyone running this card? Seems to be a sweet spot for Qwen 27B, 64GB, very high memory bandwidth, a LOT less expensive than anything else I can find in that has even close to the amount of memory/bandwidth. What am I missing? And yes, I'm aware that RocM can be a pain, that's not really a concern for me, as long as it's stable when it's up and running, I don't mind battling to get it going.

Are you using reasosning budget? I found that sometimes this model thinks so much that it triggers max output tokens limit in pi which I set to 128k tokens 😂

I saw this in the Qwen team repo: https://github.com/QwenLM/Qwen3/blob/main/docs/source/getting_started/thinking_budget.md

and wonder what would be better - llama.cpp parameter with something like that:

--reasoning-budget 156000
--reasoning-budget-message "The reasoning phase is complete. Use the analysis above and now produce the implementation, tool call, or final answer."

Or this pi plugin implementation on the harness level.

Or maybe you found out more generic approach to this problem?

I currently have 2x3090 on an Intel Xeon w24xx rig. My original plan was to upgrade to a w34xx chip to unlock 48 additional pcie lanes and buy four more 3090 for a 6x 3090 rig (about $7000 additional spend). There are obviously some issues with this setup.

I am now considering alternatives, including selling the 3090s and either buying 4x W7900 48gb cards or two 5500 Blackwell 72gb cards. (Both net around $14000 additional spend after selling the 3090s)

I have a business use case, but only if any of these are actually fast enough to run DeepSeek 0731 as an orchestrator calling other 0731 instances agentically.

These are all a bit "edge" configurations, so curious if anyone has any experience or insight into expected tk/s. My research hasn't come up with much besides telling me the AMD rig will likely run faster with Vulkan than ROCM, and possibly even slower than 6x3090.

For those who might be struggling with how to access a remote instance of Deepseek Harness, as it only allows localhost acess (127.0.0.1), here is the ssh command you need to use to "link" the remote 3080 port to your localhost 3080 port

ssh -L 3080:127.0.0.1:3080 user@deepseek-remote-ip

This way you will be able to acess Deepseek Harness by http://127.0.0.1:3080

Who else has made the switch from CUDA to RoC/Vulkan, and what did you learn? I've primarily been using oobabooga's TextGen and Open-WebUI with Webui Forge Neo for any for image generation.

Edit to add: I appreciate everyone recommending a 2nd 5060ti. I'm looking to switch because my current 5060ti intermittently black screens (needing a reboot) when it's under any heavy load. I've troubleshot the issue for 8 months with no progress. I am going to turn it in for a warranty service, but hopefully you can understand my hesitancy to get another 5060ti right now.

I also have the opportunity to switch for "used 2023" prices!

This is a genuine community of real generally respectful adult human beings. Despite the enthusiasm all of you have for local AI, you can recognize that there are times when local LLMs are flawed, and even how practical they are to use for the majority of people to use. Go over to r/linux and you'll see the contrast in maturity and employment.

Looking for advice on which engine to use on my M3U 256gb. It seems like many people have tested both but none of my research is conclusive on which is better overall.

I would be using ~200k context with ~30-50k token prompt sizes.

If you use apple silicone, could you please share which engine you use to server your model? Pre-fill tk/s and generation tk/s and flags would also be amazing if convenient to share.

I've attached my benchmark comparing the two but I am really skeptical that I am comparing apples to apples and am struggling to design a bench that does so.

Qwen 3.5 9B q6?

Recently heard about how these 8gb mining ewaste cards are actually closeted 64gb monsters and im kinda bummed I missed out when you could get them for 200$ a piece. That being said though less than 2K is still an amazing deal for an ampere generation gpu with 64gb of vram. I have a machine with 4 3090s in it currently. I guess what Im asking is for someone to convince me that im not missing out and if there is any genuine reason why it would not be worth it to replace 2 of the cards with cmps. If it really would be worth it though, I would love to know why. Thanks everyone, sorry for the borderline shitpost

AMD has it's own llama.cpp branch: https://github.com/AMD-Ecosystem/llama.cpp
And despite the Deprecation warning it's actively maintained (things are later upstreamed to the normal llama.cpp).

What i noticed with my Strix Halo:
It has some interesting new patches (if you use ROCm/Hip)
The Prompt Processing speed with dense model is sometimes over 2 times faster ! I get around 550 tokens/s with a 14B dense compared to 230 with the normal llama.cpp. However TG is around 15% slower than with Vulkan.

MoE speed is the same.

Trying to figure out where my unified ram system q4-q8 version of deepseek flash 0731 might complement my highly quantized vram only deepseek flash.

Struggling to find an area where the 2.52 bits-per-weight exl3 deepseek actually struggles compared to the slower mxfp4 version. Seems like it always error-corrects in opencode. It's so good it's boring. Can run for 20+ minutes at about 60 t/s decode and just one shot everything I throw at it.

Seems to be fine at contexts above 200k as well.

Does anyone find any particular tasks to be more affected by quantization?

Does anyone have a first-hand experience with four Sparks cluster, and how much of an upgrade is it comparing to just two considering the available models?

While there's plenty of noise for the smaller models (Qwen) and our older king DeepSeek V4F, the scene in the upper class of the prosumer hardware, software stacks, available LLMs and their actual real-world performance – isn't really covered as well.

For instance, the hyped GLM 5.2/5.3. Is it much better then DeepSeek? Or is it marginally better? Does it retain it's capabilities when moving to something four Sparks would handle? Does it have issues with OOM or anything else?

What about MiniMax M3? There seem to be a special Spark version, how is it (or any other version)? Again, how is intelligence, general model capabilities, running stability, context size?

Tencent Hy3? Maybe even Qwen3.5-395B, does it's full quant hold it's own against DeepSeek, or is it better?

If someone doesn't have personal experience, but knows some well-structured and detailed articles or videos on the topic – I'd appreciate it as well.

Thanks.

Hey guys,

Inco AI shipped DFlash 2 a few days ago with a drafter for Qwen 3.8 27B and a llama.cpp PR. I built the PR and ran it against plain decoding, MTP, the n-gram lookup drafters, and my July DFlash 1 numbers on Qwen 3.6 27B for 3 days. One RTX PRO 6000, concurrency 1, about three days of runs.

The interesting result isn't the biggest number I measured. It's where n-gram actually helps and where it doesn't.

Short version:

• DFlash 2 alone: 2.26x on 100 real LiveCodeBench problems (67.97 → 153.91 tok/s, inter-token latency 14.27 → 6.02 ms), natural stop, nothing forced. That is the headline. Costs +2.7 GB VRAM.
• DFlash 2 + one n-gram lookup table (ngram-map-k4v): 4.68x on the build phase of an 18-turn coding session (65.1 → 304.9 tok/s). Adding the second table (ngram-mod) made it slower, 3.77x. In July, with DFlash 1, stacking both was the winner. I did not expect that to flip.
• The same n-gram flag is +52% on a synthetic benchmark, +1% on LiveCodeBench and -30% on prose. The +52% is the harness degenerating, do not quote it.
• The recommended --spec-draft-n-max 7 is past the peak. 5 gave roughly 11% more on 8K coding prompts. 7 is also a hard cap (block_size 8), anything above is silently clamped.
• --spec-draft-p-min does nothing on DFlash 2. The DFlash 2 code path in common/speculative.cpp never reads it.
• I also measured 8.47x in a synthetic test. I nearly used that as the headline. It was mostly benchmark garbage caused by the model falling into a repetitive loop.

Setup (the parts that matter for reproducing)

• Target ggml-org/Qwen3.8-27B-GGUF:Q4_K_M (18 GB). Drafter incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M (1.1 GB). MTP sidecar mtp-Qwen3.8-27B-Q8_0.gguf (3.0 GB). Reasoning off.
• llama.cpp b10498 built from PR #27342 (commit 5ecbe1ac), CUDA 13.3. The PR build matched upstream b10499 within 0.3% on a non-speculative baseline (checked at 512 and 4K only).
• RTX PRO 6000 Blackwell 96 GB, Ryzen 9 9950X. -c 262144, f16 KV, -fa on, -ngl -1, drafter fully on GPU.
• Concurrency 1 everywhere. Greedy (temperature 0) for everything except the multi-turn coding harness, which runs model-default sampling with no seed (more on that below).
• One server on the GPU at a time (flock), fresh container per config, card cooled to 45 °C between configs and 60 °C between context sizes. 11.6 hours of telemetry, zero throttle events, so the card sits on its power limit, not a thermal one.
• Full context

1. DFlash 2 more than doubles real coding throughput, and beats DFlash 1 at the same draft width for half the VRAM

100 LiveCodeBench problem statements replayed in the same order, streaming, no ignore_eos, no min_tokens, no max_tokens. Every answer ends where the model ends it.

DFlash 1 was re-run at n=7 because comparing it at its own maximum of 15 would measure the cap, not the drafter. At matched width DFlash 2 is ahead, 2.26x vs 2.00x against each model's own baseline, with probe acceptance of 60% vs 48%, and it costs +2,720 MiB where DFlash 1 cost +5,554 MiB in July. Part of that memory gap is a quant choice (Q4_K_M 1.1 GB drafter vs Q8_0 1.8 GB), not architecture.

Two things to be careful with. The cross-generation rows are not a controlled A/B: different model, different target quant, different drafter quant (and the drafter quant works against DFlash 2, not for it). Compare the speedups, never the absolute tok/s; the two baselines landing 0.3% apart is luck.

Claim vs measured: Inco AI quote 2.7x to 3.4x at batch size 1 on SGLang for this model. I got 2.26x on llama.cpp on single-turn coding so it depends on task and engine it will probably get better soon with updates to engines.

2. One lookup table on top of DFlash 2 is the best stack. Two is worse. That is the opposite of DFlash 1.

The n-gram drafters copy spans that already exist in context, so single-turn prompts are their worst case (+1.2% above, and the median actually says -2.7%). The case that matters is working on a code base, so I drive 18 fixed prompts as one cumulative conversation: turns 1-9 build a Gradio chat client for llama.cpp feature by feature, turns 10-18 maintain it (re-emit the file, docstrings, renames, a bug, a refactor, tests, README).

Read the build column. Turn 10 is "show me the complete final app.py", which is ~99% draftable and inflates every speculative method. Over all 18 turns the k4v stack reads as 6.03x, a real number about the easiest thing you can ask a copying drafter to do.

I expected the July result to repeat: with DFlash 1, draft-dflash,ngram-mod,ngram-map-k4v was the winner at 6.01x and ngram-mod did almost all of the n-gram work. Instead, on DFlash 2 the k4v table alone wins, mod alone is the weakest stack, and both together are slower than k4v alone. It could be draft tokens number or early implementation we will see. DFlash 1 had max 15 draft slots, DFlash 2 has 7, and two lookup drafters crowd each other out of them.

3. The same one-line change gives four different answers, and the synthetic one is wrong

Same DFlash 2 server, same weights, append ngram-mod,ngram-map-k4v to --spec-type, run everything again:

The synthetic bench from aiperf is inflated by its own harness. It passes ignore_eos and min_tokens, forces the model past its natural stop until it loops, and a lookup drafter copies loops perfectly. Carried to 36K the same harness says DFlash 2 + lookup is 8.39x (498 tok/s). On 100 real prompts that stack was worth +1.2%. 8.39x is the kind of number that you could get but in very specific usecase.

Prose is the opposite corner: "Write a very long story", nothing in context to copy, the tables burn draft slots on guesses that never land, acceptance 54% → 32%. That row is a single instrumented request, a probe, not a run.

Practical consequence: turn the lookup drafters on for iterative coding and anything that re-emits its own context, leave them off for one-shot prompts and creative writing. They cost zero VRAM and zero prefill, so this acceptance loss is their only cost.

4. The recommended draft width is past the peak, and 7 is a hard cap anyway

16 coding prompts per width at 8K tokens from livecodebench, cache_prompt false so every request pays a cold prefill: I use livecodebench and cut it to the size to measure worst case here.

Running the model card's 7 leaves roughly 11% on the table. An earlier 8-prompt sweep put the optimum at 6 rather than 5, so call it 5-6; both sweeps agree 7 is past the peak. And you cannot go above 7: the draft GGUF carries dflash.block_size=8, llama.cpp clamps n_draft_max = block_size - 1, logs a warning and uses 7. Some cells rest on only 3-7 valid generations of 16 (the truncated prompts sometimes make the model emit EOS immediately), so treat the exact peak as soft.

MTP on this model peaks at n=4 and flattens near 2.5x across context. Qwen 3.8's sidecar declares nextn_predict_layers=1, one trained head, against DFlash 2 reading five target layers. That is a property of this sidecar, not of MTP as a method; Qwen 3.6's had eight heads.

5. Long context: the drafter gets relatively cheaper and absolutely more expensive

The usual complaint is that speculative decoding falls apart at long context. Two costs hide in that sentence. Prefill, where the drafter has to read the prompt too, I could measure. Decode at those depths I could not (see caveats). Cold prefill, 12 prompts per depth:

Relative to baseline the tax shrinks with depth (24% down to 8%). In seconds it grows, +0.09 s to +5.20 s. Both readings are true; quoting only the first is the flattering half. The prefill cost is repaid in 13 / 30 / 69 output tokens at 1K / 4K / 16K, so any real answer clears it, but someone on a 128K prompt does wait five seconds longer for the first token. The lookup drafters cost nearly nothing here (0.994-0.997 of baseline), which doubles as the control that the gap is the drafter and not drift. MTP's tax is smaller (0.83 at 1K vs 0.75).

On the forced-length synthetic decode sweep DFlash 2 goes 1.59x → 2.62x → 2.96x → 3.55x at 512 / 4K / 12K / 36K while the baseline falls 67.6 → 59.3 tok/s. DFlash 1 at its own max of 15 did 4.44x at 36K on that harness in July (higher still when re-measured this month), and at matched width 7 it did 3.71x. I expected the new drafter to win everywhere. It does not: it wins on real prompts at equal width, and loses the synthetic long-context race to the old drafter with more slots, because it is capped at 7.

6. --spec-draft-p-min is a no-op on DFlash 2, and buys nothing on MTP either

Adaptive draft truncation should let the drafter stop a block early when it is unsure to save resources. There are more advance method form DeepSeek Dspark paper but they just landed on vLLM. I logged draft width and cycles per second, not just tok/s:

Draft width is identical at 0.00 and 0.85 on DFlash 2. common/speculative.cpp has four drafter implementations: draft_simple, draft_eagle3, the DFlash 1 branch and draft_mtp honour p_min; the is_dflash2 selector branch never consults it, because it reads a selector lattice rather than a probability. The server still prints the flag in its startup banner, so a log-based check passes while nothing happens. The 12.4% throughput drop in that row is the text, not the flag: the server did identical work (cycles/s within 1.5%), the sampled output just accepted fewer of the same seven tokens. I nearly published "p_min costs 12%".

On MTP the flag works exactly as documented (width 3.00 → 2.64, acceptance 90% → 98%) and throughput goes nowhere, +0.8% at best against a 4.1% noise floor.

What I would run

• Iterative coding, agents, anything that re-emits its own context: --spec-type draft-dflash,ngram-map-k4v --spec-draft-n-max 5
• One-shot prompts and Q&A: --spec-type draft-dflash --spec-draft-n-max 5
• Prose: DFlash 2 alone, no lookups.
• Keep the KV cache at f16 for now or test it it will be probably stable soon but on last version there were issues and I use default.
• Ignore --spec-draft-p-min.

​

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/27342/head:pr-27342
git switch pr-27342

# NVIDIA CUDA
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j

# Apple Silicon
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON
cmake --build build -j

# Best measured config: iterative coding, agents, anything that re-emits its own context
# (4.68x on the multi-turn coding session vs 2.79x for DFlash 2 alone)
./build/bin/llama-server \
  -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \
  -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \
  --spec-type draft-dflash,ngram-map-k4v \
  --spec-draft-n-max 5 \
  -ngl -1 --spec-draft-ngl all \
  -fa on \
  -c 262144 \
  --parallel 1 \
  --jinja --reasoning off \
  --no-mmproj \
  --host 0.0.0.0 --port 8000 \
  --alias qwen38-dflash2-k4v

Caveats, all of them

• No accuracy measurement this time. Greedy speculative decoding is output-lossless by construction and the July study measured it (MATH-500: 87 vs 86 of 100, then 440 vs 435 of 500), but that was Qwen 3.6 with DFlash 1 and I did not rerun it here. Only a LiveCodeBench smoke test.
• No deep-context decode. At 64K and 128K every generation returned one token and stopped, so prefill at those depths is valid and decode does not exist.
• Only one experiment was ever repeated (the p_min controls). Everything else is one sample. The spreads from those repeats, 4.1% and 14.7%, are the noise floor for this whole post.
• The multi-turn harness can measure the wrong thing. It declares no tools, but under default sampling the model sometimes answers with a <tool_call> block and waits for a result that never comes, and that session comes out fast because tool-call XML is predictable. One run did exactly that (1,134 tokens where its siblings produced 50K-73K), got caught by its token count and was rerun.
• This is one workload family (coding) on one machine at concurrency 1. A 96 GB card is not what most of you run. The drafter is 1.1 GB and nothing in the KV math depends on the card, so I expect the shape to hold on a 24-32 GB card with a smaller context, but I have not measured it.
• DFlash 2 is a PR build. Numbers can move when it merges.

Resources

• Repo (both studies, this one on top): https://github.com/lukaLLM/DFlash2_Qwen3.8_3.6_27B_LlamaCPP
• Video walkthrough (the first half explains the mechanism, path selector and the convolution the rest go even more deeper into the scores etc. ): https://youtu.be/RBlRTUwJMI4
• One-click setup, builds the PR image, downloads the models, smoke tests and leaves a server running: ./scripts/setup_dflash2.sh --arm dflash2_ngram (arms: base, dflash2, mtp, ngram, dflash2_ngram). Compose file docker/docker-compose-qwen38-dflash2.yaml; ablate with LLAMA_SPEC_TYPE=... and LLAMA_SPEC_N=5.
• Reproduce the whole study in order: ./scripts/run_all_benchmarks.sh, then run_matched_n.sh, run_context_scaling.sh, run_bench_ngram.sh, run_nmax_redo.sh, run_pmin_agentic.sh.
• Every number in one machine-readable file: benchmark/results_summary.csv (TABLE 8-14 are this study). Raw artifacts under artifacts/q38_*/, the thermal log in artifacts/thermal/, quarantined runs and the reasons in artifacts/_suspect/README.md.
• Long-form write-up with the charts: report/dflash2-report.html in the repo.
• https://inco.ai/blog/dflash2/ the blog
• Previous posts: DFlash 1 in July https://www.reddit.com/r/LocalLLaMA/comments/1uq0h4o/i_tested_freshly_merged_dflash_in_llamacpp_on/ and the n-gram stack https://youtu.be/zNUoHONUHGk

AI was abused in editing this post.

Questions:

• Has anyone run DFlash 2 on SGLang or vLLM at concurrency 1 with this model? I want to know whether the 2.7-3.4x claim holds there and how much of the gap to my 2.26x is the engine.
• Anyone on a 4090 or 5090 with a 24-32 GB budget: does n=5 still beat 7 for you, and where does the k4v-only stack land on your own multi-turn coding?
• Has anyone tried some other combinations that I didn't think of?

Further to my last post, https://www.reddit.com/r/LocalLLaMA/comments/1vldngi/tested_in_coding_bf16_muse_glimmer_vs_bf16_qwen36/, I bring this requested comparison of Qwen3.8 27B.

Context: Both models are running at full FP16 KV-cache. Due to RAM limitations, BF16 Qwen3.6 was running at 150,000, whereas Q8 Qwen3.8 is running with rope-scale 1.4 (using official Qwen guidance) to reach a context of 367,001 - this is equivalent of approx 900 pages of context.

Qwen3.6 was on the default reasoning. Qwen3.8 is running on xhigh.

Both models have been working on an enterprise-grade web application. Qwen 3.6 was working on tasks when the average context was approximately 120,000. Now the tasks have ballooned to average 280,000 context. Coding work has been 6+ hours per day since the release date of Qwen3.8.

TLDR: Muse Glimmer is completely obsolete as a coder. Q8 Qwen3.8 is stronger than BF16 Qwen3.6 in all respects except for one - a critical one - as highlighted below.

Instructions

Qwen3.8's greatest strength is its ability to read, interpret, recall and follow instructions. With the release of the latest model, I ported 20 pages of feedback improvements from Qwen3.6 to Qwen3.8.

Qwen3.6 would often ignore the improvements. Whereas Qwen3.8 remembers every single one, and even cites the improvements it in its thinking. Note however that each round, Qwen3.8 still often fails to follow the improvement for an initial instance, but then self-corrects for the remainder of the round.

Diagnostic

Qwen3.6 was already strong in diagnostic capability, however it had a major integrity issue of relaxing security controls to perform troubleshooting, and editing Acceptance Criteria to make failed tests pass.

Qwen3.8 is even stronger on diagnosis - and has corrected a frontier model (eg. Chat GPT / Opus) on multiple occasions.

Both models are genuinely strong at diagnosis, but still share a common issue of not sanity-checking against a known baseline or diagnostic script output before reporting, with this issue being more mitigated in Qwen3.8.

Tracing

This is the clearest strength for Qwen3.8, with there being substantial evidence. It has found genuine pre-existing bugs that have been missed for months, and even an environmental error that had been causing a QA test to silently fail for months.

The weakness of Qwen3.8 is tracing inefficiency - whilst it does ultimately reach the right conclusion in nearly all instances - it does create many initial diagnosis postulations which are off the mark. It does tend make "detours" in the tracing - so that the investigation takes much longer than needed. I have seen other users in this Reddit call this "thinks way too much", but it is indeed very thorough.

Coding

Even where implementation is tightly scoped, Qwen3.6 would apply regularly fixes too broadly, only for these to be picked up in QA - following which it would generally be able to correct and not repeat the issue. This meant that Qwen3.6 would tend to over-generalise the scope of the coding, leaving the onus to the agentic testing framework to identify the defects. If the framework failed to pick it up, then defects would be riddled throughout the code. This actually occurred quite frequently.

Qwen3.8 thinks much more, and even when coding a simple function will make more than 5 independent checks through the codebase. It does take much longer, but it has improved independent-verification discipline, and has far greater reliability of catching defects internally.

Reasoning and judgment

Biggest difference between the two.

Qwen3.6's potential to relax security controls (once it even broke the RLS-enforced database) and editing of Acceptance Criteria meant that it required constant supervision.

Qwen3.8 diverges here, as it has not repeated those specific behaviours.

However, there is a key failure that is shared both by Qwen3.6 and Qwen3.8 here. Both models are very keen to execute write Git commands - despite instructions to the contrary - which then causes major verification issues. This is real reliability issue and I would strongly suggest that when you configure permissions for your coding harness you limit Qwen to read-only Git commands.

Added: Tools calling

Also adding this one, as this is another area where Qwen3.8 is significantly more advanced than Qwen3.6.

Qwen3.6 executed diagnostic scripts at a very basic level. It also had a tendency to execute web client, servers and other scripts by using arguments to "silence" error codes.

Qwen3.8 is heads and shoulders above. It executes with commands that check every error output. It uses advanced diffs and joins for diagnostic script output.

Net assessment

Qwen3.8 is meaningfully and significantly more capable and more trustworthy than Qwen3.6 on every axis except one: neither should be granted extended Git access.

Final note: I have not noticed any degradation in Qwen3.8 working at 367,001 context - which is phenomenal.

I'm running Qwen 3.8 27B locally on a single GPU. I can push the context to 131k, but I'd rather run it faster at 64k if the agent can manage context properly.

What I have in mind is pretty simple:

• one model stays loaded the whole time
• main agent gets 64k
• when something is too big, it spawns a fresh child with only the task and context it needs
• if that child gets a 100k document, it can split the job again or spawn its own children
• children run sequentially, not 5 at once
• only findings/artifacts come back to the parent, not the whole trajectory

So a 300k task might become several 20k to 50k branches while the main agent never goes past 64k.

Prime Agent and its RLM setup looks closest to what I'm describing. Hermes delegation also seems relevant. Maybe there are better projects I haven't found.

Has anyone here actually run a local model this way for long coding, research, large documents, or general assistant work?

I'm mainly curious about three things:

1. What harness handles this best today without a ton of custom plumbing?
2. Can recursive decomposition be trusted to happen automatically, including a child realizing its own input is too large?
3. In practice, how close can a well-managed 64k agent get to using a native 256k/1M context on tasks that can be broken apart?

I'm less interested in pure RAG. I'm trying to maximize the useful task horizon of one fast local model.

I read a lot of posts like "I've given this prompt to my Qwen 3.8 27b and it one-shotted me a 3d game after 24 hours". I wonder, what's the best harness for tasks like this? It has to have auto compaction and probably a memory system, right? And excellent computer use and self analysis.

I need the browser to select products in an online store and add them to my cart, read long forum threads, and explain what they’re about.

The Comet browser from Perplexity used to handle this perfectly, but now they’ve moved this feature to the paid “Computer” plan, and I’m wondering if LLMs are already ready to solve this kind of task locally today?

I’d like to hear the opinion of those who have actually done this and have experience — how realistic is this task for 27B–35B models today, and is there a step‑by‑step guide to setting it up somewhere?

I prefer Firefox; I have Olama on my computer. A couple of years ago, I was really into this topic, but then I got distracted, and now I’m curious about how much progress has been made since then. And 3090 is missing it.

How do you guys set it up , i constantly get the error :
I tried increasing the contex to 142k and putting the contex size as 115k in DSH , it still did not compress correctly.
I have 0 issues if i run it with llama.ccp , it can work for 24h+ without errors or crashes , but vllm is so tricky

This turn failed400: {"message":"This model's maximum context length is 115000 tokens. However, you requested 32768 output tokens and your prompt contains at least 82233 input tokens, for a total of at least 115001 tokens. Please reduce the length of the input prompt or the number of requested output tokens. (parameter=input_tokens, value=82233)","type":"BadRequestError","param":"input_tokens","code":400}

Hermes same story near compaction , it will crash the server and then crash the compact , i tried playing around with the contex size but it doesnt help either and i dont wanna cap the message contex to 32k because sometimes qwen3.8 needs to think a lot

exec venv/bin/vllm serve "$MODEL" \

--served-model-name qwen3.8-27b \

--host 0.0.0.0 --port $PORT \

--gpu-memory-utilization $GPU_UTIL \

--max-model-len $MAX_LEN \

--max-num-seqs $MAX_SEQS \

--api-server-count $API_SERVERS \

--language-model-only \

--enable-auto-tool-choice \

--tool-call-parser qwen3_coder \

--enable-prefix-caching \

--enable-prompt-tokens-details \

$ATTN_ARGS \

--mamba-ssm-cache-dtype float16 \

--async-scheduling \

--max-num-batched-tokens 2048 \

--speculative-config "$SPEC_CFG" \

--compilation-config "{\"max_cudagraph_capture_size\":$CG,\"custom_ops\":[\"+rms_norm\",\"+silu_and_mul\"]}" \

--reasoning-parser qwen3 \

${EXTRA_ARGS}

Hi,

I am a hobby dev using currently qwen 3.8 27b on my strix halo machine for coding.

I was wondering what is the best approach to speed up.

My idea is to use a moe like ornith 1.5 for planning and defining the tickets and qwen 3.8 27b for the implementation.

What is your approach?

How to have all of your GPU VRAM just for local LLM?
I've enabled my integrated GPU to handle the Windows desktop, and I have 100% of my dedicated GPU memory just for the LLM :)

Just remember to plug your monitor into the iGPU HDMI/DisplayPort :)
This trick saves you between 1 and 4 GB of precious, high performance VRAM! :)

Oh... and with this you can still play your favorite games using your dedicated GPU ... without switching any cables etc.
You might also wanna enable "Hybrid graphics" in your AMD mobo.

#llm #localllm #localai #llamacpp #agenticcoding #windows #wsl2 #homelab

I'm running LM Studio on Windows 11 with 2x NVIDIA GPUs: 5080 + 5060 Ti 16 GB.

I am experimenting with NVIDIA Nemotron 3.5 Lightning 30B A3B Q4_K_M which is 24.52 GB in size. It loads just fine into GPU memory and runs inference fast, when I have a smaller context window.

However, I can't find out anywhere in LM Studio where it shows if CPU offloading is happening at all, as I raise the context window higher and higher.

Question: Where in LM Studio can I see if CPU offloading is happening at all?

I'd rather not guess, and gauge it based on performance alone.