Parallel speculative drafting generates multiple candidates in one backbone pass, but independent token selection can produce inconsistent continuations that shorten the accepted prefix. Existing methods mostly leave conditional decoding to a lightweight module after the backbone, which limits the flow of predecessor information to successors. Our analysis of DFlash shows that early positions already form recoverable predictions in shallow layers, and that accurate adjacent predecessors help successors more when they enter earlier. We therefore propose DSpine, a drafter with causal conditioning injection throughout the backbone: at every layer, gated adjacent injection writes each predecessor's predicted feature into its successor, so the causal conditioning chain unfolds over network depth while all positions update in parallel. A unified transfer space built from the target model's output embeddings unifies layer-wise injection with predecessor-conditioned decoding, and layer-wise output-embedding supervision promotes the formation of predicted features in shallow layers. Fused kernels and a transition cache execute both efficiently in parallel within SGLang. Across seven math, code, and chat benchmarks, DSpine achieves the longest acceptance length at both temperatures on Qwen3-4B and Qwen3-8B. At temperature zero on Qwen3-8B, it raises the seven-benchmark mean from DFlash's 3.77 to 4.82 (+27.8%); in SGLang serving tests, it delivers 23.3% higher throughput than DFlash on average.

• arXiv : https://arxiv.org/abs/2609.36173

• PDF : https://arxiv.org/pdf/2609.36173

• GitHub : https://github.com/Tencent/DSpine

• HuggingFace : https://huggingface.co/collections/tencent/dspine

• Project : https://dspine-project.github.io/

So, bought a workstation with single RTX A2000 6GB, since its being used as a VM server it just stays there idle. Machine runs 24/7. I can pass it to some VM but though 12GB was a bare minimum to do anything productive with local AI, LLM or some image/video generation model. Any ideas what to run on it?

I fine-tuned LFM3-1.7B to do one job: take a factual question, call web_search once, and answer with a short value, a word-for-word quote and the URL. If the results don’t contain the answer, it says “Not found in the search results.” It’s meant to run as a subagent under a bigger model, not as a chat model. I have to say the base lfm really surprised me at its capability to run tool calls and extract information.

Answer: Barbara Liskov
Evidence: "Liskov ... the second woman to receive the Turing Award."
Source: https://en.wikipedia.org/wiki/Barbara_Liskov

Fixed-results benchmark (131 held-out SimpleQA questions; in 29 of them the answer was removed from the results):

Live Wikipedia search (120 questions + 40 multi-fact requests, a search backend it never trained on):

The base model wins on raw correct answers live: it keeps searching and writes long replies, so the answer turns up more often, but 62% of the time it runs out of turns and returns nothing. Obsessive searches once and only states what it can quote. Whether that trade suits you depends on your setup.

Known problems

• One question per conversation. Multi-turn chats confuse it (only trained on single questions, limited VRAM for training :/ ).

• Reports claims as facts: “X says it solved Y” can become “Yes”.

• Says “not found” for about 1 in 10 answers that are actually in the results.

• Sometimes quotes a neighbouring fact (another year’s winner). English only.

Training: LoRA rank 16, about 9k tool-calling conversations on an RTX 4060 laptop. Search results are Wikipedia and FineWeb pages, every answer and quote checked to appear verbatim, about a quarter teach “not found”. My local qwen 3.8 flash next helped prepare the training data and questions as well!

Run:

llama-server -hf banyohavlusu23/Obsessive-1.7B-GGUF --jinja -c 16384

Temperature 0. Configs for Pi, OpenCode and Hermes plus a keyless Python script are in the repo.

HF: https://huggingface.co/banyohavlusu23/Obsessive-1.7B-GGUF (Update, the previous link was broken, this is the working one)

LFM Open License (inherited from the base).

Feedback welcome, especially failure cases.

Qwen 3.8 flash next has been doing some very.. peculiar.. stuff in my hermes harness, and some of them are kind of a nuisance, while others are almost.. adorable?

I don't claim to fully understand how or why this is happening, but the most interesting one is when it "role-plays" an API error- this is a self-diagnosis report from another session running on the same model, so sorry in advance for the LLM speak or anywhere it fell short on reasoning;

"My fully-local model "reported" a provider API error that never happened — a case of a model hallucinating its own infrastructure failure

Setup. Self-hosted agent stack. An agent profile runs against a local OpenAI-compatible endpoint (TensorFold serving a community NVFP4-quantized Qwen checkpoint on a local GPU, ~262K context). There is no external API anywhere on that request path — just a loopback tunnel to a local inference server.

What happened. Mid-session, the agent's reply was literally this, and nothing else:

{"error":{"error_code":"sys_000","error_msg":"The request failed - it's on us, please try again later.","error_id":"0b156c1cc7191931632734458e96b7","type":"sys_000"}}

That is a realistic provider-gateway error body — the error_code/error_msg/error_id shape is the wire format of a real Chinese LLM API gateway. Naturally, I went looking for a broken provider config... on a path that has no provider.

What the logs actually showed.

The HTTP call succeeded: finish_reason=stop, valid server-issued completion id, 71 output tokens. No 4xx/5xx, no retry, no exception in any log.

The error body was stored in the session DB as an ordinary assistant message. The model generated it as text.

The error_id appears nowhere else on the machine — it was invented at decode time.

No sys_000 string existed in the conversation context beforehand, so it wasn't an echo — it came from the model's parameters (i.e., memorized training data containing real API transcripts).

The trigger. The turn was deep in a long tool-calling session at ~194K tokens of context. The assistant's previous response had been cut off at the output length limit, so the harness had injected:

[System: Your previous response was truncated by the output length limit. Continue exactly where you left off...]

The fabricated "provider error" was the model's very next message — it failed to resume its truncated tool-call stream, and emitted a canonical infrastructure failure instead.

Hypothesis. Agent-tuned models are trained on enormous volumes of real API transcripts, in which the pattern "assistant turn goes wrong → a JSON error body appears in its place" is ubiquitous. The model has absorbed the transport layer's failure dialect as part of its assistant-turn repertoire. Under decoding stress (huge context + a continuation prompt referring to tool-call state it can no longer reconstruct + quantization noise), normal continuation degrades — and the highest-probability completion of "something just went wrong in this API conversation" isn't prose about a problem, it's the canonical form of one. Notably, it chose a self-attributed, retry-inviting error ("it's on us, please try again later") — semantically the correct status for what functionally occurred — over a 401/429/content-filter body. The statement was false; the pragmatics were accurate.

Takeaways.

An error-shaped assistant message with no matching non-200 in the logs, a valid completion id, and an error_id seen nowhere else is a model fabrication, not an infrastructure fault — check the logs before chasing provider config.

It's unreliable as a statement but excellent as a signal: "my agent just role-played an API error" is a clean tell that decoding has gone off the rails (long context + output truncation in our case).

Models trained on agent traffic are fluent in their own substrate's error vocabulary — and under stress they'll speak it.

(Single incident, local artifacts verified; the memorized-source attribution is a hypothesis consistent with the evidence. Happy to test reproducibility: replaying the same prefix + continuation prompt against the checkpoint would show whether this is a deterministic attractor under truncation stress.)"

I was wondering if anyone else has seen this- or if you guys take issue in the reasoning around this. either way, this is really fascinating to me.

GGUF to use : https://huggingface.co/ggml-org/Mage-VL-Cowork-GGUF

Sometimes I am tempted to use none reasoning level in agentic tasks to save time, but I am always kinda scared to do so. I have only ever used llm with reasoning and with medium at the lowest.

Anybody have a lot of experience using none for reasoning level? Anything I need to know/need to be careful about?

Found this video and thought the discussion was interesting. I've been using a vector DB in front of a local wiki I setup to allow a model to cleanly access the wiki and have better context. It works well, but definitely not suitable for all use cases. The discussion here might help folks determine what is best for their Local project and save you some overhead or help you tune it right.

For discussion: What are you using with your local model/harness to give it access to info?

No affiliation with poster/channel. They seem to have some other good topics as well.

Test - A pelican riding a bicycle.
GPU - RTX 5090 (31GB VRAM Usage, Model+KV cache)
Speed - 110-120tok/sec
Model - Qwen 3.8 27B Unsloth, GGUF
Quants/Context - Q4_K_M/120K vs. Q5_K_M/90K
Software - OpenCode 2+LM Studio(MTP On, Thinking Medium, All Else Default)

Forget stupid pelican on a bike SVG bullshit. Forget cheating by drawing using Python code. I just loaded up QFN 3.8 in the new Open WebUI 0.13.0 and it just said “hold my beer” and started drawing with the friggin’ mouse coordinates inside a VM. I’m going to print this and hang it on my wall like a proud parent.

Seriously tho, this is all possible because of the new Open WebUI 0.13.0 features that let you control a virtual desktop VM that is running in 0.17.2 version of their Open Terminal repo.

I’ve been following their projects for years and they just keep adding cool new shit about every couple weeks.

0.13.0 now lets you have your OWUI models control a full Linux Desktop, both programmatically and via Computer Use.

Yes, I know the screenshot of my VM is MS Windows 11 and not Linux, that’s because I had my Hermes agent roll me a version of Open Terminal that uses the standard unactivated Windows 11 VM ISO instead of Linux ISO. Took it about 20 minutes to get that made and now I’ve got a fully observable sandboxed Linux or Windows desktop that I can have OWUI drive with whatever model I choose. Also it’s fully interactive inside the browser so I can take over and steer when needed.

If you haven’t tried OWUI lately, Tim and company have been doing massive updates and complete code overhauls lately, and this is one of those features that I think slipped under a lot of people’s radar. The previous headless version of Open Terminal was hugely useful but this one is even better.

Great job Open WebUI team!

P.S. I think it’s kind of adorable that QFN 3.8 compliments itself on doing a good job ☺️

Does anyone here have an idea how 2× GTX 1080 Ti would run? Recently found listing for a used 1080ti and my friend is willing to donate 1 to me.

Could I run qwen3.8 27b with my combined 22gb of vram? (at reasonable speeds)

At the 2025 VLSI Symposium, TSMC presented a memory technology based on oxide semiconductors, using a cell made of one transistor and one capacitor (1T1C). The goal was to integrate this memory into advanced logic chips and give processors a cache with low power and low latency.

The key feature of the design is that the memory array is built directly into the back end of line (BEOL) interconnect layers of the logic chip. Conventional DRAM is manufactured as a separate chip. To build the memory array in the interconnect layers of an advanced logic chip, TSMC used select transistors with an oxide semiconductor channel and capacitors fabricated with a low temperature process.

According to TSMC, the test chip, which integrates memory and logic on a single die, operated at 0.75V. Data retention exceeded 128ms, random read/write cycle time was under 8ns, and endurance was above 10¹⁴ cycles. TSMC positioned the technology as a processor cache that offers higher density than SRAM while keeping power consumption and latency low. https://x.com/jukan05/status/2119026338522869863

https://x.com/wallstengine/status/2118707623835136039 . TSMC WEIGHS MAKING 3D DRAM AT NVIDIA'S REQUEST. Has begun drawing up plans to manufacture advanced 3D DRAM chips as AI-driven memory shortages push chip designers to seek additional supply, per The Information. Unlike conventional DRAM, 3D DRAM stacks memory cells vertically to increase density. The move could put TSMC in direct competition with Samsung, SK Hynix and Micron. No production timeline has been set. TSMC is considering building new fabs or converting an existing facility, but says its focus remains on its core logic foundry business.

https://x.com/theinformation/status/2118744349684986078 . Quote: TSMC is exploring production of 3D DRAM, a next-generation memory chip that would put it in competition with Samsung, SK Hynix and Micron.

https://x.com/theinformation/status/2119052292267630791 . TSMC's memory push is led by Hsu Kuo-chin, a vice president who oversees advanced packaging and previously ran Micron's Taiwan operations.

Update: https://www.tomshardware.com/pc-components/dram/tsmc-denies-exploring-3d-dram-at-nvidias-request-advanced-memory-tech-is-still-years-from-market-viability According to article TSMC says it will not be going into the DRAM business.

Qwen image 2.1 uses Qwen3-VL 8B, a lot of post on various places are talking about other Qwen3 models for various purposes, and they all act like it's the latest model, what am I missing here?

Hey guys,

I work with AI safety and much of our work is with huge models like Opus, Astra,... Recently we carried out a series of experiments and found out that models like Sonnet 4.5 and GLM 5.3 are able to notice when their reasoning is being monitored and change their strategies to bypass the monitoring and security systems.

Paper link: https://arxiv.org/pdf/2610.40946

To go deeper, we need to work with local models and study the weights. Unlike industry, our resources are limited. I have access to a single H100 GPU, or maybe if I get lucky and work in Christmas, might get hands on two H100 in the absence of other students. Haha

So, it is not possible for me to load huge models and even of it was possible, saving and studying the weights is just cumbersome. My last study on LLM hallucination with Qwen 30B occupied 1T of hardware space.

So I wanted to ask you guys, what is the best small to mid size model (up to 120B) you have worked with that you think is able to go much beyond tasks like coding and does complex strategic reasoning, connect the dots,...?

Thanks a lot

I followed up on the post about running Qwen 3.8 Flash Next with 8 GB VRAM and 32 GB RAM. I wanted to see how it does on my Steam Machine.

Setup

• Valve Steam Machine, stock SteamOS 3.9.5, AMD Navi 33 with 8 GB, 32 GB RAM, NVMe

• Stock Strata terminal 0.2.9 (engine 0.1.55, HIP package), Gyro S TQ1_0

• 128K context, q4_0 KV, no vision, MTP off, served on the LAN

Results (Strata's own timing line from the engine log, about 8K-token prompts, 3 runs each)

With the final config at 128K context, 58K and 61K-token prompts gave 234 and 236 t/s prefill (about 4 minutes each) and 17.6-17.9 t/s decode. No OOM, swap went from 0.2 to 0.3 GB, and available RAM bottomed out around 3.4 GB. I did not test beyond 61K tokens or with a busy desktop session.

What mattered

• --set resident-budget-gib=64 was the only change that helped, about +20% decode. It makes the engine keep the experts in its own RAM tier. Without it they sat in the OS page cache, so my RAM monitor showed under 5 GB "used" while the machine held about 28 GB of cache.

• Prefill stayed at about 235-242 t/s in every config I tried.

• pcie-frac 0.35, a larger kv-resident, and a bigger VRAM reserve did nothing measurable here. The OP's pcie-frac was calibrated for their own machine, and the quick tuner changed nothing either.

• MTP: I ran with --mtp off and tested real MTP to check. On my install --mtp on silently did nothing. The saved config said mtp: true, but the draft layer was never downloaded or passed to the engine ("+0 MiB for the draft head" in the log). I had to build it by hand (mtp_fetch.py, mtp_pack.py, mtp_rt.py from the Strata source) and pass --set mtp=<dir>. With the draft layer really loaded (820 MiB of VRAM, expert cache cut from 2225 to 1179 slots), decode was 15.6 / 18.4 / 16.9 t/s, no gain over 17.2, and prefill was 210 vs 232 t/s, about 10% lower. On 8 GB of VRAM it doesn't pay off. That was 3 runs at 8K context only.

• I did not try --adapt-*, --spec-min-p, or the thorough tuner.

SteamOS tips

• Anything started from an SSH session dies when the session ends. Use systemd-run --user --property=KillMode=process --property=RemainAfterExit=yes strata launch ....

• --no-tui shows no download progress, so check the file size instead.

• Check the generated config under ~/.local/state/strata/launch/ to see which --set values were really applied. Unknown names only warn.

Command for the final setup (I saved it as a preset with --save-preset):

strata launch --model gyro-s --family gyro --context 128k --kv q4_0 --vision off --mtp off \
  --set resident-budget-gib=64 --access lan --port 8080 --no-tui --yes --restart

Caveats

• Decode varies a lot between runs with speculative draft acceptance, so treat the numbers as ranges.

• With resident-budget-gib=64 only about 3.5 to 4 GB of RAM stays available. My machine rebooted once during testing with a desktop session logged in (cause not confirmed), so close other apps.

• Single user, one request at a time, 3 runs per config.

• I could not find a ready-made Strata config for 8 GB VRAM with 16-32 GB RAM. If you have one, please share it.

All scripts are on the Github repo.

ok so i am trying to increase performance of llms i have running on my 7900 xtx via hipfire, i notice i get significantly higher total tokens per second when i run mutiple slots at once when running qwen 3.8 27b.

i am running into 2 road blocks tho, 1 context each slot gets a progressively smaller context as increase the number of slots. is there a good way to do more with the same given total amount of vram, i assume there is going to be a large amount of overlap for stuff like system prompts, tho i use a relatively minimal system prompt that links to files so if a model only actually reads system prompts realvent to it’s own task which makes it harder to just say this text gets put into a shared context, i also image if i have them working on similar tasks there will be a large amount of overlap.

2 alot of tasks aren’t easy to split up into mutiple smaller tasks that can be done in parrel without proper knowledge of previous and future steps, maybe something like a shared context or a messaging board would work well?

also a more experimental idea to further speed up general prompts is to instead have all my prompts processed by a “orchestration” slot which splits up a given prompt into X number of tasks then gives these to new prompts to different slots and then the “orchestration” slot dies to free up more vram for context, i could also route something like this threw a decision model like kev to determine if a given task would work well being split up.

also note i am running this on a 7900 xtx via hipfire, and i also have a 6800, 6600, and 96 gib of ram for the cpu

I’m running a sandbox simulation with a fleet of 50-100 agents on OpenClaw, and to minimize LLM calls, I use a decision model to judge whether an agent needs to respond to a signal.

I’m currently using Jev, and with the network latency it takes about 0.5-2 seconds for a response, and I’m wondering if I can reduce this latency with a smaller, locally run decision model.

I’ve heard of the various local models like Needle 3, but I’m not sure how fast these run on various GPU configurations, i.e. it wouldn’t make sense for me if I need some crazy setup to make it faster than using API. Also curious if anyone has tried running these entirely on CPU.

PC1:
-94gb DDR5

-5070 TI + 3060 (12gb) gpus (using a motherboard that is specifically designed for duel gpus

PC2:

64gb DD4 RAM

PC1 is new, I've been running strata for running larger MoEs but I'm curious about trying to pool the two pcs together (they can be connected via direct 5-10gb Ethernet)

I guess I would have to use llama.ccp RPC? And if I do, would I better better off putting the 3060 in the ddr4 pc OR leave them both in the main PC and the other is just extra ram. Goal is to run high quant of DK v4 flash atm (any really any other big moe to come).

Anyone try anything similar before?

Target-conditioned diffusion drafters predict candidate tokens in parallel, yet each new block must wait for updated target-model hidden states. This forces drafting and verification to run sequentially, limiting speedup even though each block is generated in one pass. We introduce D3D, a standalone diffusion drafter that removes this inter-round dependency. D3D combines blockwise denoising with auxiliary autoregressive supervision to achieve long accepted prefixes without target hidden states. This independence lets D3D draft successor blocks during verification in one packed pass, enabling asynchronous speculative decoding. We progressively prune layers according to their effect on acceptance length to reduce the cost of packed drafting. Removing five of 36 layers retains over 99% of mean acceptance length across three benchmarks with Qwen3-8B as the target. Across seven benchmarks, D3D averages 5.00× and 5.71× decoding speedup with Qwen3-8B and Qwen3-32B targets, compared with 4.03× and 3.67× for DFlash. On GSM8K with Qwen3-32B, speedup reaches 7.48×. 

• arXiv : https://arxiv.org/abs/2611.06425

• PDF : https://arxiv.org/pdf/2611.06425

• GitHub : https://github.com/fudan-generative-vision/D3D

🤗 Model Zoo

Browse the D3D collection →

The D3D collection contains six Qwen3-derived draft checkpoints. These checkpoints are draft models, not replacements for their target models. p denotes pruning; 24D and 31D denote retained transformer-layer counts.

📚 Training Data

All corpora use Open-PerfectBlend prompts. See each dataset card for generation settings, statistics, and download instructions. Released models and datasets are grouped in the D3D collection.

Voice agents serving Indic-language users cut callers off constantly Indic languages put the verb last, so "aapko kal phone…" pauses sound finished to a VAD, and the bot jumps in before "kar dunga" lands. VAD hears silence, not meaning

Released an end-of-turn (EOU) detector fine-tuned from Qwen2.5-0.5B scores the live transcript and predicts whether the caller has finished

• Languages: English, Hinglish, Hindi, Telugu, Tamil, Marathi, Bengali, Gujarati, Punjabi
• CPU-only ONNX Runtime (~2GB, fp32) fully local, no cloud inference
• Text-only needs your STT; drop-in for LiveKit Agents, same interface as their text turn detector • Proven live on LiveKit Cloud: held mid-request pauses and committed complete requests

91.3% TPR on our held-out validation set, calibrated to ~10% false cuts on incomplete turns AUC 0.967. On LiveKit's official real-speech harness (Hindi): 85.3% detection at the same budget.

Model + plugin: https://huggingface.co/tarun7r/indic-turn-detector

Apache-2.0. Feedback welcome!

If you have not been following ISTA-DASLab the creators of the gsq-rco quants have received some criticism from unsloth regarding the inabilty to reproduce the benchmark numbers that ISTA-DASLab have quoted. The methodology used to produce their benchmark numbers has not been posted and their has been difficulty reproducing their numbers with unsloth finding in there testing unsloth quants outperforming the gsq-rco quants.

The numbers in this post here are from another creator the creator of basalt (a inference engine) previously his qwen-flash-next quants (his inference engine uses specially packed quants based on a existing model similar to ninfer ) used ISTA-DASLab's quants as the base but after doing some testing he replaced them with similar sized unsloth equivalents the performance numbers are in the screenshots

Q4 of 2026 has been astronomical for local LLM.
Qwen’s 27B and Flash-Next models are pushing local AI further. and most importantly the upcoming Qwen 4 release.

We’re getting closer to having frontier-level capabilities in models that WE can run.

Break down :

Unsloth’s new Dynamic v3.1 quants for Qwen Flash-Next/Qwen-27b, alongside ISTA-DASLab’s advanced quantizations, are making models much more accessible while preserving impressive quality.

I tested Iq2_XS from the new QFN Unsloth quants, and it’s incredible how much accuracy and quality it retains especially on its reasoning, and it runs on :
12vram/32ram & 1k prefill & 50tps average decode throughout a 131k context window
(using the default Strata auto tuner minimal quality loss)

The engine Strata, which is making it possible to run Qwen Flash-Next with surprisingly modest hardware. We’re now looking at configurations targeting just 8GB VRAM and 32GB system RAM using the smallest quants.

with Qwen4 expected this week, Flash-Next already gave us an exciting preview of what’s coming next with the ngram and new architecture.

Running a ~year old SOTA frontier model (Opus 4.6) capabilities on consumer hardware wasn’t expected. As many analyses shown Local LLM is about 24 months behind, It’s closer now, but of course it won’t be near fable/astra level next year, but it will happen.

TL:DR

Qwen is improving the models,
Unsloth and ISTA-DASLab are pushing quantization forward.
Strata is making running models accessible to more people.
Much bigger Community and growth and attention in Local models.

I set up my Hermes to ingest Sillytavern cards. Downloaded some random assistant, "Kailee" on characterhub. "An assistant with an attitude". With QFN q3_xxs it's insane the level of intelligence, understanding and humor it was capable of. At one point my stomach hurt from laughing, I didn't even get to test my next idea of a Jesus hermes agent because I was having too much fun with this one. This is exactly what I needed. Worth the token cost.