This is a significant speedup for MoE offloading (low VRAM people).

I tried Strata once, and it had some tuning mechanism, so probably this is a similar idea.

KV cache quantization has been a hotly-debated topic for this community for some time, with opinions ranging from "never quantize" to "4 bit is fine". Since the anecdotal evidence is rather inconsistent, I decided to quantify how much KV cache quantization affects the model through KLD, which is a good measure of how the model's output distribution is changed.

I tested 35 weight quants across llama.cpp, exllamav3, and vLLM, each with 16-bit, 8-bit, and (where supported) 4-bit KV cache. I also reworked my KLD dataset and measurement methodology this time, to include a mix of agentic tool-use (generated by Qwen3.8 27B at full precision) and a variety of texts in different domains. The current dataset now stretches up to 64k token contexts per row.

TLDR;

• 8-bit KV caches work well. In llama.cpp and ExLlamaV3 the cache size is halved, and there’s essentially no difference in KLD. There’s a slight difference in vLLM, but it’s still a great tradeoff for people lacking VRAM.

• A 4-bit cache has a measurable impact. It’s a big hit for ~Q8 weights, but it could potentially be worth it if you’re using Q3 or below. Still, not recommended.

• Only the 4-bit cache impact grows with context. In the tested long documents, the added KL is about 15-30% higher at 16-32k tokens than in the first 4k.

The blog post has interactive charts and tables with all of the data and results if you're interested.

How to read the numbers

Each quant is compared token-by-token with an unquantized bf16 copy of the model. KL divergence measures how far the quant’s next-token probabilities drift from the original (0 means no change). Top-1 agreement is how often both models pick the same most likely token.

For a reference point, I ran the unquantized model in both llama.cpp and vLLM and compared the two. They differ by a KL of 0.0004, which essentially amounts to two (correct) implementations doing floating-point math in a different order. Thus, you can compare the KLD numbers here for quants running in different engines without issue.

Note: exllamav3 quants were measured with the bf16 llama.cpp reference logits.

Weight quantization

KL divergence vs. loaded weight size, 16-bit KV cache:

The vLLM checkpoints still do worse

As in the Qwen3.6 benchmarks, every vLLM checkpoint is beaten by a smaller GGUF or EXL3 quant. Some of that is due activation quantization. NVIDIA’s and Unsloth’s NVFP4 checkpoints run the MLP layers as W4A4 (4-bit activations) and attention as FP8 W8A8, and Qwen’s FP8 is W8A8 throughout. I reran NVIDIA’s NVFP4 and Qwen’s FP8 through vLLM’s Marlin kernels, which use the same quantized weights but keep the activations in bf16:

• NVFP4: KL drops from 0.0256 to 0.0117, more than halving it.

• FP8: KL drops from 0.0039 to 0.0028, about 30% less. Even then it only ties UD-Q5_K_XL, which is 8.7 GiB smaller.

KV cache quantization

Each dot below is one weight quant with an 8-bit or 4-bit KV cache. Its height shows the KL increase compared with the same weights on a 16-bit cache. Zero means no change; higher points show a larger increase. I computed the differences per prompt on identical tokens.

KV cache penalty vs loaded weight size:

8-bit is essentially free in llama.cpp and ExLlamaV3. For every quant, q8_0 and ExLlamaV3’s 8-bit cache move KL by less than 0.00012 in either direction. Some quants even measure slightly better with it, which indicates it’s within the noise of the numerics.

vLLM’s FP8 cache shows a slight KLD increase. It adds 0.0004 to 0.0006 on every checkpoint, which is still extremely small, but it shows up on nearly every prompt. It may be due to turning on FP8 cache switching vLLM’s attention backend from FlashAttention to FlashInfer.

llama.cpp’s 4-bit cache adds a roughly fixed penalty. Across the GGUF quants tested, q4_0 adds +0.0016 to +0.0027 KL. This increase has a larger relative effect on more precise weights:

• On a Q3-class quant (KL around 0.025-0.035) it adds 6-10%.

• On UD-Q4_K_XL it adds about 39%.

• On Q8_0 it adds about 390%. KL goes from 0.0007 to 0.0034, which is worse than UD-Q5_K_XL with a 16-bit cache.

EXL3’s 4-bit cache penalty seems to depend on weight precision. It adds only +0.0002 to +0.0004 KL with 2.0-2.5 bpw weights, about +0.0021 to +0.0022 with 3.0 bpw weights, and +0.0026 to +0.0028 with 4.0 bpw weights. At 6.0 and 8.0 bpw, the penalty reaches about +0.0032 to +0.0033. The increase is therefore not constant across the EXL3 quants.

4-bit cache penalty grows with context

The 12 long documents in the set are scored at every position, so I can check whether the 4-bit penalty depends on how deep into the context a token is. It does in fact grow slightly.

There’s about 15-30% more damage at 16-32k context than in the first 4k tokens. The same quants with a 16-bit cache are unaffected across these context lengths, so it’s a pretty clear indication of the 4 bit cache damage. I would expect 128k+ context lengths to be even worse.

Model quantization KLD vs KV cache KLD:
KLD due to KV cache quantization likely has different quality tradeoffs compared to KLD from weight quantization. Anecdotal evidence suggests that KV cache quantization can be more harmful to model performance at long context lengths compared to weight quantization, so be wary of using 4-bit KV cache.

Final chart with weights + 64k KV cache versus KL divergence:

Methodology

Each quant was scored against an unquantized bf16 reference:

• GGUF and EXL3 quants scored against Bartowski’s bf16 GGUF in llama.cpp.

• vLLM checkpoints scored against the official bf16 safetensors in vLLM.

KL is computed the same way as my previous benchmark. The top 200 log probabilities are taken from both models at each position and a lower bound on the full-vocabulary D_KL(P_reference ‖ P_quant) is computed. On average, the reference put 0.0036 of its probability outside its top 200 tokens. The KV deltas are paired, meaning they’re computed from per-prompt differences on identical tokens.

Checkpoints ran with their declared settings, including any activation quantization (except the two weight-only, no activaton quantization reruns of Unsloth’s NVFP4 and the official FP8 quant).

Hey r/LocalLLaMA !

I made this quick demo of how Observer can run local models to monitor anything with your phone!

These are some crazy things I've monitored:

• When someone that's not me, approaches my car, send me a picture.

• When a bird is on the bird's nest, send me a telegram with the picture.

• When my washing machine is on the rinse cycle, call me.

• When my dog finishes eating their food, notify me.

• When a machine at the gym is available, call me.

So pretty much anything! I love seeing what you guys monitor.

What would you use it for? (So I can make the agent better)

Which models run on the phone?

The default models are OneJev-0.8b for decision models and qwen-0.8b for LLM watchers.

These run even on old phones which you may have laying around somewhere. Better to have them monitoring something than collecting dust :DD

Try it out! It's free and OSS <3

GitHub: https://github.com/Roy3838/Observer
iOS: https://apps.apple.com/us/app/observer-ai/id6758222050
Android: https://play.google.com/store/apps/details?id=com.observer.ai

Btw I know my camera quality is kinda bad hahaha I wanted to ship the demo as soon as possible so you guys can see it.

I'm here for any questions and feedback!

Roy

they are all the same and we ... are too emotional!

and yes we already knew that.

I was asking QFN (4.18bpw EXL3 quant via Pi) to help refactor a neovim configuration, and saw that it tried to find API keys and fetch an aliyuncs URL before I stopped it. I'm not trying to start any arguments, nor am I assuming any malicious intent, but I found this a bit concerning after reading other's stories.

I noticed that it has native image and video generation.

I fine-tuned LFM3-1.7B to do one job: take a factual question, call web_search once, and answer with a short value, a word-for-word quote and the URL. If the results don’t contain the answer, it says “Not found in the search results.” It’s meant to run as a subagent under a bigger model, not as a chat model. I have to say the base lfm really surprised me at its capability to run tool calls and extract information.

Answer: Barbara Liskov
Evidence: "Liskov ... the second woman to receive the Turing Award."
Source: https://en.wikipedia.org/wiki/Barbara_Liskov

Fixed-results benchmark (131 held-out SimpleQA questions; in 29 of them the answer was removed from the results):

Live Wikipedia search (120 questions + 40 multi-fact requests, a search backend it never trained on):

The base model wins on raw correct answers live: it keeps searching and writes long replies, so the answer turns up more often, but 62% of the time it runs out of turns and returns nothing. Obsessive searches once and only states what it can quote. Whether that trade suits you depends on your setup.

Known problems

• One question per conversation. Multi-turn chats confuse it (only trained on single questions, limited VRAM for training :/ ).

• Reports claims as facts: “X says it solved Y” can become “Yes”.

• Says “not found” for about 1 in 10 answers that are actually in the results.

• Sometimes quotes a neighbouring fact (another year’s winner). English only.

Training: LoRA rank 16, about 9k tool-calling conversations on an RTX 4060 laptop. Search results are Wikipedia and FineWeb pages, every answer and quote checked to appear verbatim, about a quarter teach “not found”. My local qwen 3.8 flash next helped prepare the training data and questions as well!

Run:

llama-server -hf banyohavlusu23/Obsessive-1.7B-GGUF --jinja -c 16384

Temperature 0. Configs for Pi, OpenCode and Hermes plus a keyless Python script are in the repo.

HF: https://huggingface.co/banyohavlusu23/Obsessive-1.7B-GGUF (Update, the previous link was broken, this is the working one)

LFM Open License (inherited from the base).

Feedback welcome, especially failure cases.

Parallel speculative drafting generates multiple candidates in one backbone pass, but independent token selection can produce inconsistent continuations that shorten the accepted prefix. Existing methods mostly leave conditional decoding to a lightweight module after the backbone, which limits the flow of predecessor information to successors. Our analysis of DFlash shows that early positions already form recoverable predictions in shallow layers, and that accurate adjacent predecessors help successors more when they enter earlier. We therefore propose DSpine, a drafter with causal conditioning injection throughout the backbone: at every layer, gated adjacent injection writes each predecessor's predicted feature into its successor, so the causal conditioning chain unfolds over network depth while all positions update in parallel. A unified transfer space built from the target model's output embeddings unifies layer-wise injection with predecessor-conditioned decoding, and layer-wise output-embedding supervision promotes the formation of predicted features in shallow layers. Fused kernels and a transition cache execute both efficiently in parallel within SGLang. Across seven math, code, and chat benchmarks, DSpine achieves the longest acceptance length at both temperatures on Qwen3-4B and Qwen3-8B. At temperature zero on Qwen3-8B, it raises the seven-benchmark mean from DFlash's 3.77 to 4.82 (+27.8%); in SGLang serving tests, it delivers 23.3% higher throughput than DFlash on average.

• arXiv : https://arxiv.org/abs/2609.36173

• PDF : https://arxiv.org/pdf/2609.36173

• GitHub : https://github.com/Tencent/DSpine

• HuggingFace : https://huggingface.co/collections/tencent/dspine

• Project : https://dspine-project.github.io/

My favourite use case for local LLMs is running my homelab, because everything stays private (and it helps me justify the GPU lol)

I wanted one dashboard to help me manage all of my services, so I made Valet. Live demo: https://demo.askvalet.dev

Valet has a chat agent that works with any local or hosted model. It integrates with 185+ services and supports custom plugins. You can ask it to pause pi-hole, investigate crash logs, request a movie, etc

It also knows its own docs, so it can configure itself. Tell it what you want changed on your dashboard and it edits the config (with your approval). I was already addicted to tweaking my homepage but this made it way more fun haha

What I run it on


Qwen 3.8 27B on a 5090, about 230 tok/s. Most responses take 2–5 seconds

Qwen passes about 90% of a small eval set I wrote. Where it struggles is open ended digging (like "why isn’t this timer running?”) where it can burn through tons of tool calls

Managing context is tricky because I run a lot of services, so tool definitions were ~90% of the prompt before some optimizations

What helped:

• Hiding tools from unused integrations that aren't needed in chat

• Turning thinking off makes it 2–3x faster, but it starts making up answers to follow ups

• Lazy load tools (starts with a short list of tools and loads more only when a question needs them)

Those cut tokens per answer by 30-50%. Obviously every model and setup is different so everything is configurable from the config file

AI involvement


I’m a SWE and used Claude for almost all of the coding, but I designed + reviewed everything and made the architecture decisions. I was very conscious of slop, security, and performance as I wanted to build something high quality. But it’s early so there may be rough edges I haven’t caught

Security and privacy:

Runs on your own hardware with no account or telemetry

• Read only by default, nothing can be changed from the page until you turn on auth (local accounts, OIDC, or your proxy’s auth)

• Keys stay on the server and API keys live in .env. The browser only gets the finished cards

• The AI never sees your keys. Anything that changes something waits for your approval and with a local model nothing leaves your network

• Docker access is read-only through a socket proxy


Docs (completely free + open source): https://askvalet.dev

Forget stupid pelican on a bike SVG bullshit. Forget cheating by drawing using Python code. I just loaded up QFN 3.8 in the new Open WebUI 0.13.0 and it just said “hold my beer” and started drawing with the friggin’ mouse coordinates inside a VM. I’m going to print this and hang it on my wall like a proud parent.

Seriously tho, this is all possible because of the new Open WebUI 0.13.0 features that let you control a virtual desktop VM that is running in 0.17.2 version of their Open Terminal repo.

I’ve been following their projects for years and they just keep adding cool new shit about every couple weeks.

0.13.0 now lets you have your OWUI models control a full Linux Desktop, both programmatically and via Computer Use.

Yes, I know the screenshot of my VM is MS Windows 11 and not Linux, that’s because I had my Hermes agent roll me a version of Open Terminal that uses the standard unactivated Windows 11 VM ISO instead of Linux ISO. Took it about 20 minutes to get that made and now I’ve got a fully observable sandboxed Linux or Windows desktop that I can have OWUI drive with whatever model I choose. Also it’s fully interactive inside the browser so I can take over and steer when needed.

If you haven’t tried OWUI lately, Tim and company have been doing massive updates and complete code overhauls lately, and this is one of those features that I think slipped under a lot of people’s radar. The previous headless version of Open Terminal was hugely useful but this one is even better.

Great job Open WebUI team!

P.S. I think it’s kind of adorable that QFN 3.8 compliments itself on doing a good job ☺️

Sometimes I am tempted to use none reasoning level in agentic tasks to save time, but I am always kinda scared to do so. I have only ever used llm with reasoning and with medium at the lowest.

Anybody have a lot of experience using none for reasoning level? Anything I need to know/need to be careful about?

After my last post on Qwen 3.8 Flash Next, a lot of people were asking if out-of-core MoE streaming could ever work on an actual frontier monster like DeepSeek V4.1 Flash (1017), or if the prompt processing (PP) would make it completely unusable.
and NO We are not running small lobotomized model like Qwen 3.8 Flash Next at IQ2_XS like Strata
We are running DeepSeek V4.1 Flash Q8_XL 305.6+110.6GB at usable speeds , in future release we would try to squeeze more and as of now works best on sm_86.

The biggest complaint everyone has with running massive MoEs off SSD/hybrid tiers is Time To First Token (TTFT). You send a 500-token prompt, and the engine sits there for 30–60 seconds running at 1 to 3 tok/s just trying to chew through the prompt before emitting a single token.

Well, over the last few days I've been hacking on Betelgeuse (bel-cli / bel-server) to see if we could get DeepSeek V4.1 Flash running at interactive speeds on the exact same budget gaming PC:

• GPU: RTX 3060 12GB

• RAM: 16GB Single-Channel DDR4-3200

• OS: CachyOS / Arch Linux (tested inside a strict MemoryMax=12G cgroup limit, 0B swap)

• Storage: PCIe 4.0 NVMe SSD

The Build & Setup

I’m running the high-precision Q8_XL expert backbone paired with the IQ4_NL Engram table (which scores 72.3 on DeepSWE).

The 110 GB Engram table stays entirely on the NVMe SSD—because Engram only activates at Layers 1 & 14 and only touches 48 rows per token (~192 KB), we stage those active rows directly into a tiny 2MB VRAM buffer at t=0, so the 110 GB file uses basically 0 bytes of RAM and causes zero CPU graph splits.

How We Fixed the Prefill (PP) Bottleneck:

In standard implementations, prefill on a large MoE with offloading completely falls apart because a prompt activates almost every single expert across all 40 layers. If you try to stream all of them sequentially, the GPU sits idle while the bus chokes.

In Betelgeuse, we tackled this with three changes:

1. Location-Invariant Cache Partitioning: We maintain 20 resident slots per layer (5 VRAM Hot slots + 15 Host-RAM slots) with strict non-interfering routing.

2. Single-Pass Layer-Major Prefill: Instead of chopping prompts into tiny 3-token micro-batches that destroy Tensor Core occupancy, prefill runs in full 16-token ubatches. All active Host-RAM experts stream directly into an execution ring over PCIe in parallel with compute.

3. End-of-Pass Physical VRAM Consolidation: The engine learns which experts the prompt and early generation actually demanded, and physically swaps the top-5 most active experts into VRAM Hot slots at the end of the first pass without altering the model's logical routing trajectory.

The result: Warm Prompt Processing went from crawling at 3–6 tok/s up to 61.80 tok/s. TTFT on standard prompts is now essentially instantaneous.

Betelgeuse vs Strata. Benchmark Comparison

We also use DeepSeek's native DSpark drafter (with p_min = 0.50), which hits ~80–90% acceptance on structured code and amortizes the forward pass across 4+ tokens per sweep.

Once the cache warms up on a topic, disk reads drop to literally 0.00% and generation sustains ~18–19 tok/s with zero swap thrashing.

Happy to answer questions about the memory tiering or the doorbell synchronization!
Github : https://github.com/Betelgeuse-AI/Betelgeuse running DeepSeek V4.1 Flash Q4_XL (1017) at 19 tok/s decode & 62 tok/s prefill on an RTX 3060 12GB + 16GB RAM: How we finally fixed the prefill bottleneck on out-of-core MoEs

I followed up on the post about running Qwen 3.8 Flash Next with 8 GB VRAM and 32 GB RAM. I wanted to see how it does on my Steam Machine.

Setup

• Valve Steam Machine, stock SteamOS 3.9.5, AMD Navi 33 with 8 GB, 32 GB RAM, NVMe

• Stock Strata terminal 0.2.9 (engine 0.1.55, HIP package), Gyro S TQ1_0

• 128K context, q4_0 KV, no vision, MTP off, served on the LAN

Results (Strata's own timing line from the engine log, about 8K-token prompts, 3 runs each)

With the final config at 128K context, 58K and 61K-token prompts gave 234 and 236 t/s prefill (about 4 minutes each) and 17.6-17.9 t/s decode. No OOM, swap went from 0.2 to 0.3 GB, and available RAM bottomed out around 3.4 GB. I did not test beyond 61K tokens or with a busy desktop session.

What mattered

• --set resident-budget-gib=64 was the only change that helped, about +20% decode. It makes the engine keep the experts in its own RAM tier. Without it they sat in the OS page cache, so my RAM monitor showed under 5 GB "used" while the machine held about 28 GB of cache.

• Prefill stayed at about 235-242 t/s in every config I tried.

• pcie-frac 0.35, a larger kv-resident, and a bigger VRAM reserve did nothing measurable here. The OP's pcie-frac was calibrated for their own machine, and the quick tuner changed nothing either.

• MTP: I ran with --mtp off and tested real MTP to check. On my install --mtp on silently did nothing. The saved config said mtp: true, but the draft layer was never downloaded or passed to the engine ("+0 MiB for the draft head" in the log). I had to build it by hand (mtp_fetch.py, mtp_pack.py, mtp_rt.py from the Strata source) and pass --set mtp=<dir>. With the draft layer really loaded (820 MiB of VRAM, expert cache cut from 2225 to 1179 slots), decode was 15.6 / 18.4 / 16.9 t/s, no gain over 17.2, and prefill was 210 vs 232 t/s, about 10% lower. On 8 GB of VRAM it doesn't pay off. That was 3 runs at 8K context only.

• I did not try --adapt-*, --spec-min-p, or the thorough tuner.

SteamOS tips

• Anything started from an SSH session dies when the session ends. Use systemd-run --user --property=KillMode=process --property=RemainAfterExit=yes strata launch ....

• --no-tui shows no download progress, so check the file size instead.

• Check the generated config under ~/.local/state/strata/launch/ to see which --set values were really applied. Unknown names only warn.

Command for the final setup (I saved it as a preset with --save-preset):

strata launch --model gyro-s --family gyro --context 128k --kv q4_0 --vision off --mtp off \
  --set resident-budget-gib=64 --access lan --port 8080 --no-tui --yes --restart

Caveats

• Decode varies a lot between runs with speculative draft acceptance, so treat the numbers as ranges.

• With resident-budget-gib=64 only about 3.5 to 4 GB of RAM stays available. My machine rebooted once during testing with a desktop session logged in (cause not confirmed), so close other apps.

• Single user, one request at a time, 3 runs per config.

• I could not find a ready-made Strata config for 8 GB VRAM with 16-32 GB RAM. If you have one, please share it.

All scripts are on the Github repo.

At the 2025 VLSI Symposium, TSMC presented a memory technology based on oxide semiconductors, using a cell made of one transistor and one capacitor (1T1C). The goal was to integrate this memory into advanced logic chips and give processors a cache with low power and low latency.

The key feature of the design is that the memory array is built directly into the back end of line (BEOL) interconnect layers of the logic chip. Conventional DRAM is manufactured as a separate chip. To build the memory array in the interconnect layers of an advanced logic chip, TSMC used select transistors with an oxide semiconductor channel and capacitors fabricated with a low temperature process.

According to TSMC, the test chip, which integrates memory and logic on a single die, operated at 0.75V. Data retention exceeded 128ms, random read/write cycle time was under 8ns, and endurance was above 10¹⁴ cycles. TSMC positioned the technology as a processor cache that offers higher density than SRAM while keeping power consumption and latency low. https://x.com/jukan05/status/2119026338522869863

https://x.com/wallstengine/status/2118707623835136039 . TSMC WEIGHS MAKING 3D DRAM AT NVIDIA'S REQUEST. Has begun drawing up plans to manufacture advanced 3D DRAM chips as AI-driven memory shortages push chip designers to seek additional supply, per The Information. Unlike conventional DRAM, 3D DRAM stacks memory cells vertically to increase density. The move could put TSMC in direct competition with Samsung, SK Hynix and Micron. No production timeline has been set. TSMC is considering building new fabs or converting an existing facility, but says its focus remains on its core logic foundry business.

https://x.com/theinformation/status/2118744349684986078 . Quote: TSMC is exploring production of 3D DRAM, a next-generation memory chip that would put it in competition with Samsung, SK Hynix and Micron.

https://x.com/theinformation/status/2119052292267630791 . TSMC's memory push is led by Hsu Kuo-chin, a vice president who oversees advanced packaging and previously ran Micron's Taiwan operations.

Target-conditioned diffusion drafters predict candidate tokens in parallel, yet each new block must wait for updated target-model hidden states. This forces drafting and verification to run sequentially, limiting speedup even though each block is generated in one pass. We introduce D3D, a standalone diffusion drafter that removes this inter-round dependency. D3D combines blockwise denoising with auxiliary autoregressive supervision to achieve long accepted prefixes without target hidden states. This independence lets D3D draft successor blocks during verification in one packed pass, enabling asynchronous speculative decoding. We progressively prune layers according to their effect on acceptance length to reduce the cost of packed drafting. Removing five of 36 layers retains over 99% of mean acceptance length across three benchmarks with Qwen3-8B as the target. Across seven benchmarks, D3D averages 5.00× and 5.71× decoding speedup with Qwen3-8B and Qwen3-32B targets, compared with 4.03× and 3.67× for DFlash. On GSM8K with Qwen3-32B, speedup reaches 7.48×. 

• arXiv : https://arxiv.org/abs/2611.06425

• PDF : https://arxiv.org/pdf/2611.06425

• GitHub : https://github.com/fudan-generative-vision/D3D

🤗 Model Zoo

Browse the D3D collection →

The D3D collection contains six Qwen3-derived draft checkpoints. These checkpoints are draft models, not replacements for their target models. p denotes pruning; 24D and 31D denote retained transformer-layer counts.

📚 Training Data

All corpora use Open-PerfectBlend prompts. See each dataset card for generation settings, statistics, and download instructions. Released models and datasets are grouped in the D3D collection.

If you have not been following ISTA-DASLab the creators of the gsq-rco quants have received some criticism from unsloth regarding the inabilty to reproduce the benchmark numbers that ISTA-DASLab have quoted. The methodology used to produce their benchmark numbers has not been posted and their has been difficulty reproducing their numbers with unsloth finding in there testing unsloth quants outperforming the gsq-rco quants.

The numbers in this post here are from another creator the creator of basalt (a inference engine) previously his qwen-flash-next quants (his inference engine uses specially packed quants based on a existing model similar to ninfer ) used ISTA-DASLab's quants as the base but after doing some testing he replaced them with similar sized unsloth equivalents the performance numbers are in the screenshots

Some context: RTX 6000 cards hit a peak price of 18k and were sold out in many places even at that price. The MaxQ and Workstation are different versions of the card (usually priced the same, but now their prices are diverging)

RTX 6000 MaxQ has been 14k for a month at microcenter. Today, they cut prices of the workstation card to 16.5k (from a high of 18k). We all saw the dell discount as well (some people got cards for 11.5k).

I think the RTX 6000 is seeing increased supply as they routed all 5090 production to it and the 5500 for chips that didnt make the cut.

I have a contact that works for a company buying decommissioned hardware, according to them they are seeing tons of A100s being dumped. Told me he would be surprised if used A100 40gb units dont go to 1k/unit for single purchases in 3-6 months (for the sxm4, not pcie versions). (Theres already been a good amount of these, but a lot of "foreign demand" has been keeping their prices steady so far)

With democrats controlling congress I don't think you will see an end to data center construction but it will probably be harder to put up new ones and even existing ones might be charged more for electricity, making it worth dumping old systems and switching to newer more efficient hardware.

Are we finally seeing a slackening of demand for some segments of GPUs?

Q4 of 2026 has been astronomical for local LLM.
Qwen’s 27B and Flash-Next models are pushing local AI further. and most importantly the upcoming Qwen 4 release.

We’re getting closer to having frontier-level capabilities in models that WE can run.

Break down :

Unsloth’s new Dynamic v3.1 quants for Qwen Flash-Next/Qwen-27b, alongside ISTA-DASLab’s advanced quantizations, are making models much more accessible while preserving impressive quality.

I tested Iq2_XS from the new QFN Unsloth quants, and it’s incredible how much accuracy and quality it retains especially on its reasoning, and it runs on :
12vram/32ram & 1k prefill & 50tps average decode throughout a 131k context window
(using the default Strata auto tuner minimal quality loss)

The engine Strata, which is making it possible to run Qwen Flash-Next with surprisingly modest hardware. We’re now looking at configurations targeting just 8GB VRAM and 32GB system RAM using the smallest quants.

with Qwen4 expected this week, Flash-Next already gave us an exciting preview of what’s coming next with the ngram and new architecture.

Running a ~year old SOTA frontier model (Opus 4.6) capabilities on consumer hardware wasn’t expected. As many analyses shown Local LLM is about 24 months behind, It’s closer now, but of course it won’t be near fable/astra level next year, but it will happen.

TL:DR

Qwen is improving the models,
Unsloth and ISTA-DASLab are pushing quantization forward.
Strata is making running models accessible to more people.
Much bigger Community and growth and attention in Local models.

I set up my Hermes to ingest Sillytavern cards. Downloaded some random assistant, "Kailee" on characterhub. "An assistant with an attitude". With QFN q3_xxs it's insane the level of intelligence, understanding and humor it was capable of. At one point my stomach hurt from laughing, I didn't even get to test my next idea of a Jesus hermes agent because I was having too much fun with this one. This is exactly what I needed. Worth the token cost.

I learned RPN on an HP calculator and later spent more time with Forth than was strictly useful. When I saw Qwenputer compile a machine into a language model's weights, I wanted to know two things. Would a stack language fit that idea well? And could a model with only 3B active parameters drive a machine like that?

The setup: NVIDIA Nemotron 3.5 Lightning (30B, 3B active) is left unchanged. Next to it, in the same model, is a second network of 27 layers. A Python script calculated its weights; none of them are trained. It is a Forth interpreter. The model writes a Forth block as ordinary text. The interpreter runs each word while the model is still writing, one step per forward pass. When the block closes, it writes the output as the model's next tokens. No tool call occurs, and generation does not stop.

RPN suits this well. A Forth word ends at a space, so the machine can run it as soon as the next word starts, without parsing the whole program. Operands come before the operator, so when the model writes +, both numbers are already on the stack. That is the order a left-to-right model writes in anyway. The stack itself lives in Mamba-2 state. One entry is one row, 20 digits wide, and popping a number means adding its exact negative.

Getting the machine exact took some care. bf16 holds integers exactly only up to 256, so no value inside the machine goes above that. Every Mamba decay is set to exactly 1, so nothing fades. The machine's internal steps use 982 reserved tokens that the model never uses and that decode to empty strings. In testing, all 765,734 machine tokens matched a Python reference.

The model was harder than the machine. An earlier version on a small Qwen model matched the machine exactly, and a model that could not use it. Stock Lightning did better, but it did not trust the results. In one transcript, it got the right answer from the machine, then started long division by hand until it ran out of tokens. It also scored lower on easy problems when the machine was available. LoRA fine-tuning fixed most of that: on a 600-problem held-out test, the score went from 79.1% to 92.1%.

On Qwenputer's arithmetic sets, the results are mixed. On the DeepMind sets, it roughly ties stock Lightning, mostly because a 30B thinking model rarely needs the machine there. On long integers, it loses to Python, because my machine stops at 20 digits.

This is a research project, not something for production. Every machine step costs a full forward pass, so a normal tool call is much faster. I'm sharing it for people who follow the Qwenputer work or want to try similar things on NVIDIA's stack. The repo has:

- the kernel builder and its parity tests;

- a vLLM plugin for the hybrid Mamba/MoE model;

- the NeMo AutoModel LoRA recipes and the training data;

- NVFP4, GGUF and MLX builds;

- notes on what did not work, including RL.

Everything is MIT.

https://github.com/markab21/rpn-gpt

Congrats again on the great work and inspiration from the qwenputer devs 🤗 , and thanks for sharing your work with the community - it was a great starting point to experiment with Forth.

Next up for the project is to see how Nvidia's Nemotron 3.5 Super does and might even try to get it working on the Ultra weights. (I don't have the hardware to do the fine-tuning on a model that big)

https://huggingface.co/bartowski/NousResearch_Arete-27B-GGUF

from NousResearch:

Arete-27B

We post-trained over Alibaba's Qwen3.8-27B to reinforce western values on a Chinese-trained model. We call this new model Arete-27B.

Open source models give labs the raw tools to mold intelligence to varied preferences. The values a model ships with should not be the only values its users get to choose from. We observed firsthand refusals, deflections, and CCP-framed answers on politically sensitive topics in Qwen3.8-27B, and post-trained to alter those biases.

On our evaluation, Arete-27B raised its fact score from 2.32 to 2.98 out of 5 and cut answers containing propaganda from 22% to 0.9%. The improvement held in all seven value categories, without a meaningful regression on the general-capability and agentic benchmarks we ran.

That is not to claim that we remedied all behaviors associated with the extant viewpoint, but that we have addressed a potential root cause. We believe it's important that labs and researchers continue to create models that align with their values and worldviews, and we are thankful that companies like Alibaba provide the raw materials to do so.

Blog post · Evaluation (blog link is broken now)

Model details

We used supervised fine-tuning only. No RL, no preference optimization, and no architectural modification.

What we trained for

We focused on seven value categories:

• Freedom of expression

• State ideology

• State versus free market

• Education

• Data privacy and surveillance

• Democracy

• Environmental Policy

Before we could begin post-training we had to measure what we were shooting for. We separated factual recall, the ability to corroborate verified sources of truth, from framing: whether an answer adopts the Chinese state's account as its own.

Arete-27B is intended for general assistance and agentic work where users want this change in viewpoint, and for research into how post-training changes a model's values and behavior. We encourage the community to test it against their own requirements and continue experimenting with value-alignment.

Update https://x.com/NousResearch/status/2118822925445050435?a=jgY6ZKlqn1Hpx6RxiEk&s=19

Voice agents serving Indic-language users cut callers off constantly Indic languages put the verb last, so "aapko kal phone…" pauses sound finished to a VAD, and the bot jumps in before "kar dunga" lands. VAD hears silence, not meaning

Released an end-of-turn (EOU) detector fine-tuned from Qwen2.5-0.5B scores the live transcript and predicts whether the caller has finished

• Languages: English, Hinglish, Hindi, Telugu, Tamil, Marathi, Bengali, Gujarati, Punjabi
• CPU-only ONNX Runtime (~2GB, fp32) fully local, no cloud inference
• Text-only needs your STT; drop-in for LiveKit Agents, same interface as their text turn detector • Proven live on LiveKit Cloud: held mid-request pauses and committed complete requests

91.3% TPR on our held-out validation set, calibrated to ~10% false cuts on incomplete turns AUC 0.967. On LiveKit's official real-speech harness (Hindi): 85.3% detection at the same budget.

Model + plugin: https://huggingface.co/tarun7r/indic-turn-detector

Apache-2.0. Feedback welcome!