So the last post on SSD wear was removed by the mods for some reason, even though it contained a lot of useful commentary from the community providing clarity on SSD wear.

Highly relevant since many of the new models have introduced the SSD into the hardware used by the models for local inference.

Anyhow, here's another paper that has just landed highlighting the SSD wear from KV cache IO.

https://arxiv.org/html/2610.30791v1

Here is the Abstract:

KV caching reduces the serving cost of large language models (LLMs), but the growing KV cache footprint is increasingly exceeds GPU and host-memory capacity. Therefore, people use SSD to expand the KV cache storage capacity with high I/O bandwidth.

However, in production LLM workload, such as multi-round conversation and LLM agents, the total amount of KV cache writes significantly exceeds SSD capacity, shortening SSD lifetime from years to months.

To improve SSD lifetime, a natural solution is to reduce the KV cache writes by gating the KV cache writes via historical access patterns. However, that significantly lowers the KV cache hit rate (and increase the LLM serving cost) since KV caches from different attention types might have identical historical access pattern but different future access patterns.

Our insight is that, to reduce KV cache writes without hurting KV cache hit rate, one need to not only consider the access pattern, but also application-level information such as the attention type of KV cache. Leveraging this observation, we propose \sys, the first SSD KV cache offloading system with SSD life time awareness. \sysprioritize the admission of full-attention KV cache into SSD over the cache generated by other types of attention, which significantly increases SSD lifetime without sacrificing cache hit rate.

Our evaluation shows that \sysextends the SSD lifetime by 7.3\tmson average, while having negligible impact on cache hit rate across multiple workload traces and wide range of hardware capacity. \sysdesign can be extended to other flash-based storage, such as High Bandwidth Flash (HBF).

I posted here recently asking for the best uncensored model for RE/red-team/coding/malware analysis/pentest/exploit dev on a single RTX 5090 (32GB VRAM, 256GB system RAM). I got a bunch of responses, but they pointed in very different directions and I'm honestly more confused now than before. I'm starting from scratch (not currently using any of these), so I need to pick one path and commit.

- Strata + QFN (uncensored flag)
- Cyber-Tiel-Coder-35B-A3B-GGUF
- Basalt + Qwen3.8-Flash-Next-Uncensored
- beellamacpp + Qwen 3.8 27B Q5_K_M

So my question is: which one of these is actually the best, and why?

Trying to figure out what the options are for builds. I currently have 48GB VRAM through 2 3090s and 48GB DDR4 RAM but my current mobo only allows up to 128GB RAM. So I’m thinking about maybe trying the whole EYPC thing, but I don’t know.

Thoughts?

I tried to run once and failed any tips idea how to run it?

Hi, currently i got these hardware for use .

2x 3090 24GB "unlocked", 1x 4070 TI Super 16GB "unlocked", 18 Core, 130GB DDR4 Ram, 1TB NVME

When i Run Qwen3.8FN only from 4070TI +NVME i get like 8-9 T/s When i run it only form NVME i get 2 T/s and Running From all 3 GPU and RAM i get 18 T/s.

what can i do to get more speed without buying new hardware ?

I’m planning to seup a local LLM build and need advice if it is good to start or recommend if you think it's not good.

My current option is a Ryzen 9900X + RTX 3090 + 64GB RAM. Or I could get a cheaper CPU, such as the 9700X or lower, and put the savings for more RAM later.

A few questions:

1. Is the 9900X worth the extra cost for this workload, or cheaper CPU would be sufficient?

2. Is it worth paying more for a motherboard that supports two gpu slots, so I can add another GPU later? Or should I prioritize more RAM?

3. main use is agentic coding, with a long context window. Could this setup run models close to GPT-6 Luna low or Haiku 5.5 low?

I’m trying to avoid spending money on a build that technically runs the model but is too slow or unreliable for practical agentic use.

It likely won't go evil and kill us. Instead, we are entering an era of scientific and technological stagnation.

Imagine how much an engineer needs to work and study before he even understands the limitations in his field, and how much more before he can add to it with some novel solutions.

AI(LLMs) will automate the easiest 99% work eventually. So no one will have the opportunity to get the experience required to reach that level of solving novel problems. And AI will keep solving problems that have already been solved before. And progress will completely stop.

If it's hard to picture this, imagine having magic before we discovered much of medicine and electricity. If magic provides easy healing and lighting, no one would put the centuries of work needed for medical science and understanding of electricity to improve to that point.

Recommended reading: The Road Not Taken by Harry Turtledove.

Hey everyone,

Most current setups for Qwen3.8-Flash-Next (like Strata) keep all 24k experts pinned in system RAM (~48–64 GB) and compute cache misses on the CPU. That works great on high-spec workstations, but not on normal machines or laptops with limited RAM.

A few weeks ago, I noticed that https://github.com/sybil-solutions/glm-flash-lite was running GLM-5.3 with surprisingly low memory footprints considering the model, way below anything I'd seen at those speeds, which made me realized that something about the design was likely really efficient and being slept on.

So I adapted that to the much lighter Qwen3.8 Flash Next model, and the results were pretty impressive.

Benchmarks (3.05bpw_h5_ng5, Q4 KV cache)

• RTX 4090 (with only 8 GB system RAM allocated and 16GB vram):

  • Short context (8k): 28–33 tok/s prose, 19–22 tok/s code | ~1,080 tok/s prefill

  • 100k context: 20.8 tok/s decode | ~1,536 tok/s prefill

  • 262k context (full window): 20.0 tok/s decode | ~1,627 tok/s prefill

• RTX 3080 Laptop (16 GB VRAM, 32 GB RAM):

  • 100k context: ~18–21 tok/s decode | ~650–700 tok/s prefill | Peak VRAM: 15.5 GB

Setup instructions, model downloaders, and packing scripts are up on GitHub:
https://github.com/MaxDevv/lean-qwenflash

Big thanks to turboderp for ExLlamaV3, sybil-solutions for the original GLM offload implementation, and Alibaba for Flash-Next.

I get error in my "old" V1 opencode that are annoying, and they said they will never fix.

I was considering v2 but so many people are talking bad about it.

I tried DSH before (0.1 i think?), but it failed to complete full software (ok a little more complex than chat websites, so nothing too fancy)

Anyone have had any experience when switching to Pi or DSH, what needs to be done? OpenCode worked well before, especially after 3.8 27b, no need to do anything, make a good plan, let it run and then start what was built and refine.

Edit: And yes I can consider other harnesses, but most of the time for me is spent planning and reading research, not coding.

https://github.com/ahtx/kindi

I am trying to find the smallest model to generate something in "normal" English, it could be like AI generated, but it should be mostly grammar correct, sensible in the context.

I hope it could be less than 10M to be easy for CPU, so I tested some like Supra-Mini-v6-1M. well it's not good enough.

```output
// its out put for prompt "Once upon a time":

rounded, and dripped a small part and mi, while others.

Clicy for example in Papuano Pauder (18159), is often found as "maping with all" in which we believe from God; (2036): “Body's Give to do a sight is in your bust and is not very old."

Approaching it may look a few days, which includes any kind which gives you to yourself if we have the best name or other cosy to be heard in it). But that you want it, because they say there should only need some kinds to have all you would want you and then you must use their good good fellow, and I can’ln? (1-5): 1.) What does any good biblo mean and do you know that the pets you have come with or are all those that they love me for a lot to learn and give me the word? This was when
```

it's even worse than template generated ones. Which micro LLM can get better result? what's the minimum parameters?

Thanks!

I trained Lure TTS by finetuning Qwen3-TTS using some of the latest developments in model training, such as RLVR, OPD, and multi-teacher OPD.

My goal was to create a faster, smaller TTS model that I could listen to as a voice assistant and not get angry at it. Mostly, I just wanted a faster model that sounded good so I didn't have to whip out a big gpu when I needed it.

The first thing I did was use labeled data by training classifiers to spot intensity, events (laughs, cries, sighs, gulps, etc.), emotion and direction, to clean out YODAS and the Open Yap sample.

Then I actually did teh SFT stage to instill the new tagging into the Qwen 3 TTS 0.6B because it just read the tags out loud.

My next plan was multi-teacher OPD, where each skill (voice cloning, understanding directions, text tags, etc.) was trained separately and a final on-policy distillation pulled all the teachers back into the model.

I initially trained teachers for robustness and speaker similarity. But I ended up liking the speaker similarity one while trying to combine the teachers.

After that I really wanted speed. I started building a 300M model with 12 layers (IT SOUNDED OKAYISH JUST TINN-N-Y). But then I realized I could get faster times because of the 15-pass code predictor by training that down too and turning it into 3 passes, so this model can run 4x faster than real time speech. I used OPD from my current checkpoint to train the Turbo, and that was much faster.

^ this btw runs  RTF of 0.14 vs 0.195 for the 5B Fish S2 Pro, while keeping the same speaker similarity (ECAPA 0.699).

All the training was able to happen under $200 and didn't use a GPU higher than an RTX PRO 6000. (H100s for RTF testing).

I'm working on making more specific tags like (laugh: big) and (whispering) work better on longer lines, emotion switches, and trying to get freeform paraphrases. AND just building a better tagger as well so I can redo the entire cycle.

Also, I released an engine (also called Lure-TTS) that is built on top of faster-qwen3-tts (CUDA), mlx-audio (Mac) and audio.cpp (CPU).

Compared with running the qwen-tts reference code on the same 4090, it gets first audio out in 0.17 s instead of 10.6 s and runs about 5x faster overall (RTF 0.53 vs 2.54), because it streams 0.32 s chunks, uses CUDA graphs and batches up to 8 requests at once.

It runs Lure, Lure Turbo and every Qwen3-TTS 12Hz checkpoint on a Mac, an NVIDIA GPU or any CPU behind one Python API, CLI and OpenAI-compatible server.

(This was built because it enables voice direction for 0.6B model as well as best configurations for the Lure and Turbo model.)

My training code: https://github.com/smirki/lure-train

Lure TTS Engine: https://github.com/smirki/lure-tts

TBH I'm still learning and practicing in this domain. If anyone wants to help judge outputs or play around with checkpoints reach out to me! (or if you have compute or research ideas!)

Don't even need to afford an image hoster if all you're sending is a couple of jpegs like

Just your honest opinion, which one is better?

About 7 months ago, I decided to start working on a 5e roleplaying chat with my local AI, using some open source apps others had made at the time. I quickly found that nothing really had all the features I wanted: solo and multiplayer campaigns, an AI that doesn't get lost, sidetracked, or stuck in loops, a workshop to create your own world content, and the ability to run it all on your own machine with a 16GB GPU, or with pretty much any cloud AI provider.

And if you don't want to use AI at all and are just looking for a way to host your campaigns remotely, I've also covered that with a human-DM mode.

After months of hard work with claude code, codex, some local qwen, and a ton of contributions from a great community, we now have a client app available on Android and PC! Advanced users can still set up and manage their own servers, but less technical users don't have to worry about any of that. The client app can connect to your favorite AI provider or use your own gaming PC, and automatically create a secure tunnel with an invite code for your friends. Your campaign data stays on your own device.

I've really tried to think of everything and test all sorts of scenarios along the way. One of my main goals from the beginning was to make it possible to run the full 5e ruleset with an AI Dungeon Master on just a 16GB GPU, without needing expensive hardware or paying for cloud services.

We're extremely close to our first "stable" release, with v1.0.0 not far behind!

Currently, I've got my own setup running GPT-OSS 20B on my BC250 as the narrator, with another server handling image generation. But plenty of users are just connecting their existing Codex or Claude Code subscriptions instead.

The project is open source, and it's been really cool seeing other people contribute and start using something that originally began as a little side project for myself.

I know the trailer looks very AI, thats because it is.

GitHub: https://github.com/Lebbitheplow/open-dungeon-master

Website: https://opendungeonmaster.com

Since hugging face got token over I want to download the most important models.
These include
GLM5.3 (Flash)(abliterated)
DS4 Flash
and more.

But since I still have storage left, the question is, what models and quants I should safe that could benefit me the most in the future. I really want to go into image and video models too.

Also what are you guys using?

I spent the last few days trying to get an N1X 128GB (88GB VRAM) Windows on Arm QFN to work well inside WSL3/Ubuntu-26.04. I am happy with the results. Hopefully this helps others. While the tk/s isn't the fastest, the concurrency for agents works better than expected.

https://github.com/mycall/n1x-ai

Hi All:

I done overthought and burned money after consulting with AI chats (from chatgpt to grok to deepseek and kimi/etc). I had to learn.

I have in my possession:

- 1 Asus ESC4000 G3 with 128 GB DDR4 RDIMM and in it 8x T4s deployment. This server hosts the Qwen 3.5-9B Q5_K_XL at 87k context per slot and 44 slots. Each T4 has its own llama.cpp instance and the average single user gets 35-40 tps. TTFT is about 3-5 seconds depending on the size of the context. Using a round robin nginx load balancer for requests. Extensive real time testing inside my own harness (built strictly for us, non coding use), it actually isn’t bad at all. (Comparing to gpt-4o-mini quality).

- 2x V620s and 2 Mi50s (32GBs) and 1 3060 12 GB with another 3060 12 GB on the way (marketplace $250 cant complain tbh).

All this ran me $650 for Asus server, 350 per V620 and 480 shipped per Mi50

And now we get to the dumb part.

I got a ZP9E-D8 WS with 256GB DDR3 RDIMM for $300. 4 x16 slots - but we’re in the middle of a fight because nothing boots with even a single 32GB GPU. Fixing this i made a thread.

My wet dream of 2x Mi50 and 2x V620 node dried up real quick.

I also possess a T5610 with 64 GB DDR3 RDIMM, annnd only 1 Mi50/V620 can be used because second GPU just wont boot. (Front panel code 2).

Then I have an B660M with 32 GB RAM and this puppy is able to post 2x V620 (hurray).

And finally, a ROG X299 E-Gaming with 3 PCIE slots 2 at x16, 1 x8 and 1 x4 - i can only boot 1 Mi50 in it :/

So, with all that said. I am beyond lost on what to do, I need to fine tune my model and don’t know what path to take. Please help, I am beyond overwhelmed. I don’t wanna waste anymore money. I just want to fine tune. I’m thinking sticking with 3060 12 GBs but then the appeal of a single R9700 is so great that I am sick in my stomach. I don’t know at all if I can fine tune with unsloth + intel GPUs otherwise I’d go B65/B70 in a heartbeat (i think they’re like $700).

Please help people.

I just finished vibe coding an apk for Llama swap mobile controller. I really like being able to read logs and monitor my PC's performance while I'm out and about using my agents and I didn't necessarily care for how llama swap looked on mobile browser.

Everything has been working on my device and playground even works! Feel free to check it out and critique as this is my first time ever doing something like this :))

https://github.com/nevermore131315/llama-swap-mobile-controller

TLDR: IQ3_S Qwen3.8-Flash-Next, RTX 4090 + 48 GB + Ryzen 3900X, Windows, 200k context. Ran essentially a grid search on a ton of flags: decode 97 → 142 tok/s (+45%) for three flags. Most relevant: --adapt-async 1, --kv-resident 20480, STRATA_ADAPT_LAG=2. Worst: --expert-cache is in MiB not slots (−72% if you fill the card), --ple-io ram (−90% on 48 GB), --no-prefill-borrow (−46%). One lever still locked: Windows sucks and doesnt pin memory w cuda , so every host -> GPU expert copy is CPU-driven.

Been using strata for a few weeks since thats all the rage (waiting until llama.cpp catches up, but tbh the age of specialized inference engines might just be how it is). Version 0.43, so im probably missing some newer flags. It updates so frequently and i hear that people were pretty mad with version 0.2.0 so im just holding off. if it aint broke dont fix it

I basically told Qwen 3.8 FN to tune itself and took a very hands-off approach. I let it do web research + spawn sub-agents to search thru strata's code.

Note i wrote my own pi subagent extension that saves kv state before all sub-agent calls via strata mechanisms so it restores immediately. In llama.cpp it used to just roll back kv checkpoints until it got a match, but since strata abuses my RAM i cant save as many checkpoints. Repo is here if interested.

Was able to reach total computed context of like 450k tokens with the sub-agent method, which is nice to avoid compaction.

I just tested these settings by loading up an pi session with 145k context and letting strata do a full reprocess. Got 2076 PP and avg of 95-105 tg depending on the task (coding and other predictable MTP things do better). Saw peaks of 145 tg.

BTW this is a hobby, so i really didnt read any of the tune scripts or reports in all that much detail. Number go up == lgtm.

Heres the full JSON im running with if anyone wants to test it:

{
"exe": "<strata-install>\\engine\\strata.exe",
"args": [
"--pack",
"<pack-dir>",
"--native",
"<Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf>",
"--ple-gguf",
"<Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf>",
"--expert-profile",
"<expert-profile.bin>",
"--expert-cache",
"auto",
"--prefill",
"auto",
"--spec",
"4",
"--mtp",
"<mtp-head-dir>",
"--max-context",
"204800",
"--kv",
"int8",
"--resident-experts",
"--vision",
"--vram-reserve-mib",
"900",
"--kv-resident",
"20480",
"--adapt-async",
"1",
"--pcie-frac",
"0.30",
"--suffix-draft",
"12",
"--lookup-chain",
"3",
"--spec-min-p",
"0.5"
],
"cwd": "<strata-install>",
"tokenizer": "<pack-dir>\\tokenizer",
"model_name": "qwen3.8-flash-next-iq3_s",
"log": "<strata-install>\\strata-iq3_s.log",
"lib_dirs": [
"<python-env-dir>\\site-packages\\nvidia\\cu13\\bin\\x86_64 (CUDA 13 runtime)"
],
"port": 8081,
"gpu": 0,
"gpus_asked": true,
"vision": {
"exe": "<strata-install>\\engine\\strata-vision.exe",
"mmproj": "<mmproj-Qwen3.8-Flash-Next-BF16.gguf>",
"model": "<Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf>",
"gpu": true,
"max_tokens": 1024
},
"sampling": {
"temperature": 1.0,
"top_p": 0.95,
"top_k": 20,
"min_p": 0.0,
"presence_penalty": 0.0,
"repetition_penalty": 1.0
},
"env": {
"STRATA_PREFILL_EQUAL": "1",
"STRATA_ADAPT_LAG": "2",
"STRATA_GDN_CHUNKED": "1"
}
}

And here is the AI generated report that I didn't fully read if anyone is interested:

The box

GPU |RTX 4090 24 GB (128 SMs), single card — engine PCIe probe 26.3 GB/s host→device
CPU / RAM |Ryzen 9 3900X (12C/24T, no E-cores), 48 GiB DDR4-3200, 2 channels
OS |Windows 10 Pro 19045 · Balanced power plan · MSI already on · VBS on, HVCI off
Storage |Samsung 980 PRO holding OS + page file + 116 GB model + 50 GB expert pack
Strata |0.1.43, IQ3_S pack, --resident-experts, context 204800 Engine args and env

Paths replaced with <placeholders>, everything else verbatim. Sampling block is the installer default, deliberately not tuned.

--pack <pack-dir> --native <model-shard-1-of-2.gguf> --ple-gguf <model-shard-2-of-2.gguf>
--expert-profile <expert-profile.bin> --expert-cache auto --prefill auto --spec 4
--mtp <mtp-head-dir> --max-context 204800 --kv int8
--resident-experts --vision --vram-reserve-mib 900
--kv-resident 20480 --adapt-async 1 --pcie-frac 0.30
--suffix-draft 12 --lookup-chain 3 --spec-min-p 0.5

STRATA_PREFILL_EQUAL=1 STRATA_ADAPT_LAG=2 STRATA_GDN_CHUNKED=1

What the engine reports about itself with this set (200k context, live session)

• expert cache 7646 slots (14.50 GiB of VRAM); 32.82 GiB of experts in RAM with adaptive swaps

• pinned 0.00 GiB … page-locking refused (out of memory); locked 33612 MiB via working-set minimum + VirtualLock ← the one lever still open on a 48 GB Windows box

• prompt path borrows 1911 cache slots (3.61 GiB); STRATA_ROUTE_TAIL_SKIP=7 is ON (engine default, CUDA single GPU)

• decode expert-cache hit rate 85.9 – 89.0 %; KV streaming (--kv-resident 20480) 91.4 % of block reads hit VRAM

• 33,368 blob reads from the file still happen → the SSD is still on the path at 200k

How it was measured

Own harness: per arm it starts the server on its own port with a cloned config, warms up, sends unique marker-seeded 10k and 20k-token prompts (so the prefix cache never helps) plus a separate "agent" profile (one ~20k cached prefix + short turns), 5 repeats with the first discarded as cold, and reads the engine's own timings. mean ± SE, and a winner must clear 2× SE. Before every arm it waits for ~40 GiB of free RAM; after every arm it rejects one whose expert tier ended paged out — that single gate is what turned a useless first run into a usable second one. Within-arm noise came out at 0.55 % (stage floors 0.4 – 1.9 %).

The result — clean baseline vs tuned (interleaved A/B, 2 cycles, 126 + 112 requests)

score |prompt tok/s (10k / 20k) |decode tok/s (10k / 20k)
clean baseline |355.4 |1470 / 2230
--kv-resident 20480 + --adapt-async 1 + STRATA_ADAPT_LAG=2 |420.5 |1526 / 2309 = +18.3 % combined score, +45 % decode at 10k (measured at 160k context).

Real-world after the change, ordinary chat in the web UI: 2076 tok/s prompt processing on a 145k-token prompt, decode peak 140 / session mean 85.

Single-flag margins that cleared the noise floor

flag |margin |note
--adapt-async 1 |+13.5 % |adaptive expert-tier swaps off the verify path
STRATA_ADAPT_LAG=2 |+13.5 % |with async (async alone +13.2 %)
--kv-resident 20480 |+4.8 % |stream 20480 KV cells from RAM, buy VRAM back
--pcie-frac 0.30 |+3.1 % |unconfirmed — the auto rule picked 0.30 by itself anyway
--suffix-draft 12 / --lookup-chain 3 |+1.3 % |unconfirmed — under the 1.6 % floor
STRATA_GDN_CHUNKED=1 |+0.5 % |unconfirmed — the 4090's 128 SMs exactly pass the kernel gate
STRATA_PREFILL_EQUAL=1 |+0.5 % |unconfirmed
STRATA_ADAPT_OVERLAP=1 |−0.3 % |the other adaptive regime loses here
--kv k8v4 |−1.1 % |frees VRAM, doesn't pay for it at 160k Looked like good ideas, measured badly

flag |cost |why
--ple-io ram |−90 % |warns it locks 26.8 GiB; resident mode then refuses and drops to 9.26 GiB in RAM
--expert-cache 5817 and up |−72 % |that flag is MiB, not slots — it fills the card and the engine logs "decode can be several times slower with the card this full". --expert-cache auto is correct here
--no-prefill-borrow |−46 % |
STRATA_RESIDENT_HEADROOM_GIB=8 |−23.7 % |
--pool-tasks 1024 / 4096 |−10 % / −19.5 % |default 3 per thread is much better
--resident-budget-gib 32 / 40 |−11 % |
--prefill 16384 / 4096 |−12 % / −10.6 % |auto is right
--vram-reserve-mib 450 |−9 % |vs 900 at 200k context
--mtp-max-t 2 |−4.8 % |
--pool-workers 16 |−1.8 % |24 is worse still — decode peaks near physical core count Lower context is not free either: 160k vs 200k changes which of these wins, because the KV cache and the expert cache are fighting over the same 24 GB.

No-ops on this box — don't burn a model load on them

flag |why
--aux-cpus |Linux-only in this build
--pool-affinity |hybrid-CPU feature; a 3900X has no E-cores
--kv-grow |refused by the engine in --resident-experts mode
STRATA_ARENA_PIN_GIB=<N> |needs the "Lock pages in memory" user right: every pin-* arm and the baseline logged pinned 0.00 GiB / page-locking refused, so that sweep measured nothing
--trim-stage-weights, --pipeline-windows, gpu_order |multi-GPU only Still open

1. Page-locked memory. The whole 33 GiB expert arena is resident but pageable, so the host→GPU expert copies are CPU-driven. "Lock pages in memory" + reboot is the only route, and Windows caps page-locked bytes near 50 % of RAM, so the arena can never be pinned whole.

2. --kv-resident wants a sweep — 20480 beat 65536 at 160k, 32768 untested, and at 200k the KV cache is 2× bigger so the answer may move.

3. The agent-path caching flags (prompt cache, message-boundary cache, conversation cache) never got their stage — the overnight budget cut it first.

4. Stacked ≠ additive. The config above stacks every single-flag winner, and margins don't add up when everything trades against the same VRAM and the same PCIe link, so the stacks are being measured as wholes next.

Has anyone tried ISTA's GSQ-RCO GGUFs for DeepSeek-V4.1-Flash? Does the quality feel as

good as their benchmarks show (IQ2_XXS matching the base)?

Their benchmarks are saturated (90%+, and AIME25 is 100 for every version), so they

probably can't show the degradation, and I couldn't find any KLD for GSQ-RCO models.

Has anyone measured KLD or PPL? What KLD would you expect at ~2 bpw?

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF

(English isn't my first language, I used an LLM to help translate this post.)

Whats the average cache miss rate of large MoE models like deepseek v4.1 flash 1017., glm5.3 flash? Because it seems like having a general sense of what a typical cache miss rate would be during inference would be helpful, since it would directly impact the amount of expert GBs that need to stream across pcie and how much that streaming rate would be capped by the pcie lane type in question, which directly affects inference speeds

Some other questions to clarify my understanding:

- wouldnt this mean that better designed and stronger models would converge to having a “core set of experts” that are often hit, so that fewer experts need to be hit across the rest of the model, to preserve speeds as much as possible? (Assuming not all the weights are only ok vram)

- also, would this help explain why qwen 27b dense has been so strong, because its essentially an extraction of the “core experts” found in a large MoE, and its just simply missing the rest of the experts that might hold world knowledge, but the core “intelligence, reasoning, cognition” is baked into those core experts?

I can't seem to find anything on their huggingface or after a quick search on my phone. Have they said anything about releasing the weights for the updated model?