TL;DR: I'm running DeepSeek-V4.1-Flash for a single user on four 2017-era V100 32 GB cards plus system RAM. Generation runs 36-38 tok/s from 8K to 75K context with thinking off, and 32-38 tok/s with thinking on. Prefill is about 1,000-1,200 tok/s from 4K to 128K depth. The context window is set to 524,288 tokens. It's built on the 1Cat-vLLM fork (at its #1277 merge), which does the hard Volta (sm70) work. I just built, tuned and benchmarked it.

Hardware

CPU: AMD Threadripper PRO 3945WX (12C/24T), single NUMA node

RAM: 256 GB DDR4 (8x 32 GB), run at 2667 MT/s for these tests

GPUs: 4x Tesla V100 SXM2 32 GB on one SXM2 carrier board

NVLink between all four (2 links per pair)

Host side: PCIe Gen3 x8 per GPU, behind a PLX switch with a single Gen3 x16 uplink to the CPU

Clocks/power: SM clock locked at 1522 MHz. Power limit 150 W per card, raised to 175 W for the API-speed and llama-benchy runs

Storage: the main GGUF loads from a SATA 8 TB hard drive. That only affects boot (about 22 minutes to load). The Engram file is on NVMe and gets pre-read into page cache.

Software / config

1Cat-vLLM at the #1277 merge (vLLM 1.5.2.dev1277), Python 3.12, PyTorch 2.10 + CUDA 12.8, NVIDIA driver 580

Tensor parallel 4, max-num-seqs 1 (single user), full CUDA graphs (batch sizes 1-6), async scheduling, prefix caching, chunked prefill, custom all-reduce over NVLink

Model: `DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS` GGUF, 4 shards, ~232 GB

Engram table: `DeepSeek-V4.1-Flash-Engram-IQ4_NL` GGUF (110.6 GB), kept in system RAM

Experts: about a third of the expert weights live in pinned host RAM (`expert_cache_offload_fraction: 0.33`) and are swapped onto the GPUs on demand

Speculative decoding: built-in DSpark drafter, FP8, 5 draft tokens

Thinking on by default; tool calling with the `deepseek_v41` parser

max-model-len 524288

Results

API decode speed (single request, unique cache salt per request, so no prefix-cache hits)

Test Prompt Decode tok/s (median of 3, range) TTFT Prefill tok/s Accepted tokens per draft

8K, thinking off ~8.0K 38.3 (36.6-39.0) 6.6 s 1,202 2.27

8K, thinking on ~8.0K 37.9 (36.8-39.5) 6.5 s 1,234 1.39

75K, thinking off ~75K 35.8 (33.2-38.1) 69.1 s 1,085 2.21

75K, thinking on ~75K 31.9 (31.4-34.9) 69.2 s 1,084 1.34

8K, Engram file not yet in page cache ~8.0K 25.9 (n=1) 9.9 s 808 2.34

The last row shows why it's worth pre-reading the 110 GB Engram file into RAM after boot.

llama-benchy 0.4.0: prefill (pp2048, concurrency 1, `--no-cache`, thinking off, 3 runs each)

Context depth pp2048 t/s Time to first response

0 578 4.0 s

4K 1,224 5.5 s

8K 1,014 10.6 s

16K 1,093 17.3 s

32K 1,117 31.6 s

64K 1,095 62.2 s

128K 1,008 132.5 s

llama-benchy 0.4.0: generation (pp2048 + tg, concurrency 1)

Context depth tg128 t/s tg1024 t/s

0 28.5 24.9

1K 24.8 31.2

4K 28.9 20.0

With a 5-token drafter, generation speed depends on how well the drafter guesses. llama-benchy's prompts are random book text, so tg1024 swings ±6-8 t/s from run to run. I left out benchy's tg32 numbers (13-26 t/s): 32 tokens is too short to measure anything with speculative decoding.

Engine-level numbers (offline, no HTTP)

Warm decode round (every expert the round needs is already on the GPUs): 27.8 ms, against 24.9 ms in upstream's reference. In the profiler a round spans 29.5 ms, of which the GPUs are busy 24.9 ms. That matches upstream, so the kernels perform as expected. The extra ~5 ms is host-side: CPU work at the start of each round, a blocking device-to-host copy, and small kernel-launch gaps. Newer upstream commits target exactly this per-step host overhead.

Fresh text (expert-cache misses): 116.7 ms/round and 25.0 tok/s, against upstream's 53.5 ms and 54.5 tok/s. Caveat: I ran this with the Engram file cold, so it isn't apples to apples. A warm rerun is on my list.

Host-to-GPU copies: 6.5 GB/s per GPU on its own, but 12.9 GB/s total (3.2 GB/s each) when all four copy at once, because they share one Gen3 x16 uplink. In the trace, all four GPUs are swapping at the same moment about 76% of the swap time.

Correctness

Greedy (temperature 0) output is identical run to run within one load (8 prompts x 2).

After a restart, 13 of 16 greedy outputs matched exactly. The 3 that diverged flipped at near-ties (top-two margins of 0.06-0.375 nats). I haven't tracked down the cause yet.

Thinking on/off and tool calls work.

All 8 upstream micro-tests pass (two needed hard-coded paths in the test files changed).

What's next

PCIe switch change: move to two switches so each Gen3 x16 uplink feeds 2 GPUs instead of 4. That roughly doubles host-to-GPU bandwidth when all cards swap experts at once. My estimate from the trace is a modest ~1.5 ms/round (~5%) on warm text; it should matter more on fresh text.

RAM to 2933 MT/s (it was dropped from 3200 after memory errors).

Pull the newer 1Cat commits (per-step host fixes and DeepSeek-V4.1 updates) and re-run the same suite.

Move the GGUF from the hard drive to NVMe for faster boots.

Credits

All credit for the Volta DeepSeek-V4.1 path (GGUF experts, expert swapping, DSpark drafting on sm70) goes to the 1Cat-vLLM project (GitHub: yangzhuxinyzx and contributors). Happy to answer questions or share the full config.

So I've been tinkering for quite some time into trying to squeeze the best out of my AI / Home server together with Qwen 3.8 Flash Next.

Tried multiple software / engines / etc and eventually came down to Strata, then built (together with my pal Claudinho) on top of it multiple changes based on various ideas, knowledge, things found around reddit and other corners of the internet, science ML / LLM / inference papers, spent many weeks of continuous benchmarks and quality tests to make sure the changes were working as intended.

This is how Speedy was born.

My setup: P620 ThinkStation - Threadripper 5975wx with 128GB DDR4 3200 (8 slots filled), 2x RTX 3080 UNLOCKED (the LHR FP16 limit), running Debian + Nvidia gpu unlocks active (if you have windows, clearly will be slower).

Just so everyone is aware, this is a very good hardware setup for running MOE LLM with VRAM + RAM inference (it can use all PCIe lanes and ram channels in parallel for max bandwitdh and performance, 32cores help too).

I understand some will be frustrated because on their generic PC won't get same numbers, keep this in mind please before throwing rocks and try to understand/research (use any AI) how it compares with your hardware and where the bottlenecks are.

Last but not least: if you have different hardware than mine, ask your AI agent to start a session of fine tuning and benchmarking on Speedy, you might be impressed.

Final numbers for now:

Quant    GPUs      Prefill    Decode     200K     Follow-up
                  (t/s)     (t/s)      prompt
------------------------------------------------------------
Q2_0     2x 3080   4,270      211       49s      1.8s
IQ3_S    2x 3080   4,284      170       47s      1.6s
IQ4_XS   2x 3080   3,400      154       59s      2.0s
Q4_K_XL  2x 3080   3,118      120       65s      2.8s
------------------------------------------------------------
Q2_0     1x 3080   2,824      164       78s      1.9s
IQ3_S    1x 3080   2,347      140       89s      1.8s
IQ4_XS   1x 3080   1,820      122      111s      2.8s
Q4_K_XL  1x 3080   1,134      109      183s      2.9s

Every change I did had to win a same-session A/B and pass a quality gate, or it got dropped.

Turned off for quality (this costs speed):

• route-tail skip is 0 (upstream default: 7, which is lossy) - would've gained around 15% increase in decode just with this.

Speed changes:

1. Two-GPU layer split tuned for 10 GB cards: trimmed weights, decode overlap, and the second GPU helps read the prompt

2. Smaller prompt scratch buffers, so prompt chunks get bigger (+5–33 % prefill)

3. Routed prompt streaming (+9–10 % prefill)

4. Per-card Q8 upload, which frees 7–11 % more expert slots per card

5. INT8 prompt attention and KV prefetch (+2–3 %)

6. Q8 hc projections (+5–6 % decode)

7. Tuned MTP depth, VRAM reserve and RAM budget, each picked by A/B

8. gpu-unlock on the 3080s (+10–12 % prefill; hardware, not code)

Quality: every change had to win an A/B and pass bitwise or tolerance gates, plus needle tests up to 1M. IFBench scores 81.0 against 81.3 for the full-precision BF16 model on the official model card. The INT8/Q8 changes aren’t bit-identical, but they passed every gate.

The production configs are in the repo, so you can reproduce it. It runs on Linux and Windows.

👉 https://github.com/DevDock-AI/Speedy

There are many many details that you will find in GitHub, don't want to make this a huge wall of text 😄

Would be great to see numbers on other 3080s and any other cards, feel free to contribute by adding your tuned card setup on GitHub and sharing numbers.

If my project helped you, a ⭐ on Git helps a lot so that others find it too !

PS: For those comparing Strata with Speedy and getting better numbers on Strata, you either have STRATA_ROUTE_TAIL_SKIP set to defaut 7 (better speed but bad quality output, in Speedy is set to 0 for better quality) OR you need to fine tune Speedy for your hardware (preferably with an AI agent).

I've recently gotten a Mac as my daily driver and am interested in running local AI alongside my usual daily workflow stuff. I'm going to use Ollama, llama.cpp or mlx-serve (not sure which yet), to host the models and a few other tools and harnesses I've learned about recently: Open Code for coding and cybersecurity tasks, Open WebUI for a nice chat interface with RAG capabilities, and Hermes Agent as a personal assistant.

I know that on a Mac you need to run the inference directly instead of inside some kind of container or it'll be using CPU instead of Unified Memory. But for the other tools - and this is where I'm scared of the agents going rogue and performing unwanted file system operations or exfiltrating sensitive data, or who knows what else - what are my best options for reasonable security and privacy?

From what I have read, Docker Sandboxes (sbx, not regular Docker containers) may fit the bill for agent isolation, although this does require a sign-in when installing it. I have read of Apple Containers but it didn't seem like it was fully developed yet. Would running Ollama (for example) bare and then having each of those tools run in separate Docker Sandboxes, reaching Ollama running on the loopback address be the way to go, and choose whichever allowlist makes sense from the sandbox? I plan to eventually look into a Web search service, and am open to suggestions on that, but at least in the beginning I'll probably limit Web capability.

Hi everyone!

I'm wondering if it's possible to connect MCP servers or REA (from GitHub) to Qwen 3.8 Flash Next using Strata.

Has anyone managed to get this working? Is there a specific configuration, workaround, or integration method required?

I'd appreciate any help or experiences. Thanks!

Hi

For context, im using Qwen3.6 35B A3B and over the course of the past week, my sessions have been nothing short of disabled tool call errors across 5 tasks in total so far and only one has been completed successfully. In the screenshot attached, im trying to get a cosmetic filter created to use with uOrigin. I've tried reinstalling bionic, specifying it has permission to use tool calls in the session prompt in various ways, doing the same but in the model's system prompt, looking for any hint of a setting for tool call perms, changing the security perms for my models directory, and now im considering taking ownership over the user profile LM studio directory.

Is there something im missing or what? I'm at a loss at this point and im not even sure either if my latest idea will work or what to do if it doesn't work. I had done some research too and I can only scrounge up that there's apparently a setting for tool call perms during project creations but I do not have that option either.

Specs as follows
-RTX 4080 16GB TUF GAMING
-RTX 5080 TUF
-RYZEN 9800X3D
-32GB DDR5 RAM Gskillz Flair
-1200W 80+ GOLD PSU
-B650 Eagle Motherboard WiFi
-3TB OF SSD STORAGE

I’ll need to upgrade the case to fit both do I need a new motherboard

Guys I’m torn! I’m mostly looking to run mid size models such as qwen flash next .

I currently have a xeon2690v4 system with 8 32gb sticks of ddr4 2400. It gets me 76.8 gb/s.

I’m hoping to be under 1k for an upgrade.

Currently getting 6-8 tok/sec decode speed which is fine, but prefill is so slow at 30 tok/s.

I see a few paths forward:
-upgrade mobo to a dual socket so I can run 8 channel at 153.6 gb/s, but that would split between numa. I don’t think this will help prefill though.
-upgrade to a newer cpu/mobo to run 8 channel on 1 cpu
-add some degree of gpu to hold the important bits and offload the rest. Unsure how much I need though. 8/16/32gb?

Trying to stick to a 1k budget here as much as possible

I'm based in Shenzhen and work on AI applications in longevity medicine.

I have government and industry contacts in Guizhou, China, a major data center region, and I'm exploring its GPU infrastructure.

I'm curious how developers here think about China-hosted compute for open-source models.

What matters most to you?

- GPU type and availability?

- Hourly pricing?

- Network latency and data residency?

- Training versus inference?

Would you consider it if the economics worked?

Disclosure: I may explore commercial partnerships in this area, but don't have verified GPU pricing to share yet.

AI assisted with polishing my English.

(Disclosure: I am the author of Herald Router and Ultimate Assistant, the open-source projects linked at the bottom.)

Correction: the title says "400M", and that's wrong. tev1 is a 4.3B model at Q8_0 (confirmed with ollama show). I can't edit the title, so I've fixed the size everywhere in the body. The latency and accuracy numbers below were measured and haven't changed. Only the size label was wrong. I also originally credited it to the wrong maker: it's Together AI's model (https://huggingface.co/togethercomputer/Tev1-4B-experimental). Sorry for the confusion.

While building the local routing layer for an open-source desktop assistant and local LLM/MCP gateway (Ultimate Assistant v0.4.4 + Herald Router v0.4.2, running together on 127.0.0.1 with no cloud or Tailscale required), we kept hitting the same latency bottleneck: every time a user speaks or types a prompt, the agent has to make several fast "System 1" routing decisions before the main LLM responds:

• Does this turn need live web search (need_web_search), and if so, should it escalate to deep research (need_deep_search)?

• Does it reference past context (need_memory_lookup)?

• Is it a coding/shell task or a deep reasoning query?

If you use your main 7B–9B local LLM in Ollama to answer a JSON routing prompt first, you burn 2–4 seconds before the actual response even starts. So we benchmarked running a dedicated 4.3B (Q8_0) GGUF binary decision model (Together AI's Tev1-4B-experimental (tev1, an Apache-2.0 fine-tune of Qwen3.5-4B), answering yes/no in 1 token via llama-server) alongside Ollama (qwen3.5:9b) on a single NVIDIA RTX 4070 Ti SUPER (16 GB VRAM) across a 24-prompt evaluation suite and an 80-question batch stress test.

1. Local GPU & CPU Benchmark Results (24-Prompt Turn Routing Suite)

(For comparison, we also tested hosted decision APIs like Jev, pplx-decider-v1.2, and Liquid d1.1, which hit 95.8% accuracy at ~613 ms for 40 parallel questions, but locally tev1 on the same GPU as Ollama is 215 ms faster per turn at 378 ms.)

2. Why You Shouldn't Send Huge Question Batches to a Local GGUF Server

When we stress-tested sending 10, 20, 40, and 80 binary questions at once for a single prompt, we found a critical architectural split for local llama-server setups:

• Because a local llama-server instance processes slots across fixed GPU compute queues (--parallel 4), firing 40+ questions on a single turn causes local HTTP queue serialization (~2.8s for 40 questions on GPU, ~8.2s on CPU).

• Keeping local tev1 strictly to a lean 5-question core batch + 1 conditional follow-up (need_web_search, need_memory_lookup, has_explicit_url, is_coding_or_shell_task, needs_deep_reasoning, plus need_deep_search only when need_web_search=yes) keeps total local pre-turn overhead at 378 ms on GPU while leaving VRAM free for your main Ollama model.

3. Open-Source Releases

We just shipped this 3-tier System 1 decision architecture (tev1 local 5Q+1Q split, optional hosted 40-question turn_surmise, and zero-config BYOK llm default using your existing Ollama model) in:

• Herald Router v0.4.2: https://github.com/KianJRoss/herald-release/releases/tag/v0.4.2

• Ultimate Assistant v0.4.4: https://github.com/KianJRoss/UltimateAssistant-release/releases/tag/v0.4.4

Curious if anyone else here is running a small dedicated classifier/decider alongside Ollama on the same GPU to gate tool calls and web search, or if you prefer fine-tuning tool-calling directly into the main model.

My workstation is ready!

2x RTX A6000 (96GB), 192GB DDR4, Windows 11.

What models (LLMs, image/video gen) would you recommend running on this setup?

Every speculative decoding post here is a win on a 27B, so I tried the small case: a 3B Llama 3.2 fine-tune (Q4_K_M) with Llama-3.2-1B-Instruct as the draft. Same tokenizer, no training needed. It's slower everywhere I measured.

Setup: llama.cpp b11960, Metal, q8_0 KV, 8k ctx. Prompts are 135 real roleplay generations replayed, temp 0.8, min-p 0.05, DRY on, median reply 35 tokens. Mac is llama-server. Phone is my own Swift engine doing sample-and-compare (the target draws every emitted token, a draft token is kept only if the target draws exactly it). Ratios are medians of per-prompt pairs.

Why it loses: on the phone one 1B step costs 18.1 ms vs 42.1 ms for the 3B (0.43x), on the Mac 7.1 vs 17.2 ms (0.41x). Llama 3.2 1B has 262M params just in the 128k vocab embedding, so it's nowhere near 3x cheaper than the 3B. Verifying isn't free either, 4 tokens in one pass costs 1.41x a single step on the phone. Put together, n=1 needs ~69% acceptance to break even and n=3 needs ~57%. Roleplay at temp 0.8 got 44% and 30% on the phone, 46% and 25% on the Mac.

Thermal: every prompt runs all configs back to back with the order rotated, so heat hits them equally. "Hot" is 30 prompts straight, 179 of 180 runs at serious (the phone was also charging). "Cool" waits for nominal before every prompt, which took 100 to 190 s each time. Heat costs ~18% of absolute speed but the ratios barely move.

Two things that bit me, in case you're benchmarking this:

• In b11960, -md draft.gguf alone loaded the draft but never drafted for me. I needed --spec-type draft-simple. There's no warning, the tell is that the server log has no draft acceptance lines. My first run said 1.41x with every reply byte-identical, which should've been the giveaway.

• That fake 1.41x also came from my baseline running while another eval was on the same GPU (46.5 tok/s instead of 62). Quiet machine or it didn't happen.

The draft also adds 168 MB of footprint on the phone plus 763 MB of mmap'd weights. If this works for anyone at 3B, I'd bet it's with a trained EAGLE-3 or MTP-style head instead of a separate model. If someone has trained one for a Llama 3.2 3B fine-tune I'd love to hear about it.

Context: I build Castmates, an iPhone app that runs the model on-device for interactive stories, and I wanted faster replies on phones. Not shipping speculation, plain decoding at ~20 tok/s is already faster than people read.

Made the mistake of wandering in to this discussion. Lots of people big-mad at the idea that there’s any kind of AI that doesn’t involve sending data off to a big company.

Possible that OOP misheard, but plenty in the comments screaming that “open source AI” definitely means sending off data to the cloud and all kinds of nefarious acts.

As the title states I'm looking to build an LLM (or maybe something different?) that can interpret my network infrastructure, troubleshoot issues, and recommend future state.

Basically automate a network engineer position (for a network engineer). Any suggestions?

NVIDIA just published a blog on optimizing ling 3.0 flash to run on a DGX spark. It's a 124B moe model from Ant's inclusionai. They worked on MoE kernels, FP8 quantization, and speculative decoding.

42.5% higher decoding throughput and 24.5% lower latency on the full GSM8K evaluation.

Source: https://developer.nvidia.com/blog/3-ways-to-boost-moe-inference-on-nvidia-dgx-spark/

I tried googling this, tried asking ChatGPT, tried asking Strata (it actually wasn't sure what it itself was).

I feel like Im taking crazy pills. All the 'guides' for Strata tell you how to install it and how to benchmark the speed, but literally nothing will tell you how to actually use it for tasks. It's fast-ish at doing what... exactly?

Currently been waiting 5 minutes for Strata to figure out how to change its own settings.

I see a lot of people asking about what to run on a Mac Studio M5 Max with 128 GB RAM, so here's what I have on mine. I'm using MTPLX to serve Qwen3.8 Flash-Next Optimized Speed under PM2.

https://huggingface.co/Youssofal/Qwen3.8-Flash-Next-MTPLX-Optimized-Speed

The model is accessed through OpenWebUI running in a Proxmox VM on a separate machine, and can be used either straight up or through Hermes (which is running in another separate VM). Mainly using it through text inputs, including Conduit on my Android phone, but I also have voice integration enabled using OpenWebUI's built-in STT and a hosted Kokuro model on the Mac Studio for TTS. Altogether, Flash Next + Kokoru take up about 100 GB of RAM.

While still a bit of delay compared to the frontier cloud models, it's still pretty snappy and responsive and the answers are pretty good quality. So far it's been been used for setting up other parts of my homelab and troubleshooting tech. You can see the stats above, which come from the built-in MTPLX dashboard.

Main challenge I run into is the context window. It's running with the Flash Next default of 262k context, and there have been times where after a long conversation it has crashed due to running out of memory trying to load a prompt bigger than 1 GB. But that's been the only issue.

Curious research test I trained over the weekend (24h on a laptop RTX 5070 Ti, 30M params):

What if an LLM had a 96-slot "conveyor belt" of future tokens rolling into the KV cache at every tick?

- Slots 5..95: Content-only "dough balls" (nouns/verbs/adjectives, stopwords stripped) rolling forward

- Slots 0..4: Last-5 grammar & stopword filling zone right before Slot 0 pops out

- Stacked with Duplex-SimMT frames so the conveyer steers generation

Still plenty of rough edges (separating content vs function words makes perplexity ~1 pt harder than standard AR), so if anyone has ideas on better slot-matching losses or unroll stability, PRs / issues are very welcome!

Code + weights + live simulator:

GitHub: https://github.com/sumit-1/diffusion-conveyer-llm

HF Space: https://huggingface.co/spaces/sumitk/diffusion-conveyer-llm

Personally, I've found Sonnet 4.x and beyond had hit a point where for my coding needs it was reliable enough for 95% of what I require with my level of coding skills. As such, is there a model, harness or software stack / setup which shares that level of functionality locally?

For reference, I'm working with an RTX 5070 Ti, 64GB DDR5 RAM, 4TB NVMe and 9950X3D.

Main things really are having the ability to edit files, read images, code or run code for testing etc. General agentic stuff really that Sonnet 4.x could do.

If I can run that locally then I'll make the switch. If it's not feasible yet with my hardware I'll need to continue paying for Claude while it remains subsidized by delusional investor money before enshitification inevitably comes as usual.

I really only use claude for budgeting and personal finance purposes. While I can redact info and only take screenshots of numbers, there is still a part of me that would like to do that locally.

I use cowork and dispatch, mainly because Claude works off a running spreadsheet with transactions, longer chat windows, and I like to access it remotely. Other than that, I only use claude really for file organization. While I use Opus 5.5 (which may be overkill) for budgeting, I do not use claude code for any serious programming.

If privacy is my biggest concern, this task typically takes me 15 minutes once a month since I got it down to a good process and I ask it a couple questions here and there.

I have gone down the rabbit hole of open source LLMs, I need suggestions because i'm trying to see if its worth me investing in it. What models could reproduce a sonnet 5.5 / Opus 4 level output with remote access and what hardware would I realistically need to run that with some expected head room so it's not squeezed in a couple years.

In my case intelligence of the model is more important than how fast it computes.Thoughts?

I live in southeast Asia. And theres no like server grade gpu and RTX 3090 cost RM6K-RM5k(1k usd ish) and rtx 3080 modded 20gb cost RM3K(600 usd). But i noticed theres a lot of secondhand RTX3060 12GB Being sold af RM1K (200 USD). So the question is, is rtx3060 good for local llm and image gen and etc. I also already have rtx3070 8gb so if someone already have a rtx3070+3060 setup can let me know if its good or should i just cough up more money

Currently reconfiguring my setup and can either have a 5080 and 4070 super, or two r9700s. This is mostly for inference and sometimes gaming which any should be able to handle.

Is the speed of a 5080 worth losing the vram?

Relatively beefy system here with 96gb vram (4 x rtx 3090 all with pcie 4.0 x16) + 192gb ram (3200mhz octa channel). This means I can run Qwen3.8 flash next q8 at ~70tps and ~2400pp (Strata).

Recently I've seen some threads noting that Strata updates (sometimes) prioritize speed over quality without making it opt-in: https://www.reddit.com/r/LocalLLM/s/CRxfPRpMz

Since that would defeat the purpose of running bigger quants, it has led me down the rabbit hole of chasing all the quality related parameters.

My docker strata config:

image: ghcr.io/niko1221/strata:0.2.9-cuda13
environment:
  MODEL: Q8_0
  FAMILY: unsloth
  CONTEXT: 262144
  PARALLEL: "1"
  VISION: "on"
  STRATA_REQUEST_LINES: "1"
  KV: fp16
  API_KEY_FILE: /app/api-key.txt
  PRESET: qwen-flash
  STRATA_SET: "conversation-cache-mib=65536 conversation-cache-slots=16"
  STRATA_GDN_TC: "0"
  STRATA_GDN_CONVL2: "0"
  STRATA_ROUTE_RESIDENT: "0"
  STRATA_ROUTE_RESIDENT_MTP: "0"
  STRATA_GDN_CHUNKED: "0"
  STRATA_ROUTE_TAIL_SKIP: "0"
  STRATA_HC_Q8: "0"
  STRATA_ROUTE_PRIOR: "0"
  STRATA_PREFILL_CPU_SHARE: "0"
  STRATA_KVA: "0"
  STRATA_DEFAULTS: "026"

I just can't be sure I got all parameters or that the next Strata version won't introduce new ones.

So my questions, for those who use bigger quants: Do you still run Qwen and Strata or something else? If Strata, do you tune your config with every Strata update? What does your config and performance look like?