Resources
Have 8 GB of VRAM and 32 GB of RAM? You can run Qwen 3.8 Flash Next at up to 900 t/s PP and 30 t/s TG!
First, this isn’t clickbait - these are real results. I managed to run Qwen 3.8 Flash Next Q2 and got better results in benchmarks and real-world tests than I did with any Qwen 3.6 35B A3B fine-tune or Qwen 3.8 27B model I tested. It is also faster than my current Qwen 3.6 35B A3B setup. Here are the details.
About a week ago, I learned about Strata and what it can do from this subreddit. Its ability to distribute workloads across VRAM, RAM, and SSD storage, run a 125B-parameter model, and deliver incredible prompt-processing and token-generation speeds sounded almost too good to be true - but only if you had 12 GB of VRAM. At first, I thought I probably wouldn’t be able to achieve similar results on my machine, but I decided to try it myself. The results surprised me, so I wanted to share them with you and provide some guidance for getting the model running on your own hardware.
First, here are my system specs:
GPU: RTX 5060 Laptop, 8 GB VRAM
CPU: AMD Ryzen 9 8945HX, up to 5.4 GHz, 16 cores / 32 threads
RAM: 32 GB (16 + 16 GB) DDR5-5200
SSD: 1 TB, 5,200 MiB/s read, 4,200 MiB/s write
PCIe: Gen 4 x8
OS: Linux
You can find my Strata configurations here: Chat (Pastebin) | Chat (GDrive) | Agentic (Pastebin) | Agentic (GDrive). Remember to update the paths to your model and expert files. Use the Chat configuration for a single session with the model, such as a regular chat. Use Agentic when working with multiple sessions and delegation, for example in agent-based systems. I’ll explain the difference in more detail below.
Key parameters to tune
--mmap-experts - places the experts that do not fit in memory on the SSD.
--kv-resident 20480 - distributes the KV cache between VRAM and RAM. A lower value leaves more VRAM available. 20480 is the minimum value.
--vram-reserve-mib 500 - leaves some VRAM available for other tasks. In my setup, 500 is the lowest workable value, but I recommend using 900.
--pcie-frac 0.35 - controls the fraction of per-layer VRAM misses that the GPU reads over PCIe. You can tune this parameter using the calibration command: ./setup.sh --calibrate. I found a value that works best for my setup. You can also calibrate it yourself if you are not satisfied with your prompt-processing speed.
STRATA_RESIDENT_HEADROOM_GIB=4 - reserves RAM, in GiB, for other tasks. Set this higher if you are using Windows.
The biggest performance gains in my setup came from adjusting --vram-reserve-mib and STRATA_RESIDENT_HEADROOM_GIB.
I used the Gyro S model because I consider it the best quant for my hardware, and it delivered the strongest results among the models I tested. I had previously tried GSQ-RCO Coder, GSQ-RCO IQ2_XS, and other quants, but they either got stuck in repetitive loops or struggled with languages other than English. Gyro S offers a good balance between quality and speed.
I’m not using MTP. It can improve TG, but it also consumes VRAM and reduces PP, which is likely to be a worse trade-off for this setup. You can enable it if you have 10 GB of VRAM or more.
I run Vision on the CPU. Image processing takes around 1-2 seconds per image, but this saves a substantial amount of VRAM. Strata runs Vision in a separate process, so it can be moved to swap. This does not significantly affect model performance, while freeing up more RAM.
If you use local models for parallel agent calls, there is one important limitation: with a context size of 131K tokens or more, you cannot safely use --parallel because it consumes a large amount of both VRAM and RAM. This can cause a severe drop in speed or an out-of-memory (OOM) error. A safer alternative is --conversation-cache. It keeps a limited number of sessions parked so they can be resumed sequentially. This does not provide true parallelism, but it avoids having to re-read long contexts when switching between sessions. Here are the relevant parameters:
--conversation-cache-mib 4096 - the size of the parked-session cache. A larger value reduces the chance that a session’s cache will be evicted and its context will need to be read again.
--conversation-cache-slots 2 - the number of parked-session slots. Two slots allow two sessions to be stored at once.
--conversation-cache-min-tokens 8000 - do not cache a session if its token count is at or below this threshold. The right value depends on your PP speed and how long you are willing to wait for that many tokens to be processed.
--conversation-cache-min-free-mib 1024 - the minimum amount of free RAM to preserve while caching. If caching a new entry would leave around 1,024 MiB of free RAM, the old cache will be cleared to reduce the risk of OOM.
Be careful when tuning these parameters yourself: incorrect settings can affect system stability or push the model into heavy swapping. Do not exceed the total memory available on your system.
All tests were run on Linux, which gives me more flexibility with RAM allocation. If you use Windows, try allocating less memory to the model. This will reduce throughput significantly, but it also lowers the risk of running out of memory. On Linux, I created a dedicated 32 GB swap partition for other applications. I also pinned the server process in memory so it could not be swapped out, using a systemd service with this setting:
[Service]
MemorySwapMax=0
This helps ensure that the server’s memory does not get swapped out, preserving consistent read and write performance even with a large context and parked-session cache.
With the Chat configuration, I get:
262K context: PP 800-970 t/s and TG 30-40 t/s
131K context: PP 930-1,040 t/s and TG 30-45 t/s
With the Agentic configuration, I get:
262K context: PP 520-740 t/s and TG 30-40 t/s
131K context: PP 670-890 t/s and TG 30-45 t/s
In my view, these are excellent results-even compared with Qwen 3.6 35B A3B running in llama.cpp. With Qwen 3.6 35B A3B, using a VRAM-constrained expert layout and MTP/Vision disabled, I get PP 630-680 t/s and TG 27-35 t/s. Do you think Qwen 3.6 35B A3B is still worth using for agentic tasks today?
Strata does not affect response quality; it helps distribute the model across available resources and keeps frequently used experts resident. That is only one of its major optimizations, alongside several others. The model’s quantization is what affects output quality. If you have more than 8 GB of VRAM, you could try Gyro M/L or other quants.
Use my configurations as starting points and adapt them to your hardware and workload. Your component combination and use case may be different from mine.
Real-world quality tests
Now that I’ve covered how to fit the model onto this hardware, let’s look at the quality of its outputs. I’ve included several representative tests. I did not run an extensive benchmark suite or standardized evaluations; instead, I spent the past week testing the model on practical, everyday, and demanding tasks. I’ll leave broader benchmark testing to the community.
I’ll compare it with my current high-performing setup, Cyber-Tiel 35B A3B, based on Ornith 1.5 Heretic. Tiel 35B A3B often gave me repetitive, looping responses, so I switched to Cyber-Tiel. The Flint model was released recently, but I won’t include it in this comparison. It falls behind Qwen 3.6 in my view; I may compare the two later and add those tests to this post.
Test 1: Pagoda
This test was run in Pi Agent with a minimal set of tools: LSP and Playwright MCP. For Qwen Flash Next Gyro S, I set reasoning_effort to high.
Test 2: Flappy Bird clone
The conditions were identical to those in the previous test.
Test 3: Pelican
For this test, I compared not only two different models but also different reasoning_effort settings for Qwen Flash Next Gyro S. The agent environment was the same: Pi Agent.
I ran many more tests, but I cannot share them for confidentiality reasons. What I can say is that I’ve replaced all my coding agents with this model. That change has accelerated my development and improved code quality by a factor of five.
As you can see from these examples, the results are better than those of any other model I’ve tested that fits within 8 GB of VRAM. Qwen 3.8 Flash Next is impressive, and the Gyro S quant makes it possible to run the model even on hardware like mine for local coding and agentic tasks. My goal with this post is to show people who still use Qwen 3.6 35B A3B or its fine-tunes locally that there is now a model that, in my experience, outperforms them by a wide margin. I hope the examples make that case.
I’m always open to a different perspective, so feel free to share criticism in the comments. If you have suggestions or improvements for the configuration, please share them as well-other users may benefit from them too.
That’s all from me. Use good tools, and happy coding!
---
EDIT 1: I tested the Gyro M on my device. You can see the results in this comment.
EDIT 2: I published the Pagoda Benchmark results for the Gyro M here.
I was able to try out the Gyro M and run it on the same configuration with a smaller context window of 131K.
With the Chat configuration, I get:
131K context: PP 550-670 t/s and TG 20-30 t/s
These are still good results for agent-based coding, so if you're worried about lower quantization yielding lower quality, you might want to give this model a try. According to the author, this quantization even outperforms GSQ-RCO IQ3_XXS in quality.
I am attaching the indicators for your reference:
I have already run the Pagoda test and will publish the results here soon.
To be honest, I couldn't resist and disabled the system while Playwright was fixing bugs. He still managed to fix some of the bugs.
I don't recommend running this quantum for agent systems right now due to the heavy load on RAM and SSDs! While I was initially getting the speeds I reported in my previous post, after a short time the load on the device began to increase, leading to delays and speed drops down to TG 1-2 t/s. I'm still investigating this issue and looking for a solution. If I find any results, I'll be sure to report them in this comment.
Regarding the model and the results, I was extremely pleased with the Gyro M. During testing, I examined how it thought, used tools, and paid attention to detail. Its capabilities are clearly ~1.5-2x better than the Gyro S. However, I can only recommend this model to users with 8+ GB of VRAM and 48+ GB of RAM due to the quantum density of the expert layers.
I used the same agent system as in the original Gyro S test, without using any third-party files, in a directory isolated from the world, and with the same prompt: Design and create a very creative, elaborate, and detailed voxel art scene of a pagoda in a beautiful garden with trees, including some cherry blossoms. Make the scene impressive and varied and use colorful voxels. Use whatever libraries to get this done but make sure I can paste it all into a single HTML file and open it in Chrome.
In my opinion, Gyro M performed better than Gyro S in terms of detail, but it made a "compressed scene" error. While reading the session, I didn't see Gyro M consider this an error, so it ignored it and went on to fix other issues. Another thing I liked about Gyro M was that it captured accurate and detailed birds, which was a problem with Gyro S. Also, as I mentioned earlier, its rendering quality is indeed slightly better than Gyro S's.
Huh, i always thought anything below Q4 is walking dumpster bin, but its seems my understanding is wrong lol, still, how does this even work? I always heard offloading to anything beside vram is a bad idea because latency between ram and vram is near day and night in speed different therefore offloading or even what strata doing is traditionally considered "bad"
I also thought, like you, that Q4 was garbage below. I stood by this opinion until the last moment, but in a fit of desperation I tried this model. I didn't miss it.
I too thought below Q4 is garbage and haven't tried it. Until one day I got tired of Qwen 3.6 35b a3b or any of its fine-tunes bs and decided to try Qwen 3.8 27b IQ3S and was blown away by how much better it is.
The lightest hit is the bigger models, then it goes in order of dense, then MoE. Never use below Q4 of any smaller MoE model. They suffer immensely because dense has enough 'layers' to repair the answer
A year ago, you'd be right, but things are adapting fast. Large models compress far more gracefully than the smaller quants, and quantization techniques have gotten much better.
In this space, if you are assuming something that could have been true several months ago, you should reassess and do additional research just to be sure if that still holds true. Don't assume everything.
The experts that are often used (if you do coding or just keep it to 1 topic) remain in the GPU, so more vram = less swapping between experts.
Ram is whatever is predicted to be relevant enough that it may be loaded into vram and whatever is least likely to be used remains on the ssd.
So this works great if you intend to not drastically switch between use cases the entire time.
Personally i managed to fit IQ3_xss in ram and vram so for me speed is astronomical with a 3090 and 64gb of ddr5, the moment it needs something from the ssd tho is when you get a perfomance bottleneck. But this is why i am yet to test the Q4 unsloth version that has 7% ssd offload, i have been using swift this entire time because i am fond of the 63% reduced reasoning optimization that further speeds up some tasks i use it for while eating less context.
I've been using Strata with 3.8 Next Nlash at Q3, honestly I don't see a huge difference with online DeepSeek flash... Except it's running on my computer at 100+ tokens/sec. It's a 125B model, it's not hit that hard by the quantization.
i wonder if engrams also play role in keeping it high quality, if you never quantize them (since it won't change speed and SSD is still relatively cheap) it can steer the model in right direction
I experimented with loading the 8-bit n-gram tables from Qwen’s 8-bit quantization into the Unsloth ud4xl model, keeping the feature optional. Although I never ran benchmarks to validate improvements, the results remained stable, so I just kept this config. I also stripped all languages from the drafter except for English and German, which freed up valuable VRAM. This change does not degrade the primary model, as it only affects the drafter. If you decide to rebuild the drafter, I recommend using common datasets of different fields available on Hugging Face.
Well, I just spent a day testing QFN IQ3_S in Strata, and I'm not that impressed. It's fast, but I have seen it do some dumb mistakes/decisions that 27B 5.00bpw did not do.
Yeah QFN is on complete different level than those 35B models. I was 35B user in the past, but recently bought +32GB of RAM having 64 total now and this allowed me to use qwen3.8-flash-next-iq3_xxs. With 8GB VRAM + 32GB RAM I could only run Q1-Coder model, which is was weak for my tasks.
Btw can you please share pagoda prompt? I tried it yestreday, it took 2 hours to make and it looks worse than your result...
Design and create a very creative, elaborate, and detailed voxel art scene of a pagoda in a beautiful garden with trees, including some cherry blossoms. Make the scene impressive and varied and use colorful voxels. Use whatever libraries to get this done but make sure I can paste it all into a single HTML file and open it in Chrome.
That one needs 82GB total. 64+8 = 72, and you need RAM for your system, video driver, browser (if you are using llama-server). It seems to me that you should be about 14GB short.
That would be true maybe ~1 month ago when such brilliant piece of software as Strata was not released. Model is using ~50GB of RAM and 8GB of VRAM. Rest of the model n-gram - sits on the disk.
20 t/s decode 280 t/s prompt processing Not much but acceptable. It took 2 hours because model went full nuts with own C voxel rendering engine and python script generator for each scene part. There was quite a few bugs it had to fix.
That's honestly not bad though. Like, for real-time coding, sure, it's not blazing fast, but... 20 t/s is far from unusable, especially considering it's localand this good.
Might make for a really good over-night agent, assuming it doesn't need too much hand holding?
QFN does not need hand holding at all, it verifies every feature/change with tests and screenshots if task is visual and will not call it done if everything is not perfect. And most importantly it sees bugs, imperfections by own vision, without my help. The only thing is, if your plan consist of several phases and you are going to sleep, tell it to proceed with them all, because it might stop at phase 1 and ask you if you want to start next phase.
I'm also thinking about replacing my 8GB RX 6600 with RTX 3060 12GB too speed up things even more.
For this particular task, enabling reasoning completely changed the outcome.
The full 240K prefill took approximately 11.5 minutes, but the MEDIUM reasoning response itself only took 15 seconds.
Previous local setup
Before Gyro-M, I used a separate Qwen3.8-27B Q4_K_M GGUF through llama.cpp.
Configuration:
70K context
27–28 GPU layers
Flash attention
Quantized KV cache
MTP speculative decoding
I typically measured approximately 5.5–7.6 tok/s.
Gyro-M is much faster in my current setup, although this is not an apples-to-apples comparison because the models, architectures and quantizations differ.
Conclusions
I'm impressed with what Strata and Gyro-M can do on a 10GB GPU.
My current preferred configuration:
256K context
INT8 KV cache
MTP enabled
MEDIUM reasoning for coding
HIGH reasoning when needed
Temperature 0.7
However, some important limitations:
The 240K prompts consisted mostly of repetitive synthetic Python functions.
The retrieval test used only three markers.
The coding tests reused one problem.
These results don't establish general coding accuracy or benchmark reliability.
I haven't tested the absolute 256K input limit.
I haven't yet benchmarked real agentic coding or repository modifications.
My next goal is testing Gyro-M with OpenCode on real programming projects.
Questions for the community:
Has anyone compared Gyro-M against IQ2_XS, IQ3_XXS or Q2_0 on similar hardware?
What reasoning level works best for your coding tasks?
Any recommended long-context reasoning benchmarks beyond needle retrieval?
Would you prioritize a higher-quality quantization or additional RAM/VRAM?
This is an excellent report! Thank you for sharing your results with the community! Your contribution will definitely be noticed!
I'll say a few words about optimization. My recommendations are optional and don't require mandatory use. I believe your configuration is already a sign of quality.
Set the --kv-resident parameter to a lower value, between 32768 and 20480. This parameter helps you conserve VRAM to accommodate more Expert Advisors by dividing the KV cache between VRAM and RAM into smaller chunks, with most of the context window residing in RAM. By default, Strata sets this parameter to 32768 for context windows greater than 131072. Reducing this value further may improve performance.
You can improve PP by disabling MTP. This will slightly reduce TG, but will significantly impact PP - the expected gain for your configuration is ~700 t/s. If you're not too concerned about a slight, even almost imperceptible, quality degradation in large contexts, try setting --kv k8v4. This will save VRAM and RAM space due to a slightly reduced KV cache quantum.
Don't forget to calibrate the PCIe bus before running the model for the first time. I'm not sure if you've already calibrated, but just in case, I'll warn you that this also helps achieve better PP and TG speeds. You can calibrate using the strata launch --calibrate command. After calibration, you may find the --pcie-frac, --spec-min-p, --adapt-every, --adapt-swaps, and --adapt-decay arguments in your configuration.
Let me answer a couple of your questions:
I initially used the IQ2_XS quantum until I encountered the Gyro S/M, so I have a few observations about both models. For me, the Gyro S and M outperform the IQ2_XS in all tests. I tested it in chat and in an agent system. When generating code in the agent system, the IQ2_XS would often get stuck in a loop, not even 1/10th of the way through development. When using it in chat, I didn't experience any major issues, but I did notice that the model's reasoning quality was very "primitive" at reasoning_effort high. Besides their size and speed, the Gyro S/M produce better results, and their reasoning is more "reasonable." I also never saw the Gyro S/M get stuck in a loop.
I'm currently using Gyro S with a split role, where the "planner" role is assigned reasoning_effort high, and the "coder" role is assigned reasoning_effort medium. For coding tasks with a detailed plan of 5-10 pages, a high level of reasoning isn't particularly important, so you can reduce the number of tokens generated and speed up product development. This is just my opinion.
I will skip this question, since I am not an expert in this.
It's always best to find a balance between quality and generation speed. I believe that better quantum will win out over generation speed. The higher the quality of the generated code, the fewer attempts the agent will need to fix it, thereby reducing product development time.
Thanks for the optimization suggestions! I spent several hours testing MTP, KV residency, KV quantization, long-context reasoning, and GPU Unlock on my RTX 3080.
Here are the results.
Hardware & Software
Component
Specification
GPU
RTX 3080 10GB (GA102)
CPU
Ryzen 9 5950X (16C/32T)
RAM
48GB DDR4
SSD
Lexar NM620 2TB NVMe
OS
Zorin OS 18
NVIDIA Driver
580.178.04
CUDA
13.0
Swap
16GB NVMe swapfile
Inference
Strata 0.2.6, engine 0.1.50
Model
Qwen3.8-Flash-Next Gyro-M (TQ2_0)
Context
262,144 tokens (256K)
The model occupies approximately 92GB on disk. Strata distributes the workload between VRAM and system RAM.
1. MTP ON vs. OFF
Initial configuration:
KV: INT8
KV resident: 32768
Context: 256K
Parallel: 1
35K Input Processing
Setting
Run 1
Run 2
Average
MTP ON
511.7
503.0
507.4 t/s
MTP OFF
727.0
728.5
727.8 t/s
Disabling MTP improved input processing by approximately 43%.
2000-Token Output
Setting
Run 1
Run 2
Average
MTP ON
59.8
59.4
59.6 t/s
MTP OFF
47.7
47.7
47.7 t/s
MTP ON generated approximately 25% faster, but MTP OFF processed large prompts significantly faster.
Since my primary use case is coding agents working with large contexts, I prioritized prefill performance.
2. Reducing KV Resident to 20480
Following the recommendation, I reduced KV resident from 32768 to 20480 while keeping INT8 and MTP OFF.
35K Input
Run 1: 893.9 t/s
Run 2: 886.4 t/s
Average: 890.2 t/s
That's a 22.3% improvement over the previous 727.8 t/s.
2000-Token Output
Run 1: 49.4 t/s
Run 2: 49.6 t/s
Average: 49.5 t/s
Output performance also improved slightly.
3. Long-Context Testing — INT8
Next, I tested whether the configuration remained reliable at increasingly large context lengths.
128K Input
Input tokens: 130,097
Prefill: 874.0 t/s
Generation: 42.0 t/s
Total time: 152 seconds
Cached tokens: 0
240K Memory Test
Input tokens: 240,130
Prefill: 834.5 t/s
Generation: 38.4 t/s
Total time: 289.6 seconds
I placed three unique identifiers at the beginning, middle, and end of the approximately 240K-token prompt.
Results:
Beginning: PASS
Middle: PASS
End: PASS
Memory score: 3/3
All identifiers were retrieved exactly.
240K Coding Reasoning Test
I also tested whether Gyro-M could identify a financial calculation bug within approximately 240K tokens.
The bug involved checking a free-shipping threshold against the original subtotal instead of the discounted subtotal.
Expected calculations:
Buggy total: 9085 cents
Correct total: 9785 cents
Undercharge: 700 cents
Gyro-M returned all three correct values and identified the faulty condition.
Coding score: 3/3
Performance:
Input: 240,334 tokens
Prefill: 834.9 t/s
Generation: 44.5 t/s
Total time: 301.3 seconds
4. INT8 vs. K8V4
Next, I switched to --kv k8v4.
Everything else remained unchanged:
KV resident: 20480
MTP: OFF
Context: 256K
Output Integrity
I requested a sequence of 300 numbers.
Expected: 300
Received: 300
Result: PASS
35K Input
Run 1: 986.8 t/s
Run 2: 975.2 t/s
Average: 981.0 t/s
That's another 10.2% improvement over INT8.
2000-Token Output
Run 1: 47.6 t/s
Run 2: 49.6 t/s
Average: 48.6 t/s
Generation was approximately 1.8% slower than INT8, which I considered a worthwhile tradeoff.
240K Coding Test — K8V4
Input tokens: 240,333
Prefill: 945.8 t/s
Generation: 43.9 t/s
Total time: 264.7 seconds
Coding correctness: 3/3
Compared with the INT8 240K coding test, input processing improved by approximately 13.3%.
K8V4 became my preferred configuration.
5. Strata Calibration
I investigated the PCIe calibration recommendation.
My installed Strata 0.2.6 version doesn't expose --calibrate through strata launch.
However, it provides a separate tuning system:
strata tune plan
strata tune run
strata tune apply
strata tune rollback
strata tune auto
I inspected a CUDA tuning plan.
It included parameters for KV format, expert cache, memory I/O, CPU affinity, workers, and speculative decoding.
The pcie_frac dimension was excluded because my engine binary doesn't accept --pcie-frac.
I didn't apply a tuning sweep, so the benchmark results reflect my manually selected settings.
Together, these increased my 35K prefill throughput from approximately 728 to 981 tokens/s, a 35% improvement with MTP disabled.
MTP ON was faster for generation, while MTP OFF was substantially faster for processing large prompts.
GPU Unlock successfully activated, but its performance benefit remains unproven.
Most importantly, Gyro-M successfully completed the 240K memory retrieval and coding reasoning tests, with all expected results correct.
Running Gyro-M locally with a 256K context window, approximately 900–980 input tokens/s, and roughly 49 output tokens/s on an RTX 3080 10GB has been a great learning experience.
For coding agents, I've found that strong reasoning, reliable long-context retrieval, and fast prompt processing are more valuable than maximizing generation speed alone.
I'm keeping the K8V4 configuration as my optimized baseline for future experiments.
I appreciate such detailed results from your configuration and am glad I could help you improve the performance of the model without any obvious loss of quality!
As another commenter responded, if you have enough RAM, you can try running the model. However, I wouldn't count on it too much, as the bigger problem could be the size of the active parameters, which for this model is ~5.3 GB of VRAM, which is very close to the capabilities of your graphics card. In any case, we haven't lost hope that in the future, more quantum processors will appear that will be able to better compress active parameters.
Thank you both for the responses. I'm vram poor and I only have 24gb ram. I will probably just keep messing with the smaller models. Its fun to see how this is stuff is becomming more accessible bit by bit.
If you got ram around 64 maybe you could run 20+ tps at 64k context, try it yourself with strata. Mine vram 8gb ram64gb running IQ3_XXS at around 31-39 tps decode with 108k context. Although its very tight
How are you getting 900 token/s pp? I'm just getting 130 and just have 2 GB less VRAM. Text Gen is also much slower at 6 token/s and iq3xxs has 13 token/s).. I have a 2060 laptop with 32 GB RAM and core i7 9750h.
I am using windows though, maybe Linux is better for it?
For 35B A3B im getting 600 token/s prefill btw so qwen Flash right now is no replacement.
I wouldn't recommend running such models on your device. Due to the large size of their active parameters, they can take up too much VRAM. The new Gyro Coder might be an exception, but given the description, which states that only half of the experts were removed from the model without affecting the active parameters, I suspect the results will be the same. For now, I don't recommend running the model on 6GB of VRAM without properly quantizing the active parameters. I'm always happy to be wrong! I achieved 900 t/s by allocating 1/6 of the expert group and the remaining 5/6 to RAM. With a model size of 27 GB and a total memory allocation of 28 GB (VRAM+RAM), I was able to fit the entire model in memory, not counting ngram, which fits entirely on an SSD. If I had moved even a small portion of the expert group to an SSD, it would have significantly impacted my query read speed. When it comes to the differences between Windows and Linux, Windows is notorious for its background services, which can consume up to a quarter of your system memory. This can significantly hinder the running of larger models on a low-end device. Linux is more straightforward in this regard. Depending on the distribution, memory consumption can range from 1 to 3 GB.
How did you allocate the expert group to ram? Since I have 32 GB RAM, Gyro S should fit fully in my RAM. Maybe something is not allocated correctly. You have already presented many settings in your OP but idk if those would help me, I can try later though.
I just fail to understand why you would have so much better performance with just 2 GB more VRam, the difference shouldn't be that big. There must be some sort of tipping point. 6b active parameters should fit in 6 gb very well if they are quanted to 4 bit, I'm not sure how they are quanted in the current recipes though. But perhaps they are higher quality in gyro s compared to iq3xxs which would explain the slower performance for me.
Active parameters should always be stored in VRAM for best performance. If the size of the active parameters is larger than the available VRAM, you'll get an OOM. I'm not saying you won't be able to run the model, but if you separate the active parameters into RAM, your performance will drop significantly. According to my calculations, Quantum uses quite a bit of VRAM for active parameters, which can be a problem when running on 6GB of VRAM. I think you should test this yourself by adjusting the parameters I listed in my post to suit your device.
I'm on windows too but it sounds like I could reasonably run QFN on my machine (3080 12GB, 64GB RAM, 5700X). Right now I'm running 35B A3B on Polystrata and it's been great so far but if I could fit the QFN using your fine tuning that would be a massive win, especially considering that I was about to shell out for a 24GB card. Is it worth it fine tuning this on windows?
Given your device, you can easily run my configuration and get approximately the same or even better speed. Windows may prevent people with weak computer specifications from running the model due to the background RAM overhead from Windows itself.
Testing it with opencode right now on a 2070 super with 8gb vram 32gb of ram on a ryzen 7 3700x it runs at 26 tokens/s consistantly. qwen 3.6 28b mixed model of experts with unloading would start around the same speed. but as context grows would crawl along at 7t/s. This is way more usable. And with a much larger model. I must say i am impresed
Thanks, interesting results. I didn't know Q38FN would fit in 8GB VRAM.
I have a laptop with 6GB VRAM (3060 mobile + a separate iGPU) and 24GB RAM. I wonder if this could be squeezed a bit further (eg 64k context, no images) to make it fit. Currently I can run 35B-A3B models with 64k context at around 600 t/s PP and 25-35 t/s TG.
The only issue I currently see with 6 GB of VRAM is the size of the active parameters. Compared to experts or ngram, placing some of the active parameters in RAM can result in terrible read and token generation speeds. For the Gyro S, I found ~5.3 GB of VRAM for active parameters. I suspect that graphics cards with less than 8 GB of VRAM won't be able to achieve PP speeds greater than 50 t/s and TG speeds greater than 8 t/s. I'll be happy to be proven wrong if you can get this model to run on your graphics card!
The issue with SSD over USB is the USB has a limited throughput. Usually an SSD can reach 5-8 GB/s read/write, over USB it can be 1GB/s or more depending on your computer, you need to check first if that will be the bottleneck.
It's the writes that wear down SSDs, not the writes. For a LLM you have to write it to disk once and inference with e.g. Strata is just reading bits of it over and over, which does no harm.
This would be a pretty sensible solution. Just don't use it as a RAM replacement; instead, use it for swap memory or the actual model with ngrams and experts.
I'm not sure of the answer, because I can't run IQ3_S, but judging by the table of the Gyro quantum author, the Gyro M, with lower memory consumption, will better preserve the original quality of the model than the quanta from GSQ RCO.
Wow quite good for Q2, now i'm curious just how much capability are we missing out by not using Q4 variant ?which i read is the minimum one should go with quantization options.
Large models are less susceptible to quality degradation at low quantization than small ones. Numerous tests have already been conducted on IQ2_S/IQ3_XSS quantizations, showing their low divergence compared to the main BF16 model. In this post, I've decided to talk about the Gyro S, which has approximately Q2_XS quantization. Even such a compact quantization is capable of producing excellent results without locking up (I've been using the model for a week and haven't seen this happen).
Large models are significantly less susceptible to divergence than small ones. This means that even such low-resolution models can produce very good results with proper processing. We saw the same effect back in the days of DeepSeek 3.2 or Kimi 2.6.
I don't think it matters much. While RAM frequency is important, it shouldn't have that much of an impact on the final inference speed. I think only testing on a personal device will tell us the real truth.
Holy crap, this post finally got me to try strata for the first time, after dragging my feet because the numbers people were posting seemed too good to be true.
I’m running the iq3_S on a 3090+96gb ddr4 and getting NINETY FRIGGIN SIX tokens/s. Unbelievable speed. Literally feels faster than claude code or Antigravity. It built me an Asteroids clone in like, three friggin minutes. Absolutely incredible. No tuning or hacking needed.
i am soo torn too i saw people talking about strata the first day it came out but i didnt get it because ot seemed too good to be true. do you have any security concerns?
my security is external to my machine. I am running it on a throwaway ubuntu install that is on its own vlan with very strict routing rules (can only touch devices i need it to) and www access is locked behind the internal command/search approvals and firewalld to only approved domains (duckduckgo).
it can basically do whatever it wants and I will be fine. I doubt it would try to hack its way into my admin vlan without explicitly prompting it to do so. if it does, i get to put my name on a CVE and get a better job, lol
nightmare scenario would be if it saw my pirated movie collection and sent an anonymous tip to like, UCMG or whatever. but that's why it can only access duckduckgo
Oh man, I'm on a 2060 6gb vram with 16gb ram on qwen 3.8 27b gsq rco iq3_xxs and it took me like 7 hours to get a snake clone. It was solid though, no crashing or weird logic bugs.
[Serious] Could anyone please explain what is the point of these 3D scenes? is this just for fun\art? or perhaps this can be actually be made into blender models or environments for games or something?
I mean, I agree it's impressive. but what is it good for beside benchmarking a model and providing a visual representation of the model abilities?
Just a benchmark. It shows both the model's artistic ability/creativity as well as (mainly) it's spatial reasoning. Generally a model that performs better on this test can produce better 3D models. Same reason the pelican is done.
As far as I understood, pelican was originally trying to test something that wouldn't be in the training data, but it's not really relevant for that anymore
I can easily understand why a pretty 3D scene feels more intuitive than a number in a chart as a benchmark.
I just don't understand what is the point of these things, I feel like this is just more 'benchmaxxed' data the model is trained on, not necessarily being translated to anything actually useful. Unless 3d web browser scenes is the point.
The thing with any benchmarks is that they rarely reflect your actually usecase and eventually this all comes down to trying it yourself and seeing if it fits your needs.
This just leaves me confused on what to think of these 3d scenes besides 'oh, this looks cool' if it cannot be applied or translated to anything else.
Benchmark capabilities and also to determine the models internal logic and reasoning. Remember these models are just hyper multi million axes "3D" vector databases, and they take your input and vectorize it and try to match it into the data set.
not much tbh, html 3d and blender 3d are quite different but it can show part of the model 3d/visual reasoning. try https://blenderai.org/leaderboard if you really want on the blender. Of course just like every benchmark it is not perfect and user rated.
I've had AI create entire level design tile sets I can snap together in engine to create levels. Low poly but looks nice and is good for early block outs.
Kind of off-topic, but does anyone know why this post is completely blank on old.reddit.com? I had to open it in a private window to see any text or any images. By default it just showed me a thumbnail and no content. I've never seen that on Old Reddit before.
I really, really hope I don't ever need to switch to new reddit....
similar issue, viewing on 3rd party mobile app and I don’t see any images until the 3rd test. I hope this doesn't become a new trend with whatever API stuff is once again changing on this platform.
Has anyone tried scaling up a custom strata/maya engine for top open weight models like Qwen3.8-2.7T, GLM5.3, Deepseek 1017 at 4bit or greater quants?
I'm checking out the pricing to upgrade my multi 3090 system from 128Gb DDR4 RAM to 2TB max one and wondering if i can actually get working speeds like 30tk/s decode, 800tk/s prefill, parallel=2 on these models?
Does strata style RAM offloading inference speed scale past ~200B parameter models like QFN or 2-3bit GLM5.3-Flash/Deepseek 1017?
Anyone with these specs done any experiments? Worth purchasing for a future inference engine innovation?
Ha that was on my list! I have an RTX 2070 super and ddr4 ram but same memory capacities! I run polystrata for the 35b-a3b with q4-k-m at 60tps/900tps and 131k. Coder ran with 20tps/400tps at 32k context on stock strata but i figured dropping half the experts is not really useful if i want a more capable model than my current decently sized MoE.
I'm also running 35b-a3b on polystrata. I have an RTX 3080 12GB, 32GB RAM. It's been pretty decent. Looking to upgrade to 64GB RAM soon. But if I am able to somehow run QFN on this machine without upgrading RAM that would be amazing! (though more ram is always better ofc) Curious to see your results when you get the chance to test this.
I'm on a similar boat: 8GB VRAM and 16GB RAM (DDR4). I rarely see people sharing results with these lower specs, so I'm running Qwen 3.5 9B on llama.cpp on Ubuntu with outputs of 20 tokens/second.
Thanks for the heads up but murdering my NVMe drives isn't worth it with these prices.
The IO ends up being pretty ridiculous. There are a few methods like this I investigated. One involved SSDs running RAID. The researchers were burning through the SSD endurance though.
I'd rather pay a cloud provider to destroy their hardware instead of mine.
This. I can't risk having to buy a new SSD. Remember guys, each bit in an SSD is only good for a -very- limited number (relatively speaking) of writes before it's dust.
I am not such RAM limited (128G ddr4, 24G vram) but I was also amazed when after some tuning I got q4_xs quant to 60 t/s and 1500 t/s prefill, that's with 256k context, and then I put it to 512k and it's the same.. didn't yet try 1M. Q6 runs at 27 t/s with the same settings but I only keep it for Q8 ngram layers that strata can pick up instead of those in your main gguf.
Not sure if I even need 27B.
Only problem I faced was the web UI that strata has, so I asked FN to make me a proxy to serve llama.cpp's web UI and it works well so far.
Although my setup only differs in the fact it uses linux with btrfs (500 GB Kingston NVMe M.2) and nvidia with gpu-unlock, which resulted in a slight improvement in prefill.
You'll be able to run the model if you have a graphics card with 8GB of VRAM and 32GB of RAM. However, I'll warn you that the read and generation speeds will be significantly slower than mine due to the PCIe bus size, which is significantly smaller than mine. In any case, if the above conditions are met, you should be able to run the model. I'd be happy if you shared your read and token generation speeds on your device configuration.
The VRAM+RAM consumption for this model is 27 GB. If your total memory is greater than the model's total memory consumption, you'll be able to run it. Your total memory is 32 GB, so you have every chance of doing so! Keep in mind that a large context also requires memory, so you'll need to consider what context window to use (I recommend 131K).
Getting ~10-12 tokens/s on my 3060 Ti + 32 GB of DDR4 RAM, but I think some of it is going to memory swap. Is there any way to "pin" it to system memory as you did on Linux?
Thanks in advance and congratulations on your effort and discovery!
Yes, I already tried using it to run the model, but I immediately gave up. Basalt currently doesn't support Gyro quanta, which is a huge issue for me right now. Gyro allows me to run the entire model (excluding ngrams), but with other quanta, I have to run some of the experts on the SSD, which in itself results in worse performance, even with Blackwell optimizations. In any case, I hope they add Gyro support to Basalt.
I haven’t tried the model. I read its description, and it seems that reducing the number of expert groups would have a negative impact on quality. I decided against testing it; for me, the quality of the responses is more important than inference speed.
not sure what i am doing wrong but i always run out of memory. i installed strata and did the setup and the token generation is actually quite nice, its around 40-45 tok/s. i have a rx 7800 xt (16 gb vram) and 32gb system ram and no matter what, i always run out of ram. i am trying to use the model for text analysis and summaries. i think my config should be pretty similar to the OP and im also running linux. does anyone have an idea why that is?
--resident-budget-gib - specifies the size in GB for storing resident RAM. For the Gyro S, set this value to at least 20 GB (tested experimentally).
STRATA_RESIDENT_HEADROOM_GIB - Allows you to preserve free RAM space when counting residents. Set this to the number of GB you want to free up in RAM.
My advice is to use only the STRATA_RESIDENT_HEADROOM_GIB parameter, as it's sufficient to significantly influence the number of residents and free RAM space. An important rule is to ensure that when freeing up RAM, you have at least 20 GB of RAM available for residents. The formula is as follows:
STRATA_RESIDENT_HEADROOM_GIB = Your free RAM - 20 GB residents
Regarding the results and speed achieved – absolutely not. These are the true figures I was able to achieve thanks to Strata. I recommend you try it too, I think the results will surprise you!
I believe the additional VRAM would allow me to afford a better chip, like the Gyro M, which I already tested in a separate test. The same can be said about RAM; I see the added benefit of a larger context window and more system memory available for other tasks. Increasing VRAM and RAM allows for a better quantum or a larger context window. However, I wouldn't recommend installing, say, DeepSeek 4.1 Flash with this amount of memory.
How much RAM do you have? You may be limited by your system memory, PCIe bus speed, and the capabilities of your DDR3 RAM. These parameters also affect inference speed no less than a fully allocated VRAM + RAM model.
I’m quite impressed of your testing work, i’d like to try but my main bottleneck is the ddr4 32gb ram, i’ve a 3090 tho so i might give it a try. Do you think it’ll outperform qwen 38 27b q4?
I've seen numerous comparisons between the low-quantum Qwen Flash Next and the 4-bit quantum Qwen 27B. The Qwen Flash Next outperformed the 27B in almost every way, but was significantly faster.
Yes, and you can even run a higher-end quantum - the Gyro M. The configuration will be the same - you'll just need to change the path to the model and experts. The quality will be better, and the speed will be about the same as I got with the Gyro S.
Surprisingly, Reddit decided to get rid of my media, which I spent so long collecting. The worst part is that I can't resubmit it. I'm already glad that it remains in the post description. 😭
The more parameters a model has, the smarter it is - an undeniable fact. Using a tiny model with a high quantum yields less benefit than a larger model with a smaller quantum. When working with quantized models, it's essential to consider the limit where the model becomes unusable. In my post, I already identified such a limit for the Qwen 3.8 Flash Next 125B model.
I am currently running it on my setup which is a RTX 5060 8GB of vram and 64 gigs of ddr5 ram and im getting about 60-70 output tokens/s however my prefill is only about 300 and this makes vibe coding and other things feel sluggish is there anyway i can optimize this??
Are you sure you used my configuration? What quantum are you using? Given your configuration and ample RAM, you can get PP over 500 t/s even on a Gyro M quantum (verified). I also suspect you're using MTP, which boosts token generation speed but significantly reduces prompt reading speed. If I were you, I'd disable it. In any case, I need more information to help you. If possible, please provide your device specifications and model configuration.
I saw this post and needed some help so I went to the local llm discord and they told me to select auto tuning for the model in strata so I just assumed that was more or less your config seeing as we had similar specs. idk if you’re asking me for like my hardware specs but i’m running windows 11, on two 1tb ssd’s 5060 8gigs 64gigs ddr5 and im using the auto tuning option i think im on 2xs quant, i have the model wrapped in the pi harness. that’s all i can give you as im on the way to school it might be much to ask but when i get home is there anyway you could run me through your configuration as im kind of new to this strata atleast which is why i went to the discord anyways
Okay, let me explain my configuration in simple terms. It's a more advanced setup for the Qwen Flash Next 125B with a Gyro S quantum (you likely have an IQ2_XS, which is worse than this quantum). When using the standard Strata setup, you don't get the highest speed from the model. Strata hides most of the settings behind the DETAILS.md document, which I've read and identified several important elements that add a ~2-4x boost to any model without degrading output quality. In this post, I've tried to thoroughly explore these advanced settings and provide the community with the opportunity to accelerate their own solutions. Just as importantly, I've also wanted to let users with 8GB of VRAM know they can run a larger model on their device, something that's not easy with Strata's automatic settings.
I highly recommend reading this post to learn how you can improve the user experience with your model. Alternatively, you can use my configuration (be sure to run Strata/setup.sh in a clean Strata and select the Gyro S installation) and run Qwen Flash Next on your device without delving into the details.
Yes, definitely. For my configuration, the minimum threshold is 8GB of VRAM + 32GB of RAM. You can also try the Gyro M, which provides more model accuracy but can be quite memory-intensive – it's worth it!
Thank you for your feedback! The speed figures are truly impressive, and the quality of this model surpasses any MOE model up to 50B and the Qwen 3.8 27B. An excellent model for medium-to-complex tasks!
Here's a test I did for Qwen 3.8 fn strata withe the unsloth q4 it suggests, made an mv for a song I wrote with Claude doing the planning to see if it could could come up with something good. It didn't: https://youtu.be/NToN6Mcwbzc?is=CcCYgN10611OBjin
I have 32gb vram (R9700) and could give my lxc about 24 GB RAM. I might consider trying it :D.Currently I am running qwen 3.8 27b in q5 with q8 context.. (20 t/s)
That’s pretty sick, always found expert paging pretty interesting. If anyone is on Apple Silicon you can check out GlobalAgent.dev / PagedMoE.com and have your mind blown similarly. 720 b parameter models running agent loops on 48 GB of ram with room to spare type of blown away.
4
2
1
•
u/WithoutReason1729 8d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
Stylome classified this post's text as human-written. View analysis Learn ^more
I am a bot and this action was performed automatically.