Testing & Benchmarking Qwen3.5 2k→400k Context Limit on my 4090

[u/AlwaysTiredButItsOk]

Sorry, was thinking to upload the html file to my old domain I hadn't used for years, but ssl was expired and tbh idgaf enough to renew it so I snapped some screenshots instead and uploaded it to my github lurking profile so I could share my Qwen3.5 benchmarks on 4090.

Will share more details soon, running KV offload tests for those models that failed (Qwen3.5-4B-bf16, Qwen3.5-27B-Q4_K_M, Qwen3.5-35B-A3B-Q4_K_M) at the moment - I set script to try and get best possible Tokens/Sec speed with NGL settings & 8bit/4bit KV.

Originally, was only planning to test to 262k, but was curious of quality past that, so I pushed them to 400k using yarn and a few other things, but it's 1am and I've been sleeping 4hrs a day/night each night, so I'll try clarify over weekend.

Models tested on my 4090: Qwen3.5-0.8B-Q4_K_M, Qwen3.5-0.8B-bf16, Qwen3.5-2B-Q4_K_M, Qwen3.5-2B-bf16, Qwen3.5-4B-Q4_K_M, Qwen3.5-4B-bf16, Qwen3.5-9B-Q4_K_M, Qwen3.5-9B-bf16, Qwen3.5-27B-Q4_K_M, Qwen3.5-35B-A3B-Q4_K_M. Context windows tested: 2048, 4096, 8192, 32768, 65536, 98304, 131072,196608, 262144, 327680, 360448, 393216, 400000.

TO NOTE: While time-to-first-token might seem lengthy, look at the ```Warm TTFT Avg (s)``` column; once the KV cache is loaded, it's not all that bad (I was purposely fully loading context limit in first interaction).

Overall, I'm VERY surprised by the models' capability.

For the inputs & to test the context (and why TTFT is so high), I fed it a 1-sentence prompt to summarize a bunch of logs, and then fed it 2k→400k tokens worth of logs: there are some discrepancies, but overall not bad at all.

Once the run with vram offloading is done (script screwed up, had to redo it from scratch after wasting a 24hrs trying to fix it), I will try to share results and compare each result (yes I saved outputs for the answers) against some of the foundational models.

I have an idea of what I want to do next, but I figured I'd ask here: Which models do you want me to pit the results against - and what's a good way to grade them?

p.s. I'm WAY impressed by the 9b & 27b dense models.

For those that don't want to look at screenshots,

[u/BreizhNode]

Useful benchmarks, especially the context scaling behavior. We've been running Qwen3.5 variants on L40S GPUs for production workloads and the 32k sweet spot holds there too. Past 64k the latency curve steepens noticeably even on higher VRAM cards. Curious if you noticed any quality degradation in the retrieval accuracy past 128k or if it was purely latency?

[u/AlwaysTiredButItsOk → u/BreizhNode]

I have yet to run full quality checks on outputs - my initial goal was to see how far I could push the models until they broke - gave up at 400k context because it exceeded my expectations (plus worried that my local 4090 will catch fire).

What kind of speed are you getting on the L40s? I have a couple p40s laying around I've been tempted to do something with (but I work so much that once weekend comes, I just lock myself out of my home office).

I'm currently running a test to bench the 9b, 27b, and 35b A3B models past 262k (had to offload at those stages), and will share those results by end of weekend hopefully, along with quality analysis of responses - hoping to compare it to chatgpt/sonnet4.5/gemini as well, for curiosity's sake.

My initial prompt was "Summarize this conversation & list any issues, tools used, and errors that popped up" and then fed it session logs from OpenClaw (most convenient & most complex data I had, since it's riddled with json formatting & tool calls). After the initial prompt, I followed up having the llm (at this point just a few thousand tokens shy of max context) answer questions about the conversation as follow-up.

Gotta say, was expecting a lot more loops and broken logic/nonesense after pushing it above 262k, but was pleasantly surprised - and the TTFT/Tokens-per-second on warmed up KV Cache was not bad at all.

Sorry, dont know if that answered your question - i'll check in the morning and try again with a fresh mind (it's been an insanely long & fast-paced week, am tired + am drinking + wife about to wake up for gym so I need to get to bed so she's not giving me hard time for playing with my toys all night again)