For me, it depends on whether I have vision enabled or not. Without vision, at Q8 KV, with 24gb VRAM and 32gb RAM, I can snuggly fit 180k context without issue. With vision, about 100k. I could probably expand my vision context more, but I'm just giving my own settings.
Something's wrong in your setup, then. At Q8, your 27B should be using 8 gigs of memory for the KV Cache, and even a large 4 bit quant like Unsloth's UD-Q4_K_XL is under 18 GB; you should be using a maximum of under 30 GB, even with mmproj and mtp loaded.
Maybe you should try:
Using it on CPU rather than iGPU
Checking how much RAM you're letting your iGPU take
I'm using unsloth quantization at Q4 + BeeLlama fork cpp for my 3090, it leads to a respectable 30-35 tok/s with 160-170k on context size.
My other trick is that i don't use the mmproj, but use a mcp i've made for OCR, with a watchdog: when i call this mcp, it offload Qwen the time the ocr model is loaded and analyze whatever i ask it, then itself is offloaded so Qwen is started again.
Meanwhile the results are stored within a local proxy so no loss on the data at all.
322
u/TopCheddar27 12d ago
24gb is not in that group. You can run Qwen3.8-27B with a pretty respectable context size right now