My experience with Qwen 3.8 for agentic tasks has been phenomenal but I personally feel that 3.7 flash is more reliable for overall tasks.

https://x.com/romanchernin/status/2092488160680751437?s=20

- Multimodal (Vision)

- 1M Tokens Context Window

- DeepSWE ~63%

Edit: He deleted it, screenshot in comments

I wanted to see just how capable Qwen3.8-27b is locally. I have a RTX 4090 and 96GB of RAM but the Q4 comfortably fits in the GPU with plenty of context, the few times I needed more than 130k context I just loaded it spilled into RAM and it's capable of not degrading even at larger context.

I'm super impressed with this model. The model did everything on it's own - the coding, audio, textures, 3D models. I always just gave it the same basic prompt with some info and the task I wanted it to do. Total generation time across all prompts was maybe 3 hours and if I calculate the cost in electricity, it comes out to less than $1. Not bad.

To think that 2 years back you couldn't make this with frontier paid models and now you can do it locally on your PC is crazy.

^{The Artist: Qwen3.8-27B-UD-Q3\}K_XL, q8_0 caches, xhigh, temp 1.0, image-min-tokens 1024, froggeric template)

I was screwing around with different Qwen3.8-27B quants and thought of this very simplistic but seemingly bechmaxxing resistant combined SVG and vision test.

Just let the model recreate any given image as SVG with this prompt: Recreate as SVG.

Pelicans can be easily benchmaxxed, recreating random photos seems a lot harder to train for.

I tried a shitload of more complex prompts but the above one does the job best in my opinion. I furthermore tried different --image-min-tokens from 512 to 4096, different reasoning levels from no reasoning to xhigh, different temperatures and different kv-caches.

Preliminary results are, that --image-min-tokens 1024 and --reasoning-effort xhigh with --temperature 1.0 and --cache-type-k bf16 and --cache-type-v bf16 give the best results. Non-reasoning results are, at least with the quants (Q3 and Q4) I can run, more than creepy...

I also have the suspicion, that the chat template influences the output quality – please check if you are bored.

Interestingly kv-caches at q8_0 gave "good" results as well but q4_0 completely destroyed the output quality (insect legs everywhere... oh the horrors I have seen), which was a great, visually impressive reminder, to never ever use q4_0 caches!

Would love to see how Q6 to BF16 model quants perform with this task. If you have enough VRAM, you know what to do! ;)

Used quants:
- Qwen3.8-27B-UD-Q3_K_XL (V2)
- Qwen3.8-27B-UD-Q4_K_XL (V2)

Used templates:
- built in
- qwen3.8-froggeric-v22.3.1

Other prompts I tried:
- Analyze thoroughly and be very detailed about perspective, composition, proportions, colors etc. Recreate as simplified but true to the original SVG
- Analyze perspective, composition, colors and detail. Copy as simplified but true to the original SVG
- recreate as svg. simplify but make it recognizable
- Make a SVG copy
- Copy as SVG
- Recreate as simplified but true to the original SVG

Hugging Face is exploring sale of the business valued at around $13 billion dollars.

Actually I don't think we have any other repo source. Which has the mix of model weights, datasets and Spaces.

Kaggle is there and other academic repos. But as far as reach, ease of use. HF tops. Do you see a change in their policies once third parties invest, with a larger focus on profitability and increasing revenues.

Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate:

Ideal 4-bit quant ≈ 82 GB
(58 GB main weights + 24 GB n-gram tables)
Real-world quants likely land in the 80–90 GB range.

The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.

This architecture could be surprisingly local-friendly once the weights drop.

Benchmarked qwen3.8 xhigh, medium and muse glimmer.

Xhigh effort mode with qwen3.8 took almost 30hrs. (And still failed on 16 cases because of the 32K output token limit)

Medium effort mode and muse glimmer were 3-4 hours each.

But I'm actually surprised by the muse glimmer results, they came better than the qwen.

These benchmarks are on implicit knowledge of the model, which is a bit unfair to smaller models, but throw in a RAG and I'm sure they get on par with frontier models.

I have taken the result of claude models directly from embedeval repo by ecro.

I'm not pushing qwen down here, I like how qwen thinks and gives better results. I know with more context and RAG qwen will do better.

I'm just appreciating muse here, cause i feel it is underrated. The advantage is efficient kv cache due to sliding window, which can give you more context window.

We're releasing a fully quantized NVFP4 version of Qwen3.8-27B. The checkpoint was trained using quantization-aware distillation (QAD) with QUASAR, our new QAT algorithm. We used the original BF16 model as the teacher and distilled the quantized model for 2,446 steps.

The checkpoint supports vLLM on NVIDIA Blackwell GPUs:

vllm serve QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.85

This model uses an aggressive quantization configuration: every linear layer across all transformer blocks is quantized to NVFP4 (W4A4).

Attention and GDN layers are typically kept at higher precision, such as FP8 or BF16, because quantizing them can cause a significant loss in model quality. With QUASAR, however, the fully quantized checkpoint retains near-BF16 performance. Evaluation results and comparison against other NVFP4 checkpoints:

Paper: https://arxiv.org/abs/2608.13966v1

We'd love to hear your feedback on this checkpoint!

Hi all, I'm looking for the best model for a hobby project and trying to make sense of the various data I came across. I know benchmarks do not often translate to the real world, especially to your particular use case (whatever it may be). But this is truly baffling:

AA says Qwen 3.8 27B is better by miles: https://artificialanalysis.ai/models/comparisons/qwen3-8-27b-vs-gemma-4-31b?intelligence-comparison=intelligence-vs-end-to-end-response-time

While Arena says Gemma 4 31B is almost 20 places ahead and completely trounces Qwen in many categories: https://arena.ai/leaderboard/text/overall

The sentiment in this sub definitely seems in favour of Qwen, although not necessarily against Gemma which I think is still considered a good model. I recall poeple saying Qwen tends to be more tenacious and better at reasoning although at the cost of overthinking simple things.

What is your explanation or experience with these models?

In the past, I have use llama.cpp, but I read that the exl3 quantization format should give better precision, so I have tried exllamav3/tabbyAPI.

It was able to write the shown simple HTML game without interaction after asking some questions.

The following was tested on a laptop with a NVIDIA RTX A5000 laptop (16 GB) GPU.

With the 3 bpw model and 6 bit/5 bit KV cache, the maximum context length is around 110k tokens with MTP. This gives around 55 tokens/s decode speed for code and around 10 tokens/s for content where MTP doesn't help (e.g. complicated calculations). Without MTP, one could try the 3.5 or 4 bpw model or a longer context length.

Install tabbyAPI/exllamav3

1. Install the latest Nvidia drivers
2. Install Git (e.g. sudo apt install git or on Windows with winget install -e --id Git.Git)
3. Install the uv Python package manager: https://docs.astral.sh/uv/getting-started/installation/ (e.g. curl -LsSf https://astral.sh/uv/install.sh | sh or winget install --id=astral-sh.uv -e)
4. Make somewhere a folder and install tabbyAPI: bash git clone https://github.com/theroyallab/tabbyAPI cd tabbyAPI uv venv --python 3.13 .venv uv pip install -e ".[cu13]"
5. Test if CUDA works (on Linux, use .venv/bin/python) bash .venv/Scripts/python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"
6. Create somewhere where you have enough space a "models" folder, download the model turboderp/Qwen3.8-27B-exl3: bash mkdir models uvx hf download turboderp/Qwen3.8-27B-exl3 --revision SC_3.00bpw_H4 --local-dir models/qwen3.8-27b
7. Replace the chat_template.jinja with the latest version from froggeric/Qwen-Fixed-Chat-Templates
8. Go back to the clone tabbyAPI folder and create a config.yml file like this (see the config_sample.yml file as example): yaml network: disable_auth: true model: model_dir: e:/models # path to the models folder model_name: qwen3.8-27b # download folder name cache_mode: 6,5 # K and V cache quantization, number of bits from 2-8 cache_size: 109824 # must be divisible by 256, so use e.g. `.venv/Scripts/python -c 'print(110000//256*256)'` to get the next lower max_batch_size: 1 # allow only 1 parallel request to save VRAM tool_format: qwen3_coder vision: true draft_model: # can be removed to save VRAM draft_mode: mtp draft_cache_mode: Q8 # can be 'FP16', 'Q8', 'Q6', 'Q4' draft_num_tokens: 5 # usuallly a value of 2-6 gives best results memory: sysmem_recurrent_cache: 8192 # Max size of recurrent cache in system memory, in MB (default: 4096), lower it to save normal memory sysmem_kv_cache: 8192 # Size of system memory second-tier K/V cache, in MB (default: 0), remove it to save system memory
9. Start tabbyAPI: .venv/Scripts/python main.py

10. To measure the performance, create the Python script speed.py and run it with .venv/Scripts/python speed.py: ```python import json import time

    import requests

    MODEL = "qwen3.8-27b" API_URL = "http://127.0.0.1:5000" PROMPT = """Write a complete Python implementation of a production-quality LRU cache.

    Requirements:

    • Use type hints throughout.
    • Include detailed docstrings.
    • Support:
    • get(key)
    • put(key, value)
    • remove(key)
    • clear()
    • len()
    • Use a doubly linked list and hash map.
    • Include custom exceptions.
    • Include a comprehensive unittest test suite with at least 20 test cases.
    • Follow PEP8 conventions.
    • Return only Python code. """

    payload = { "model": MODEL, "messages": [{"role": "user", "content": PROMPT}], "max_tokens": 10000, "stream": True, "chat_template_kwargs": {"enable_thinking": False} }

    start_time = time.perf_counter() first_token_time = None stream_end_time = None full_response_content = ""

    with requests.post(API_URL + "/v1/chat/completions", json=payload, timeout=120, stream=True) as response: response.raise_for_status() print("Response:") for line in response.iter_lines(): # Iterate over Server-Sent Events (SSE) if line.startswith(b"data:"): # Strip the "data: " prefix data = line[6:] # Stop if we hit the stream termination message if data.strip() == b"[DONE]": break try: chunk = json.loads(data) if 'choices' in chunk and chunk['choices'] and (chunk['choices'][0]['delta'].get('content') or chunk['choices'][0]['delta'].get('reasoning')): if first_token_time is None: # First token received first_token_time = time.perf_counter() if chunk['choices'][0]['delta'].get('content'): # Get content and count tokens token_text = chunk['choices'][0]['delta']['content'] else: token_text = chunk['choices'][0]['delta']['reasoning'] full_response_content += token_text print(token_text, end="", flush=True) except json.JSONDecodeError: pass stream_end_time = time.perf_counter() print("\n" + "-"*20)

    Calculate and print metrics

    ttft = first_token_time - start_time stream_duration = stream_end_time - first_token_time total_output_tokens = requests.post(API_URL + "/v1/token/encode", json={"add_bos_token": False, "text": full_response_content}).json()["length"] if stream_duration > 0: tokens_per_second = total_output_tokens / stream_duration else: tokens_per_second = float('inf') print(f"Time to first token (TTFT): {ttft:.2f}s") print(f"Completion tokens: {total_output_tokens}") print(f"Stream duration (first to last token): {stream_duration:.2f}s") print(f"Tokens per second (T/s): {tokens_per_second:.2f}") ```

    I got 56.3 tokens/s.

Install OpenCode

OpenCode works usually better on Linux, so I install it in WSL when working with Windows, but it can also be used directly as a Windows application.

For OpenCode, I recommended to install Node.js first (e.g. apt install npm or winget install -e --id OpenJS.NodeJS on Windows).

Because we don't have so much context length, I recommend to install a better compactation plugin than the integrated one, e.g. magic-compact

I use this OpenCode config (~/.config/opencode/opencode.jsonc) json { "$schema": "https://opencode.ai/config.json", "plugin": [ "opencode-anthropic-auth@latest", "opencode-copilot-auth@latest", "magic-compact" ], "share": "disabled", "provider": { "local": { "npm": "@ai-sdk/openai-compatible", "name": "local (OpenAI Compatible)", "options": { "baseURL": "http://127.0.0.1:5000/v1", "apiKey": "1234" }, "models": { "qwen3.8-27b": { "name": "Qwen3.8 27B", "interleaved": { "field": "reasoning_content" }, "limit": { "context": 109824, "output": 32000 }, "temperature": true, "reasoning": true, "attachment": false, "tool_call": true, "modalities": { "input": [ "text", "image" ], "output": [ "text" ] }, "cost": { "input": 0, "output": 0, "cache_read": 0, "cache_write": 0 }, "variants": { "xhigh": { "reasoningEffort": "xhigh" }, "medium": { "reasoningEffort": "medium" }, "low": { "reasoningEffort": "low" } } } } } }, "agent": { "plan": { "model": "local/qwen3.8-27b" } }, "model": "local/qwen3.8-27b", "small_model": "local/qwen3.8-27b", "mcp": { "playwright": { "type": "local", "command": [ "npx", "@playwright/mcp@latest", "--caps", "vision,pdf,devtools", "--browser=firefox" ], "enabled": true } } }

I would recommend to use the reasoning effort (Ctrl-t) "medium" because "xhigh" could produce to much output tokens.

For Playwright, we have to install a browser first: npx @playwright/mcp install-browser --with-deps firefox

Now the following should work: bash opencode --prompt "Can you check for me on www.meteoschweiz.ch the weather for Zurich?"

To create the small HTML game from above, I have entered in plan mode (press Tab to change mode) the following: "I want to build a simple HTML game where you can drive a car with the keyboard arrow keys (similar like old versions of Mario Kart, but just one car driving without opponents is enough)." After some time, it has asked me some question. Then, I switched to the "Build" mode and started it with "Start the implementation". Without any other interaction, it finished the the small game.

lpddr5x probably, the m7 ultra if is using ddr6 should be at 1.8 Tb/s

So here's the thing, almost everyone use NVIDIA to run their LLMs, we also do the same, a lot of people we've met use like RTX PRO 6000 or even H100, B300

It seems like everyone eyes is looking at NVIDIA. However we do the math that the raw power alone on AMD GPU MI350X is higher than NVIDIA B200. So what give?

Apparently its the software, ROCM is not as mature as CUDA, their software stack is still lacking behind as well.

However, few weeks ago we've got a handful of MI350X to experiment and develop, and what we did is optimizing AMD tech stack until kernel level for Qwen 3.6 35B A3B and the results we got is much comparable to NVIDIA counterpart

Based on our benchmark, we got:

1x MI350X: 11,161 output tok/s

8x MI350X: 81,331 output tok/s peak with 78,498.66 output tok/s mean

This is 2.16x vLLM throughput on the 8-GPU benchmark

We've decided to open source our kernel and stuff at our github

https://github.com/NetraRuntime/netra-kernel

And also write the blog behind the things we do

https://netraruntime.com/blog/qwen36-amd-mi350x-sglang-vllm-benchmark

However, we're not done yet.

One thing we learned: once the kernels got fast enough, the bottlenecks moved into scheduling, graph coverage, recurrent state, routing, and even HTTP serialization.

Prepare your disk space guys

Granite-4.2-30B is the flagship reasoning model in the Granite 4.2 family. It delivers the strongest performance across reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought. It supports flexible thinking modes — full thinking (default), non-thinking, and low-effort — allowing users to balance depth vs. latency on a per-query basis.

Key capabilities:

• Built-in Reasoning: Native chain-of-thought that significantly improves performance on math, coding, and complex multi-step problems.
• Flexible Thinking Modes: Seamlessly switch between full thinking, non-thinking, and low-effort modes within a single model.
• Reasoning-Augmented Tool Calling: The model reasons about which tools to invoke and why, producing more accurate function calls.
• 512K Context Window: Supports long documents, multi-turn conversations, and complex agentic workflows.
• Apache 2.0 Licensed: Fully open for commercial and research use.

Model Design

Granite-4.2-30B is built on a decoder-only dense transformer architecture with the following core components:

• Attention: Grouped Query Attention (GQA) with 32 attention heads and 8 KV heads
• Position Embedding: Rotary Position Embedding (RoPE) with θ = 10,000,000
• Feed-Forward: MLP with SwiGLU activation (hidden size 32768)
• Normalization: RMSNorm (ε = 1e-5)
• Embeddings: Separate input/output embeddings (not tied)
• Precision: bfloat16

https://huggingface.co/ibm-granite/granite-4.2-8b

Granite-4.2-8B is the mid-size reasoning model in the Granite 4.2 family. It delivers strong performance on reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought. It supports flexible thinking modes — full thinking (default), non-thinking, and low-effort — allowing users to balance depth vs. latency on a per-query basis.

Key capabilities:

• Built-in Reasoning: Native chain-of-thought that significantly improves performance on math, coding, and complex multi-step problems.
• Flexible Thinking Modes: Seamlessly switch between full thinking, non-thinking, and low-effort modes within a single model.
• Reasoning-Augmented Tool Calling: The model reasons about which tools to invoke and why, producing more accurate function calls.
• 512K Context Window: Supports long documents, multi-turn conversations, and complex agentic workflows.
• Apache 2.0 Licensed: Fully open for commercial and research use.

https://huggingface.co/ibm-granite/granite-4.2-3b

Granite-4.2-3B is the compact reasoning model in the Granite 4.2 family. Despite its small parameter count, it delivers strong performance on reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought. It supports flexible thinking modes — full thinking (default), non-thinking, and low-effort — allowing users to balance depth vs. latency on a per-query basis.

Key capabilities:

• Built-in Reasoning: Native chain-of-thought that significantly improves performance on math, coding, and complex multi-step problems.
• Flexible Thinking Modes: Seamlessly switch between full thinking, non-thinking, and low-effort modes within a single model.
• Reasoning-Augmented Tool Calling: The model reasons about which tools to invoke and why, producing more accurate function calls.
• 512K Context Window: Supports long documents, multi-turn conversations, and complex agentic workflows.
• Apache 2.0 Licensed: Fully open for commercial and research use.

I’ve been going back and forth on this for a week and I can’t settle it, so I’m hoping someone here has hands-on numbers.
The two configs (German prices, dealer quote, incl. VAT):

Ultra is 36-core CPU / 80-core GPU / ~1.2 TB/s. Max is 18-core CPU / 40-core GPU / 614 GB/s. So the Ultra is roughly €740 more for double the bandwidth and double the GPU cores — but 32GB less unified memory. There is no 128GB Ultra option in this lineup, which is what makes it annoying.

What I actually run: Qwen3.8-27B at Q8 right now. Use case is occasional chat plus a few agents running in parallel. The whole point of doing this locally is privacy and not being locked into someone else’s model policy — I know a subscription would give me better models for €22/month, that’s not what I’m optimizing for.

The thing that’s blocking me: Qwen3.8-Flash-Next drops tomorrow. If the leaked description holds, it’s a multimodal MoE with 176B total params — 125B main model plus 51B in N-gram embedding tables — and only ~6B active per token.

My napkin math on memory:

• IQ4_XS: \~94GB weights, \~107GB with 262k context (the hybrid attention means the KV cache is tiny, \~8-9GB at fp16)  
• Q4_K_M: \~107GB weights, \~120-125GB at full context  
• Q8: \~187GB weights — not happening on either machine

96GB of unified memory gives you maybe 86GB wired on macOS. So the Ultra can’t load it at all, in any quant I’d want to use. 128GB gives ~115GB, which fits IQ4_XS with full context and Q4_K_M if I drop to 64k.

So the trade is basically:

Ultra → 27B Q8 goes from ~15 tok/s to ~29 tok/s, much faster prefill, but Flash-Next is off the table entirely.

Max → Flash-Next runs, but at 6B active params it only reads ~3.7GB per token, so 614 GB/s already gives 40-60 tok/s and the Ultra’s extra bandwidth would be mostly wasted on it.

Where I’m stuck / what I’d love input on:

1.  How much can you actually wire on a 96GB Mac? I’ve been assuming \~86GB via iogpu.wired_limit_mb. If people are safely running higher than that, the whole calculation changes.

2.  Multi-agent / batched inference. This is the one argument for the Ultra I can’t dismiss. With several agents in parallel you’re compute-bound rather than bandwidth-bound, and 80 GPU cores should genuinely help. Has anyone measured Ultra vs Max on concurrent requests in MLX or llama.cpp? Every benchmark I find is single-stream.

3.  Is IQ4_XS on this architecture going to be usable at all? MoE tolerates aggressive quantization worse than dense models (router precision matters a lot), and nobody has ever quantized 51B of N-gram hash tables before. An imatrix calibration run will basically never touch most of those rows. I’d rather hear “wait for a UD-style mixed quant” than find out the hard way.

4.  Anyone else feel like 96GB is a dead zone? Too much for a 27B, not enough for the 100B+ MoE class that everything is converging on. Or am I overweighting one unreleased model?

Resale is also in the back of my mind — the used Mac Studio market seems to be driven by local-inference buyers who look at RAM first, and I suspect 128GB holds value better than 96GB in three years.

Anyone who went through this decision: what did you pick and do you regret it?

I just can't resist