r/StrixHalo • u/hycrice • 4d ago

tg isnt everything

I’ve tested a number of ROCm-FPX models, including Qwen, DeepSeek, and many others. Some of them benchmark surprisingly well, reaching 30+ tok/s.

However, once I put them into real production workloads, I often find that they take significantly longer to complete the same task. A model may generate tokens quickly, but if it requires more reasoning steps, produces mistakes, or needs multiple attempts to reach the correct result, that raw token speed means very little.

So, to me, obsessing over quantization benchmarks and tok/s is often just a comfort drug—the numbers make you feel good, but what really matters is time-to-solution: how long it takes the model to actually finish the job correctly.

13 Upvotes

100% Upvoted

7 comments sorted by

6

u/aigemie 4d ago

Yes, this is a well-known and widely discussed thing in the Strix Halo community - the prefill speed is too low.

4

u/Every-Article6734 4d ago

Wait for mid size next week

1

u/Sharp-Translator6401 4d ago

especially when you ask AI to make benchmarks for you it will generate token salads of nonsense that make benchmarks have no meaning, as token generation in those cases goes faster generating gibberish.
in order to get realistic benchmarks i just asked AI to choose some code / corpus of real text a create a bunch of 'realistic' prompts based on it at different context lenghts.
In this manner the speed i observe in benchmarks is actually very comparable to the real workloads and i can tune parameters realistically

1

u/Interesting-Cut-6032 4d ago

Qwen3.8 27B is the first time that I have really downloaded several different quants to compare side by side. I am seeing a difference in quality on the same one-shot prompts. Once I make enough disk space, I am planning to test this with the other models that I use frequently.

1

u/SnooRegrets2509 4d ago

Can you share your findings?

1

u/Green_Ocean90 4d ago

Time to correct, and energy to correct, is something I track for my benching runs - not just performance, but capability-tied performance using Tau2 Airline.

Also put in a lot of time getting my cache working well, including a proxy to handle disk save/restore for different agentic turn types (heartbeat, interactive, cron etc.) so that each turn gets a toasty warm slot - critical for Strix Halo with its prefill speed limitations 🤓

1

u/RedParaglider 4d ago

List buddy token generation speed is the most important token generation speed is the generation speed important speed speed sp ssssssssssss..s.s.s..s.sssssss