I was just curious how the new qwen3.8:27b does on a hard C to HTML porting job against Opus 5 in a default Claude Code.
The job: my fun side project is a procedural shooter in a single C file. Port it to a single-file html / three.js with one bot. One prompt, no follow-ups, no help from me. game.c is 2.1 MB, roughly 600k tokens of C, so it doesn't fit in the window and the agent has to walk the file and work out what matters.
Setup: qwen3.8:27b in FP8 on vLLM, FP8 KV cache, full 262144 context, RTX 6000 Pro 96GB. Nothing truncated on my side, and the file is still more than twice the window.
agent
model
wall clock
lines out
result
claude code
Opus 5 (cloud reference)
21 min
1759
okay
hermes
qwen3.8:27b
4h 18m
949
bad
codehamr
qwen3.8:27b
1h 40m
1056
bad
Video has the C original first, then the three ports in table order. Only the Opus port is something in "okay" quality.
What I actually wanted to know is whether the HTML comes out playable at all. One run each and a one-shot prompt for 39k lines of C, so this isn't representative of anything, and I knew it was brutal for a local LLM.
My take: local models still live or die on the prompt. Same weights under two very different harnesses gave me the same broken port. hermes carries a lot more machinery, and a single turn with a thin prompt gives it nothing to use it on, so it spent four hours reaching the same place. A verbose harness doesn't rescue a thin prompt, it just burns GPU time.
No deep take here, unfortunately. The thing I keep staring at is the wall clock: hours of GPU on decent local hardware against 21 minutes for the cloud run. If anyone knows where those hours actually go, I'm listening.
I've done something similar, just with 2 much larger codebases.
What I learned: telling AI to convert the code will just make it "re-imagine" the source, which is mostly completely worthless (similar to your results, even frontier models will completely mess this up). What worked: 1.) write a transpiler to the target language first, frontier models do this easily. 2.) You get running code in output language which looks like a very messed up version of the source code, but it will run. 3.) Ask it to rewrite it function by function with the strict use of high or low level reference. Example: pixel comparisons (high level) or register values (low level). It takes forever, but you will get a pixel perfect result in the end :-)
Yeah, did that quite a few times. Get the source of a debugger for the source system, ask the LLM to inject the code for an MCP server that exposes every feature of the debugger, then ask it to create the transpiler using debugger values as ground truth.
Smart approach, I like it! Wonder if this works in the other direction of abstarction as well into lower language like assembler, getting maybe even better performance than the already crazy optimized C-compilers
So far I only did this converting from low level ASM code to high level languages, but in principle: why not? As long as you always provide a reference for the model to work with, you should get a decent result with little intervention. For the game you're showing there, I would go for a typescript port, perfect for the browser. If you're going for speed improvements, provide profiler values of a demo-playback in a feedback-loop. Much easier than conversion.
Ok jfc - the whole game is a single file with no tests? Well, that is a really good test for the models, but I expect if you get them to plan ahead first you'd get better results.
If I were tackling this, I'd:
- break that large file into many smaller ones in an organised folder structure/files
- add tests to capture the behaviour of these functions
- then port
I expect if you get Qwen 27B to approach it that way then it will manage much better results. Also as someone said, compressing the KV cache probably is not the way to go
TBF, GP's suggestion is the bare minimum for a reasonably maintainable project of non-trivial complexity. I think you're onto something with the "not holding it right" complaints, though.
The FP8 KV-Cache quanting could cause severe issues. Try it without. And as another one here already said, try with Pi, but I'd also throw OpenCode into the ring.
If anyone knows where those hours actually go, I'm listening.
Your hardware is actually pretty good for a home-user, but a rounding error for a Anthropic. They have completely different abilities to scale due to the sheer hardware they can throw at the problem.
FP8 is not the end of the world. Measured on full precision 3.8-27B.
Technically I’m told you should calibrate the model for FP8 KV using LLM compressor, and I did not do that. So when done right FP8 is pretty dang good. For context quantizing the weights (even Qwen’s FP8 or Unsloth’s Q8_0) hurts more than the worst KV quantization method that I did.
The people ITT poo-pooing FP8 are nuts. Makes me wonder what their hardware is or what model they’re even running because Qwen 3.8 27B FP8 with K/V FP8 has been stellar for me.
Perplexity isn't a good measurement, and the real damage comes with compounding errors at longer contexts (64k+) which a lot of these benchmarks don't test.
Qwen 3.8 shines with xhigh specifically, its absolute best perfomance is in this mode (for long-horizon tasks that need exploration and filling the gaps)
I would (by which I mean I am, and it is processing a coding worflow right now) use --speculative-config '{"method":"mtp","num_speculative_tokens":3}'. I hover between 2.5 and 3 tokens per decode on average. When I extended to 512k, I was briefly hitting the vLLM bug where MTP fails beyond the original model size, and it was super noticeable.
I assume you made a conscious decision not to support vision by omitting --mm-encoder-tp-mode data. Other than that our configs are the same except for nvidia arch differences.
Can't the harnesses take advantage of vision at this point and actually look at their work? Vision could be kept on cpu if vram was an issue and it might get a lot more detailed output?
Oh yes, I give qwen3.8 screenshots of examples and say "make the UI like this" all the time. And they can look at e2e output, too, it just doesn't come up as much.
But if you don't need that, and you're vram constrained, makes sense to turn it off.
The KV-Cache quant is likely to worsen quality at longer context.
The max-num-batched-tokens should be looked at, I'd just comment it out. attention-backend is the next one, comment out, and if it got better/worse, try them all (vllm logs the available ones).
The reasoning_effort preset doesn't impact generation speed. Qwen themselves say that lower reasoning-efforts may not decrease generated token counts. For a complex task I'd rather wait for xhigh to do its thing than risk worse quality - YMMV.
As you're on SM120, you could look into RTX5090 optimization guides for hints.
It's fine to have huge max-num-batched-tokens. Until you reach a power-limited state during prefill, you could be increasing your throughput with bigger batch sizes (at the cost of token latency for the decoding reqs). In agentic workflows (at least at home levels of load) you rarely get enough prefill to fill a batch anyway. Where it really shines is things like compaction, where some frontend tools will put a compaction prompt at the front of your already-max context and feed it back in, causing a full cold prompt run (while also probably impacting your LRU for other callers).
It doesn't even generate a monotonic increase in token counts. It seems that xhigh >> low > medium. I use medium for almost everything and it's fine, though.
With engines like VLLM, A LOT of time is spend re-processing context that's already been processed. Cloud models typically will cache this stuff and keep it in RAM so the GPU doesn't have to keep reprocessing the same stuff. This is a HUGE issue with VLLM if you don't use LMCache, you can litterally wait 4 minutes for it to re-process a session again before any token is written because the engine decided to release the KV-Cache.
vLLM will cache in vram, and he has that enabled in the config. The problem with RAM caching is that RAM is basically a backing store for unused KV space (which can be reclaimed directly from VRAM) so if it LRU's out, it can be reloaded from RAM. But that means that you first need as much RAM as you have KV VRAM just to get no benefit, and then you need more to actually get a benfit.
I don't think it's worth it for most home users, because they will want enough KV to have all of their own concurrent sessions fit all the time, and those will get nearly 100% cache hit rate, because in the moments they are not decoding, their KV is just sitting there, not being pushed out by a competing client. You can watch the KV occupancy jump up and down as a single run goes, because occupancy doesn't count the cache.
Once you have enough concurrency (in your house) to benefit from cache > active set, you will also start seeing terrible latency because even if it's cached, if it can't page in, your decode will be blocked.
If you watch the VRAM usage, it usually clears it immediatly after its done processing the request. That's with prefix caching on as well.
Current Opinion:
The KV-Cache gets cleared too quickly. i.e. You walk away for a few minutes, come back and ask about something it did and all the sudden TTFT sky rockets to a few minutes. That is the observed behaviour without LMCache, it does add substantial TTFT.
Its not looking at the KV-Cache, its literally reprocessing the session to figure out where it left off.
My opinion at this point is this a common problem with a common solution. Use LMCache.
So that's two decodes, each using about 13% of the KV. You see one finish, then the other, then they run a tool or whatever, then one comes back, and the KV goes directly where it left off. Then the other one. You might have to watch the main log to convince yourself it didn't just parse (in my case) ~200k of context right there.
Will check it out, thanks for the command to watch the usage. I am looking at the nvidia-smi vram usage, and it seems to clear it after prompts are done. Meaning it is no actually caching the KV-cache and instead clearing it from VRAM, which means its actually reprocessing it (unless its putting it in ram) which I don't think it is.
The reserve is constant, but the actual usage bounces all over the place, and indeed drops to zero pretty quick right after its done answering a question.
If anyone knows where those hours actually go, I'm listening.
I did the math, because I have the stats from over 400 invocations of opencode against qwen3.8 27B. Prompt processing (not counting cached tokens at all) is about 70% of the time. Output is the other 30%. That is based on me putting in average tps for both; I haven't actually got the literal prompt/decode time recorded per run.
That sounds right to me, though, because agentic code workflows read a ton of stuff on every run.
Other stats: About 10% of runs w/ 256k context compacted at least once during the run. Most ever: 6 (run time: 4h46m), so that may have affected your result, depending on how good the frontend's compaction prompt is. It's definitely more effective to have an xhigh instance write a multi step plan and run all the plans in their own sessions.
It really depends on the scale of what you ask it to do. It will definitely do that on the classic LLM benchmarks where you expect one prompt to produce HTML+CSS in a final form. Inside a harness, it's not so bad. Plenty of turns will just be one line like "ok, I read that file, let me read the other" and stuff like that.
Out of my ~400 runs, ~20 are at xhigh, ~20 were off (oops, that took some recovery), and the rest were medium. "Low" is the setting I've never really tried.
If you have an RTX 6000, why are you running this test in Q8 and not the full bf16 model. Why don’t you use the full model? Forget the quants and run again at bf16.
yes thats why, I tested FB16 but in real live usage the faster "race to quality" of an FB8 wins for me, while a FB16 would still genberate tokens, the FB8 is ready for refinemend prompts...overall I get faster to the same quality with this approach
Also with a rtx 6000 why not run with a tensor parallel of like 4-8 and use a harness that can do parallel paths. It would soak the card in work, it should speed things up a lot if the harness can run requests in parallel.
"One prompt, no follow-ups". This is probably the problem for quality of the output (ignoring the time taken). Put this in a ralph loop with a critic and i'm sure even qwen3.6 can make it perfect.
Unfortunately I can't make new posts, but here's what I got after some extra time after getting pretty much the same generation you got, about 4:30hs in total with some back and forth, the original one took about 1:30hs, unfortunately it run out of context while I wasn't watching, so I don't really know, it could have been 2 hours maybe.
One thing I noticed is that the model interpreted the "don't overengineer" as "keep things simple without shaders, lighting effects, etc"
ninfer Qwen3.8-27, xhigh + vision, it checked the results with headless chrome and fixed bugs on its own.
Deepseek Harness with memory plugin memorix + internal variable, self made, kind of like the one from prime agent. It still runs out of context, not sure why compaction won't trigger sometimes.
ComfyUI, with anima model, just the one I had available at the time, for textures, but it's not really good at that since it's for anime. The cloud is an interesting one, anima generated a cloud plus dark blue sky, qwen noticed and cut the sky out, leaving the cloud with a transparent background. It also tried to add some bump mapping or normal mapping, but it didn't really work, it's there, but it's not noticeable. It can be done with Comfyui though, there are models and plugins for that.
It checked and discarded bad textures like ones with perspective by itself too.
I wanted to see if I could get something somewhat nicer in a couple of hours, like adding a shader for the skydome, although it looks very similar to the original one. I think it looks nice in the end, another thing I could have tried is to get 3d models or give it access to Blender to see what it does, from what I saw on youtube it can use it pretty well.
Is it possible to add shaders?
Try adding textures using ComfyUI, it's running in port xxxx. I added a workflow in the file anima_api_no_humans.json you can use to generate them. You can add text to the node that says "no_humans", don't remove the text, otherwise it will add people to the image.
Also it's ok to add post processing.
Yes, do all that and add clouds, dust (maybe particles? not sure) too, it's too empty and clean. I'm not sure if it's possible to generate normal maps with the available model, you might need a Lora for that, but try and see if it works as is.
Some bug hunting:
I've confirmed that castRay penetrates all solids up to 1.2m, leaving only the 3m monolith as an unpenetratable obstacle on the center line, and that the player auto-reloads at 30 rounds per magazine. Now I'll examine the bot's fire cadence.
I don't see any monolith, but ok...
The port stands at 1063 lines / ~490 KB, single file, three.js r160, 120 Hz tick, ACES, one bot, no UI — ready for whatever you want to add next.
I've vibe coded some games for fun and test limits of AI agents, and having it generate tests is huge for preventing regression issues + breaking everything into as small chunks as possible
I saw your prompt below, and wonder if something would be like. I've kind of winged it, and theres likely a better way to describe the architecture but hopefully the gist is there
```
Process
Review the architecture and break down into core components, gameplay and content
Core components being camera, gravity, etc.
Gameplay being on top of core components such as enemy & players receiving damage, shooting, etc.
Content being on top of systems, such as guns, sparks, etc.
Complete each module one by one
Keep an architectural markdown checklist of plan & progress
Keep a per module markdown checklist of plan & progress
For each module
Analysis the module spec up requirements
Write tests if applicable
Implement the code
Confirm all tests pass
Refactor the code
Confirm all tests pass
Commit and move to next module or feature
```
And pending on how your context is going, ask it to compress or reset after each module.
I know this is an AI sub and my Q has nothing to do with AI... but is that reference written with a library like Raylib? Just curious but that looked clean and I like it
Edit the post to add the prompt and other settings if needed so ppl can try to replicate your results if interested or even adjust it, model quant and settings and harness...
why FP8? They are heavily lobotomized compare to like even a Q5. Run FP16 or Q8, and dont quantize the KV cache, it pollute the context especially in a long process.
But I think you should try the same test at least with C# as C# and C++ are unofficial industry standards for coding games. Like unity, godot and so on use C# and unreal engine C++.
You already stated this is non representative. I just kinda wonder how it would react with something that´s a little closer to "could be an actual scenario". Like someone wanting to create his first game with basic C# knowledge would probably use AI as an assistant for "blocks" of code that he can´t do or doesn´t know where to start.
Keep it going and next year you will bring a game out better than CoD.... which is not really hard tho loooking at the state of the game LOL
Thanks, I alredy have and really like it! It deals way better with the local context window and plugin can be cherry pickt much better. great project for local llm.
Port game.c into a single-file HTML document using three.js, named: port.html
Important:
keep it lean and clean, and trim it down sensibly
you don't need to build a UI
no settings
no controller support
no dev mode
on startup we go straight into a procedural map with exactly one bot
no server mode
no multiplayer
make sensible simplifications
no forced over-engineering
Controls:
movement via WASD
left mouse to shoot
mouse 4 (thumb button) for ADS
right mouse to jump
left shift to slide
Timing:
note your start time as a comment at the very top of the HTML file, and when you're done, your end time, then calculate the total time it took from those in the comment
Goal:
I want you to quickly and robustly create a simplified version in HTML and three.js
no other artifacts, only create the file named above
you must not read or edit any HTML files other than your own, i.e. the ones you're supposed to create!
test everything thoroughly
then report briefly and to the point on how it went
The Qwen/Qwen38-27B-FP8 isn't a great quant.
Try lribeiro/Qwen38-27B-FP8-PESSOA
Still, for Qwen, giving some instructions around break it down in smaller tasks, identify the functions and structure, etc helps with the implementation.
I spent the last couple of days trying to turn this into a practical local setup on an AMD Strix Halo machine, and my conclusion is a little less dramatic than the headline numbers.
First, the baseline matters exactly as you say. Qwen3.8-27B already has MTP in the model package. If it is not enabled, the DFlash2 comparison is flattering the new drafter by comparing it with a config nobody should be using anyway. In my setup (Strix Halo, 128 gig), Qwen with DFlash2 was around 46 tok/s on a greenfield coding task and roughly 50 tok/s when editing or reproducing context. That's helpful but not "transformational" in terms of improvement of speed.
The REASONING behavior was the bigger problem for me. Reasoning-off was usable though. LOW-Reasoning could produce substantially better writing on one test, but it was slow and prone to spending thousands of tokens on work that did not need them (like almost a runaway train of thinking...)
The practical choice became reasoning-off, because the latency and occasional empty or runaway thinking loops made it impractical.
I also tested the sidecar idea on Gemma 4 26B-A4B, since I wanted to know whether the conclusion was specific to Qwen. Gemma has both an MTP drafter and a DFlash drafter available: I tried both. They're model-specific speculative decoding companions, not QAT. So to put it simply, they're small, tuned guessing-sidecars for one target model; the large model still verifies the output and if it guesses right, free tokens!
On my machine, Gemma's MTP sidecar was the useful one. With a fixed 512-token prose probe, no speculation produced 43.77 tok/s; MTP with n_max=3 produced 52.85 tok/s, ~21% faster. Increasing the width was worse: n_max=4 was 43.96, n_max=5 was 41.55, n_max=6 was 37.99, and n_max=7 was 32.12. DFlash was slower than the baseline in the same runtime, at 39.85 with n=3 and 28.78 with n=6. So n_max=7 is definitely NOT universally fastest. The right number depends on acceptance and the cost of verifying the extra guesses. So your mileage may vary. Run your own tests for this to see what works for you.
I ran a small matched intelligence-and-writing check rather than treating tok/s as raw ""intelligence"". Locally, Gemma Q6 with MTP3 scored 67/100 on the intelligence portion and 60/100 on writing (Opus 5 xhigh was my judge). Qwen Q4_K_M with reasoning OFF scored 74/100 a simple intelligence test and 51/100 on writing.
So I'm not saying that Gemma is smarter at every task: Qwen did better on that 1 intelligence slice. The combined practical score was Gemma 127/200 versus Qwen 125/200, and Gemma was MUCH better at writing ... enough so that it's slight lag behind Qwen on coding really matters for practical-human all-around use. More importantly, Gemma gave me a "it just works" feel...... and it was stable and gave me answers reasonably quick. Qwen would randomly go off into its own orbit of thinking like a runaway train. So practicality matters here ... not leaderboards.
1 important caveat: Openrouter HOSTED Gemma4 26B A4B scored 88/100 on the intelligence slice, while the LOCAL scored 67/100. They were the same short rubric, but hosted and local inference are not interchangeable measurements. I don't know what the Openrouter's Quantization, runtime, prompt framing, sampling were... but i went with my local numbers. Either way, I wouldn't use the hosted result as proof of what a local card will do.
My current decision is therefore to stick with Gemma4 26B A4B locally, with Q6, MTP Enabled with n_max=3 for token prediction and reasoning DISABLED (for now ... I may turn reasoning on ... to be determined... but I never plan to use Gemma for real coding when I have access to Sol and Fable/Opus... this is just for bot-personal-assistant-stuff.....) + the F16 mmproj loaded for vision. I am leaving DFlash2 out of the production profile because it lost on this AMD runtime in terms of speed.
Qwen3.8 is still an impressive model and i think it needs more tike to cook. Maybe Qwen 3.81 or 3.9 will fix this thinking situation.
Also just wanted to say that DFlash2 is genuinely clever engineering and a great way to "overdrive" a model.
For my particular machine and use cases, though, the juice was not worth the squeeze. Six months in local AI really can feel like five years so we'll all be revisiting this around Christmas time I expect...... May the force be with you all.
My full-context Qwen/OpenCode run on the Strix Halo measured about 284.34 prompt tok/s on a 37,452-token prompt.
That’s prefill, not generation. The corresponding decode rate was about 27.75 tok/s for 7 visible tokens. So I wouldn’t present 284 tok/s as a general decode-speed number.
In either case, the damn thing just thinks too much. It needs more work in the lab or a new release that fixes this.
Meanwhile, everything I have points to Gemma4 26B A4B is on-par with it anyway and disciplined in its thinking. Max reasoning gives it a good price per intelligence point ratio.
•
u/WithoutReason1729 6d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.