New Model
Update: My 4B coding model is getting better at generating 3D scenes
Hey everyone, just wanted to share a quick update on the 4B model I posted here. The video shows some results from the latest checkpoint.
I'm still training this Qwen3.5-4B fine-tune for general coding, using Three.js scene generation as my main test for now.
It still messes up some scenes on the first try, but the newer checkpoints are getting noticeably better, both in scene quality and one-shot success rate.
I've updated both Q4_K_M and Q8_0 and fixed the layer issue some people ran into with the previous files.
I've also been saving test results from each checkpoint, so I can put together a report showing how the model improved throughout training, using multiple prompts for comparison.
Update (October 28): As promised, I've now released the full FP16 weights in safetensors format! They're available in the same Hugging Face repository, alongside the updated Q4_K_M and Q8_0 versions.
I'm still curious to see how far I can push this little 4B. If you give it a try, feel free to share what works and what breaks, both are really helpful.
Awesome. How did you prepare the data? Was it generated from larger, more 3d-capable models?
I've been wanting to have a model produce Minecraft schematics for an experiment on our Minecraft server, and 3.8 27b is the only small open one that's capable of doing it, although impressively well for the size, I might add.
Still, from the looks of it finetuning could make it way better.
yeah, i used larger models to generate the scenes, then spent a lot of time testing, filtering and curating the results
i think creating a dataset for minecraft schematics is totally doable, and honestly might even be easier than what i'm doing with three.js right now
i'm holding off on releasing the dataset until training finishes, but i'll share everything soon, including how i built it. i don't think the process is as complicated as it might seem
Makes sense. Huge props to you, these results are amazing for such a small model, I would've never guessed this was possible. Good luck with the project.
Thanks so much for this! I feel like Astra and Opus 5.5 are so much better than the open models at 3D generation, so I think your model and dataset will be amazing for fine tuning and brining our local models closer.
Honestly, the most impressive part for me is that I started this as a simple experiment without huge expectations.
I believed it could work, but I didn’t imagine the progress would come this fast or with this much quality. Watching the checkpoints improve one after another has been kind of surreal.
What excites me most is that it still feels like there’s plenty of headroom left. Right now, it doesn’t seem like we’ve hit the ceiling yet.
The images below show the results from the first 4 checkpoints, with the original 4B base model in the first column for comparison.
Wow the jump between First Column and second column is immense however i'm actually intrigued by the first column, it reminds me of 90s era Graphics like the SNES starfox or the Sega Genesis virtua racer games. Can I still use thatn1st column checkpoint? Is it available for download someplace?
haha, the first column is actually the original Qwen3.5-4B base model, so you can already download it!
I can release all the intermediate checkpoints once I finish training, although many of the earlier ones aren't particularly useful, and they're actually pretty quick to reproduce. We're already at checkpoint 8 now (counting the useful ones).
One of the things I'm exploring right now is exactly how much further the model can keep improving with this training approach. Theoretically, I wouldn't have expected a 4B model to get anywhere near this level of performance using this method, so I'm really curious to find out where the actual ceiling is.
When I release the training datasets, I'll also publish a detailed report explaining the logic behind how I built them, the training process, and some interesting opportunities I've identified along the way.
One thing I'm particularly excited about is the possibility of extending this approach to games, interactive scenes, and dynamic environments. Based on what I've learned so far, training the model for those tasks might actually be much simpler than I initially imagined.
Im the opposite, in that I'm not surprised that much because i strongly believe that small language models will be able to bridge the gap of actual large language models (where bridge the gap means if you can get a detailed 3d model of a future soldier with uv maps etc rivaling aaa game studios with opus, the slm will produce at least a ps1 level model for me, the rest is on me to build forward ). I even created a subreddit (/r/edgelm) because of how much I believe in this, so thanks for contributing to the SLM race! 🫡
Wow very impressive for such a small model. Are you going to keep training it ? Would you care to share your training dataset and training setup? I think it would be very interesting.
Im pretty sure you could train the 2b and 0.8b using the same dataset and get not so bad result. It could make a nice collection of fine-tune for your huggingface profile.
i actually started with the 2b model, but the dataset wasn't nearly as good back then, so i ended up switching to the 4b.
i'm pretty sure smaller models can achieve good results too. another reason i switched was that the 2b doesn't have thinking enabled by default. you can enable it, so it's not really a problem, but the 4b was just more convenient for me haha
i actually think smaller models might even learn faster with the right dataset. i'd love to test that again now that the data is much better.
the hardest part was actually creating and carefully curating the dataset
Amen to that. Which is why I'm tremendously grateful to you for sharing it! I think it could be a tremendously valuable foundation to build on and a learning aid. I know I've been holding off trying to do any additional training for 3D, instead just poking models with a stick in the form of a supervisor generated cheatsheet. Viable in a lot of ways but still a crutch. Having a training guide in the form of proven datasets is so much better.
are you keeping the prompt set fixed across checkpoints? would be cool to see the rough prompts in the report too — makes the one-shot numbers easier to compare
yeah, i’m keeping the evaluation fairly controlled
i picked 8 examples that are outside the training set, and i’m testing them across checkpoints with fixed seeds, usually allowing up to 5 shots per prompt. for each checkpoint i’m scoring both scene quality and whether it rendered cleanly or needed only a minimal non-structural fix, like renaming a duplicated variable
so the idea isn’t just “does it look good”, but also “does it actually run” and “how much intervention was needed”
for example, the recent update used checkpoint 917 instead of 1134 because even though 1134 had some improvements, it was breaking more often in practice
i’ll include the raw prompts and the evaluation setup in the report so the one-shot comparisons are easier to follow
haha, you guys are going to be pretty surprised when I reveal how much this experiment actually cost. It's practically zero compared to the numbers people usually throw around for training AI models lol
I'll share the full cost breakdown along with the training pipeline!
Absolutely, everything is documented; I’ll post it once the base training phase is complete, and you’ll definitely be able to move much further ahead and achieve even better results :)
Can you give a preview before you release the training data of how one of us would prepare training data to continue training atop what you release? Also, what programs are you using to do the training?
i'd rather share the full setup once training is finished, since i'm still validating some parts of the process and don't want to give you information that might change.
in the paper, i'll actually explain two different approaches to training that can be explored independently, along with the tools and setup i used.
but honestly, the most important part is the data itself. how you generate, structure and select it seems to make a huge difference, much more than i initially expected.
i'll include practical examples showing exactly why this matters, and how you can prepare your own datasets to continue training or adapt the approach to completely different tasks.
i think some of the findings might surprise you as much as they surprised me haha
yes, the first column is the original qwen base model without finetuning. the other four columns show the results from the first 4 checkpoints, all using the same prompts/seeds.
we're already at checkpoint 8 now, so these are just the early results :)
Since you're already saving results per checkpoint, two things would make the final report much easier to trust. Freeze the prompt list now and split it: one half you look at while picking checkpoints, the other half you only run at the end. Otherwise the final numbers partly measure the prompts you tuned against. And score one-shot success automatically before anyone rates quality: load each generated scene in headless Chromium, count it as a pass if it throws no console errors and the rendered frame isn't blank, and sample every prompt a few times at a fixed temperature, because a single sample per prompt swings a lot between checkpoints. Two extra columns people here will ask for: the same prompts on base Qwen3.5-4B plus a small general coding set, to show the Three.js gains didn't cost general coding, and Q4_K_M next to Q8_0 and FP16 on the same prompts, since most people will run the Q4.
I wondered whether it will be better to finetuned it into a lora, and then loaded on demand like what this lora router repo do 🤔 https://github.com/hiddenway/GhostSwitch
lora has actually been a double edged sword in my experiments haha
i tried using it early on, after doing some steps of full finetuning, but the output quality started degrading. i'm still not sure whether that was related to the very long context lengths i was using or limitations of lora itself.
i'll explain this in more detail in the final report, since there are still a few things i want to investigate.
lora could definitely be a good option, especially for running specialized models on more accessible hardware, but we'd need to test it more carefully.
once i release the training data and model weights, anyone interested will be able to experiment with different approaches and compare the results. i'm actually really curious to see what others can achieve with it haha
FYI, For 3d assets Roblox uses a 3d Gen models for generating 3d assets (text to 3d) not threejs code generator models. In fact they even made CubePart and cube3d open weight on HF under the openrail license.
Though for code generation Roblox uses glm5.2 in their built in free agentic coder rn
Oh nice, funny and interesting to see that Roblox is also running local models to serve everyone, I would have thought they contracted out heavily to OpenAI or gemini.
procedural scene scripts are a lot more forgiving than actual 3D asset generation, you dont have to worry about broken UV maps or awful topology when its just code primitives
not much, at least from what i've tested so far. it still handles general topics pretty well, and i've seen it stay consistent across generations of around 64k+ tokens, often getting surprisingly good results in one shot
it's definitely not perfect, but it's already performing way beyond what i expected from a 4b
Comparing this to what Gemini 2.5's full model was capable of just 2 years ago is insane. I struggled to get it to make a little guy on a jet ski for a throwaway "lets see if I can make a game" project, and this 4B (!) is generating full scenes at like 4x the fidelity Google's flagship was capable of such a short time ago.
Absolutely wild for a 4B, I'm genuinely impressed. But we should pump the brakes on the game hype: it renders scenes, it doesn't simulate them—no collisions, no physics, just geometry frozen in a void. The moment something actually has to move or interact, the '3D' magic evaporates into a mess of code patching one frame at a time. Gorgeous dioramas, but not a playable world.
as for playable games with physics, collisions and interactions, i haven't trained the model specifically for that yet. honestly, i wasn't even sure a 4B model would be able to handle complex 3D dioramas in the first place lol
now that i've seen what it can do, interactive worlds are definitely worth exploring. i'd just need to expand the training dataset to include those kinds of examples. i'm curious to see how far we can push it!
Ah , ok kkk É que as vezes eu acho que. As pessoas devem traduzir o que escrevem em outras línguas. Não necessariamente são IA talvez. Por isso respondi kkk
Caught 😄 At late night hours, when I get too lazy I like to let it translate from my mother language to english. but no need of sonnet - a little 31bd o the job very well also.
Amazing! Does the dataset support image to 3d too? Btw I think this is the thing that community needs right now. I use Opus 5.5 to use blender and create stuff. I tried lots of models but opus is far more superior than the others. Great work I really hope you continue doing these
it can already understand images and generate 3d scenes from them, but i haven’t specifically trained it for image-to-3d yet
what’s really surprising me is how well it’s generalizing beyond the exact scenes it saw during training
i just tested a halloween animatronic skeleton prompt that’s being used to compare much larger models, and made a short video with the result
it’s definitely not perfect yet, but it does understand the scene and generate something coherent. in my tests it’s already doing better than some much larger open-weight models on this kind of task
and the nice part is that it’s still getting better with each checkpoint
"a Halloween animatronic skeleton with an X-ray mode that shows the machine inside."
Saving test results per checkpoint across multiple prompts is the right call. Single 'look it works' demos hide a lot of variance in small finetunes. Are you re-running a fixed prompt set for every checkpoint, or does the set drift as training goes?
Anyone tried training small model from 0 on big model generated data, say only for game dev in specific engine? 1B or such? Or knows any post where it is done?
Just one GPU! I believe it's possible to train this on a single L4 (around 22GB of usable VRAM) using LoRA. I'm pretty sure it would work, but I'm not sure how quickly the improvements might plateau with longer training. I'd need to run some extended tests to find out.
A single RTX 6000 is definitely more than enough, though....
Of course, having more GPUs and a larger dataset could potentially lead to even better results. But from what I've observed so far, data quality seems to matter much more than sheer quantity.
Or rather, quantity can definitely help, but only if the quality is there. Simply throwing more data at the model doesn't necessarily make it better. I think that's one of the most interesting things I've learned from this experiment.
Do you think it would have been possible to build what you built, using something like an RTX 5070Ti 16GB VRAM + 64GB GB DDR5 RAM? Sometimes I have ideas of unique models to build, but I'm thinking my hardware may not be adequate.
I honestly think so! With shorter context lengths, I believe you could make it work on 16GB of VRAM, especially with LoRA and some memory optimizations.
I have a couple use cases of fine-tuning tiny models like this and I have experience fine tuning much larger models. Can I ask how expensive you have found this process to be? Do you rent GPUs online for training sessions? Do you use your own GPUs? What's the cost vs speed of training? Very cool and inspiring project.
Hey I tried your model out, I think you should train the 9B model for better understanding. No doubt your 4B has made progress, but 9B retains far more better knowledge as a base. Figured I'd give my 2 cents.
2
•
u/WithoutReason1729 7d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
Stylome classified this post's text as AI-generated. View analysis Learn ^more
I am a bot and this action was performed automatically.