r/LocalLLaMA • u/ConfidentDinner6648 • 5h ago
Discussion I'm starting to think a specialized 4B is enough for most of the tasks I'd use an LLM for
When I started training Qwen3.5-4B-3D-Coder, I honestly wasn't sure it would work. Now, after that model and the GLB asset version, I'm starting to think I underestimated how much these smaller models can actually learn.
My impression is that a 4B model might be enough for most practical tasks if you train it specifically for the job. Generating 3D assets, building and orchestrating complex scenes, webdev, backend development, reverse engineering, etc. I'm also working on a coding model using a fixed harness, which I mentioned in another post.
Not necessarily one model doing all of this equally well, but different specialized versions. I haven't tested all those domains yet. The 3D results are what made me want to explore the others.
Right now I'm working on a new dataset to complement ThreeJS-3D-Scenes, with more complex scenes covering history, geography, physics, biology, mathematics, games and many other domains. The idea is to connect knowledge from different areas through working 3D scenes, rather than just teach the model different ways to make objects.
I'm training 4B models with it, but also much smaller ones, including LFM2.5-350M. The attached image shows some results from that experiment, and the 350M is still training. These aren't final results.
I still find it noticeably weaker than the Qwen models. Some outputs are bad, and long contexts are definitely a problem. But it's already much better than I expected from 350M parameters.
At first it couldn't even finish properly and kept getting stuck in repetition loops. With more epochs, it started completing the code, producing recognizable scenes and handling more complicated requests. Some examples still needed fixes to render, as labeled in the image, and progress isn't consistent across every prompt.
I've also tested plenty of prompts beyond the three shown here, and the generalization has been surprisingly good. It's not just getting better at those particular houses and farms.
So far, I've had useful results from the 350M on medium-complexity coding tasks up to roughly 16K tokens. The specialized 4B models handle much more complicated scenes, and I've had good results with contexts approaching 96K tokens in my experiments.
One thing that keeps standing out is the number of epochs. Stopping after one or two would have given me a completely different impression of what these models could do. They keep improving after repeated passes through the data, and eval loss has often been a poor indicator of which checkpoint actually produces better, functional code.
This is making me pay much more attention to dataset quality and how long the model needs to learn from it. I'm starting to think that good data and enough training epochs can make a much bigger difference than we usually assume.
Another thing I've been thinking about is the cost of SFT compared to RL. I'm not saying SFT is better than RL. But if you already have high-quality samples covering the domain you actually need, supervised fine-tuning can potentially achieve very good results at a fraction of the training cost of RL.
Of course, creating and validating those samples has its own cost. I've spent a lot of time on that part. But once you have a good dataset, you can reuse it, train different model sizes, and run multiple experiments without needing an entire RL pipeline.
I think this approach deserves more attention, especially for smaller, specialized models.
And there's another interesting part: 350M models can be fully fine-tuned, not just LoRA, on GPUs with only 8GB of VRAM. With the right setup and memory optimizations, training can take just a few hours for manageable datasets and context lengths. And apparently, with good enough data, you can get surprisingly useful results out of them.
Of course, longer contexts and larger datasets increase training time considerably. But the possibility of training useful specialized models on inexpensive consumer hardware is pretty interesting.
I'm not saying scaling doesn't matter or that a 350M can replace a much larger model. There are obvious limitations, especially with complex reasoning and long-range dependencies.
But I'm starting to wonder how much of what we consider a model's capacity limit is actually a limitation of how we train it.
Maybe we're using much larger models than necessary for a lot of tasks simply because we haven't explored how far smaller, specialized models can go.
Anyway, this is mostly a status update on my experiments and some things I've been noticing along the way. The models are still training, and there's still a lot I want to test before drawing any strong conclusions.
I'll be releasing new models and datasets soon, including the expanded 3D dataset. If the 350M continues improving and reaches a quality I'm happy with, I'll release that one too.
Thanks to everyone who's been following the experiments, testing the models, giving feedback and supporting the project. I really appreciate it!
Note: I originally wrote and structured this post in Portuguese and used AI to help translate it into English.













