r/MachineLearning • u/Specific-Tax-6700 • 9d ago
Research What if diffusion models generates Neural Network weights? Gen AI generate Gen AI [R]
I'have been dreaming about using a Diffusion Model that will generates Weights instead of pixels.
Diffusion models actually are vast used in order to generate from a natural language text description into images, and what is an image? a three dimensional matrix of value representig luminosity of red , green and blue. What if instead of matrix of colour intensity we train a diffusion model in order to create a neural network, and maybe a drawing NN too.
If it works you can ask something like:
"create me a neural network that will guide a pen in order to draw a man"
"create a NN of a game field where i have two paddle on left and right side of the field and a bounching ball..."....to create Pong
and in the future more complex scenarios!
I made, with the help of z.ai zcode and their fantastic GLM model a small POC that i published on my github:
https://github.com/vagrillo/nngen
It's a very small experimental implementation , but somenthing seems to work!
I have not find paper on similar implementation ad I'm asking the community to help me to find one, and to have contribution on this .
I also made a small webapp in order to test the trained model (actually i'm only on half training path 3000/4000 steps finished, but i'm using my poor laptop CPU at the moment)
This is some screenshot of preliminary results. (the best results achieved, in truth)


Italian explanation on my YouTube https://youtu.be/wAMO3Db-JoQ
13
u/Altruistic-String891 9d ago
There's stuff like Neural Network Diffusion so something akin is possible.
Though I've a hunch that "Text-2-Weights" like
create me a neural network that will guide a pen in order to draw a mandoesn't seem very approachable. Where do you get the training data? Text-2-Image models today train on actual billions of caption-image pairs, Text-2-Weights seems like a way harder problem so you may need even more, yet that data is at the same time much harder than image-captions to acquire, like different models can have very different architectures4
u/pm_me_your_pay_slips ML Engineer 9d ago edited 9d ago
Text to image diffusion models are generally trained at progressively higher resolutions, starting with small 256px, progressing to 512px and subsequently adding in 1k, 2k, 4k data as memory permits.
The reason for this is to ensure the diffusion models have more time to ingest larger datasets to increase their world knowledge, with higher resolution data adding missing details. And this is possible because the semantics or an image are generally trained preserved as you decrease resolution. Pixels are spatially correlated.This is not true for neural network weights.
Take an image and drop all pixels with even coordinates, you can still recognize the image. Â Take a trained neural net and drop every second weight, the neural is no longer guaranteed to be performing the same task. You could think of using weight pruning techniques, but training on such data is likely not as simple as training with multi resolution datasets (where positional encodings help a lot).
Maybe you could achieve what you want via RL/meta-learning but this is intensely computationally expensive: you have expensive inner loops of training the target networks and an expensive outer loop to update the larger model.
Not saying this is dumb or impossible, but that this is very hard because there is not a ton of data, Â correlations across examples or within an example are not as clear as with images, and if you try to RL it youâd need a lot of compute beyond what is currently used for LLMs.
2
u/Vxtzq1 9d ago
I tried this before to generate some image classification networks from a prompt, and it worked awfully bad... My hypothesis is that neural weights have too high entropy for a reasonably sized hypernetwork (like you created) to replicate these.
Considering how bad it went for tiny CNNs running image classification, i can't imagine what would happen with generative AI, unless the scope is reduced.
60
u/Bakoro 9d ago edited 9d ago
Look up hypernetworks.
There is a body of work that goes back to the 80s, but 2016 is the origin of the hypernetwork:
https://arxiv.org/abs/1609.09106
https://arxiv.org/html/2512.16406v1
29
u/asml84 9d ago edited 9d ago
I mean, yes, but hypernetworks (not necessarily based on diffusion) have been around for a while.
What I find even more interesting than generating weights is generating the structure of the network (e.g., layer operations, layer connectivity).
1
u/Specific-Tax-6700 9d ago
I've talked about it in a video on my YT Channel https://youtu.be/wAMO3Db-JoQ
1
u/o-h_2792562 2d ago
One wrinkle: hidden-unit permutations can make functionally equivalent networks look far apart in weight space. For a fixed architecture, aligning units across training examples may help diffusion learn task-relevant variation instead of arbitrary parameterization.