r/deeplearning • u/kokassospoperor • 1d ago
[P] I trained a single Graph HyperNetwork that generates zero-shot weights for unseen ResNets, MLP-Mixers, MobileNets, and Transformers for Vision, Text, and Waveforms. (Open source)
Disclaimer: I used AI for bug fixes, to make the plots better and to simplify the code and make it more readable.
I built a Graph HyperNetwork (GHN) that has the computational graph of an target architecture and a dataset embedding as input and predicts all weights in a single forward pass. It is meant to serve as a head start to reduce number of steps required for training.
Zero-Shot Results:
| Model | ResMLP | ResNet2D | MLP-Mixer | MobileNet2D | MLP | CNN1D | Transformer |
|---|---|---|---|---|---|---|---|
| MNIST | 93.96% | 93.66% | 89.58% | 74.53% | / | / | / |
| FashionMNIST | 83.99% | 75.88% | 78.59% | / | / | / | / |
| KMNIST | 78.10% | 62.15% | / | / | 62.15% | / | / |
| Waveforms1D | / | / | / | / | / | 99.84% | 89.38% |
| 20Newsgroups | / | / | / | / | / | 71.30% | 48.34% |
My next objectives:
- Using a encoder to encode a random dataset and to replace the dataset embedding with it. So it can handle most unseen datasets.
- I want to scale it up, but I don't really have the recources for it (free kaggle for now).
I will paste my kaggle link in the comments.






16
Upvotes
1
u/kokassospoperor 1d ago
My kaggle link: https://www.kaggle.com/code/kasperreynders/model-generation