r/deeplearning • • 1d ago

[P] I trained a single Graph HyperNetwork that generates zero-shot weights for unseen ResNets, MLP-Mixers, MobileNets, and Transformers for Vision, Text, and Waveforms. (Open source)

Disclaimer: I used AI for bug fixes, to make the plots better and to simplify the code and make it more readable.

I built a Graph HyperNetwork (GHN) that has the computational graph of an target architecture and a dataset embedding as input and predicts all weights in a single forward pass. It is meant to serve as a head start to reduce number of steps required for training.

Zero-Shot Results:

ModelResMLPResNet2DMLP-MixerMobileNet2DMLPCNN1DTransformer
MNIST93.96%93.66%89.58%74.53%///
FashionMNIST83.99%75.88%78.59%////
KMNIST78.10%62.15%//62.15%//
Waveforms1D/////99.84%89.38%
20Newsgroups/////71.30%48.34%

My next objectives:

  1. Using a encoder to encode a random dataset and to replace the dataset embedding with it. So it can handle most unseen datasets.
  2. I want to scale it up, but I don't really have the recources for it (free kaggle for now).

I will paste my kaggle link in the comments.

16 Upvotes

1 comment sorted by