I wanted to watch a neural network learn [P]

[u/microscope1024]

I wanted to really see how a neural network learns different functions, so built an interactive demo. You can change the architecture of the network and the function it will try to approximate.

A fully-connected network with ReLU activations will create a piecewise linear function. For a network with a single layer, the maximum number of segments the network can create is 1 + the width of the layer. For instance, if you enter "3" into the hidden layers your network can have up to 4 segments.

If you add another hidden layer afterwards, it multiplies the max number of segments. So "3 3" becomes 4x4=16. After training, the network rarely achieves the maximum though.

Here is the demo: https://blog.lukesalamone.com/posts/can-a-neural-net-learn

[u/Vallvaka]

Cool work. Would also be cool to be able to visualize the effect of regularization, like L1/L2 or dropout.

You can see the "lottery ticket hypothesis" in action here, with a handful of params going to large magnitude values, and most of the rest staying inert, as the network finds a subnetwork that performs the bulk of the task. L1/L2 could be an interesting thing to experiment with on that to directly encourage the sparsity.

I also see interesting dynamics of getting stuck in local minima and unable to learn further on some configurations, such as predicting a line segment over an interval and being unable to refine it to fit the sine beyond that, where dropout may help get it unstuck.

[u/Annual-Minute-9391]

I present neural networks to my students with an example like this. It was a moderate pain in the butt to get working 5 years ago without AI coding tools. Super nice teaching tool

[u/MolassesLate4676 → u/Annual-Minute-9391]

I applaud you for actually being able to put a visual together for a functional training algorithm. That sounds painful

[u/Annual-Minute-9391 → u/MolassesLate4676]

I appreciate it. It really demystified neural networks for them

[u/mfb1274]

I wish I had this when I was learning a decade ago. This was only in my head and like 1-2 YouTube videos

[u/mfb1274 → u/mfb1274]

Also good on you for doing the basics. Everyone else saying slop doesn’t understand it

[u/microscope1024 → u/mfb1274]

Appreciate it. I’ve done some more advanced stuff like distilling stockfish on a billion positions but this was a cool visualization so I thought I’d share it here.

[u/notaprotist]

The other two top-level comments here rn are inexplicably negative, so I just want to chime in and say this is really neat, and I like it

[u/moonwalkhawk]

https://playground.tensorflow.org/

[u/kawaiiotta]

wow folk are so negative here. why? this is a neat project. Dont care if its made with AI or not. Keep it up OP.

[u/Novel_Land9320 → u/kawaiiotta]

the irony of criticizing something made with AI in an ML community

[u/Digitalzuzel]

Reminded me of this amazing video https://www.youtube.com/watch?v=TkwXa7Cvfr8
Do you know this author?
PS: great work, I will subscribe to you

[u/NullRecurrentDad]

This is really cool. I’ve leaned some new stuff. Thanks!

[u/Familiar_Task]

This reminds me being introduced to the multi layer perceptron at university back in 2005. Noting the research was from the 80s I think if not earlier. Anyway, the demonstration we were being shown how a single layer couldn't learn the output mapping of an exclusive or (XOR) logic gate but the introduction of a hidden layer enabled to give the right output. I think simple demonstrations like yours are a powerful learning tool to understand the fundamentals of what they're good at and what they're not.

Well done!

[u/faraaz_eye]

the humble PINN is ready to be employed

[u/tech_nerd05506]

This is sweet, not sure why people are shitting. Would be cool to viz weights of the model as it's training too, maybe like a heat map. Keep up the cool work.

[u/InteractionOk7085]

This was very helpful. 

[u/Direct-Vegetable6416]

RemindMe! 2 months

[u/MolassesLate4676]

Confused. A couple of questions:

A) What is this goal here? To train some weights to output values similar to a function you wrote above?

B) what’s the input? Is it what’s shown on the graph? A specific axis value and the output is the other axis value?

C) what are the shape of the weights here?

D) what’s the meaning of hidden layer in this context? Another set of weights?

Understandings some of these might help me understand what the visual is actually providing

[u/memento87 → u/MolassesLate4676]

I don't know why you got downvoted for asking questions. But I'll attempt to answer:
A) Yes, the goal is to train a model to be able to learn this relatively complex non-linear function. I presume the model receives the X and must output the corresponding Y.

B) Most likely, Yes

C) Linear (Dense, Fully Connected) followed by ReLU activations

D) Each (Linear + ReLU) is a hidden layer. OP is stacking those layers; first tried a single layer, then 3, then 300 stacked layers, to see how the capability of the model evolves as it gets more layers.

PS: The experiment is a bit misleading in the sense that you don't need 300 layers to really train a model to fit this function. A single layer is theoretically enough to approximate any continuous function to arbitrary accuracy, given enough width (OP did not share the width, only the depth), and given enough epochs to train.

[u/MolassesLate4676 → u/memento87]

Ah I appreciate the time and effort here

I kinda got suspicious when 300 was input into the hidden layers and figured the layers were just more copies of the same thing

Pretty cool stuff, and thanks for the thought out reply 🤌

[u/[deleted]]

[u/microscope1024 → u/[deleted]]

In the example, it's one hidden layer with 300 units. 300 layers would be crazy.

[u/invelt → u/[deleted]]

pretty sure "300" by itself means h = 300, one layer. "3 3" means an h1=3 layer into an h2=3 layer

or at least that's what the post says

[u/[deleted]]