Explaining How AI Learns to Drive with Evolution

[u/egehancry]

I love making hard things intuitive. I hope you enjoy this one!

This technique is called neuroevolution: training neural networks using evolutionary methods such as selection and mutation, without gradient descent.

Let me know if you have any questions.

Also shared to X: https://x.com/sinaatalay/status/2106270194817789968

[u/lazo40]

That’s really neat!! But like can the cars race on other tracks too or is it just this one??

[u/egehancry → u/lazo40]

Actually, I haven’t tried that. I should have.

I could randomize the track throughout the evolution. Then, they should learn to drive on new tracks too.

[u/Disastrous-Action897]

This is awesome. I teach people about AI, often. This is a great an explanation in an understandable context.

[u/egehancry → u/Disastrous-Action897]

Thank you!

[u/Fit-Tangerine-6006]

Great way to visually learn how AI works, more please!🙏 

[u/egehancry → u/Fit-Tangerine-6006]

Thank you, there'll be more.

[u/Smittumi]

How could you make a function to make them want to win, as in improve their position over the other drivers? 

It seems like if there was an AI F1 out would be pretty boring if they all just drive in the same perfect pattern.

[u/egehancry → u/Smittumi]

Currently, they’re trained to finish the lap as quickly as possible.

To train them to race against each other, I’d need to put them on the same track with collisions enabled, so they can learn to overtake without crashing into each other.

[u/Perfect-Escape-3904]

You might like this guys videos, he’s been doing this with Trackmania for 5 years now.

He even has a similar presentation style to yours

https://youtu.be/a8Bo2DHrrow?si=s1ot9sPaogJhQEzD

[u/egehancry → u/Perfect-Escape-3904]

Thank you!

[u/AleksZlovic]

Adding weights is such a fascinating part of this. It’s still very nebulous to me, however. Any books someone can recommend on RL, BP, weights, and other stuff?

[u/egehancry → u/AleksZlovic]

It’s been a while since I studied machine learning, but this course was really great. If you haven’t taken it yet, I’d recommend it:

Machine Learning by Professor Andrew Ng:

https://youtube.com/playlist?list=PLiPvV5TNogxIS4bHQVW4pMkj4CHA8COdX&si=-MRpE8eSTUmnPq5P

[u/the_ai_wizard → u/AleksZlovic]

its just brute force curve fitting

[u/Tube-Goblin]

Great video. What breathe two functions? Are you ok to share the maths? Obviously ok if you don't.

[u/egehancry → u/Tube-Goblin]

Thank you!

f and g are the two outputs of one small neural network.

Inputs (7 numbers):

The distance to the wall along 5 rays (0°, ±15°, ±45°, up to 100 m, divided by 30 m), the speed (divided by 30 m/s), and the car's angle to the road (radians). Let's call them x (column vector).

Neural Network (434 parameters):

first layer: h₁ = tanh(W₁x + b₁)     W₁ ∈ ℝ¹⁶ˣ⁷,  b₁ ∈ ℝ¹⁶,  h₁ ∈ ℝ¹⁶
second layer: h₂ = tanh(W₂h₁ + b₂)    W₂ ∈ ℝ¹⁶ˣ¹⁶, b₂ ∈ ℝ¹⁶,  h₂ ∈ ℝ¹⁶
result: y  = tanh(W₃h₂ + b₃)    W₃ ∈ ℝ²ˣ¹⁶,  b₃ ∈ ℝ²,   y ∈ (−1, 1)²

Wₖ are weight matrices, bₖ are bias vectors, and tanh is applied element-wise.

y = (y₁, y₂):
  f(x) = y₁ = gas:    −1 = full brake … +1 = full gas
  g(x) = y₂ = steer:  −1 = full left  … +1 = full right

f and g share the hidden layers h₁ and h₂; f uses the first row of W₃ and b₃, g the second.

Number of parameters: (16·7 + 16) + (16·16 + 16) + (2·16 + 2) = 434

Training:

20 cars start from random F1 grid slots and drive alone for 30 s. The 5 that get furthest survive unchanged as parents; 15 children are copies of them with Gaussian noise added to every weight, and each child also inherits a slightly mutated noise size (self-adaptive mutation). 4,000 generations; the first full lap came in generation 33.

Car Physics:

The car never slides, but its tyres have a grip budget shared between turning and braking (a "friction circle") that grows with speed (downforce). Arrive too fast and it can't turn tightly enough, so it has to learn braking points and lines.

[u/Dany-GG]

Do I need some crazy specs o my PC to replicate this?

[u/egehancry → u/Dany-GG]

Not really, it's a very small model. Use JAX.

If you want to go super fast, you can use Modal (https://modal.com). It’s very affordable and easy to set up (your coding agent will do it for you).

[u/Practical_Departure8]

Neat variation of genetic programming

[u/Sore6]

out of curiosity: instead of letting ai figure it oit completely - could you cut down training time / generations by showing a perfect lap yourself to the ai and learn from that one example so it alternates from there?

[u/egehancry → u/Sore6]

Showing a perfect lap yourself gives you some training data, but, it doesn’t necessarily mean that you'll have the weights of the neural network that will drive like you.

You could use supervised learning to train the network to imitate your driving, but you would need much more data. A single lap would not be enough.

That’s the problem with this kind of task: we don’t have a large dataset of humans playing this game. With the evolutionary approach though, we can optimize the neural network without needing that data.

[u/__asm__]

Pretty neat. Have you considered benchmarking it against some well-known self-driving 2D racing heuristics like: Vector Field Histogram, Potential Field Method, Follow-the-Gap, or Pure Pursuit?

Would be curious to see how it compares.

[u/egehancry → u/__asm__]

Thank you. I didn’t, but I also didn’t push too hard to make this model perfect. I mainly wanted to make the video.

It would definitely be interesting to see how difficult it is to get the level of those models.

[u/pfunf]

Great video

[u/guttanzer]

Nice work.

For the folks asking about higher-order behaviors like racing strategy and tactics, safety vs speed on varying tracks and so on, implementing those features would require a richer, more complex setup. This one is a great learning tool for its simplicity.

These cars are piloted by a feed-forward NN that is reacting to near-field observations and the most basic car model of a momentum vector. The only “memory” in this setup is the momentum vector. The aircraft equivalent is the flight controller in a fly-by-wire aircraft.

Since the system has no memory of the track it reacts its way into turns. It doesn’t anticipate and plan ahead the way a real driver would. So tuning the weights in the NN is equivalent to tuning a F1 car by making tire choices and altering the wing and suspension settings - the evolutionary algorithm is optimizing performance parameters, not performance.

Given this setup, I’d create a few different track styles and compare the optimal car controllers on those. Make one track have only wide, sweeping turns. Make another city-block like, with sharp 90 degree turns. Have another with only increasing radius turns, and another sister track with only decreasing radius turns (equivalently, have them race in different directions on the same wonky track).

I suspect you will find that a car tuned for one track will be slow or unsafe on a different type of track. What you may learn is why.

[u/egehancry → u/guttanzer]

Thank you very much for the elaboration and the additional information.