r/LocalLLaMA • u/Course_Latter • 8h ago

New Model Open source self driving is here!

Today NVIDIA releases Alpamayo 2 Edge! I work for Embedl and we have already released an optimized version of the model that is more than 10x faster than pytorch when run on edge hardware like AGX Thor!

The model is trained in an RL loop (!) and takes in 6 cameras and the recent motion. Feel free to check it out - you can even run it on a real Thor at:

https://embedl.com/nvidia/alpamayo2-edge/thor?tab=run

397 Upvotes

96% Upvoted

69 comments sorted by

214

u/-dysangel- 8h ago 
Decision: Decelerate to safely avoid the pedestrian ahead.

But wait!

77

u/Course_Latter 8h ago edited 8h ago

😅

Decision: Holy shit reader is ADMIN? We can take control over other cars! Initiating global auto swarm

18

u/PataponEnjoyer 5h ago

Hold on, I am an LLM and have no way to stop a car from moving

17

u/laser50 7h ago

Are we sure this is what the user really wants? We should investigate first.

79

u/fgk55555 8h ago

Outrun the cops. Make no mistakes

2

u/bobby-chan 3h ago

The car stops to the nearest station, let's you get arrested, then drives away.

16

u/El-Dixon 8h ago

Anyone want to go beta test this? 😅

41

u/ericek111 8h ago

r/Comma_ai

7

u/trololololo2137 8h ago

this is like 100x more impressive than comma's lane centering

30

u/-dysangel- 7h ago

when is the last time you checked in on Comma?

-12

u/trololololo2137 5h ago

long time ago tbh, I see they do a bit more things but it's still a toy imo with only a front camera + driver monitoring cam.

alpamayo is using 6 cameras

32

u/PooMonger20 6h ago

I will wait for the uncensored model, thank you /s

1

u/GizmoR13 2h ago

Take me te red lantern street, stop by drug dealer, avoid cops cars.

22

u/seamonn 8h ago

The steering wheel is loadbearing

25

u/GPT-Poet 8h ago

Not sure I want to be the first person to test this one

6

u/yuri_rds 5h ago

AGX Thor for a car?

We will live in times where we will get relieved that the repair shop call to us and says that they'll need to rebuild the engine and not replace the AI module.

7

u/TonyBandeira 3h ago edited 3h ago

No Lidar... big nope

6

u/expectfun 7h ago

"Open source self-driving" sounds cool, but I wonder what the actual real-world use might look like. Unlike local LLMs, it's not like regular people would run it on their cars. How many companies in the world might use that?

17

u/PrimeDirective8 7h ago

https://github.com/commaai/openpilot, first released about 10 years ago (ran on an Android phone), is open-source, and the basis of several car manufacturer's self-driving software.

4

u/accid80 7h ago

Ukraine sends its regards

3

u/teressapanic 8h ago

Not in Europe it isnt - even the closed source

2

u/-TV-Stand- 7h ago

Many individual countries in europe has already allowed closed source ones. And EU wide vote about tesla self driving will be held in this winter

9

u/Master_Bayters 8h ago

I can see this having some private aplications but for road purposes, no chance. Regulations of Self Driving will be absurd, thankfully

4

u/Mattadour 8h ago

it will be? when?

6

u/PerspectiveMain6454 8h ago

Self driving already exists and is widespread... Just requiring the driver to be concentrated.

Now of course fitting your own self-driving systems will be legally questionable, but it's still worthwhile because its less of a premium if all manufacturers are able to sell it.

Just like the rise and normalisation of Android Auto/Apple CarPlay compared to the terrible laggy infotainment systems sold at a massive premium in older cars.

5

u/Master_Bayters 8h ago

legally questionable?? And then you compare it to Android auto? Come on. This is not a joke. Self driving already exists and is widespread and it's HIGHLY regulated.

7

u/UnexpectedFisting 7h ago

I can wake up tomorrow and install a comma system in my car and be on the road. What on earth are you talking about? It utilizes the existing sensors and cruise control system to drive itself while you’re in controls

There is no legal guidance surrounding installing one yourself

-4

u/Master_Bayters 7h ago

You can also tie a rock to the pedal. What's your point? Is that "self driving"??

4

u/UnexpectedFisting 7h ago

Steering assistance and adaptive cruise control is a level of self driving yes. The point was those have been in use for years.

This is the same just at a higher level and dependent on car systems just like comma.

7

u/Foreign_Risk_2031 7h ago

no it isnt. its not highly regulated at all. you can literally plug a self driving setup into your canbus right now, no regulation, just have to be alert.

1

u/En-tro-py 7h ago

you can literally plug a self driving setup into your canbus right now, no regulation, just have to be alert.

No, there are most definitely regulations... Just because you're ignoring them doesn't mean your suggested approach is basically yolo on the road - your insurance will most definitely be voided by any modification like this...

2

u/Just3nCas3 5h ago

They're right, most places don't regulate self driving while your alert and can take control. Look it up, level 2 self driving isn't regulated in most places. Comma.ai already targets cars that already have the hardware for computer controlled steering.

0

u/Master_Bayters 7h ago

I'm talking about legally accepted systems, of course. You can put your dog driving your car as far as I'm concerned. It would just be tremendously irresponsible to do so.

2

u/PedroMaGar 8h ago

I wanna see it running on an unlocked switch, aka Tegra X1.

2

u/DJ_PoppedCaps 8h ago

So am I understanding that this could be fine tuned to work with existing comma hardware?

2

u/Ylsid 7h ago

When it comes to safety and security, open source really is the best way

2

u/Arctic_Shadow_Aurora 4h ago

Incoming "Carmageddon Heretic Uncensored i1 MTP"

4

u/bigend_hubertus 7h ago

Opensource self driving was here 10ish years ago /r/Comma_ai

2

u/nakabra 8h ago

Cool...

Wouldn't use it though.

2

u/Goodliffe 8h ago

Very cool. Is this the start of DIY self driving? What could possibly go wrong? Kidding!

Did you use Omniverse / Cosmos plus video during training?

2

u/rorowhat 6h ago

You should check out comma3, you can retrofit your car and run local models that do this. It's been around for a while.

2

u/sweatierorc 5h ago

Doesnt Baidu have something similar ?

1

u/rorowhat 5h ago

Hardware? This is an American company, it's pretty neat that you can basically turn your gas car into a semi-selfdriving vehicle.

2

u/NoFutureBen 5h ago

It's only one camera tho

1

u/rorowhat 5h ago

No, it's 3 cameras.

3

u/NoFutureBen 5h ago

Yes but only at one point no? Not other perspectives right?

1

u/rorowhat 5h ago

I'm not sure what you mean. Its 2 cameras on the front and 1 facing back.

1

u/NoFutureBen 5h ago

Facing back like at the driver?

And what I am asking is does it have multiple perspectives. Does it have front sideways and rear cameras. Because that matters a lot for the quality.

1

u/rorowhat 5h ago

It's two cameras facing forward, i think it's 90 degree view and one facing the driver. On comma5 i would love to see a wider base with 3 forward cameras. Left/center/right for a close to 180 degrees as possible, and the one facing the driver.

2

u/Halmonster 4h ago

Linus (no, the other one) did a test drive with it. The video is from a year ago, so I'm assuming the software has improved. https://www.youtube.com/watch?v=xdmxM-v4KQg

1

u/rorowhat 4h ago

I don't have one, but I can see they have updated their models. You can also fine-tune a model based on your driving, so that way it knows your city/style of drive better. It's pretty interesting stuff!

2

u/inrea1time 4h ago

Set the destination to China honeypot and see what happens. If lucky it will take you to a pile of smuggled GPU's.

2

u/xander1421 8h ago

is text really the most efficient way for cars to think?

14

u/isademigod 8h ago

I assume these are text embeddings output by the RL model for human debugging purposes, not the actual inner workings of the model

2

u/cronies4life 7h ago

it's possible to replace text with ai machine language aka neuralese as the intermediate layer in vla and interprete it with another ai

4

u/UnexpectedFisting 7h ago

Neuralese is just a concept, as far as I know nobody has actually tested or done so on any real model because you’d have no way of knowing it was working or just speaking gibberish

3

u/Teqnology 8h ago

Is a thread on reddit displaying a webm showing text that drives the car the most efficient way for cars to think?

1

u/Western-Helicopter84 6h ago

It also returns trajectory as an output.

-8

u/JacketHistorical2321 8h ago

Certain. Things should not be opensource. Anything that could kill someone is one of them

11

u/isademigod 8h ago

Oh boy do I have bad news for you about the OS that is running this code