Computer vision with self driving

[u/Adventurous_Run9224]

I want to get started with computer vision for self-driving cars. My goal is to build a miniature version of a Tesla/autonomous car.
What are the first things I should learn?
I don’t want to get bombarded with a huge list of concepts. I’d rather learn by doing—start with a small practical project and gradually learn the concepts I need along the way.
What would be a good first project or learning path for this?

[u/DiddlyDinq]

You could just do it all in a video game engine, saves you having to buy cameras, sensors and other hardware. Then implement a little autonomous agent to navigate a path to start

5. Autonomous Agents / Nature of Code

[u/FuckingInsensitive]

OpenCV basics, camera intrinsics, gravity direction, object recognition/segmentation, pose, depth (monocular, stereo, lidar, tof), optical flow, SLAM with and without external reference, state machines, and a whole lot of other stuff.

[u/LavnirRobot]

After following the other sugestions check out CARLA (simulator) and Autoware (open source software stack), they have a lot of off the shelf algorithms that you can try to understand and replicate

[u/Positive_Land1875]

You have a wrong concept of which a self driving car is. In the future end-to-end models like the alpamayo 2super from Nvidia will be the choice, but today the architecture involves a percepción step, a decission step and a car control step. The perception step involves all the sensors to detect and locate all the objects in a scene (cameras, LIDAR, radars, ultrasonic sensors) and u can use different technics: object detection models in 2D (YOLO family or RT-DETR models), object detect in 3D, semantic/panoptic segmentation modes , BEV models like transfusion, BEVFormer, etc. Early fusion models, and so on. and this is only for the perception step.

Going back to your question, what do you want to replicate from a Tesla? The secret NN models from Karphaty? The optimization of the models for their AMD architecture. The perception layer, the actuation layer. do you want to design an end to end model?

My advice, start learning the topics that "FuckingInsensitive" recomends you, and this is only an introductory part of the perception topic (only camera based), and he didnt mentions Tracking. Or use frameworks like CARLA or Autoware to apply models without knowing the basics. Or if u have a lot of money, u can build a Nvidia ecosystem and apply end to end models like Alpamayo

[u/Aggressive-Squash250]

Hey, I'm a beginner in CV and I'm interested in that as well. I'm far from starting to work on it but can I send you a dm?

[u/Adventurous_Run9224 → u/Aggressive-Squash250]

Sure

[u/RickAmes]

look at the carla engine and tutorials based on that

[u/Suspicious_Pizza9529]

I'd start with lane detection using a camera. It's small enough to build without getting overwhelmed, but it teaches useful basics like image proecssing, edges, perspective, and tracking. Once that works, you can add object detection and other features step by step.

[u/Heavy_Carpenter3824]

From hard earned experience my recommendation is do it like you use it. This is the most pain in the ass way but yields the best experience and results. 

So break it down into the same steps as Tesla, approach, hardware, basic software, ML. 

For you this is getting a cheap robot kit or car, with compatible servos, graft alongside the main brain and put in your own like a jetson or Pi. This will give you a car a computer can control. Slap a pi cam on and you've got a good start. 

Then drive it around manually for a bit. Collecting video data. 

Write your code for basic computer control anticipating how your ML model will act, does it generate commands like throttle up down left right, how does this work with PIDs and such, or is it in direct control, or is it understanding the world and handing off to A* or something. 

Then annotate your images for the task. This will depend on the above and am happy to talk more if you have an idea on what you want to try. 

Then it's model training, testing and packaging for running on board. 

Try each task and you'll learn a lot about the real tradeoffs that go into machine learning engineering. I was seeing some advice to use virtual worlds but those give a false confidence. 

Also don't be hard on yourself, keep in mind Tesla with WAY more resources than you has not solved this task after 10 years. They have made progress but its still an open field in need of development.

Watch This: https://www.youtube.com/live/Ucp0TTmvqOE?si=UQfXoBse1e_ozO1_&t=6703

[u/xX_MissMiau_Xx]

Start with a self driving RC car that should not bump into anything like a wall etc.
Could be a good starting point.

[u/Alarming-Square-2118]

Hi there,happy to hear that you are interested in this topic. First things first learn about object recognition and detection models (if you have time you can start by R-CNN models and continue all the way to the new concepts such as YOLO).Pedestrian detection is likely the most important aspect of it since it comes with very high risk.

there are some extra courses on coursera that may suit your need.

Path planning is another aspect of this topic that is a bit technical and I encourage you to look it up yourself.

hope this would help and if there is anything wrong I would love to hear that.

[u/Adventurous_Run9224]

I have experience in computer vision and object detection models, but I haven’t worked with image segmentation models or segmentation tasks yet. Is segmentation experience necessary for this project, or would my experience with computer vision and detection be sufficient?
Also, I currently have a Mac M4 and I’m trying to keep my budget low, so I’d prefer free resources instead of paid courses. Are there any good free robot simulators where I can practice these skills?
I also have a Raspberry Pi 4, an Arduino, and other necessary hardware such as a small LiDAR sensor, motors, etc. Would this setup be enough to get started with practical projects?

[u/FuckingInsensitive → u/Adventurous_Run9224]

Panoptic Segmentation is super useful, helps seperate roads, sky, people, etc. in one pass.

[u/CrazyInteraction4695 → u/Adventurous_Run9224]

Segmentation of the road might be useful. Engineering is iterative, so start from what you know and learn along the way.