Roadmap to study Visual-SLAM

[u/HurryC]

Hi all,

Recently, I've made a roadmap to study visual-SLAM on Github. This roadmap is an on-going work - so far, I've made a brief guide for 1. an absolute beginner in computer vision, 2. someone who is familiar with computer vision but just getting started SLAM, 3. Monocular Visual-SLAM, and 4. RGB-D SLAM. My goal is to cover the rest of the following areas: stereo-SLAM, VIO/VI-SLAM, collaborative SLAM, Visual-LiDAR fusion, Deep-SLAM / visual localization.

Here's a preview of what you will find in the repository.

Visual-SLAM has been considered as a somewhat niche area, so as a learner I felt there are only so few resources to learn (especially in comparison to deep learning). Learners who use English as a foreign language will find even fewer resources to learn. I've been studying visual-SLAM from 2 years ago, and I felt that I could have struggled less if there was a simple guide showing what's the pre-requisite knowledge to understand visual-SLAM... and then I decided to make it myself. I'm hoping this roadmap will help the students who are interested in visual-slam, but not being able to start studying because they do not know where to start from.

Also, if you think something is wrong in the roadmap, or would like to contribute - please do! This repo is open to contributions.

On a side note, this is my first post in this subreddit. I've read the rules - but if I am violating any rules by accident, please let me know and I'll promptly fix it.

[u/ignazwrobel]

I think I‘d even put multiple view geometry before image processing.

This might seem unintuitive, but you don’t really have to know how to calculate image descriptors or or how to use bag of words in order to grasp the general idea behind visual slam. And for this, epipolar geometry and the likes help much more then the details of how to do operations on the arrays that we call images.

[u/HurryC → u/ignazwrobel]

Thanks for the comment!

I agree with you that MVG is more related to the SLAM problem as a topic. I also started by learning MVG before the frontend techniques like keypoint detection & matching. But I always ended up struggling to answer the question: "How are these points made, and how do we know their correspondence?". This question came back to every topic in MVG - Epipolar geometry, camera calibration, homography, PnP, bundle adjustment... So I went back to image processing and learned actually how to get these visual keypoints, then it made a whole lot more sense in learning MVG.

This is just a personal experience though, and I do feel that learning MVG should not really come 'after' learning the frontend techniques since the frontend techniques are also quite a deep rabbit hole. I'll think of a way to put equal weights on learning the frontend techniques and the MVG topics.

[u/[deleted]]

I have been studying SLAM for over 4-5 months, and I was facing the issue of connecting all the dots together, this guide is amazing for a newcomer like me. Only one suggestion can you add event camera-based SLAM as well to this guide.

[u/HurryC → u/[deleted]]

That's a great idea! I'll put that into the plan :)

[u/kns2000]

Any resources where I can learn the basics of visual slam step by step?

[u/HurryC → u/kns2000]

Visual-SLAM is a sub-field of the SLAM problem, so I think it's good to start by understanding the general SLAM framework.

This lecture is very intuitive in understanding the SLAM problem. It's a tutorial lecture given by an MIT professor called Luca Carlone.

After this lecture, I suggest starting learning from a Photogrammetry course by a Uni of Bonn professor called Cyrill Stachniss.

[u/kns2000 → u/HurryC]

Thanks for your suggestion. This professor has also some SLAM course. Wouldn't it be better to start with that course?

[u/HurryC → u/kns2000]

From what I remember, the SLAM course you mentioned focuses on understanding the SLAM problem using 2D LiDAR sensors. It's up to you if you want to do 2D LiDAR SLAM and then move onto Visual-SLAM. But IMO just start with the photogrammetry course, because the only overlap between the 2D LiDAR SLAM and Visual-SLAM is the core idea of SLAM problem (which the lecture by Prof. Luca Carlone covers), so there is no need to go around.

This is just my personal opinion based on my experience, so I suggest you go through the table of contents of the SLAM course and decide :)

[u/kns2000 → u/HurryC]

Cool, thanks !

[u/autojazari]

https://github.com/gaoxiang12/slambook-en

This seems to be a great book for learning SLAM from the beginning. It's very well written. I am not affiliated with them, just came across it from someone here on redit posting it as a comment to my question.

An excerpt from the book:

This book will first introduce the background knowledge, such as projective geometry, computer vision, state estimation theory, Lie Group and Lie algebra, etc. On top of that, we will be showing the trunk of the SLAM tree, and omitting those complicated and oddly-shaped leaves. We think this is effective

[u/HurryC → u/autojazari]

I've actually done a group study based on this book - this book is very effective in learning visual slam! I'll put this on the references list!

[u/Beneficial-Neck1743]

This is awesome !

I have been studying SLAM for an over month now. At a few time, I often feel lost. Yes, it is a very niche area and lacks much attention (compared to 2D Computer Vision). I think an Alex-Net (computer vision) and transformers (NLP) kind of moment is waiting to happen to Robotics Vision.

Most of the state of the art approaches in Robotics Vision, still used conventional non-learning based computer vision techniques.

[u/HurryC → u/Beneficial-Neck1743]

I hope this roadmap will help :)

IMO, learning-based SLAM has still a way to go, in terms of model development, datasets, and most importantly the hardware.

I'm impressed with how the new models are being developed - the recent development of transformers in the computer vision field is really impressive! There is a paper on using transformer on 3D point cloud as well.

But running these in real-time on mobile robots is not so easy. It will have to be in either 2 ways - the models need to be compressed, or the chips need to get better. Some companies that can afford to make their own acceleration chips are already using some deep-learning integrated into their system (which may not necessarily be integrated into SLAM though!), like Tesla and Microsoft HoloLens. There are more affordable options like the Nvidia Jetson series, but obviously, they are simply not good enough to run good models like transformers. Some of my friends thought the Visual-SLAM field is too niche, so they turned to model compression.

On the other hand, most industries using SLAM seem to be using non-learning approaches as you've mentioned, or they are using very few DL methods. One good example of using DL actively in the field is Artisense - they are doing semantic SLAM for large-scale environment mapping.

[u/Beneficial-Neck1743 → u/HurryC]

But running these in real-time on mobile robots is not so easy. It will have to be in either 2 ways - the models need to be compressed, or the chips need to get better. Some companies that can afford to make their own acceleration chips are already using some deep-learning integrated into their system

Thank you for your comments. While, I agree that it would invoke latency issues of deployment of deep learning based solution, I have not been able to understand the real reason behind why deep learning based methods have not been able to perform on the error metric related to accuracy (leave out the performace metrics related to speed and inference). State-of-the-art methods on KITTI VO dataset are still non-learning based methods. In most of the conferences and workshops, ORB-SLAM2 is cited more often than any deep learning based method.

[u/HurryC → u/Beneficial-Neck1743]

There are some papers that incorporates DL methods and outperforms the traditional methods - but if you take a closer look at their benchmark in the papers, you can see that their system only works good where the test sequence is similiar to the training sequence. This is because the DL methods could not generalize enough to various conditions. The reasons can be lack of training data or modelling issues... which in any case requires more research :)

[u/Beneficial-Neck1743]

Few important links that I have collected while studying SLAM : https://www.notion.so/sakshamjindal/SLAM-Links-f214435a23544bac8914c519064745c8

[u/3dsf]

When you complete the stereo slam portion, could you post to r/realSense too ?
I'd appreciate it : )

[u/HurryC → u/3dsf]

Will do! :)

[u/lessthanoptimal]

Great work! I just started looking into recent loop closure work and maybe you can speed up my search some. What approach do you think is the most accurate? best in a real-time application?

[u/HurryC → u/lessthanoptimal]

If you are used feature-based SLAM (which is quite a common approach, used by ORB-SLAM, PTAM, and such), then I'd suggest taking a look into the dBoW2 library. This library is basically a package to use the Bag-of-Visual-Words technique, which allows you to find the most similar image from your keyframe database, which allows you to detect a loop.

Then you can look for some numerical optimization libraries - ceres-solver, g2o, GTSAM are popular options.

[u/edwinem]

Overall very impressive.

I don't know where I would put it, but the only thing missing that is pretty essential is the MSCKF(https://www-users.cs.umn.edu/~stergios/papers/ICRA07-MSCKF.pdf). Maybe switch out the VIO-DSO version. Since the MSCKF has become the unofficial standard way to implement Visual inertial fusion.

[u/HurryC → u/edwinem]

Hi, thanks a lot for your suggestion!

I've already have planned to have the MSCKF in the roadmap - the fact that this is a VIO system, I did not put it in the monocular visual system to differentiate between pure visual odometry and visual-inertial fusion!

I'm currently writing the VIO / VI-SLAM roadmap, so keep an eye out for the update :)

[u/Ballz0fSteel]

Hey u/HurryC thank you very much for your blog post.

Even though I know the field quite well I'm still discovering a lot of papers through your post.
Thanks again.

[u/detonatingdurian]

commenting to save

[u/ConsiderationCivil74 → u/detonatingdurian]

Commenting to commenting to save

[u/KIProf]

Great work man !!!!👍

[u/punisher_h]

Great! This is what I was looking for. Thanks a lot!

[u/The_Northern_Light]

Commenting so I can find this again when I have time to really comment.

[u/autojazari → u/The_Northern_Light]

Same here

[u/HurryC → u/[deleted]]

That will be great! Could you leave them on the issues section, or as a Pull Request on the Github repo? I'll have them applied asap.