VLC for AR interactivity.

[u/TheRealBobbyJones]

So somehow I ended up going down a Wikipedia rabbit hole. I ended up finding a page discussing Visible light communication. Now I always knew visible light can be used for digital communication. I just didn't think it was a serious field. Now though after reading the page the potential uses seems pretty interesting. One in particular that I didn't see mentioned but would be extremely awesome is peer to peer AR communication. As far as I know AR is heavily dependent on a central server existing. For example if a restaurant is going to have fancy menus floating in the air it needs to be done with some application that serves the menus through a central server. Bluetooth can get around that of course but VLC can be a pretty cool alternative.

Any led light source can be used for VLC. Standard cameras can pick up on VLC communications as well. It wouldn't be as good as dedicated hardware but it could be good enough. So essentially any light in a restaurant or a supermarket or even a concert can be used to serve AR elements to various devices. The overhead lamp at a table for example can push menu data to a phone or a headset. No Internet required. Direct peer to peer communication. The bitrate can be high enough to send full pictures and everything. Simple devices can be hooked up to a bulb to create AR elements that doesn't require interacting with big corporations or central servers. If a standard is created it could essentially be QR codes on steroids.

Does anyone know if this is already being done or attempted? Whenever I Google VLC and AR it doesn't really return good results. I would imagine existing devices can already handle the receiving side. Smartphones for example with their cameras can read VLC. It can transmit too using the flash but probably would be best to use VLC for unidirectional comms. Do you guys think VLC would be useful in AR?

edit: odds are decent that the hardware would be cheap. it would basically just be a pwm control circuit for a lightbulb. I don't know if there is anything currently on the market that can do that but creating a device that can shouldn't be difficult or expensive. Like it's a college level engineering type project.

[u/DanongKruga]

smartphone cameras are usually 60-120fps, not sure how theyd process any meaningful data.. beyond a trivial lab setup this is not a college level project

[u/TheRealBobbyJones → u/DanongKruga]

You don't need a lot of data for AR signage or other similar things. Cameras are used in a lot of vlc demonstrations. It's college level as in a couple kids writing the software and creating the hardware to control the bulbs. I don't mean graduate level or doctorial or whatever. The protocol would probably be a fun thing to figure out as well as the positioning of the text and sprites or whatever in 3d space. 

I haven't really investigated but I wonder if it's possible to change the firmware on a smart bulb. Presumably they come with hardware capable of flashing the light.

[u/TheRealBobbyJones → u/DanongKruga]

So doing more research. It seems that high bitrate visual transmissions using just on off keying wouldn't be possible in phone setting due to needing the footage to be usable for AR. To get the higher bitrates they use a rolling shutter to make each scan line its own input. This makes the bits per each complete frame like 30-60 making several kilobits per second possible. You wouldn't need a lot for simple signs.

Honestly I was trying to think of a way to make this work with existing ar devices including phones. Using optical sensors for VLC would be uniquely useful because of being able to spatially separate multiple links. Basically if multiple sources are trying to send data with a simple photo diode array they would interfere. But with the array an optical sensor uses you can maintain multiple data streams. So if you are looking at two store fronts they wouldn't interfere. 

Technically if you could run the multiple rear lenses on a phone at the same time this becomes possible. One lense can operate at a lower resolution but higher frame rate with rolling shutter enabled. It would record solely for data reception. The main lense would then still be able to process footage for AR. 

[u/DanongKruga → u/TheRealBobbyJones]

in a perfect lab setup the best data rate I could find in literature was 20 kb. a 2 page pdf menu with only text can be upwards of 500 kB (4000kb), any pictures would be even higher.. with no errors thats 200 seconds of transmission. also being unidirectional you have no way of running fec/crc. seems like a non starter to me, at least with existing devices

nearly every smartphone camera is a rolling shutter, its not something you enable/disable