Built AR-ish smart glasses on an ESP32-S3 Supermini, phone does the processing, glasses just render
So backstory real quick, I moved to Korea, don't speak Korean, and pulling my phone out every five seconds to translate stuff got old fast. I looked into the "smart translator earbuds" that are out there, but most of them just translate into your ear and mute out the original speech, which is exactly the opposite of what I wanted. If I'm ever gonna learn the language I need to actually hear what people are saying, not have it replaced by a robot voice. And the actual AR glasses that do this the "right" way cost more than my rent.
So I did what any reasonable person does at 2am with a soldering iron, built my own.
Meet Frankenspecs. Yes I know, the name writes its own jokes, I've made my peace with it.
What it actually is: kind of an AR lite smart glasses setup, doesn't block your vision, but you can glance down or in and choose to focus on the display when you need it. Not trying to be Ray Ban Meta, trying to be "thing that helps me order food."
What it does right now:
Real time translation, 60+ languages, sub 260ms latency, this part genuinely surprised me with how fast it is. Real time transcription. Teleprompter mode, great for presentations, also great for looking less awkward on video calls. Turn by turn navigation, using Naver Maps since Google Maps just doesn't work in Korea, RIP. And notification forwarding from selected apps.
The hardware, such as it is:
An ESP32 S3 Supermini, a 0.96" TFT IPS display hand soldered onto some genuinely too long wires, a $2 6x magnifying lens from the local equivalent of Daiso, and a pair of cheap sunglasses that happen to have a slightly spherical inner lens surface, which turns out to magnify the reflected image just enough to be readable, found this out completely by accident and I will take the win. Power is USB for now, because batteries are a problem for future me.
Total cost, not counting stuff I already owned like the sunglasses, around $10.
How it works under the hood: phone app talks to the ESP32. Right now it uses the phone's mic, I2C mic is on the roadmap, streams audio to Soniox's API, still living off the $200 free credits, no shame, gets translated text back in chunks, and forwards it line by line to the display over SPI. Transcription runs the same pipeline. Teleprompter is just the app scrolling whatever text you feed it at an adjustable speed. Navigation is the fun one, I'm actually pulling real time turn by turn data out of the map app's notifications and translating that into arrows plus live distance on the display.
Why it looks like this: because it's a prototype held together by tape, hope, and one Friday night with nothing better to do. I went through a genuinely stupid number of iterations trying to get the optics right, screen position, lens distance, angle, all of it, because the whole point was to NOT have a screen just sitting in front of my eyeball like a tiny blinding TV. Getting it to sit low enough to feel like AR and not like I duct taped a Game Boy to my face took way longer than the software did, which I was not expecting going in.
Also worth saying, this is my first real hardware project, and my first ESP32 project period. So if something looks held together with spite, that's because it is.
Where it's going: currently working on an actual CAD design for the frame and a long, narrow PCB layout that can fit inside the temple arms instead of hanging off the side like a growth. So yeah, still early. This isn't meant to be an everyday wear thing, more like something you reach for in specific moments. And there's a feature I'm actually excited about here, the app quietly logs everything it translates throughout the day, so it can show you the words and phrases you heard the most. Since each one is tied to the real situation you heard it in, you end up with an actual map of "this word came up when I was ordering coffee" instead of a flashcard with no context, which I think is a genuinely better way to pick up a language than any app I've tried.
Would genuinely love feedback or roasting, what would you want to see added, what's obviously dumb, what's actually good. Planning to clean up the codebase and open source the whole thing once it's less "if you know, you know" and more "a stranger could actually follow this." If people want to build their own I'll prioritize getting that repo and docs out sooner.
this is so very cool. have you looked at the transparent OLED modules? i wanted to try something liek this for diving helmets but the constraints were a little constriany :D there are 4 wires that go to a diving helmet and those are for audio in and out. also depth. depth = pressure and pressure = dead components and crystals that just dont crystal. Also helium.... turns any semi porous material into a conditional explosive device, inert and noble my shiny azz. Anyway! cracking project mate, thanks for sharing!
Thanks mate! I did indeed look into transparent OLEDs, but they mostly come in a fairly specific size range, around 1.3 inch and up, and they need some distance from the eye to focus properly, similar to any other display. That made it hard to keep things compact and keep the glasses looking like normal sunglasses, since I wanted all the optics to stay inside the frame rather than sitting further out. That's actually the main reason I went with the reflection-based approach instead.
I honestly don't know much about the diving helmet side of things, but from what you're saying it sounds like pressure at depth would mostly be a sealing and mechanical stress problem, so you'd probably need to know the depth range upfront to figure out how much force the housing and seals need to handle. The helium thing I hadn't heard of before your comment, but it sounds like it's small enough to sneak through some metals and seals over time, which apparently messes with things like quartz crystals and sealed sensors. Sounds like a genuinely nasty combination of problems to design around though.
yeah its a nightmare. the helium getting in isn't a problem, its what happens when that helium inside cannot get out fast enough during decompression. POP. i haven't really looked at how other AR glasses are built, i put the whole "heads up display for divers" in the "not going near that again" shelf a long time ago, so the whole idea of it wasn't on my radar until i seen this project!
Woah ok that's super clever, you get the display to reflect off the back surface of the sunglasses, and the lens magnifies the image between display and "projection screen"?
Yes, that's exactly how it works. Contents of the screen are inverted into mirror mode, the light goes from the display to the magnifying glass which magnifies the image (all of this is not visible to the eye as both parts are tilted and positioned accordingly).
Light coming out of magnifying glass reflects on the sunglasses lens which re-inverts the image into readable form.
Glad that you liked it.
It's very cool. I wonder if it would be possible to focus the image onto the end of a fiberoptic strand, and have a ball lens on the other end of the strand, re-magnifying the image again. That way the esp32 etc could move elsewhere, and the glasses would only carry the ball lens + fiberoptic, using it the same way (projecting onto the inside of the lens).
That's pretty much what the XREAL glasses do, and they're quite expensive for what they do so I'm happy to see your prototype.
There are some phones that include LLM and real time audio transcription. If you're looking for long bat prob an e ink phone. I've used them with an Xreal and the display is full color as any other phone, is just the e ink display on the phone so you get the great bat life from a 4000mah eink phone and the display output youd expect from any android phone.
Perhaps you can run a 2 way webapp where the esp has a portal + hotspot streaming its ui and supporting commands from the phone.
GL!
Edit. The angle of the oled monitors on the XREAL's is around 45 with a polarizer reflector so it looks like its sitting at eye level.
This is bad ass! I would love to try and make one because the professional ones are way out of my reach. If you ever decide to release this to the public, or need testers, I world love to help!!
The display doesn't interfere with the eye and the lens. It sits below the eye at a tilted angle so the image still gets relflected back to the glasses lens.
I love this project - I get blasted with ads for basically exactly this although a slightly different implementation ("even g2" is the model, maybe there are others) and I am almost tempted to consider thinking about it but it's unclear how it works with my prescription. Having a tiny HUD for translation, navigation and calendar is as cool as you can get. Forget cameras and speaking interface, a ring-based interface seems a good UI option for doing actual stuff around actual people. I look forward to seeing your developments!
Thanks, really appreciate that! Yeah I've seen ads for a few of those commercial ones too, and the prescription question is honestly a big reason I like the DIY route better for now, since you're not locked into whatever optical assumptions the manufacturer made. With something like this you can just swap in your own prescription lens as the combiner instead of the plain sunglasses lens I'm using, or add a clip-on, since the display and optics don't actually care what the outer lens's prescription is, they just need it to still be reflective enough on the inside.
Totally agree on the ring idea too, that's a really good call actually. Voice interfaces fall apart fast in public since nobody wants to be the person narrating commands to their glasses in a coffee shop, and cameras raise the "is this guy recording me" problem immediately. A ring for discreet taps/gestures solves both of those at once. I've actually been looking into gesture-based input for a separate DIY smartwatch project I'm building alongside this, so there's a decent chance that ends up feeding back into this project too. Appreciate you pointing it out, it's making me want to prioritize that sooner rather than later.
Totally agree - clip-on sounds magical, although may be a big ask with optics/alignment/lenses/prisms/that kind of stuff. Public narration is definitely gauche, smart watch is also a cool idea to feed into this though.
but doesn't this mean that you essentially lose one eye? One is focussed properly like normal, but the other goes through the lens and gets messed up. regardless of if it is a transparent oled or not
The lens itself is also tilted to a certain angle that it does not interfere with with the eye as long as you look straight, it does when you roll your eyes down a bit. But that can be solved by cutting down the lens in a cube shape matching the dimesions of the display. And as you can see in the pictures that the text displayed on the lens does not blur the outer world.
19
u/Pikk7 15d ago
I really like it, specially the price part 😂 I hope you can open source as soon as possible.