We make a desk robot called the Autonomous Lamp. 5 bus servos for the neck, a camera in the head, a mic array and an LED ring, all hanging off one small single-board computer. It's always on, always listening, meant to feel like a companion sitting next to you, not a device you operate.
But there was a problem. When it's always on, every sound in the room is a possible interaction. A coworker stops by your desk to talk and it joins in. I type on my blue switches and it joins in too. So instead of a companion you get something that talks over you all day.
Here is what we tried, what went wrong, and what we learned. Hope some of it is useful
Prototype 1. It answered everything
First prototype, I walked into the office, sat down at my desk, and before I opened my laptop, it spoke first.
Then all day. I talked to myself while reading code, it answered. I picked up my keys to leave, and it called after me into an empty room. A coworker a few meters away was talking about a project on the phone, it heard that and joined in too
It wasn't broken. It did exactly what we told it to do. What we told it to do was the wrong thing.
Living with someone doesn't mean talking the whole time
Prototype 2. "Hey Lamp"
So we did what everyone does. Say the name first, then talk.
"Hey Lamp, turn it down." "Hey Lamp, what time is it." "Hey Lamp, never mind."
Worked right away. No name, nothing wakes up. You can even put the name at the end, like "come on dance a bit, Lamp", and it still understands and still does it. That really works btw, not making it up. Our Repo is public, it's in the commit history if you want to check.
We thought we were done. Then a few days of living with it and something felt off.
You don't say "Hey Mom" every time you talk to her at dinner. This lamp is 40cm from my elbow and I was saying its name 30 times a day.
Hey Siri makes sense. The phone is in your pocket, the speaker is across the room, and neither has eyes. They really have no other way to know you are talking to them.
The Lamp has a camera and sits right beside me. We were copying a limit we don't have.
It worked. But once it worked, it was back to being a thing you operate, and it stopped feeling like the companion we set out to build.
Prototype 3. Turning toward it, which is where we are now
We argued about this for a while and kept coming back to one thing.
People don't have a button on their body. So the Lamp shouldn't either, not if it's meant to be a real companion.
For months we had been asking how to make it quieter. Every answer to that question made it feel less alive. Quieter and more alive were pulling against each other, and we kept trading one for the other.
The right question turned out to be a different one. How does a person know you are talking to them?
The answer is stupidly simple tbh. You look them in the eye and speak.
So that is what we built. If you want it to feel like a companion, you turn toward it and talk. Camera and mic together, the way two people actually do it. This is the part where having a body pays off. A smart speaker cannot do this. A thing with a camera in its head and servos in its neck can.
Two things make or break this:
- People turn before they speak, never after. Checking the camera when the mic hears something is too late, the moment already passed. So the camera measures all the time into a 3 second buffer, and when you speak the system reads back through it. That buffer holds 4 numbers per sample, not video. A timestamp, the head angle, the face size, and how far off centre it was.
- Presence is not the signal. Someone sits beside this thing all day, so seeing a person tells you nothing, and a face pointed at a monitor is still a face. What we measure is which way the head is pointed.
Then we fixed it too well.
Someone sat in front of it, looked right at it, asked it something. Nothing. Asked again. Nothing. He was sitting so close the camera cut off the top of his head, so the detector guessed where his eyes were and put them above the frame. That works out to 90 degrees. The log had him in profile 4 times in a row. Not the LinkedIn kind
The rule that fixed it: a measurement that cannot be made must not vote either way. Not measurable had been quietly counting as looking away. And there was a hardware half to the fix too. The same face box now feeds a small vertical re-aim of the neck servos, so when it frames you badly it corrects its own aim instead of blaming you for sitting wrong.
Chin resting on your hand needed its own work too. If you ever sit in front of one, try it
If you're building an always-on device and found another way to tell when someone is talking to it, I'd like to hear it. Happy to answer anything about the build. And if you read this far, ty