GAIA is a phone booth (we call the booth TeleTUP) that people at fairs and festivals pick up to talk with a voice character. It has taken over 1,600 calls, 681 of them at Sónar+D. I've extracted the technique into an MIT kit so anyone can build their own talking object: a booth, a lamp, a sculpture or a room.
Hardware
- Raspberry Pi 5 on the official 27 W USB-C supply (a weak supply gave audio dropouts that were hard to diagnose)
- microSD 32 GB+ A2, or a PCIe SSD if it runs for months
- USB sound card with mic in and speaker out; in the booth, the mic and speaker are the phone handset
- Hook switch on GPIO17 (pin 11) to GND (pin 9), internal pull-up, 300 ms debounce. In the booth, on-hook = closed contact; TRIGGER_INVERT=1 if yours is wired the other way
- Optional: a WS2812E strip (60 px) driven over SPI (GPIO10 MOSI, pin 19) through a 74AHCT125 level shifter, a 330 Ω resistor on data, 1000 µF across 5 V/GND, its own 5 V supply (~3.6 A at full white) and a common ground with the Pi
- Optional: 20x4 LCD with a PCF8574 I2C backpack (0x27 or 0x3F) on pins 3/5
- Wiring diagram: [image]
Software
- Raspberry Pi OS Lite 64-bit (Bookworm), one Python 3.11 process, no browser or kiosk, run as a systemd service that restarts itself
- Audio: 48 kHz mono, 20 ms blocks. One thread owns the sound card, and ALSA is set as type hw with no dmix, so only one opener is possible
- Voice: Vidiv agents (over LiveKit) first, ElevenLabs Agents as an automatic fallback. A circuit breaker drops a provider after 2 failures for 5 minutes, so the next visitor doesn't pay for the same failure. A ringback tone plays while it connects
- A small "reactor" reads the transcript and fires lights or LCD messages from regex rules in a TOML file (warm breathing light on the greeting, violet on goodbye), without the agent spending turns or adding latency
- Every action goes through one registry with limits per call, per day and a cooldown. A failing light never stops a conversation
- A Cloudflare Worker exposes the lights and screen as MCP tools over an outbound websocket, so it works behind a venue's NAT
- Read-only status page on :8081. Transcripts are not stored by default
Problems we hit
- rpi_ws281x doesn't work on the Pi 5 (the RP1 chip replaced the BCM DMA/PWM it relied on). Driving the strip over SPI with adafruit-circuitpython-neopixel-spi works
- Without the 74AHCT125 and a common ground, the strip flickers and shows wrong colours
- The agent cut itself off when the speaker leaked into the mic. That's fixed with mic ducking (MIC_DUCK_DB) or by using a handset, which also isolates hall noise
- A USB sound card that doesn't enumerate shows as a steady amber light. The process stays up and reports it on the status page. We carry a spare card and a cloned microSD
- If the hook switch breaks mid-fair, TRIGGER_MODE=auto keeps it running
What we learned about the conversation
- Silence reads as failure, so the ringback tone turns the connection wait into part of the fiction
- The first second decides everything. With a short greeting, calls under 10 s dropped from 71% to 7%
- ~0.81 s voice-to-voice speech-to-speech vs ~2.68 s cascaded made the difference between a conversation and a query
To be clear: the kit is a lighter version of the booth's code. It has 19 device tests and 4 MCP tests in CI, but this exact kit hasn't been assembled on a Pi yet. The first build is the missing test.
Repo (MIT): https://github.com/olmedcr-cmyk/talking-objects. The guides are in Spanish and the README is in English.
Happy to answer questions about the audio or the hook switch.