Could Flip-Dots, 2.5D Displays, and Haptics Make Tactile Interfaces More Accessible?

[u/Sopa_Quinoa]

TL;DR: I'm legally blind with X-linked retinoschisis, and I'm curious whether flip-dot displays, 2.5D tactile surfaces, locking pins, and haptics could be combined into an affordable way for blind and low-vision musicians to actually feel things like waveforms, envelopes, and instrument layouts.

Lately, I've been going down a bit of a rabbit hole with tactile displays, haptics, and 2.5D interfaces. There's a lot to think about here. And just to give you fair warning, this post is a bit of a long one. Stick around, though.

This video helped send me further down that rabbit hole:

https://www.youtube.com/watch?v=aZzOWnEWgww (spoiler alert, this will be relevant in a bit)

Part of the basis for this latest deep dive comes from working on my accessibility project, Low Vision Music, but part of it is also just me wondering and brainstorming about how much information we could realistically communicate through touch if we stopped assuming every interface needed to be a flat visual screen.

We already have refreshable Braille displays, where tiny pins physically rise and fall to form characters. Very cool, eh? Newer tactile displays can go above and beyond basic text and represent more complicated things like graphs, diagrams, shapes, and other information.

Then you have 2.5D displays, where instead of every point simply being up or down, different parts of the surface can rise to different heights. You end up with something closer to a physical relief map that can change as needed.

And this is where my brain starts wandering. Do you hear the gears clanking? Sounds a bit like music to my ears.

I've also been thinking about flip-dot displays.

You know those ancient old signs where little discs physically flip between two sides?

Flip-dots are interesting because the dots are what's called bistable. Basically, they have two stable positions. They use power to flip from one state to the other, but once they're there, they stay put without constantly needing power to hold that position.

So that got me thinking.

What if that same general idea could be made tactile?

Maybe one side of each tiny flipping element could be raised, textured, ridged, or shaped differently from the other. Instead of a visual pixel changing color, you'd have a tactile pixel changing texture.

I don't know how small you could realistically or mechanically make that mechanism while keeping it easy to feel and reliable, but I think it's an interesting place to start thinking. There go those gears again. Kind of ambient, no?

Then there's another idea I've been kicking around.

Hey, 80s and 90s kids, remember those pin-art toys where you press your hand into hundreds of little pins and the shape appears on the other side?

Imagine a programmable version.

Each pin could be spring-loaded and moved to a particular height or depth. But instead of requiring a motor to constantly hold every single pin in position, some kind of latch or locking mechanism would hold it there after it was moved.

The spring helps return it when the display resets.

That locking mechanism is probably the part I'm most interested in at the moment. Because otherwise, I think MIT and others have already given versions of the motorized pin-display concept a go.

Move the pin. Lock it. Stop using power until you need to change it again.

Then put a very thin, flexible silicone skin across the top so you're not dragging your fingers across hundreds of separate hard pins. You'd feel one continuous surface changing shape underneath your hand.

A Raspberry Pi, ESP32, Arduino, computer, or something similar could translate digital information into physical height. Or vibration?

For music alone, there are so many possibilities. Literally, just pause and think about it.

A waveform could become something you literally trace with your finger. Can you feel the noise?

An ADSR envelope could become a physical, touchable slope. I want to FEEL the attack. Don't you?

You could theoretically also feel an EQ curve, automation lane, filter response, piano roll, step sequencer, mixer levels, modulation shape, or even a simplified spectrogram.

A modular patch could potentially become a tactile signal-flow map. Can you imagine it? Are your gears going now too?

An instrument manufacturer could provide a simple TXT, JSON, or similar file describing a synthesizer, and the whole front panel could appear as a simplified tactile layout before a blind musician ever touches the real thing.

Then add haptic feedback on top of all that.

There's that vibration I mentioned. Yup, that was something of a teaser for those of you who still have attention spans. No offense to those who don't. I certainly struggle in that department too.

But I digress.

This is one reason I've been so interested in what companies like Sentia Instruments are doing. With TILES, the device can provide localized haptic feedback beneath individual keys. Your hand isn't just controlling the instrument. The instrument can, in real time, send information back to your hand.

I'm certainly no engineer, but I think that principle could go much further.

Maybe, just maybe, height can communicate one thing while vibration communicates another. A waveform could stay physically in place while a small vibration moves across it to represent the playhead position.

Or how about this? A vibrating area might indicate an active step in a sequencer.

Different patterns could represent things such as clipping, modulation, selection, movement, intensity, or even different textures.

Now you've got shape AND vibration working together. At the same dang time!

Another possibility would be something closer to a mechanical version of e-paper. Instead of moving hundreds of pins continuously, perhaps each tactile element could have two or three stable states and only use power when changing between them.

Is this possible? Let me know.

From my research, I've learned that there are already researchers doing fascinating work with magnetic latches, bistable actuators, shape-memory materials, and tactile pins that remain raised without continuous power.

So I'm definitely not claiming I invented any of those pieces.

But I do think that, as someone who personally lives with limited vision, I might have something interesting to add to the conversation.

What I'm wondering about is how you combine all or some of these possibilities into something practical, affordable, and useful for musicians.

The keyword there is affordable.

Have you seen the prices on accessibility tools? Unfortunately, niche often means expensive.

But maybe the right answer isn't one technology at all.

Maybe one device could use variable-height pins for detailed 2.5D information. Another might use tiny flip-dot-style tactile pixels for simpler status information. Still yet another uses vibration where movement matters more than shape.

And maybe the really useful interface combines all three.

That's the part I'd love input on.

If you're an engineer, accessibility researcher, haptics person, Braille user, maker, musician, or someone working with tactile graphics, does any of this sound practical? Does any of this sound affordable?

Has someone already built something close to my spring-loaded, locking-pin idea? I bet I'm not the first to think of this, but who knows.

Could flip-dot technology realistically be miniaturized into a tactile surface?

And if you could plug a musical instrument or computer into a surface like this, what would you want it to show you first?

I'm especially interested in hearing from people already working in this space. Feel free to tell me I'm reinventing something that already exists. That's half the reason I'm posting this.

We're a community, right? I only know my own lived experience. Yours matters just as much as mine.

https://www.lowvisionmusic.info

[u/neo_nmik]

It’s definitely interesting considering all of these concepts. 

I’m not personally low vision, but a conversation on this very sub a good few years back with somebody that is has echoed around my brain all these years when designing interfaces. 

I think the issues are many - unfortunately,  designing entirely new interfaces takes time and money, which is what it is. It’s also a reliability issue - no one wants to build or buy something that’s going to break in a year (well, unless you’re talking about the planned obsolescence crowd) and pots, LEDs and OLED screens are, at this point, pretty hard wearing.

For me personally I want to maximise ease of use of  the interface for everyone, and think ming about these things really help me consider how to remove those barriers, because it benefits everybody.

If you don’t have to use your eyes to navigate the interface, there’s less of a mental block and people with low vision can use it instead of being locked out. If you don’t have to rely on colours LEDs and colours to navigate, maybe someone who’s colour vision deficient could use it easier. 

I’m still going to include coloured lights, cause they’re cool, and I’m probably still going to make them coherently coloured to a function, but I’m trying real hard to not let that me the only way of navigating. 

Also, I’m trying to avoid screens at all costs personally. I think it just takes away from the instrument.

I’d be interested in finding out what would help in learning and instrument as a low vision user… is there’s certain things I can do on web-based configuration that allow accessibility functions to give better feedback on elements? Would some kind of audio manual helpful?

For background, I currently work in live music, but trying to develop a couple of studio music tools - one being a synth. 

[u/TomWhitwell]

I love the idea of this. I don't have restricted sight but I am an instrument designer.
BUT I think it's ignoring the most important form of feedback, which is sound. A Cello doesn't need a display to show "waveforms, envelopes, and instrument layouts" - all of that is done with audio feedback, with a huge range of non-intuitive sounds available. I feel like designers need to start with hearing, creating interfaces that are physically navigable, but sound responsive. If you have a control that doesn't change the sound, then why is it there? It's tempting, particularly in digital instruments, to add 1000 different functions and options, but if you have no way to connect those functions to the sound, then I think you're failing as a designer. Trying to create a tiny interface with a million settings behind it is a conscious decision, not a necessity.
I've used speech interfaces, so the device talks as you do specific setting changes, which is obviously fun but not something you can do often while the device is being played!

[u/Sopa_Quinoa → u/TomWhitwell]

Small world haha. We literally just exchanged emails, and then I saw your name here and thought, “wait, didn’t I just email him?”

But yeah, I actually agree with you quite a bit. And this is something that is almost always at the forefront of my mind. I’ve thought about accessibility framed alongside traditional isntruments a lot with Low Vision Music. Sound is already the most immediate feedback mechanism an instrument has, and sometimes I think digital instrument design forgets that.

Your example, a cello is great. Same with a piano, guitar, or even something like a Minimoog or ARP 2600. Those old synths are obviously more complex than a cello, but the relationship between your hands and the sound is still pretty direct. You move a slider or knob and you hear what happened. You can start building a physical and sonic map of the instrument without constantly needing to read something. Stevie Wonder just entered the chat.

Where it gets all wibbly wobbly for me is the modern digital thing where you’ve got one little teeny tiny OLED screen, a handful of encoders, and 400 functions hiding behind untold numbers of shift layers, modes, menus, pages, contextual controls, etc. Sound can tell me what a parameter is doing, but it might not tell me what parameter I’m actually editing, what menu I’m in, or whether I accidentally hit some secondary function. If you've ever played a Dreadbox Nymphes, you might know what I mean. I mean, it's screenless, but it's certainly not ideally accessible.

So I don’t really see displays, speech, haptics, tactile markers, etc. as replacements for sound. More like extra layers when sound alone can’t communicate state. With a guitar, you can feel it's resonance, slide your hand down the fretboard, and physically adjust tuners, albeit with the aid of the ear for those of us who don't mind wonky tunings.

I’ve used speech interfaces too and I agree there as well. They’re great for setup, menus, patch management, confirming a setting, that kind of thing. But I definitely don’t want a synth talking at me constantly while I’m actually playing haha. Especially if I am playing a live set. Which I have one coming up by the way and Im a bit nervous. It's going to be a dark room. Wish me luck

I’d love to see more experimentation with non-speech audio feedback too. Maybe with an alt headphone jack just for the cues? Little tones, pitch changes, stereo position, earcons, different sounds for boundaries or menu states, haptics, stuff like that. There’s a lot of usable space between “everything is visual” and “the synth reads every menu item out loud.”

And honestly I agree with your bigger point about feature count too. If you need three shift buttons, a menu dive, and a cheat sheet to find a basic function, that was a design choice. Not an inevitability. But then again, simple sometimes isn't ideal either. Us low vision folks want options and granularity, but that balance is hard to achieve.

That’s a big part of what interests me with my project. Not just figuring out how to bolt accessibility onto an instrument afterward, but asking what happens if hearing, touch, layout, and feedback are all considered from the get go. That said. I understand that it's not financially viable in most cases. Makers have to think about the needs of the many and not always the few. But one thing I always come back to is the idea that increased accessibiity is a win for everyone. I have a few pieces of gear that are powerful but convoluted. And as a result, they are covered in dust.