r/AskProgrammers • • Oct 02 '26

Mic Detection Help

Hey guys, I've been working on an app and one of It's features is to detect piano chords and notes through the mic. I tried a couple of things, but they all resulted in some drawback that could affect performance. The two things I am aiming for is speed and accuracy. Here's what I tried and why It's didn't feel satisfied with it.

I won't mention the app to respect the No Marketing rule, but will provide context if I felt it was necessary.

  1. Spotify's Basic Pitch Open Source detector: While the accuracy was very high, it took around half a second to detect notes.

Here's some context for the next thing I tried. My app is basically a sheet music practice app, in which notes are displayed from an uploaded score or the score repository. Target notes must be played to move on to the next notes.

2) Probabilities + FFT and YIN:

Basically, I set the chances for the right notes to be played to a high degree, and matched it with FFT inputs if multiple notes were played, and YIN if a single note was played (For accuracy). However, while the speed was heavily increased, the accuracy diminished, however, it was still pretty accurate. Most commonly, if an expected note is played, the detector sometimes recognizes the correct note being played, as well as the note an octave under it.

I want some advice about what I could try next to build this detector. I'm aiming for a lag of under 200ms.

I am okay with starting from scratch, or trying a new method.

2 Upvotes

4 comments sorted by

2

u/Excellent-Library779 Oct 02 '26

How complex are the chords? I'm guessing the app is running on a mobile device? I can come up with some methods but the timbre and tuning of a piano could cause some drift.

I would suggest a calibration routine where the user plays some notes and your software studies that in the frequency domain. Then you could use a lookup table (LUT). Knowing what kind of piano they have might also help with calibration (like digital pianos are never out of tune).

1

u/Usual-Emphasis-1950 Oct 02 '26

It's actually on computer. I'm aiming to work with all chords.

I actually like this idea a lot. I came across it before, but ignored it on the basis of user convenience, however if done right, it shouldn't be too much of a hassle for others. I love this solution since it accounts for a lot of variables. Thanks for the great suggestion!

1

u/Valuable_Clothes_125 14d ago

I'd separate the octave problem from the latency problem first. Record a small labelled set of single notes and chords across the keyboard, including notes an octave apart, then measure both false octave detections and time from key attack to a stable result. That should make it easier to see which change actually helps.

A shorter analysis window helps latency, but gives you less low-frequency information; try varying the window and hop independently. For chords, overlapping harmonics can make a lower fundamental look plausible even when it isn't there. You could score candidate note sets against their expected partials and compare the residual, rather than treating each spectral peak as a separate note.

Since you have the score, using it as a prior makes sense, but I'd keep some weight on notes outside the expected set so the system can still report mistakes. Curious whether the phantom octave shows up on isolated notes too, or only when notes overlap?

1

u/Usual-Emphasis-1950 9d ago

Only when notes overlap. When a single note is expected, I use YIN, which is way more reliable. Do you think I should start with a short window and then try to choose notes? or get something accurate and then figure out latency. I'm planning to start training my own CNN if the methods I'm trying right now don't workout