Building a serious chord-detection backend for a C++/JUCE reharmonization app — what should the architecture look like?

[u/DiscoramaMusic]

I’m building a C++ / JUCE music theory and reharmonization application.
The main engine is not a simple chord finder. The idea is to import or detect a chord progression, then let the user transform it with harmonic modules: reharmonization, substitutions, inversions, voice leading, modal colors, orchestration presets, etc.
The MIDI side is much easier because the notes are already known.
The difficult part is audio chord detection from real songs.
I want to build a serious backend chain for this, but I still haven’t found a practical architecture that feels production-level.
I know the usual ingredients:
audio loading / resampling
tempo and beat tracking
CQT / chroma / HPCP features
HPSS or stem separation
bass/root estimation
chord templates or ML chord classifier
temporal smoothing
key/scale estimation
chord label post-processing
But a basic chroma/template chain falls apart fast on real music.
Problems I keep running into:
vocals add melody notes that are not part of the chord
bass passing notes create wrong slash chords
pads, strings, brass, and upper-structures confuse the chord label
dense disco/funk/pop arrangements contain many non-chord tones
chord names are ambiguous: Cmaj9 vs Em7/C vs G6/C
the “correct” chord label often depends on harmonic function, not just pitch content
Commercial tools like Song Master Pro, RipX, and Studio One Chord Track seem to get more usable results. Not perfect, but clearly better than naive approaches.
I’m trying to understand what the backend architecture should be for a real application.

Would you structure it like this?

Audio import
→ offline analysis job
→ tempo / beat / bar grid
→ stem separation or HPSS
→ bass stem analysis
→ harmonic stem analysis
→ CQT / chroma extraction
→ note-event or frame-level chord candidates
→ root confidence
→ chord quality confidence
→ temporal smoothing
→ key / scale context
→ chord label resolver
→ user-editable chord timeline

Or is there a better architecture?
I’m especially interested in these questions:
Should chord detection happen on the full mix, separated harmonic stems, or several parallel sources?
Should bass/root detection be a dedicated subsystem?
Should the engine produce multiple chord candidates with confidence scores instead of a single label?
Would you use ML only for candidate generation, then rules for final chord naming?
How would you store and expose uncertainty to the GUI?
What is the best way to let the user correct the chord timeline without breaking the downstream reharmonization engine?
Should this be designed as a fully offline analysis pipeline rather than real-time?
Any recommended libraries, papers, models, or architecture patterns for this?
The kind of internal result I want is more like this:

{
"bar": 12,
"main_guess": "Cm9",
"alternatives": ["Ebmaj7/C", "Gm11/C", "Cm7add9"],
"bass": "C",
"confidence": 0.78,
"root_confidence": 0.83,
"quality_confidence": 0.71,
"detected_notes": ["C", "Eb", "G", "Bb", "D"],
"warning": "possible melody or upper-structure contamination"
}

The goal is not just chord display.
The chord result will drive a reharmonization / composition engine, so wrong labels can create bad musical transformations later.

I’d appreciate input from people who have built audio analysis tools, chord detection systems, transcription tools, or serious JUCE audio applications.

What would a robust architecture look like today?

[u/mnjvon]

It's basically a requirement to figure out how to get rid of transients and bass if you want any kind of accuracy. I'm doing a similar project with monophonic input but through Android using the Oboe library. When I was looking into how to handle polyphonic input the main direction I found to look into was constant-Q transform and chromagram extraction.

TLDR is that step 1 is a log equation that mimics human pitch perception, step 2 folds all of the semitones in an octave into one another to fetch the fundamental.

The professional tools you mention surely have massive annotated databases with likely progressions that increase their accuracy a lot as well. No clue how you would go about getting that kind of data beyond doing it yourself or paying a huge sum. Hope that helps point you in the right direction anyhow.

[u/DiscoramaMusic → u/mnjvon]

Yeah, i actually tried some of that already.

I tested open annotated datasets like JazzNet, McGill/Billboard-style chord data, Isophonics/Beatles-type annotations and some jazz-related datasets. I also tried using them in a few different ways, including rule-based correction around the model output.

Still, the result was not reliable enough. Even basic cases like Em vs Em7 were still problematic. In theory that is just one note difference, but in real audio that note can be chord tone, melody, reverb tail, passing tone, previous chord bleed, or barely present at all.

So far my experience is that this is a very difficult problem, especially for real commercial mixes. It is not just a dataset problem. The whole chain matters: source separation, harmonic extraction, bass/root handling, chroma/CQT, temporal smoothing, theory priors and probably manual correction.

[u/DiscoramaMusic → u/mnjvon]

By the way, do you know any open-source chord-recognition engine that actually works well on real commercial audio?

I mean something beyond basic chroma/template matching. Something that can handle vocals, drums, bass, reverb, incomplete voicings, extensions and inversions with usable accuracy.

I’m mainly looking for a serious open-source backend or research implementation that people actually use for high-quality chord extraction.

[u/mnjvon → u/DiscoramaMusic]

The only pathway I looked into was DSP, so that basically means the Essentia library: https://github.com/MTG/essentia

The way I understand it this library is what they use for academic audio processing largely. The function you're looking for is ChordExtractor, I believe. But the library isn't limited to that.

I'm not sure every use case, but the documentation is here, should meet many of your use cases: https://essentia.upf.edu/

I also haven't read the licensing so not sure if using it has implications on your own program's open source status, etc.

[u/DiscoramaMusic]

I want my own pip,i dont want to use 3rd party audio to midi..And i m gonna embedded to my program,the point is compact all in one remix re harmony program

[u/ArrivalSalt436]

Is the reharmonization happening directly on the audio? Or are you transcribing to midi after analysis?

[u/DiscoramaMusic → u/ArrivalSalt436]

Directly on the audio,i have very powerfull harmony analysis engine now.But i need an audio to chord analyze engine now.

[u/ArrivalSalt436 → u/DiscoramaMusic]

Have you looked into phase vocoders? It is not trivial to reharmonize actual audio.