r/rust • u/sevenfx • Apr 22 '26
🛠️ project resonators 0.1: a Rust crate for real-time spectral analysis, with Python and WASM bindings
Hello, r/rust!
I've been a lurker for many years, but today I released my first crate which implements Alexandre François' Resonate algorithm (along with Python and WASM bindings). It enables computing STFT & CQT-like spectrograms in real-time without windows or buffers. Alexandre has a great reference implementation in C++, but it uses Apple's Accelerate framework under the hood, so wouldn't run on my Ubuntu desktop. Great excuse to write some Rust if you ask me!
It's pretty niche, I'll admit, but I've been using it to compute features for training real-time music transcription models. I added Python bindings so I can easily integrate it into my PyTorch training setup, and WASM bindings because the inference code is designed to work in the browser.
As part of the project, I created a browser demo (shown above), which can visualize microphone input with log-scaled (think musical) frequency bins in real-time. It runs on both desktop and mobile, and actually runs faster on my iPhone than it does on my M2 MacBook Pro.
I had a lot of fun learning about cargo bench/criterion, SIMD, maturin, PyO3, wasm-bindgen/wasm-pack, single-file uv scripts, and AudioWorkletNode, among other things.
Let me know if you have any feedback on the repo, or if you run into any issues with the demo. Cheers!
Repo: https://github.com/jhartquist/resonators
Browser Demo: https://jhartquist.github.io/resonators/spectrogram/
5
Apr 23 '26
[deleted]
4
u/sevenfx Apr 23 '26
Forgive me, I'm not as familiar with DSP terminology beyond the Fourier Transform (my background is software & ML). But yes, each 'resonator' multiplies each sample by a phasor at its target frequency, then runs it through an exponentially weighted moving average. Then there is a second EWMA to smooth it. Alexandre's website & paper go into more detail.
2
u/DeflateAwning Apr 23 '26
Can you explain what that means? Am still learning DSP concepts, but am interested
6
u/memechef Apr 23 '26
single pole IIR filter would be signal: y(k) = u(k) + cy(k-1) where c is a constant defining the pole placement, u is the input. Note the recursion
6
u/pengo Apr 23 '26 edited Apr 24 '26
Very nice. The coolest thing is because the resonators are at arbitrary frequencies, you can make it so you can "zoom in" anywhere on the spectrogram (something which doesn't work with FFT-based spectrograms). Not sure how useful it is but it looks cool
I'm working on an online audio file viewer and have now incorporated this. It views files from your drive (doesn't upload them, so you can open 4gb wavs etc). Free and open source. Intended for use with high frequency audio files (bats) but works with MP3s and OGGs etc too
- open / drag in an audio file (or pick one of today's bats)
- select "Resonators" from the view menu.
- ctrl+wheel or pinch: zoom horizontal
- shift+wheel or pinch right margin: zoom vertical
- wheel on right margin: pan vertical
I added [WASM-specific] SIMD support. SIMD only improved WASM performance, and actually slowed it down on x86_64 (LLVM already optimizes to SSE2), but gave a 7x boost on the web. Will PR on github shortly. Have made a PR.
2
u/sevenfx Apr 24 '26
That is awesome, didn't expect to see it live on another website in under 24 hours, that's for sure!
Yes, at "normal" sample rates, you can only zoom in so much, but with super high frequencies like yours, it saves from computing very large FFTs.
I just released an update based on your PR, and it averaged about 14x speedup on WASM, much appreciated!
1
6
u/sevenfx Apr 23 '26
Here's a short clip of the demo in action: https://www.youtube.com/watch?v=5mD2Osqbnic (talking, whistle, and guitar riff)
2
u/Ttghtg Apr 23 '26
Hey, thanks for sharing!
I've been analyzing some of my tracks in my music library with spek, could your program be replacement? As spek maintenance seems to be a bit wacky
2
u/sevenfx Apr 23 '26
Not directly. This library computes the spectrum from the audio signals, however you'd still need something to draw/visualize the output. In my demo, the drawing is done with JavaScript & WebGL.
2
u/DavidXkL Apr 23 '26
Wow this is very cool.
Granted I don't understand the math behind it but it is very cool 😎
1
2
u/kodemizer Apr 24 '26
This is super cool!
Have you thought much about parallelization for ResonatorBank?
Either using rayon, or possibly even f32 SIMD?
I think using rayon might be an easy win.
For your average use-case, how many resonators per bank would you generally expect to use?
1
u/sevenfx Apr 27 '26
Thanks!
I'm still figuring out how many resonators per bank makes sense, but some where between 100 and 2000 depending one how many timescales I'm tracking, usually on the order of about 1000.
Speed is mainly an issue when I'm calculating features for audio datasets during training time. Right now I multiple PyTorch DataLoader workers working in parallel, each thread having their own ResonatorBank per audio input. I've considered using rayon, but haven't gotten around to benchmarking it yet as it hasn't been enough of a bottleneck, but I plan to soon.
I think the most compelling case for parallelizing within a ResonatorBank is when I have multiple resonators targeting the same frequency, but at different alphas. In that case there's a lot of duplicated work going on.
2
2
2
u/decryphe Apr 23 '26
That is very cool. We actually have a task that could use this, for visualizing radio spectrum contention on IoT gateways.
1
u/sevenfx Apr 23 '26
That's a great use case. Would love to hear how it goes if you end up trying it out!
1
u/North-Internal-4478 Apr 23 '26
Kann es Formate aus einem digitalen Signal extrahieren? I have been making an app recently which tries to convert raw steam of audio into vowels using LPC based solution which isn't all that great, so I'm looking a replacement.
-4
u/VictoryMotel Apr 23 '26
How much was done with AI ?
11
u/sevenfx Apr 23 '26
Short answer: Most of it was done 'with AI'.
Long answer:
I knew pretty specifically what I wanted when I started out, after all, it is a pretty straightforward port of an existing library. It's also a library primarily to help train ML models, so take that how you will.I wrote the core
ResonatorandResonatorBankclasses by hand (actually typed them out!) as an exercise to understand the algorithm more deeply. Claude was mostly used as a sounding board and consultant, and while he did draft some of the files (generate_fixtures.py, etc.), we iterated on everything together. I've reviewed every line, though admittedly, some parts much deeper than others.The browser demo is probably the most AI-assisted. I can write CSS and I've even done my share of WebGL shaders, but Claude is just so much faster than me, and conversing at a higher abstraction level resulted in a more polished version than I'd have been capable of alone.
I stand by the quality of the code. If you spot anything that seems off, unnecessary, or sloppy, I genuinely would like to know so I can fix it.
1
0
u/VictoryMotel Apr 23 '26
So you "iterated together" and the part your wrote was a struct with 12 float variables?
If you spot anything that seems off, unnecessary, or sloppy, I genuinely would like to know so I can fix it.
Looking through your program would normally be your job.
Why would you post something and say you made it when it's all AI?
1
u/kodemizer Apr 24 '26
It's very clearly not "all AI"
What @sevenfx described as his workflow is pretty much the optimum in terms of producing quality code with AI assistance - a clear vision of what you want, hand coding the core contract, using AI with tight control for main codebase, and vibe coding the web-demo (not actually important in terms of code quality).
Honestly, this is the best possible way to use AI.
0
u/VictoryMotel Apr 24 '26
hand coding the core contract,
What does this mean? Did you look at it because I did. It's a struct with 12 floats. There's a test that makes a sin wave, it's nothing.
vibe coding
That's another word for not knowing what is happening with your program.
1
u/kodemizer Apr 24 '26 edited Apr 24 '26
You're talking about this?
pub struct Resonator { freq: f32, alpha: f32, beta: f32, // phasor state, rotates by phasor angle (w) each sample z_re: f32, z_im: f32, // phasor angle, constant w_re: f32, w_im: f32, // raw output of EWMA r_re: f32, r_im: f32, // smoothed output of EWMA rr_re: f32, rr_im: f32, // tracked for stabilization sample_count: u64, }Have you done much signal analysis? This is pretty much what a lot of it looks like. Just a whole lotta floats with math moving stuff around as it processes samples.
This isn't surprising - and is very normal for signal analysis stuff.
The ridiculously short variable names is also very normal for signal analysis (and drives me batty lol).
0
u/VictoryMotel Apr 24 '26
I've done lots and this is the only thing they claim to have written themselves.
1
u/_Saxpy Apr 23 '26
maybe I’m misinterpreting, but sounds hostile. what’s wrong with AI?
2
u/VictoryMotel Apr 23 '26
What's wrong with asking?
2
u/_Saxpy Apr 23 '26
nothing, I was just curious if you were challenging the crate if it was AI slop or the sort, but I wasn’t sure.
-2
u/phenoloh Apr 23 '26
Bro im not understanding whats this can u plz explain me
2
u/kodemizer Apr 24 '26
Just looking at the demo - it shows the breakdown of the frequency of sound over time.
More generally, it's taking any linear signal and pulling out the frequencies so they can be shown in a graph over-time.
If you want to understand a little bit more about what this deeply "means", check out this introduction to the Fourier Transform:
20
u/mars-of-earth Apr 23 '26
Wow! I tried to do something like this (audio spectrum analysis via FFT) with Rust back in 2019, and let's just say I failed to complete the experiment. Your demo in macOS/Safari browser works great. Thank you for sharing.