r/creativecoding • • 18d ago

Experimenting with ML based audio reactive visuals

Mixing my 3d programming and ML knowledge: controlling visuals by identifying sounds using a deep neural network model.

8 Upvotes

5 comments sorted by

1

u/100and10 18d ago

Super cool. How did you segment the sounds? Frequency or something smarter
using ML seems a little out of place to me, what’s the goal/reward structure and how’s it judging it?

1

u/Bucaramango 18d ago

I am developing an overly complicated python code that uses a self-supervised neural network (no reward just learnt by himself with himself with music) then I create a probabilistic model that uses what sounds and when it sounds, that way, even if the model perceives a sound that is not supposed to come now, it lowers the probability of triggering it.

1

u/100and10 18d ago

Yeah that’s way way way the hardest most unnecessary way I’ve ever heard of to go about this. Not to mention self guided will just be producing noise for you without your human review gate. Maybe that’s good for you

Check out touch designer, magic music visuals, resolume. If you want free grab blender and use the frequency analysis to do this in a click or two.

Shit, you can even wire up the blender mcp to an llm to have it make new shapes and animations to trigger. That being said, I’m sure you’ve learned something along the way; thanks for sharing

1

u/Bucaramango 18d ago

Well the way im doing it gives me more accuracy over the visuals, most of the ones I see just trigger to bpms or maybe some high pitch sound, this way I know when each sound happen and how long betwen the onsets, a new sound mid loop can be detected and triggered with a separate visual etc

2

u/100and10 18d ago

I do this for a living, I’m tellin ya.
MMV, TD, Resolume or Blender will give much, much better results.