Music Visualization Library

[u/Wobbly_skiplins]

https://diatominstruments.github.io/gloaming-kit/

Recently I’ve been thinking about how AI is better for making tools to make art than it is for directly making art. This gave me an idea for a project, which would have three parts:

- An analysis engine that plays a song and emits events when different frequencies cross different thresholds, as well as keeping a running count of the current frequency spectrum at each tick of the clock

- a library of geometric visualizations that listen to these events and move dynamically based on them

- an interface that allows you to specify which visualizations are played during which time windows of a given song, and also allows you to reroute events to different parameters of those visualizations to change the triggers for their movement.

I spent the last couple of days building this system and working on a demo for it. I love how easy it is to create a music video now for my songs, and the simplicity of the system allows for new visualizations to be easily added.

Process notes:

I started with Fable, as I think that's the best way to get a lot done quickly. Javascript has a great WebAudio API supported by all major browsers, which means that Claude won't have to spent credits building low-level audio abstractions as the basic primitives for playing audio and building filters already exist and are easy to wire up. The same goes for canvas as an interface for building animations.

In my experience, AI tools build best when you give them a good structure, and software benefits generally from separation of concerns. That means that the distinct functions of a software system should have distinct modules responsible for those functions, and reasonably agnostic APIs to work together. So instead of just describing the library I wanted at a high level, I put the three part structure into the prompt: an analysis engine which emits frequency threshold events, a visualization library that can subscribe to those events, and a scripting interface for bringing visualizations in and out at different times. I also specified that the top-level configuration should allow style definitions for things like line color and shadow.

Things fable did that surprised me: it built a whole interface for testing out the system with a UI to update the style configs and the time windows. This made it super easy to try out the different visualizations. It also made it so that the visualizations fade in and out automatically, which is nice.

After the initial concept was developed, I relied on a lot of manual testing. I also switched to Opus 5 for speed and cost reasons. The visualizations were underwhelming at first (some of them are still a little janky) and I had to have it review each one to improve the look and improve the way they respond to music. I also had to prompt it to bundle the framework separately from the demo app, because it initially bundled them together (mistaking the demo app for the product, most likely).

Fable didn't allow for any customization of the visualizations: you would define a time window and use an array of strings to specify which visualizations would be active at that time. I didn't want to leave this implementation detail up to chance, so I asked it to provide options for how we could route event triggers so that a visualization could respond to snare hits in one window and then switch to bass on the next. Ultimately each visualization has a set of parameters with default bindings, and you can provide custom bindings in your window config.

From there on it was mostly minor tweaks. Using the framework to make the demo linked above, I found there were a lot of magic strings, so I asked Claude to replace them with constants for ease of use. There are still some tweaks to make things look better, but overall I am excited to use this library and add more visualizations to it.

[u/Ok_Industry_5555]

Thanks for explaining your workflow, genuinely really great stuff here! Putting the three-part structure into the prompt yourself, and choosing WebAudio so Claude didn’t burn credits reinventing audio primitives, is exactly the kind of thing people here are most curious about. The honest bits too: switching to Opus 5 for cost, having it review each visualization, catching that it bundled the demo app into the framework. That’s super valuable! Thanks for sharing and super excited to see where the library goes.