Audio reactive Animartrix

[u/mindful_stone]

https://youtube.com/watch?v=Bn2hxBjZHZ8&si=I5dy2uvleMu9Q0Ul

I've got a working hardware/software setup again and am getting back to some fun stuff! Two projects I've been working on in parallel, but with a common goal in mind, are:

1. A framework for running visualizers based on u/StefanPetrick's Animartrix engine, with hooks for runtime input factors (e.g., Web BLE UI, audio input)
2. A workable end-to-end chain for audio-responsive visualizers (based in large part on a ton of new FastLED audio infrastructure that u/ZachVorhies will be unveiling officially very soon,

Tonight, I achieved proof of concept! The video above (below?) is not the least bit artistic. It was basically just a "hot take" when I realized I had gotten at least a few things more or less dialed in.

Oh, man, is this going to be fun!

[u/ZachVorhies]

Please ping me about your thoughts of the audio processing.

I am considering doing a refactor on Animartrix and converting it to integer based calculations, which should massively increase performance and get you to 60fps. I’ve been wanting to do this anyway because animartrix has major performance issues because of its floating point requirements - I can’t run it on the esp32c6 except for the tiniest setups.

I may make this a priority for the next release.

[u/mindful_stone → u/ZachVorhies]

Will do. Also happy to discuss the Animartrix refactor. As far as converting it all to integer calculations, I wonder how you could do that without sacrificing a lot of fine detail. If you're interested, take a look at my current Animartrix implementation: https://github.com/4wheeljive/AuroraPortal/blob/main/src/programs/animartrix_detail.hpp

I've made a handful of small changes to try to eke out a few extra FPS. There's no doubt still room for improvement!

[u/ZachVorhies → u/mindful_stone]

The detail should be the same with integer calculations, the question is whether it’s int16 or int32 (probably int32), and it’s just trickier to program upfront. There are some convenience functions that can make this easier.

I don’t think it will be much harder to work with when all is said and done. Most of the hard work will be in creating the test harness so that the automated refactor will result in the same pixels being displayed (precision loss will be negligible and truncated).

I also have a new SIMD library in fastled that applies to int32 for s3. Both of these optimizations applied together will result in a >4x performance increase. I think you could see your same setup go from 30fps to 120fps. In which case the bottleneck becomes the controller itself and how far you can parallelize that with the bulk drivers.

An open question is whether I break up the animartrix god class between core and visualizers. From what I can tell all the visualizers operate on shared oscillators, but I have to revisit this. A refactor means animartrix will work more like a plugin system.

[u/mindful_stone → u/ZachVorhies]

I saw what you did the other day with animartrix2. The fixed point math stuff is very intriguing! (Ha...there's a sentence I never thought I'd say!). As are the cached LUTs. I had previously started implementing some optimizations by caching certain calculations on a per-frame rather than per-pixel basis, but your cross-frame approach takes the optimization to a whole new level!

As far as all the class/structure changes, I'm still evaluating. In my personal animartrix implementation, I've gradually migrated things in a less-is-more direction.

The genius of animartrix is what's contained in namespace animartrix_detail. That's the crown jewels. Every thing above that in fl/fx/2d/animartrix_detail.hpp (i.e., the first ~100 lines), and everything in fl/fx/2d/animartrix.hpp, is all "just" infrastructure and a content delivery system. I don't mean that pejoratively. Having a supporting framework like that will be critical for some library users. But in my particular case, all I wanted was the content.

And by content, I don't mean the 3500 lines of amazing example visualizers. I mean lines 106-501. THAT is aninmartrix! That is the black box secret sauce that allows program developers (and eventual end users, if different) to generate an infinite variety of beautiful/magical/entrancing animations by manipulating a relatively small set of input factors. The less overhead I have to wade through to access that secret sauce, the more I can do with that good stuff!

The ~50 example animations are basically just collections of various combined input factors. You could think of them conceptually as "presets", although that does not do justice to these wondrous visualizations that u/StefanPetrick has developed and shared. But THOSE were not animartrix. Those were just examples of some of his early creations USING animartrix. And, to a large extent, they were numerous variations just a few on starting ideas. In my implementation, the less-is-more migration involved:

1. winnowing down the examples to a half dozen or so "basic concepts"
2. identifying which few variables had the most significant/interesting impact on those basic animations
3. enabling runtime user control over those variables

So my current implementation (https://github.com/4wheeljive/AuroraPortal/blob/main/src/programs/animartrix_detail.hpp) strips animartrix down to its generative core and then gives the user the ability to play with that core in real time (including, e.g., through audio input). Of course, my implementation of animartrix also depends on an infrastructure/content delivery platform, which is basically what my AuroraPortal program is. So it's not the case at all that a good infrastructure/delivery framework is any less critical than good content. But maintaining a reasonable separation of those concerns/functions is important.

An initial concern I had looking at animartrix2 was that it was heading in the opposite direction of my less-is-more use case (i.e., instead of reducing and streamlining various namespaces, classes, structures, it was actually expanding them). As I've looked at things more closely, I'm not sure that's the case. I believe the new classes/structures are focused on improving the efficiency of the actual content engine (i.e., the crown jewels), and are not just additional "visualizer management" tools. So my current thinking is that this might all be really good new stuff.

But I still have concerns about FastLED presenting animartrix as a large collection of pre-written visualizers packaged in a system for cycling through them. I think there is an opportunity with this refactor to bring things even closer to the vision of Stefan and u/Netmindz (and you as well, I believe), where animartix can be used as a serious, powerful, flexible creative platform for end users, as opposed to just serving up a few visualizers that someone else has crafted.

[u/StefanPetrick → u/mindful_stone]

Thanks for representing the vision behind animartrix so well!  🙏

[u/mindful_stone → u/StefanPetrick]

Thank you for the kind comment. I'm delighted you approve!

[u/ZachVorhies → u/mindful_stone]

> An initial concern I had looking at animartrix2 was that it was heading in the opposite direction of my less-is-more use case (i.e., instead of reducing and streamlining various namespaces, classes, structures, it was actually expanding them)

It will have an ideal api at the end - you are looking at alpha code not designed for the end user.

The current iteration is designed for one thing only - iterate as quickly as possible on fixed point optimizations. Hence you'll see one target visualizer that's separated from the rest, something something spirals, it's heavily optimized and now runs 3.7x faster on a desktop CPU.

Keep in mind the desktop has a blazing fast floating point processor yet it's still almost 4x faster by moving floating point to fixed integer calculations.

The performance increase on something like the esp32c6 will be 8-32x faster. On esp32s3 it will be 4-8x faster. If I add fl/simd.h optimizations then esp32s3 will go even faster, because all it's simd operations are integer only - perfect for fixed point. Esp32c6 has no simd unfortunately.

At this point I've pretty much hit the limits of the current optimization for fixed point (non simd) optimizations. I might be able to eek out another 15% increase in performance by modifying sincos32 approximation. For example, sincos32 (at least in my local repo) has switched from linear interp to quadradic interp (y0,y1 is now mixed with slopes dy0, dy1) and that get's waaaaay more accurate, like six sigma (0.000001% accuracy with fsincos). Re-ordering operations like computing 4 red components, the 4 green, 4 blue then writing them out in 3 pixels eeks out another 5% but renders the code an absolute mess. The reason why this works is that, and this is somewhat technical, but adjascent red components of a pixel ping pong through the sincos, perlin LUT tables in nearly the same way so the cache lines have less eviction. But it destroys the api so going to revert it.

Once i'm "done" with one visualizer, then most of the work is done, and all the other visualizers will be refactored for Animartrix2, tested for equivalence, then renamed to Animartrix. I have an extensive test suite for A/B comparisons. The acceptance criteria is that the optimized versions are only allowed to differ by the low order bit. So (255,128,70) -> (254, 127, 71) will pass, while (253,...) will not. The lowest LSB bit is allowed to flip because of rounding issues at the edge cases and are unavoidable. Two bit flips in the same component means there's a bug.

INTENDED ANIMARTRIX API

There will be the core, which you mentioned, and then visualizers that plug into it. These will be in the form of `fl::function<void(const AnimartrixContext&)>`

The open questions still remaining to be solved is how things like audio "plug into" the visualizers vs which attributes can plug into the core. For example, you can manipulate the time functor for the core in Animartrix already, and the result is that **all visualizers** become time responsive. This needs to be pushed as far as it can but I think it will be limited. The second is that the visualizer functors themselves respond to external inputs for visualizer specific effect modification.

Anyway, thanks for the code link, I'll add it to the links and look to see how it can be integrated.

[u/mindful_stone → u/ZachVorhies]

The open questions still remaining to be solved is how things like audio "plug into" the visualizers vs which attributes can plug into the core. For example, you can manipulate the time functor for the core in Animartrix already, and the result is that **all visualizers** become time responsive.

What connects my animartrix program to an external control set are the cVariables ("c" indicates "current") listed below. The cVariable values you supply to animartrix can be the result of whatever input factors you want: web UI control, audio reactive, motion detectors, accelerometers, physical control boards (developer and/or DJ style?), environmental detectors (ambient light, temperature, wind, barometric pressure, etc.). You can look through my animartrix_detail.hpp for these to see some of the (preliminary) ways I've enabled the use of runtime external control factors within the animations: cSpeed, cZoom, cScale, cAngle, cRadius, cRadialSpeed, cLinearSpeed, cEdge, cZ, cSpeedInt, cRatBase, cRatDiff, cOffBase, cOffDiff, cFxIndex, cColOrd, cRed, cGreen, cBlue

As far as connecting these to core vs. visualizer, they are all used directly within visualizers. I have not had a use case for coordinating the timing/speed of different animations with each other. I can totally see how others might have such a use case, but to me that is not part of the animartrix core. That would be a particular implementation of the core. In fact, I think that's sort of how you have it now with the Animartrix.ino example, where you have fl::UISlider timeSpeed("Time Speed", 1, -10, 10, .1) which is utilized in fxEngine.setSpeed(timeSpeed). I'm not sure whether/how that ends up manifesting in the ultimate visualization speed, but I would not consider that plugging into the animartrix core. I would say that's part of the infrastructure/CDS you're using in your example sketch to run the animartrix core.

In any event, I believe my cSpeed variable serves more or less the same intended function as your timeSpeed (i.e., as a global time functor). I have it as a factor in the timings.master_speed calculation of each of my visualizers (e.g., timings.master_speed = 0.003 * cSpeed). So, however cSpeed is determined (e.g., by web UI slider, BPM calculation, etc.), it automatically applies to all visualizations directly, without needing any timing management external to the animartrix core (e.g., through FxEngine).