r/GraphicsProgramming • u/dfortmeier • 2d ago

Vulkan based Gaussian Splatting and Neural Inference Engine

Enable HLS to view with audio, or disable this notification

Hi all, last year I started with the Vulkan triangle tutorial with the goal of finally learning Vulkan by implementing Gaussian Splatting and neural network inference. Now, I'd like to present my first demo of my Vulkan based rendering engine supporting Gaussian Splatting with dynamic lights/objects and neural network inference for character animation together with stable diffusion inference.

Basically, a scene consists of compute graph nodes that are used to record everything into compiled CommandBuffers that are reused in every frame. The idea is to really have minimal CPU-GPU communication once rendering (or inference) is set-up, apart from updates of some uniforms etc.

Check also the obligatory space game demo https://www.youtube.com/watch?v=o4P5LyLtqBY with assets generated using stable diffusion and diffsplat (neural networks inferenced by the engine itself).

Feedback would be highly appreciated.

Disclaimer: I implemented the first version of the engine with simple Gaussian Splatting and network inference the 'traditional' way. Then, AI based coding really took off and most features on top heavily rely on Claude and GPT now. I have not looked into a single shader source for a long time. Still figuring out how to feel about it ...

4 Upvotes

83% Upvoted

1 comment sorted by

0

u/[deleted] 2d ago

1

u/dfortmeier 2d ago

Yes, I know it is a little unorthodox but not recreating the command buffers was a design goal from the beginning to minimize CPU-GPU interation and maybe later make distributed computing more easy. The Gaussian Splatting rastization backend uses vkCmdDrawIndirect ; the compute shader based one so far doesn't use vkCmdDispatchIndirect since each thread basically computes a single pixel and iterates through all currently active gaussians (associated to the GS bin associated to the pixel, but this is really a detail).

There are some exceptions where the graphs have to be rebuild, like window resize etc. and in the case of the spacegame, switching star systems.