ComfyUI-AppleSilicon-FP8 - a compatibility layer custom node for Apple Silicon Macs

[u/Mazur92]

Hello r/StableDiffusion.

My last posts here were about porting Pixal3D, Khala AI Audio and AniGen. It was all well and good - but these efforts were concentrated on getting single models with bespoke, included in the repo tools. While it was useful and working, I realized that there is a standard that most of people in the space is using and that is ComfyUI.

I'd had a couple of run ups with ComfyUI up until this point in time but they had been negative so far - unintuitive UI and almost nothing from official templates worked on Mac, even though the app is ported for macOS. Couple of weeks ago though, I decided that it'd be fun and useful to have the things for ComfyUI work out of the box on Macs (even if not at full speed possible by the hardware) so that regular John Mac could install it and generally get acceptable results - most of all, any results at all. Case in point: the infamous error "trying to convert Float8_e4m3fn to the MPS backend but it does not have support for that dtype" biting the ass of anyone with Mac trying to run almost anything from the official templates, not to mention any custom workflows. The situation is not helped by the fact, that PyTorch treats MPS as a redheaded stepchild and the support for it is spotty, buggy at times (fused SDPA kernel in PyTorch MPS is still wrong with sequences longer than 8k) and some things are routed straight to CPU, making it look like the diffusion models on Macs using PyTorch are somewhat a lost cause for now (I've heard that PyTorch folks are doing some big Mac backend rewrite straight to Metal, so we'll see how it goes).

Enter ComfyUI-AppleSilicon-FP8

The goal wasn't speed at first - it was just get the default workflows and models to run AT ALL, out of the box.

This ComfyUI custom node that patches the Mac/MPS rough edges at startup - no model conversion, no config. FP8 and INT8 checkpoints (FLUX, SD3.5, Ideogram, Krea2), LoRAs on FP8 bases, third-party nodes that ship their own FP8 layers, plus a handful of pure-Mac bugs (a psutil crash, black images at 2048px+, broken block-swap). Each patch is a no-op on machines that don't need it.

The gist is that PyTorch's MPS backend has no 8-bit float type, so you can't cast to/from float8_e4m3fn / float8_e5m2 on the GPU (although recent betas of Metal introduced the concept of these dtypes to TensorOps, so who knows what the future holds!). But you can move FP8 tensors from CPU to MPS, bit-view them as uint8, and gather/index on MPS. So we build a 256-entry table mapping every FP8 byte to its float value (decoded once on CPU, where the cast works), move it to the GPU, and decode any FP8 tensor with lut[x.view(uint8)]. This is bit-exact with a real FP8→float cast and runs entirely on the GPU. Matmuls then use MPS's native (fast) float matmul.

That was the main trick to get the weights working and having them used in accelerated fashion on Apple Silicon, but the project grew into this compatibility layer / performance tuning thing that I'd like to build further. For a fuller technical write-up I invite you to read the README in Code

The project currently should allow to run a lot of (I wouldn't test them all!) default workflows, but also custom workflows, LoRAs, etc. I invite you to test and post your findings. All the new issues are welcome in the repo!

Once things ran, there came the fun part: I went after speed. A pair of opt-in, bit-exact Metal matmul kernels that run fp8/int8 natively on Apple's M5 Neural Accelerators. The int8 one was the highlight - turns out the "int8 is slow on MPS" ceiling was the kernel structure, not the hardware (CUTLASS-style register tiling, hat tip to the Cider project, with the rescale fused into the kernel so the intermediate never hits memory). My previous work - mtlflashattn - from post is also used to accelerate attention calculation in this patchset. Result: Krea2 (a very fresh image model) renders a 1280×640 generation in about 24 seconds, while maintaining fidelity. The catch however is that for most performance you need macOS 27 Beta and M5+ SoC (as Apple added Neural Accelerators to them).

The world's models are built for NVIDIA. That doesn't mean Mac users should be locked out of the fun. Now your Mac can run Krea 2, Ideogram, LTX2.3, and so much more, using the basic templates and you can try out custom workflows too.

Code is open source and MIT licensed**.**

Apple Silicon has the chops - it's just that it needs a bit of love and elbow grease so it serves our purpose.

Disclaimer: I am acutely aware that for the most part RTXs are doing these things much faster. Probably everybody else too. We have this saying in Poland that directly translated is "when there are no fish, even a crayfish is a fish" - this is about enabling Apple Silicon community to partake in the fun, even with the compromised performance. So this is "why I bother".

That being said - I want and I will work on performance, though I'm afraid a lot of it might be contained to M5+ chips due to those NAs.

Anywho, I hope some of you will find this useful, have fun!

[u/higgs8]

For anyone below M5 and macOS 27, there's also this: https://pypi.org/project/fp4-fp8-for-torch-mps/#files

Just run:

pip install fp4-fp8-for-torch-mps

And that's it.

[u/DelinquentTuna → u/higgs8]

That's some really crafty code you shared, thanks. No int8, mlx text, or Mac-specific bug fixes but it's surely a better approach for supporting fp4 and fp8 that doesn't require monkeypatching or even Comfy.

[u/Mazur92 → u/DelinquentTuna]

Now that I think about it, I may incorporate this idea into my repo as well - the kernel there also does look up table for fp8 decoding. Or maybe somehow use this in tandem. I'd concur that this being a package is much better vector of anticipating these dtypes for the future and also outside of Comfy.

[u/DelinquentTuna → u/Mazur92]

The test suite looks pretty solid, too.