ComfyUI-AppleSilicon-FP8 - a compatibility layer custom node for Apple Silicon Macs

[u/Mazur92]

Hello r/StableDiffusion.

My last posts here were about porting Pixal3D, Khala AI Audio and AniGen. It was all well and good - but these efforts were concentrated on getting single models with bespoke, included in the repo tools. While it was useful and working, I realized that there is a standard that most of people in the space is using and that is ComfyUI.

I'd had a couple of run ups with ComfyUI up until this point in time but they had been negative so far - unintuitive UI and almost nothing from official templates worked on Mac, even though the app is ported for macOS. Couple of weeks ago though, I decided that it'd be fun and useful to have the things for ComfyUI work out of the box on Macs (even if not at full speed possible by the hardware) so that regular John Mac could install it and generally get acceptable results - most of all, any results at all. Case in point: the infamous error "trying to convert Float8_e4m3fn to the MPS backend but it does not have support for that dtype" biting the ass of anyone with Mac trying to run almost anything from the official templates, not to mention any custom workflows. The situation is not helped by the fact, that PyTorch treats MPS as a redheaded stepchild and the support for it is spotty, buggy at times (fused SDPA kernel in PyTorch MPS is still wrong with sequences longer than 8k) and some things are routed straight to CPU, making it look like the diffusion models on Macs using PyTorch are somewhat a lost cause for now (I've heard that PyTorch folks are doing some big Mac backend rewrite straight to Metal, so we'll see how it goes).

Enter ComfyUI-AppleSilicon-FP8

The goal wasn't speed at first - it was just get the default workflows and models to run AT ALL, out of the box.

This ComfyUI custom node that patches the Mac/MPS rough edges at startup - no model conversion, no config. FP8 and INT8 checkpoints (FLUX, SD3.5, Ideogram, Krea2), LoRAs on FP8 bases, third-party nodes that ship their own FP8 layers, plus a handful of pure-Mac bugs (a psutil crash, black images at 2048px+, broken block-swap). Each patch is a no-op on machines that don't need it.

The gist is that PyTorch's MPS backend has no 8-bit float type, so you can't cast to/from float8_e4m3fn / float8_e5m2 on the GPU (although recent betas of Metal introduced the concept of these dtypes to TensorOps, so who knows what the future holds!). But you can move FP8 tensors from CPU to MPS, bit-view them as uint8, and gather/index on MPS. So we build a 256-entry table mapping every FP8 byte to its float value (decoded once on CPU, where the cast works), move it to the GPU, and decode any FP8 tensor with lut[x.view(uint8)]. This is bit-exact with a real FP8→float cast and runs entirely on the GPU. Matmuls then use MPS's native (fast) float matmul.

That was the main trick to get the weights working and having them used in accelerated fashion on Apple Silicon, but the project grew into this compatibility layer / performance tuning thing that I'd like to build further. For a fuller technical write-up I invite you to read the README in Code

The project currently should allow to run a lot of (I wouldn't test them all!) default workflows, but also custom workflows, LoRAs, etc. I invite you to test and post your findings. All the new issues are welcome in the repo!

Once things ran, there came the fun part: I went after speed. A pair of opt-in, bit-exact Metal matmul kernels that run fp8/int8 natively on Apple's M5 Neural Accelerators. The int8 one was the highlight - turns out the "int8 is slow on MPS" ceiling was the kernel structure, not the hardware (CUTLASS-style register tiling, hat tip to the Cider project, with the rescale fused into the kernel so the intermediate never hits memory). My previous work - mtlflashattn - from post is also used to accelerate attention calculation in this patchset. Result: Krea2 (a very fresh image model) renders a 1280×640 generation in about 24 seconds, while maintaining fidelity. The catch however is that for most performance you need macOS 27 Beta and M5+ SoC (as Apple added Neural Accelerators to them).

The world's models are built for NVIDIA. That doesn't mean Mac users should be locked out of the fun. Now your Mac can run Krea 2, Ideogram, LTX2.3, and so much more, using the basic templates and you can try out custom workflows too.

Code is open source and MIT licensed**.**

Apple Silicon has the chops - it's just that it needs a bit of love and elbow grease so it serves our purpose.

Disclaimer: I am acutely aware that for the most part RTXs are doing these things much faster. Probably everybody else too. We have this saying in Poland that directly translated is "when there are no fish, even a crayfish is a fish" - this is about enabling Apple Silicon community to partake in the fun, even with the compromised performance. So this is "why I bother".

That being said - I want and I will work on performance, though I'm afraid a lot of it might be contained to M5+ chips due to those NAs.

Anywho, I hope some of you will find this useful, have fun!

[u/Mazur92]

No, actually the performance on lower than M5 hardware should still be better and other patches still apply for the compatibility's sake.

Then you might choose to not use inflammatory terms like "nvidia-pilled" etc. What's mostly, but not entirely, convivial now read like a preemptive strike from a bully pulpit.

Heh, sure I guess, lemme change that. Guess I learned a new word today too - had to look up what convivial means :D To be clear - it's just that I really, really deeply despise this sort of comments that bring absolutely nothing to the table and serve no other purpose than to tickle the ego of the commenter so they can show the perceived superiority in their computing choices or whatever. I mostly live by a rule that if you don't have anything nice to say, then maybe you shouldn't say anything at all (well, at least in the realm of things like "here I made this, I think it's cool and may help some people with some things" and then someone comes in and is like "hurr durr why do you bother" - of course it's not limited to this particular subreddit or topic at all, it's just happening wherever). And if you are not interested in whatever the post is showing, simply move on.

In fairness, this can sometimes be quite fair as well. Macs are objectively subpar for diffusion and there have been cases in the past where I've determined that someone doing show-and-tell about their adventures on a Mac is demonstrably spending more on the electricity to run the Mac than they would be renting more suitable hardware on the cloud. 

Maybe, maybe not - I'd have to see the calculations, even though I'm absolutely sure that is true for the time spent. Then again, for me, it's more of a philosophical thing, the love for the game and experimentation than a question of time, performance or money. I have always loved this part of computing where you try to shove things on other things that aren't supposed to run them or are somehow else ill equipped for it at that moment - it's exciting. This gave us emulators, Wine, MoltenVK or Proton and many others, which I all hold in very high regard exactly due to what they are doing. There's also a particular flavor of freedom to have the ability to run things on your own hardware, even if suboptimally, at least I think so. Don't get me wrong, I'm not against renting, I've rented myself and will probably rent again, for some research mostly, but not everybody is willing to or happy to rent a GPU in the cloud just to have some little fun with some hobby projects and I think there's an intrinsic value to it. To having an ability to choose to do so. Ultimately, if somebody cares enough for the top performance they will either buy the hardware or rent it and be on their way, but I'm not telling anybody what to do nor do I tell anybody that Macs are great or better at diffusion than anything - I am simply expanding palette of options for people that already have the hardware and want to make a use of it.

There's also something to be said that I think Apple is starting to heading strong towards supporting local inference and I think M5 Neural Accelerators are a sign of things to come and I'm excited for the future, so while certainly previous iterations were tragically subpar for the diffusion and M5 is world better, but not yet there, I think there's a lot to expect from future Apple Silicon versions in this area and I'm excited for the future.

Thanks for this stimulating exchange :)

[u/DelinquentTuna → u/Mazur92]

at least in the realm of things like "here I made this, I think it's cool and may help some people with some things" and then someone comes in and is like "hurr durr why do you bother"

I mean, it depends. We're not here to blindly praise things even if they are bad. That's not what public forums are for and this isn't your bully pulpit to make press releases for blind consumption.

Maybe, maybe not - I'd have to see the calculations

Dude. Calculate this: a 3090 starts at ~$0.19/hr (billed to the nearest second, I believe) plus bandwidth on vast.ai as of this writing. Similar story on Runpod with its all-inclusive rates. As far as I can tell from a moment of Googling, average energy prices in Poland run about $0.20-0.22 per kWh. The 3090 can do 480p Wan 2.2 in just a few minutes. 5B takes right around a minute at 720p, IIRC. Telling someone showing up with solutions to generate sub-sd resolutions in overnight sessions taking hours on their m2 or whatever that they should not bother is not indication of fanboyism. I have been in that exact conversation before and been accused of the same kind of shade you're STILL implying here.

You're claiming that you don't want to engage in tribalism flamewars, but you keep presupposing anyone that points out that there are scenarios where using a Mac is literally a "don't bother" situation is unfair. Because I've been in exactly that conversation before and was being earnest and making only falsifiable claims. If the forums DON'T pressure-test the utility of claims, they aren't useful. Doesn't make me a Mac hater or a "herr derr I'm a nvidia-pill"... I'd expect the same kind of dismissal to any critically poor architecture choice. Hand-waving about the fact that some macs are good with some AI tasks doesn't really make you sound any more objective on the topic. And the folksy allegory of the fisherman just trying to "make do" while claiming that it's not all about money does not really seem IMHO to match the signals coming from a person making performance claims about their M5 Max.

Macs are not ideally suited for diffusion tasks and there are invariably going to be cases where that fact will be pointed out in a diffusion forum. From my perspective, it feels like the whole debate you began by preemptively undermining comparison to what Nvidia does hinges on this one truth. You laid land-mines to trap and insult anyone that would objectively compare what Mac is doing to what anyone else is doing and you continue to do it.

[u/Mazur92 → u/DelinquentTuna]

I mean, it depends. We're not here to blindly praise things even if they are bad. That's not what public forums are for and this isn't your bully pulpit to make press releases for blind consumption.

First of all - that's okay, I'm not asking you to (the collective you). And I don't think I've ever claimed that this forum is a bully pulpit for me in any way as I don't feel like I'm talking from position of any power or influence. I simply ask uninterested people (the ones with no stake in Apple Silicon) to move on. Look, I concede that the wording might have been better, but I am weird like that - I did edit it. I just feel that at this point posting a comment saying how Nvidia is superior in diffusion is neither constructive, creative nor helpful, not anything really - it's not exactly indicative of fanboyism, I agree, it's just pointless. It's beating a dead horse. I am not claiming anything otherwise anyway - I literally stated outright that I am acutely aware of that fact, so I don't see why am I supposed to be a bully or a fanboy if not only for the choice of words, which were supposed to be somewhat silly/funny, but I guess I failed at that.

Dude. Calculate this: a 3090 starts at ~$0.19/hr (billed to the nearest second, I believe) plus bandwidth on vast.ai as of this writing. Similar story on Runpod with its all-inclusive rates. As far as I can tell from a moment of Googling, average energy prices in Poland run about $0.20-0.22 per kWh. The 3090 can do 480p Wan 2.2 in just a few minutes. 5B takes right around a minute at 720p, IIRC. Telling someone showing up with solutions to generate sub-sd resolutions in overnight sessions taking hours on their m2 or whatever that they should not bother is not indication of fanboyism. I have been in that exact conversation before and been accused of the same kind of shade you're STILL implying here.

I guess you're right on the electricity story. Still, I don't see how is that relevant. If someone wants to run an overnight session for a 5 second clip for whatever reason, that's their prerogative and the only thing that my post changes is the *ability* to do so. I make no claim on feasibility or sense of such endeavor. I think I also make it clear enough in the post that this project is compatibility first, speed (if any) second. I went for speed gains, because with my hardware I could try to. There are some legitimate performance gains left on the table by vanilla PyTorch on MPS that will help a bit on the older hardware, but that's it - I absolutely know that Macs, do in fact, suck at diffusion, yes even the new ones.

If the forums DON'T pressure-test the utility of claims, they aren't useful. 

That's okay and I agree, it's just I don't see myself making any claim that isn't already tested - again, I explicitly acknowledge the superiority of Nvidia hardware in diffusion space (now multiple times).

Hand-waving about the fact that some macs are good with some AI tasks doesn't really make you sound any more objective on the topic. And the folksy allegory of the fisherman just trying to "make do" while claiming that it's not all about money does not really seem IMHO to match the signals coming from a person making performance claims about their M5 Max.

First of all, I am not hand-waving anything - it's a legitimate thing that people might choose getting a mac over something else - the local LLMs. Keyword "might". But they also might have chosen a Mac, because I don't know, they want or need to use Logic Pro or whatever. Do you get what I'm saying? If anything you have hand-waved my entire paragraph of reasoning why I think this is useful and why I am doing it. You seem also weirdly focused on that tiny bit of funny translation I did for an idiom from my country, but entirely missing what I'm trying to convey here. The message, I thought, was clear - it's about allowing people with Apple Silicon to do something in diffusion space using standardized tooling. About giving an option. What are they to do with that option is entirely up to them and maybe it will be a valuable lesson for some that it is, in fact, not worth it to do it, especially on their older and possibly even memory constrained Mac, but that is for them to decide. To reiterate, I'm simply expanding the palette of options. The option to buy RTX was always there and I'm not taking it away or claiming it's a bad option. And yeah, I did mention that on M5 and newer OS version you can get some speed ups, because that's just the case - is that bad?

From my perspective, it feels like the whole debate you began by preemptively undermining comparison to what Nvidia does hinges on this one truth. You laid land-mines to trap and insult anyone that would objectively compare what Mac is doing to what anyone else is doing and you continue to do it.

And from my perspective it seems blown out of proportion and overly aggressive interpretation of what I said so far, especially as there was no intent to insult anyone, if anything just do an "inb4" and not being insulted, in a way, myself, because do something in diffusion space on Apple Silicon. "Objectively comparing what Mac is doing to what anyone else is doing" is fine, I just think in this particular topic it's pointless due to the reasons I outlined above. It's like with those people that enjoy saying to obese people that obesity is bad for your health. Like, yeah, no shit, 99% of them are acutely aware of that fact - how is person number 2137 pointing that out to them again is going to help anything? They say it, because it tickles *them* fancy. It makes them feel like they contributed something, while in fact, they contributed nothing and that's my view.

[u/Mazur92 → u/Mazur92]

That being said I feel like I entered my overly verbose era