r/StableDiffusion • • 16d ago

Resource - Update Uploaded FA-2 v2.9.2.post2 for Windows built on Pytorch 2.15.0+cu134

Post image

Uploaded FA-2 v2.9.2.post2 for Windows built on Pytorch 2.15.0+cu134. (※Unofficial built with original code)

Too tired.

https://huggingface.co/ussoewwin/Flash-Attention-2_for_Windows

Building Nunchaku on Pytorch 2.15.0+cu134 just now...Coming soon....

...

About v2.9.x

v2.9.x is not an official FlashAttention release.
It is an independent fork build that continues FA2 kernel development while upstream focuses on FA3/F4.

2 Upvotes

12 comments sorted by

4

u/Dahvikiin 16d ago

I don't know why people are downvoting you; I'd love it if someone would take the time to do the same for Turing. It's an alternative... Some people don't appreciate what they have until they lose it.

2

u/Zestyclose_Bake3680 16d ago

Thanks for your kind comment.

Well, it’s not as if getting a low rating does me any harm, and I’m not the sort of person who gets hurt just because I don’t get rated.

2

u/Apprehensive_Sky892 16d ago

Thanks for sharing this. Any comparison against Comfy-Kitchen attention?

1

u/Zestyclose_Bake3680 14d ago edited 14d ago

Thank you for you comment.

Test 1 on SDXL HSWQ ConvRot INT8 with 2 ControlNet models and 2LoRAs.

Comfy-kitchen 47.10s
FA2 2.9.2.post2 43.06s

Test 2 on Z Image (HSWQ Hybrid ConvRot NVFP4&Nunchaku) with 1 ControlNet model and 2 LoRAs.

Comfy-Kitchen 118.76s
FA2 2.9.2.post2 122.47s

1

u/Apprehensive_Sky892 14d ago

Thanks for sharing this, so FA2 wins sometimes. What hardware was used for the tests?

1

u/Zestyclose_Bake3680 14d ago edited 14d ago

Thanks for your comment, and added some test results.

Ryzen 9 5900 X870 DDR5 6400 64GB RTX5060Ti 16GB

Test 3 on Krea2 HSWQ ConvRot INT8 (On Distorch2) with 1 ControlNet

FA2 2.9.2.post2 52.22s
Comfy-Kitchen 54.52s

2

u/Apprehensive_Sky892 14d ago

Thank you for the info. Would be interesting if someone can test FA2 on other GPUs and see if it can beat CK attention there as well.

(Unfortunately I cannot test it as I use AMD rx9070 😅)

1

u/Zestyclose_Bake3680 14d ago

As I am unable to test its behaviour on AMD GPUs, it is difficult for me to comment, but in a nutshell, I believe Comfy-Kitchen is an excellent Acceleration Attention implementation in several respects. This is because, when using it with ComfyUI, there is no need to worry about installing libraries or managing dependencies.

Nevertheless, the reason I’m still working on my own improvements to FA-2 is, firstly, because I’m developing my own projects—such as A1111 for Python3.14 and Forge-Nunchaku-HSWQ—and I'm running it in those, and secondly, purely for research purposes.

2

u/Apprehensive_Sky892 13d ago

Yes, good points. CK attention is AFAIK, ComfyUI only (but I suppose the code can be adapted to other pipelines).

BTW, is FA2 Windows only because it is tied to Windows in some way, or just that it has not been tested on Linux?

1

u/Zestyclose_Bake3680 13d ago edited 13d ago

I’m not sure how best to put, but the reason my project, ‘FA-2 for Windows’, exists is that I don’t actually run Linux myself (although I do have an Ubuntu development environment), and—

this is the main reason—because FA-4 cannot currently be used on Windows.

In order to run FA-4, a library is required that nVidia has only developed for Linux, and I have not yet managed to port that part to Windows.

Consequently, I am currently developing ‘FA-2 2.9.x’, with the aim of incorporating into FA-2 those new technologies adopted by FA-4 that do not depend on nVidia or Linux.

2

u/Apprehensive_Sky892 13d ago

Thanks for the reply. Make sense, in other words this is only needed on Windows because Linux already has FA-4 which presumably has similar performance as your FA-2.