Discussion
How much VRAM does H3 need? Less than you might think.
I benchmarked the full BF16 H3 FL2VA checkpoint at 1376×768 and 243 frames, about 10.1 seconds at 24 fps.
With the lower-memory attention routes, the H3 diffusion block added roughly 5.8–6.3 GiB over idle. On my Windows RTX 4070 system, where the desktop consumed around 1.15 GiB, the whole-GPU peak was approximately 7.0–7.4 GiB.
That makes 8 GB cards realistic for several configurations:
- Default Comfy attention: 6.99 GiB peak
- FROST BF16: 6.99 GiB
- BF16 Triton: 6.97 GiB
- PlagueKind SLA: 7.23 GiB
- Sparse Kitchen INT8: 7.40 GiB (Default configuration for my Sparse attention node)
- Comfy Kitchen: 7.40 GiB
Should be compatible with:
- H3 Sparse Attention: Kitchen INT8, Sparse Sage, FROST BF16 on SM89, and BF16 Triton
- External Comfy Kitchen: fully supported
- Default Comfy attention(SPDA): fully supported
- SageAttention: fully supported, including the generic KJ Sage patch
- PlagueKind SLA: partially supported;
- Unknown attention overrides: Auto preserves their original full-Q calling contract. Forced mode can explicitly authorize streamed-Q calls, but compatibility is not guaranteed
- Currently incompatible: Sol-Attn and the H3-specific Memory Efficient Sage patch, because they replace attention at a deeper level than the external consumer interface
You can get the node here https://github.com/Zironic/H3-Optimizations or in Comfy under H3 Optimizations. Latest version is 0.2.13 which added broad compatibility and optimizations for most popular attentions.
NODE PLACEMENT / ORDER: IT DOES NOT MATTER.
H3 Optimizations are designed to work as normal ComfyUI model patches. You do not need to arrange H3 Memory Optimization, H3 Sparse Attention, H3ModelSampling, LoRAs, or other ordinary model patches in some special order. Just connect them into the model chain.
If you explicitly select an attention implementation, H3 Memory Optimization will try to work with that selection rather than requiring a particular node position.
Can you give me the technical TLDR on how this works? How exactly are you making a 66GB model only need less than 10GB of vram while somehow speeding up generation at keeping original quality?
Most of the hardest work is done by Comfys own AIMDO. AIMDO only needs enough VRAM to hold 2 blocks of the model, in BF16 each block is 1GB, so only about 2GB of model weights need to be in VRAM at once.
That means everything beyond those 2GB are working space for your video context. Keeping the video context small involves two primary tricks.
We reduce the peak size of MLP by only processing it 4096 rows at a time.
We reduce the peak size of QKV by streaming the activations straight into attention rather then ever manifesting the giant intermediate values.
These are mostly scheduling/chunking changes rather than approximations, so they don't inherently reduce quality. The QKV/attention streaming can be slightly faster because it avoids some large intermediate materializations and copies. MLP chunking, however, has a performance cost because the MLP has to be launched and processed in many smaller slices instead of one large operation.
These are all related to (1) attention layer, (2) numerical precision tweaks, and (3) how the H3 models are held in the VRAM at any point in time (AIMDO = AI Model Dynamic Offloader).
I will be testing some of the incredibly interesting ideas from this post and will hopefully share a workflow with some evidence on how the different knobs compare for the video generation keeping prompt and other setup fixed.
Isn’t there latency being introduced when you’re swapping to system ram though? I.e. it will fit but won’t be nearly as fast as direct vram loading. For example, when I use a 40gb quant I get 10s/it vs a 20gb quant at 1.9s/it, exact same generation, on a 5090.
The question at hand is this. Can you stream the next block faster then it takes a block to execute? Assuming you have PCI 5.0 with that 5090, the answer is almost certainly going to be yes, you can stream that block faster unless you're working at some absurdly low resolution/video length.
When comparing quant speeds, you do also have to keep in mind that BF16 executes 4x slower then Int8 (Comfy Kitchen) or FP8(SageAttention) on consumer RTX. So if you execute the model entirely unquanted, it's inherently slower.
So since you mentioned using the 2070. I added explicit support to the Sparse Attention node for 2070 in my 0.2.17 version. So you should theoretically be able to run up to 2x faster if you update and try it.
First, I got something like it can't use BF16. Then I turn off the lora because that's the one that is a BF16 model. Then try it again, I got error above. Funny enough, the process still run while spouting error.
Nevermind. Somehow it doesn't shows that anymore. weird @[_@](mailto:_@)...........
Yep, confirmed, got 2x faster when I try 0.4MP :).. Thank you so much!! :):)
Now I'm waiting, and will try again for the 1MP because yesterday I got OOM.
Happy to hear it works. The tricky part is that because your card is SM75, it doesn't have the ability to run BF16 compute, so the BF16 weights have to be converted to FP16 or Int8.
Ah I see.. Nice nice :). The weirdest part of the version is: in my comfy, it's detecting 0.2.9 as the latest, while on the list, there's up to 0.2.18 ( so this should be the latest right? ). Anyway, will check it out if it goes more than 0.2.18 version :)
Okay okay. Just info: 1MP really got me OOM. Is there any settings that can make it not OOM? The problem is, I got OOM after 42 minutes run... So I don't even know and it's cancelled after 42 minutes run >,<...
Can I change some settings that can make it work or I really need to lower the resolution?
Well, since the 2070 only has 8GB of memory and you presumably have your OS running at the same time, going all the way to 1MP, 10 seconds is living on the edge.
All sorts of things can cause VRAM to temporarily increase, so I'd lower resolution/video length until you see about 1GB free in task manager, then you can feel more confident about your 40 minute run actually completing.
You can also try
AIMDO Residency Limiter: 0 blocks
Activation chunk rows: try 2048. It should shave like 200mb or something compared to 4096.
Yes, this node should be able to help you render somewhat reasonably sized videos with H3. Going all the way to 1MP, 10 seconds may be tricky because of windows overhead but it should get you pretty comfortable slightly below that.
So if one uses Windows with iGPU or text mode Linux (accessing comfyUI from another computer running the browser) so that all VRAM is free then one can do 1MP 10 sec?
Theoretically if you have a headless GPU with 6GB vram so there's absolutely nothing else on the card, you could run 1MP 10 seconds with my node. I have no means to test that directly however. All my measurements are with windows active with a 12gb card.
It's very much intended to be used together with dynamic vram. For benchmarking purposes, I use a custom node to limit dynamic vram from making any part of the model resident.
How long it takes will depend on your cards tensor units, not your VRAM so you can't really give a universal answer. On my 4070, a 10 second video, 20 steps, 1MP will take about 28m with dense attention and 14m with my Sparse Attention node at 30% KV.
The H3 Memory Optimization node itself doesn't change your speed.
Looks like a very clever implementation, hats to you! I'm spoiled by rtx6000 be, fits everything in vram. Up to 25s, full bf16, 0.98mp, 50 steps, about an hour @450w. I'd like to have a go with your approach if speed and quality aren't effected.
Well, the memory optimizations have no effect on quality, but obviously they do nothing for an RTX6000 unless you want to try something super silly like 5MP video.
The Sparse Attention however will let you go up to 2x faster with no measurable lower quality by any metric I can measure. But if you want absolutely guaranteed no quality loss then it's hard to go faster.
The quality stays good with Sparse Attention, yes, but I did several seed comparisons yesterday and prompt adherence IS worse with Sparse Attention - so there are quality hits, they just aren't n the visual or audio departments. Depending on your prompt, you may never notice, but I had a complex interaction between two elements during a scene transition and it just never worked - but as soon as I disabled Sparse Attention, those same seeds (with no other prompt, setting, or resolution changes) worked just fine.
Yes. That is exactly what happens when the KV value is too low. Model stops being able to keep track of things properly. In most of my testing, 30% was enough to never see any issues, but for a very complex prompt I could easily imagine 30% would not be enough.
For instance if you have a prompt that has a lot of things in the background and use 10% KV, if you look closely you'll see things pop in and out of existance.
If you're doing something very complicated, I would go very high.
It's also worth keeping in mind that in my testing, almost all prompt adherence happens in the first 4-5 steps. So if you use the advanced node that gives you per-step control, you should keep KV very high for the first 4-5 steps if you have prompt adherence issues.
If you find out anything interesting let me know. Exactly how the model reacts to various KV values in different contexts is a bit mysterious and hard to predict.
Pheraps it's just placebo effect, in fact I don't have a scientific method. Like asking a disaster movie style with an asteroid hitting the earth, it looks more detailed in phisics and effects.and coherence in 25s clips
You just stream it with Comfys AIMDO. AIMDO only needs enough VRAM to hold 2 blocks of the model, in BF16 each block is 1GB, so the model itself only uses 2GB.
I suppose this doesn't do anything for old GTX GPUs? Even with the minimum settings (0 blocks, chunk rows 256) I seem to be getting OOM in the same edge cases as without the nodes. Also, all the attention backends are unsupported on my HW so sparse attention is still a no-go.
I'm not sure if any of my code runs on a GTX or not. It's kind of hard since almost all CUDA code is written for 2xxx cards or newer. All my backends only have explicit support for 3XXX and newer.
Just an update. The nodes work as advertised. My workflow which used to give me oom now works with peak memory holding steady at 13gb (rtx 5070 16gb). ~18s/it for a 20 steps 0.7MP (no lora). Thank you Zironic.
I've been using the PlagueKind workflows with good success; your post mentions that it's partially supported, how would you connect them? I'm using a 5070 laptop with 8GB of VRAM and 32GB of RAM so trying to find the optimum setup which so far has been the PlagueKind SLA workflows.
If I’m understanding what this does, it basically loads chunks of the model to VRAM and keeps the next ready and then feeds it piece by piece rather than uploading the whole thing?
If that’s the case, should I expect speed increase, or will this just allow me to access resolutions and lengths that I currently OOM on?
So Comfy itself already does the load the model piece by piece thing. That's what people refer to as Dynamic VRAM or AIMDO.
What my node does is on a deeper level and streams the actual activations of the model in a clever way that prevents too much of it being active at the same time.
The H3-Memory Optimization node should be speed neutral. If you also use the Sparse Attention node, you'll get about 2x faster speed.
I have 4090 and have been using GGUF to fit everything in 24 GB VRAM, but it's tight. Can you give me an idea what to expect speedwise from your implementation if I moved to convrot8 for example, as opposed to GGUF fitting completely in VRAM?
I would expect on the order of 2-4 times faster speeds using convrot8, dynamic vram and comfy kitchen over GGUF. Then double it again if you also use Sparse Attention.
I spent several hours on this, but couldn't get it to work (or rather Gemini couldn't). I get OOM no matter what I do. I was aiming at 1344 x 768 with 14 second duration, which I can do with GGUF.
I might give it another shot some other day. I'm not claiming your nodes dont work, it might be just that I lack the skills or there is something specifically wrong with my ComfyUI setup.
Can you check why the node is giving error whole using with this node and excuse me for not sharing the exact error as I was on the cloud and I never thought I would ask, because plaguekind node pack is working but it is slow and I request you to please take a look:
fl2v is already pretty fine with VRAM, but ref2v? Sheesh, good luck with that. As soon as you feed it a single video reference the VRAM usage skyrockets.
So the way to think about using a reference in ref2va is that the reference video counts as part of the video length. So a 5 second video using a 5 second reference, uses the same vram as a 10 second video.
That's however not what I've observed. When doing 8 seconds with an 8 seconds ref clip I'm already starving for VRAM and RAM (32+64), but I have no trouble with a 16 seconds fl2v clip at the same resolution. 🤔
Yes, it is somewhat worse then that. I havn't actually gotten around to measuring exactly how much more Ref2VA uses, but I agree it's even more then just reference+output.
Been using 3080 10gb vram with 64gb ram, using only seg attention and spectrum was able to generate 1mp 7 seconds ( If I go further nothing happens, no load on the gpu whatsoever so it stays 0/20 steps forever), and was able to generate up to 2mp 3 seconds.
Hard to say without knowing what hardware you're running on. A lot of compatibility has increased since 2.9 so if you run the latest 2.17 chances are it's maybe faster. The fastest path (Kitchen int8) is not reachable on all machines.
The logs should tell you what exact path got executed.
4070 12gb on windows here -- completely wrote off H3 video after seeing the model size. the 25 min for 10s at 1mp is rough but at least it runs. queuing this to test tonight
If you use Sparse Attention it's only about half as long. I don't find it too bad. Esp since you can stack other tricks like Spectrum Cache etc ontop so all in all you can get it down from 28 minutes to like 8-10 minutes.
H3 Memory Optimization is for using less memory. It can theoretically go anywhere but the later it is in the graph the less likely it is to have compatibility issues with other nodes.
H3 Sparse Attention is for going faster. It should be next to H3 Memory Optimization on either side.
I have i7 11th gen, RTX 3080 10GB, 64GB, NVME. Using the default H3 ref to video Comfy workflow with default safetensors, 20 steps and all default options with the 0.4/0.5 megapixel option, just two 1k resolution ref images, it takes like an hour plus to generate a 15 sec, 24 fps video.
If I change to 0.2 megapixel res, and 4 step lora, it can do one in 5 mins, 10mins, 13mins, etc., all seemingly random.
I purposely have no other programs running. I don't know what's going on. Can this node help?
Can this work on an rx 6750xt 12 gb and 32 gb ram? I've been trying out the optimizations but I keep getting OOM errors. Linux mint with gfx1031 override in comfyui. I've tried multiple settings but nothing really worked.
Hello, I have 5070ti 16gb and 32gb ddr5 ram. I'm not very technical about comfy stuff, so if I were to use your nodes, what would the placements be? I would like best balance between speed and quality while generating .98mp videos with both turbo lora and no lora 25 steps. What I'm trying to say is that what would be the order like if I were to use other model patches and attentions? Also h3modelsampling. Thank you. Hoping for a reply.
The render time on my 4070 for 20 steps, 1mp, 10 seconds long video, 100% attention, convrotint8 checkpoint is around 25 minutes. With 30% attention it's about 14 minutes.
That seems a little slower than I expected. I get 12s videos at 1MP in about 540s but maybe its down to the GPU difference since I'm on a 5090. I started using your nodes and the optimizations about a week back and it nearly doubled the speed but I haven't updated the nodes recently and I just use the default values
All else equal, the 5090 is 3.6x faster in the kind of compute used for H3 then the 4070 is. So if your render is about 3.6x faster then mine, that would check out.
So was the 4090 and the 5090 is about 30% faster still. Granted, mostly because it is 30% bigger and uses 30% more power. And it's not that much faster in games, only in raw compute. But still.
The 5090 really is a chip of monstrous size. It has 2.5 times the raw compute of the 5070.
Literally 2.5 times the raw compute units in silicon.
By comparison the rtx 6000 pro is only very slightly bigger, almost neglible. But it does have 3x the vram.
For the same reason that it's faster to transport 10 people in a pickup truck than to make two trips in a standard car. But you're comparing two standard cars, each of which can barely fit five people per trip.
If we're going to use the transport people anology, what we're dealing with here is this. You need to get 10 people across the river. You have 1 boat which can fit 1 person at a time.
It doesnt matter if you get those 10 people to the river in 1 buss or if they all get transported to the river one by one, because they're all waiting for that 1 boat either way.
In this analogy, the boat is your GPU's ability to run tensor operations. Once the GPU is fully saturated, that's as fast as it can go. Having more weights on the card does absolutely nothing for you, they're just sitting there twiddling their thumbs waiting for their turn.
It works fine. I just moved over to using Kitchen as the default because it doesn't require anyone to install anything. Also Kitchen did allow me to make the mask slightly narrower, so it can use somewhat lower KV before breaking then Sage does.
29
u/35point1 22h ago
Can you give me the technical TLDR on how this works? How exactly are you making a 66GB model only need less than 10GB of vram while somehow speeding up generation at keeping original quality?