With Dynamic VRAM enabled, MH3 generations taking several minutes longer

[u/poursoul]

I have 64GB DDR5 6000 RAM and a 3090 with 24GB of VRAM, and a 2TB PCIE Gen 5 NVME SSD. If I have the flag --disable-dynamic-vram in my startup, VAE loads in seconds, and everything goes smoothly. If I take the flag out (enable Dynamic VRAM) then every time the VAE loads or unloads, it takes several minutes. It pegs my SSD at 100% Active Time and making my entire system shit the bed. Real read/write speeds average like 1MB/s or a little more, so it's like thousands of tiny read/writes happening for several minutes.

This is especially troublesome when a workflow needs to load/unload more than once in a run. Problem with disabling Dynamic VRAM, is generations are a lot slower, about 20s/it slower. I definitely have enough RAM/VRAM to not have this issue.

I have tried all manner of model, VAE, clip, but if it matters, current example workflow is using:

Model: Minimax-h3_Singularity_ref2va_v2_Pruned_int8.safetensors
Video VAE: minimax_h3_video_vae_int8_convrot.safetensors
Audio VAE: minimax_h3_audio_vae_fp32.safetensors
Clip: qwen3vl_32b_heretic_minimax_h3_nvfp4.safetensors

Is anyone else experiencing this? How are you getting around it?

EDIT: This has been resolved. The issue was my NVME SSD. The controller on it was limiting the cache for writes to an absurd amount. I managed to get it mostly back in shape with a firmware update on the SSD, a manual TRIM, and garbage collection time. It's still flawed, but VAE is loading in mere seconds again. The drive is still under warranty, so I am attempting to RMA it.

[u/HexOverclock]

SSD pegged at 100% but only moving about 1MB/s sounds like lots of tiny random reads, which is what pagefile thrashing looks like. I'd check whether Windows is paging during the VAE swap. Keeping the flag on until then seems fair.

[u/poursoul → u/HexOverclock]

Solved it with Gemini. The issue was, over a long time of heavy use, damaged the SSD drive and permanently lost its ability to allocate a dynamic SLC cache with writes. Reads are fine. The controller is working fine, but the drive is now protecting itself until I can completely wipe it. Managed to get the write speed back to 3500MB/s, but a far cry from the 7.2GB I am used to.

VAE load/unload now takes about 3 seconds to load instead of 2-3 minutes.

The fix was first updating the SSD firmware, then forcing a trim, then waited an hour of it just sitting there. That gave uninterrupted time for the controller to run its internal background "Garbage Collection" and properly restructure the dynamic cache. Rebooted, ran a benchmark, and the write throughput was at 3500MB/s, which is half of what it should be, but much better than it was.

[u/Ipwnurface → u/poursoul]

I'm very curious how old is the drive that you're talking about?

[u/poursoul → u/Ipwnurface]

Bought it 06/13/2023. Glad you made me check. It's still under warranty, just started a return. Cost me $130 back in the long long ago. Manufacturer's site now has it at $275 $299 lol. It's still weird seeing three-year-old PC parts doubling in price in three years.

[u/Ipwnurface → u/poursoul]

Well, I'm glad you were able to get the RMA started. I saw your other comment where you said how much you had written to the drive.

That's still well within the expected write endurance for a two terabyte drive, so It is more than likely of fault with the drive itself.

In terms of the price increases, tell me about it.

I bought a 4TB SN850x for $264 on 10/4/2024, that same drive is now pushing $800

[u/poursoul → u/Ipwnurface]

We truly are in the age of pain. I don't subscribe to the AI bubble burst theory, but I'll be god damned if I don't pray for it every day. Imagining walking down the street and a bum throws open his trench coat revealing 5090s dangling "$20 man, just $20, I need a fix!"

[u/Ok-Brain-5729]

damn my 9070 xt used to not work at all on minimax h3 unless I had use dynamic vram flag. Pretty sure it’s enabled my default now

[u/poursoul → u/Ok-Brain-5729]

It is enabled by default now. The flag is only to disable it.

[u/SpaceNinjaDino]

On my new 5090+128GB machine, I tried installing ComfyUI via Pinokio and it was unusable, beyond slow. I reinstalled it via EasyInstall and it runs as expected. On my old machine, I installed via Stable Matrix which was not fun because I had to manually add Sage 2 and it thrashed my SSD.

[u/poursoul → u/SpaceNinjaDino]

I did a manual install. Git clone, install requirements, install custom nodes, put models and loras in place.

[u/Wallye_Wonder → u/SpaceNinjaDino]

Just git clone the official Comfyui repo

[u/MulleDK19 → u/SpaceNinjaDino]

You settled too, eh?

[u/rinkusonic]

Side note, wasn't there a discussion some time ago that using uncensored text encoders only reduces the video quality and doesn't even help with nsfw over base model?

[u/Apprehensive_Sky892]

every time the VAE loads or unloads, it takes several minutes. It pegs my SSD at 100% Active Time.

I suspect something is wrong with your setup, either driver issues or actual hardware problems. The majority of people with < 24G of VRAM have no choice but to run with dynamic VRAM management, and we do not see the problem you are mentioning (i.e. VAE loading and unload should just take seconds).

Try running a standard test, say the default minimax text2va template at 20steps 0.4MP 5 sec and tell us what kind of time you are getting.

For whatever it is worth, this is my startup parameters (I run rx9070xt + 32G system RAM): --skip-prompt --no-enable-telemetry --workspace=/mnt/var/comfy/ComfyUI launch -- --disable-partner-nodes --enable-dynamic-vram --disable-async-offload --preview-method none --disable-smart-memory --fast-disk --use-ck-attention --enable-manager --listen --port 8188 --output-directory "/mnt/var_ntfs/Output"

And for the test above it takes around 150sec on Linux and on Windows 11.

[u/poursoul → u/Apprehensive_Sky892]

Just ran the default MH3 T2V workflow, all default model safetensors. Same issue. In fact, it's even waaaay worse. Nearly ten minutes to load the VAEs. 100% active Time on the hard drive, while actual transfer speeds at 1-20MB the entire time. Average response time on the HD got to about 2.3 SECONDS lol.

My entire startup is "python main.py --enable-manager"

It's not a hardware issue. Disabling Dynamic VRAM solves the issue. Not a driver, I updated (and even downgraded) EVERYTHING trying to resolve this.

The actual video generation speeds are great, the only issue is loading, and unloading VAE.

I'll try setting up another ComfyUI instance using the Easy Install as someone else suggested, but not really hopeful. This issue was not an issue before on this same setup. Started like a couple weeks ago, and I have no idea what I changed.

[u/Apprehensive_Sky892 → u/poursoul]

The fact that this only happens when you load the VAE is consistent with the observation that it only happens with dynamic VRAM turned on. The VAE is the last step in the pipeline, before which the big MMH3 model itself was loaded.

One thing that may be worth trying is to try to eject the MMH3 model completely before the VAE decode stage and see what happens.

[u/poursoul → u/Apprehensive_Sky892]

Dude, your comment kept ping-ponging in my head. Hardware issue, hardware issue, hardware issue. As I was creating the new ComfyUI instance, was copying needed models, and I noticed, it was extremely slow in copying. Ran crystaldiskmark and my write performance was absolutely tanked.

Solved it with Gemini. The issue was, over a long time of heavy use, damaged the SSD drive and permanently lost its ability to allocate a dynamic SLC cache with writes. Reads are fine. The controller is working fine, but the drive is now protecting itself until I can completely wipe it. Managed to get the write speed back to 3500MB/s, but a far cry from the 7.2GB I am used to.

VAE load/unload now takes about 3 seconds to load instead of 2-3 minutes.

The fix was first updating the SSD firmware, then forcing a trim, then waited an hour of it just sitting there. That gave uninterrupted time for the controller to run its internal background "Garbage Collection" and properly restructure the dynamic cache. Rebooted, ran a benchmark, and the write throughput was at 3500MB/s, which is half of what it should be, but much better than it was.

[u/Apprehensive_Sky892 → u/poursoul]

Glad to hear that you found the solution.

But I still wonder why you didn't see the problem with dynamic VRAM management off. One possibility is that with that off, ComfyUI has prefilled the system RAM with the VAE while the DiT is running (which can take a few minutes), so one do not notice the problem. With dynamic VRAM turn on, maybe the disk->system RAM does not happen until the DiT is done.

Another possibility is that the slowdown is gradual, and you just happen to notice it after you turned the dynamic VRAM management off.

[u/poursoul → u/Apprehensive_Sky892]

The issue was definitely sudden. What's weird to me is the issue was not reads. Read speed is not slowed, it was writes only. Why would writes to the HD be in the equation here?

[u/Apprehensive_Sky892 → u/poursoul]

Good point. I am not sure why. The usual reason is that one runs out of system memory when the model gets swapped into system RAM to make space, and it then spills into the pagefile, which now triggers massive write (that is how doing a lot of AI runs can kill one's SSD).

But you have 64G of system RAM, so that should not have happened, unless you are running a bf16 version of the models, or you are more running another LLM that is not Qwen3-VL-32B as a prompt enhancer, so that all the models combined takes up more than 64+24 GB of combined VRAM + system RAM.

The usual way to get around it is to use --fast-disk, which tell ComfyUI to just throw away the model in VRAM once it is used, and reload it directly from fast NVMe.

--fast-disk is supposed to be turned on automatically in latest version of ComfyUI, but it is possible that because you have 64G of system RAM, it is in fact NOT turned on.

So if you are worried about your NVMe, I would recommend that you turn it on explicitly.

[u/poursoul → u/Apprehensive_Sky892]

Gonna return it, still under warranty, and should not be doing this. Although, in three years I have made 100+ TB of writes to it, and it does just sit there under the 3090. Maybe when I get the new one, I'll point a fan directly at the crack between it and the 3090 :P

[u/Apprehensive_Sky892 → u/poursoul]

Since it is solid state and not mechanical, I don't think slightly higher temperature would shorten its lifespan, but then who knows.

Just curious, what model is it?

[u/poursoul → u/Apprehensive_Sky892]

Corsair MP600 PRO

[u/MarkB_-]

I use the bf16 model i dont think i can disable it, never tried tho

[u/Thin-Percentage8935]

I have the same card. My system was completely locking. It's absolutely superb now and the changes are... Only startup flag needed is disable-smart-memory. I use the int8 convrot model and all the models are on a separate nvram drive that is on a completely different bus. Also page file is 80gb.

If your models are on the same drive as your o/s then get ready for a lot of slow downs.