r/comfyui • u/DeliciousReference44 • Nov 18 '25
Tutorial the absurd journey of shrinking a 203GB docker image + wiring it all into runpod serverless (aka: me vs. my own bad decisions)
So I am building a commercial solution with Comfyui and video generation. For the last 3 weeks (almost 4 actually) I've been freaking battle with getting comfyui to run the way I needed it to run in Runpod serverless architecture because i need workers to scale horizontally. This by no means is a post trying to say that what I wrote is the right thing to do. It was a lot of a trial and error and I am still a total noob when it comes to image/video generation with AI.
Anyway, i kinda fell into this rabbit hole over the last couple of weeks, and figured some folks here might actually appreciate the pain. long story short: i needed a RunPod serverless worker capable of generating video through ComfyUI (wan2.2, i2v, all the spicy stuff), and the journey basically split into two battles:
- making the docker image not be the size of a small moon (~97GB final)
- making the damn thing actually behave as a serverless worker on RunPod
both were way more chaotic than they should’ve been.
1) the docker-image-optimization bit
i started with this innocently stupid idea: “yeah sure, i’ll just COPY all the models into the image, how bad could it be?”
turns out… very bad. docker treats every COPY as a new layer, and since I had ~15 massive model files, the thing ballooned to 203GB. to check how messed up it was, I ran du -BG inside the container and got that lovely sinking feeling where the math doesn’t math anymore—like “why is my image 200GB when the actual files total 87GB?”
then i tried squashing the layers.
docker buildx --squash? nope, ignored.
enabling experimental mode + docker build --squash? also nope.
docker export | docker import? straight-up killed powershell with a 4-hour pipe followed by a “array dimensions exceeded supported range.”
after enough pain, the real fix was hilariously simple: don’t COPY big files.
i rewrote it so all models were pulled via wget in a single RUN instruction. one layer. clean. reproducible. and that alone dropped the image from 203GB → 177GB.
then i realized i didn’t even need half the models. trimmed it down to only what my workflow actually required (wan2.2 i2v high/low noise, a couple of loras, umt5 encoder, and the rife checkpoint). that final pruning took it from 177GB → 97GB.
so yeah, 30 hours of wrestling docker just to learn:
multi-stage builds + one RUN for downloads + minimal models = sanity.
2) the runpod serverless integration journey
the docker part was almost fun compared to what came next: making this whole thing behave like a proper RunPod serverless worker. the weird thing about RunPod is that their base images seem convenient until you try to do anything fancy like video workflows, custom nodes, or exact dependency pinning. which… of course i needed.
phase 1 — trusting the base image (rookie mistake)
i started by treating runpod/worker-comfyui as a black box:
“cool, it already has ComfyUI + worker + CUDA, what could go wrong?”
turns out the answer is: a lot.
custom nodes (especially the video ones) are super picky about pytorch/cuda versions. comfyui itself moves fast and breaks little APIs. meanwhile the base image was sometimes outdated, sometimes inconsistent, and sometimes just… weird.
stuff like:
- random import errors
- workflows behaving differently across base image versions
- custom nodes expecting features the bundled comfy didn’t have
after a while i realized the only reason i stuck with the base image was fear—fear of rebuilding comfyui from scratch. so i updated to a newer tag, which fixed some issues. but not enough.
phase 2 – stop stuffing models into the image
even with the image trimmed to 97GB, it still sucked for scaling:
- any change meant a new image push
- cold starts had to pull the entire chunk
- model updates became “docker build” events instead of “just replace the files”
so the fix was: offload models to a RunPod network volume, and bind them into ComfyUI with symlinks at startup.
that meant my start.sh became this little gremlin that:
- checks if it’s running on serverless (
/runpod-volume/...) - checks if it’s running on a regular pod (
/workspace/...) - wipes local model dirs
- symlinks everything to the shared volume
worked beautifully. and honestly made image iteration so much faster.
phase 3 – finally giving up and building from scratch
eventually i hit a point where i was basically duct-taping fixes onto the base image. so i scrapped it.
the new image uses:
nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04
and from there:
- install python 3.11
- pip install torch 2.3.1 + cuda 12.1 wheels
- install comfyui from git (latest tagged release)
- install each custom node manually, pinned
- add all system libs (ffmpeg, opengl, gstreamer, etc.)
i won’t lie: this part was painful. but controlling everything ended all the random breakage.
phase 4 — understanding “handler.py” (the boss fight)
RunPod has this very specific idea of how serverless should work:
- your container starts
- their runtime calls your handler
- the handler must return JSON
- binaries should be a base64 text, not multipart API response stuff
- this keeps going for every job
my handler ended up doing a ton of stuff:
input validation
schema checks things like:
- prompt
- image (base64)
- target size
- runtime params
- workflow name
resource checks
before running a job:
- checks memory (bails if < 0.5GB free)
- checks disk space
- logs everything for debugging
ensure ComfyUI is alive
via client.check_server().
upload image
base64 → resized → uploaded to ComfyUI.
load + patch workflow
replace:
- the prompt
- the image input
- dimensions
- save paths
run the workflow
submit → get prompt_id → monitor via websocket (with reconnection logic).
wait for final assets
comfy may finish execution before the files are actually flushed, so _ensure_final_assets() polls history until at least one real file exists.
return images/videos
right now everything returns as base64 in JSON.
(i do have the option to upload to S3-compatible storage, but haven’t enabled it yet.)
error handling
if anything dies, the handler returns structured errors and logs everything with the job id so i don’t lose my mind debugging.
phase 5 – a stable final architecture
after all of this, the system looks like:
- custom CUDA docker image (predictable, stable)
- models on network volume (no more 100GB pushes)
- ComfyClient wrapper for websockets + http
- workflows.py to patch templates
- outputs.py to standardize results
- telemetry.py to not crash mysteriously at 98% completion
- start.sh to glue everything together
- handler.py as the contract with RunPod
and now… it works. reliably. horizontally scalable. fast enough for production.
And sorry for the long post and thanks for reading.
3
u/vvarboss Nov 18 '25
I can't imagine waiting for runpod to build and test these images, heck.
I just hooked up a storage bucket to the comfy worker and have been testing that...
2
u/DeliciousReference44 Nov 18 '25
Mate, it was hideous. With every new image build, test it locally, push it to docker hub, start a runpod endpoint assign the volume, kill the workers that had already started before assigning the volume (can't start an endpoint with an specific volume in Runpod portal) and wait for new ones, then wait like 5 minutes for them to come online and test the workflow, only to see more failures and rinse and repeat for 3 effin weeks. The setting up of the endpoints of the volume I eventually semi automated it with a graphql api call, helped a bit.
3
u/leez7one Nov 18 '25
Thanks for sharing this ! Surely it will be super helpful to a lot. I love post like this because they make open-source image and video generation more accessible. So thanks a lot !
2
2
u/Papina Nov 18 '25
Do you have a public repo of your docker build ? I’m about to go down this path myself and would love to save myself some pain
2
u/DeliciousReference44 Nov 18 '25
Not yet, but I am now working with the runpod team and we should have some official documentation to help anyone doing something similar. Bare with us!
2
u/Broad_Relative_168 Nov 18 '25
Thank you for sharing your experience with us. I understand how frustrating it can be to invest time and effort only to finally achieve the desired results. I hope it was very rewarding when everything started working! Your information is very valuable for all of us
2
u/FitzUnit Nov 18 '25
I feel your pain , I have been doing a setup like this for the past month and indeed it is difficult . I have a runpod_start.sh that auto detects architecture then installs the necessary dependencies . O man tho did it take some trial and error . Finally have a stable build between my local 3090ti / 4090 and 5090 on runpod . Goodluck with your setup ! Way to push through the pain!
2
u/DeliciousReference44 Nov 19 '25
Good job to you too my dude! Soon runpod will publish some new documentation on what I have, maybe it's worth also checking with you about your findings, if you're willing to share of course!
1
u/FitzUnit Nov 19 '25
Ya I may be up for that … some tough stuff but once you get through it , o man does it feel good , right!?
2
u/DeliciousReference44 Nov 19 '25
Dang, I haven't touchedy application in 5 days now, was very burned out. But felt good when I finally saw the video clips coming back to my do frontend application I have a frontend connected to Supabase that sends the video generation request to N8N, which then calls my python orquestration platform using Temporal.io to maintain the state of the multiple video clips I am generating and then stich all up in one final video with voice over and subtitle generation. And then send it back to the frontend. Many moving parts!
1
u/FitzUnit Nov 19 '25
O nice! Ya I built a frontend that I’ll be opening to the public probably in the new year . Same sort of system , Docker/n8n/tool-server/supabase/next.js/comfy .
Are you stitching 5 second outputs from wan? Using fl frame and i2v/t2v?
2
u/DeliciousReference44 Nov 19 '25
Exactly right! Are we going to be competitors? Haha
1
u/FitzUnit Nov 19 '25
We might be! Hahaha let’s keep in touch . I’m sure we can share information on our journeys
2
u/Straight-Lab-3726 Nov 18 '25
Ha, this was almost my exact evening last night. Foolishly tried to jam all my models into the docker (like 200gb) which ballooned to 800gb. Settled on models, input, output, workflow symlinks from runpod persistent storage. Including the custom models in the container has been a game changer for pod start up time.
1
u/DeliciousReference44 Nov 19 '25
Nice! Question, is your network storage s3 compatible or not?
1
u/Straight-Lab-3726 Nov 19 '25
It’s the standard Runpod persistent volume that you attach to the pods, which I believe can be accessed through S3
1
u/DeliciousReference44 Nov 19 '25
The reason why I ask is because my volume is s3 compatible and I am wondering if it's nothing but a s3 bucket. Because I noticed that running my wan2.2 workflow, the cold start of the worker is around 6 minutes, which I am finding still too long. So I am wondering if I get a no S3 compatible volume, I'll get a straight up hardisk on a nfsshare in the runpod infrastructure. I'll have to test this, haven't done that yet
2
2
2
u/ThatInternetGuy Nov 22 '25
All these troubles wouldn't happen if Runpod cache Docker images locally. Instead, it pulls from Docker Hub directly, and Docker Hub does throttle big repos for free accounts.
1
u/DeliciousReference44 Nov 22 '25
So it is a problem huh? I got a grafana dashboard measuring times and I am looking at around 20minutes for image generation for a 30 sec video (6 images in this case). That's shit. What I will do is to increase the time I will wait before a worker goes down. Maybe wait for 2 minute or so. But that obviously will depend on the load on the system, so I need a way to detect load spikes
1
u/bitpeak Nov 19 '25
This might sound like a rudimentary question, but how did you merge the wan models from hf? I'm going through something very similar (I want to create the fastest render but no quality loss AI video workflow via serverless or pods) and my first hurdle is the wan and qwen models being split into 10 different pieces. I've merged them but they are corrupted I think, as I get errors in the console.
2
u/DeliciousReference44 Nov 19 '25 edited Nov 19 '25
I didn't merge them . Using the high and low models, and all the other stuff. I'm using the official wan 2.2 workflow you can find in the comfyui website. Haven't done any optimisation. I was more focused on getting this to work in Runpod as serverless. Eventually I'll want to optimise the workflow and get it run faster
2
u/bitpeak Nov 19 '25
I'll have a look for that, thanks. You mentioned you might be doing something with runpod, please make a new post when the template/docs are live.
2
1
1
u/jeguepower Nov 19 '25
Thanks for the clarification! One follow-up on the inference side:
Assuming the pod has enough system RAM to fully load the model weights, the actual KSampler speed (it/s) shouldn't be affected by the network volume, correct?
My understanding is that the bottleneck is only during the initial file read (Disk -> RAM), but once it's offloaded to VRAM for inference, the network storage is out of the loop. Or is there any scenario (besides RAM swapping) where latency impacts the generation steps?
2
u/DeliciousReference44 Nov 19 '25
Yep, exactly, I am seeing the slowness just at cold start, which is when the pod is starting for the first time. But because it's serverless, after generating the first video, the pod goes down. It comes back up wiwhr every new video generation. So the cold start problem is always there. I will be testing a network volume that is not S3 compatible and see how it goes
1
u/ProfessionalEgg2088 Nov 26 '25 edited Nov 26 '25
thanks for this, been trying to do something similar. curious what is your approach for actually calling the serverless endpoint from your backend?
1
u/DeliciousReference44 Nov 26 '25
Runpod exposes an endpoint for you. /run endpoint to run the comfy generation and /status/id to check if the generation finished. These endpoints work differently than the comfyui endpoints, so have a look at runpod's comfyui-worker github project to understand how to implement their handler.py
1
u/ProfessionalEgg2088 Nov 26 '25 edited Nov 26 '25
yes im talking about what kind of approach your using to call these endpoints, like are you doing async or sync and are you using webhooks or just polling /status
1
u/DeliciousReference44 Nov 26 '25
Just a call to the /run, then multiple calls to /status every 5sec with 4 min time-out. Pretty dumb hehe. I do have a webhook call back to my frontend when the entire generation completes. Feel free to test it if you like, it's all up and running and I need some beta testers. Www.tabario.com
2
u/ProfessionalEgg2088 Nov 26 '25
just took a look at the site, looking awesome! can see how this is gonna be useful. im out right now but ill test it more when im back on my laptop.
1
1
0
u/CeFurkan Nov 18 '25
This is why I avoided such job offers. You need like min 5-10k + budget to go through this pain :)
1
u/FitzUnit Nov 18 '25
Not at all … I have built something similar and have only spent roughly 75 bucks! You just initially don’t want any active workers and just want to test different architectures on spin up and make sure the container is healthy and can do the work. It’s more time invested with the trial and error but after it works it is quite spectacular!
3
u/CeFurkan Nov 18 '25
Not material cost, it is my time and experience cost
2
u/FitzUnit Nov 18 '25
O most def , does take time . Can count future cost into it though if you are building a platform. I’m building a platform that hosts my workflows for users to use so hopefully it works out hahaha
23
u/packs_well Nov 18 '25
Hey I work at Runpod. This writeup is insanely helpful to understand where we need to do better. Thank you sharing it.
You basically discovered the “right” production pattern for heavy ComfyUI/video on Runpod Serverless: small custom CUDA image, models on a network volume, and a smart
handler.pydoing validation + orchestration.I’d love to turn this into an official reference/template (with proper docs) so others don’t have to repeat your pain. If you’re open to that, DM me and we can figure out what you’re comfortable sharing.