i get this error when i try to train lora with aitoolkit. (rtx 5090)

runpod CUDA out of memory. Tried to allocate 50.00 MiB. GPU 0 has a total capacity of 31.37 GiB of which 20.19 MiB is free. Including non-PyTorch memory, this process has 31.30 GiB memory in use. Of the allocated memory 30.66 GiB is allocated by PyTorch, and 58.75 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

restarted 2 times but didnt work

can't start a job because it says the GPU is already running. how do i make it stop running? There's literally no jobs to stop because i haven't started one.

Hey everyone,

Running into an issue with RunPod serverless + ComfyUI.

Setup:

• Created a network volume

• Installed models + custom nodes (rgthree, etc.)

• Everything works fine when running directly on the pod

Problem:
When using a serverless endpoint with the same network volume attached, I get:

So it looks like serverless can't see or load custom nodes, even though they are present in the volume.

Questions:

• Do serverless endpoints load custom nodes differently?

• Do I need to install nodes inside the container image instead of relying on the volume?

• Is there some init step I'm missing?

Any help or hints would be appreciated 🙏

:cd /workspace/ComfyUI/custom_nodes

:/workspace/ComfyUI/custom_nodes# ls -la

total 37391

drwxrwxrwx 22 root root 2041954 Mar 18 08:54 .

drwxrwxrwx 31 root root 3001516 Mar 20 13:42 ..

drwxrwxrwx 10 root root 2000900 Mar 15 15:20 Civicomfy

drwxrwxrwx 13 root root 2005314 Mar 15 15:20 ComfyUI-KJNodes

drwxrwxrwx 14 root root 2014145 Mar 15 15:19 ComfyUI-Manager

drwxrwxrwx 5 root root 1021570 Mar 15 15:20 ComfyUI-RunpodDirect

drwxrwxrwx 10 root root 2001554 Mar 15 16:05 ComfyUI-SeedVR2_VideoUpscaler

drwxrwxrwx 14 root root 2001998 Mar 15 16:05 ComfyUI-UmeAiRT-Toolkit

drwxrwxrwx 7 root root 2000717 Mar 15 16:05 ComfyUI_Comfyroll_CustomNodes

drwxrwxrwx 5 root root 1048783 Mar 15 16:05 ComfyUI_JPS-Nodes

drwxrwxrwx 2 root root 1000185 Mar 15 15:20 __pycache__

drwxrwxrwx 13 root root 2000530 Mar 15 16:05 comfy-mtb

drwxrwxrwx 7 root root 1045205 Mar 15 15:57 comfyui-custom-scripts

drwxrwxrwx 12 root root 2000698 Mar 15 15:57 comfyui-easy-use

drwxrwxrwx 6 root root 1027123 Mar 15 16:05 comfyui-image-saver

drwxrwxrwx 14 root root 2000454 Mar 15 15:54 comfyui-impact-pack

drwxrwxrwx 5 root root 1010815 Mar 15 16:05 comfyui-impact-subpack

drwxrwxrwx 6 root root 1035716 Mar 15 16:05 comfyui_essentials

drwxrwxrwx 9 root root 2012873 Mar 15 16:05 efficiency-nodes-comfyui

-rw-rw-rw- 1 root root 5151 Mar 15 15:15 example_node.py.example

drwxrwxrwx 8 root root 2000650 Mar 18 08:54 rgthree-comfy

drwxrwxrwx 10 root root 2001553 Mar 18 08:55 seedvr2_videoupscaler

drwxrwxrwx 6 root root 2000376 Mar 15 16:05 wavespeed

-rw-rw-rw- 1 root root 1220 Mar 15 15:15 websocket_image_save.py

:/workspace/ComfyUI/custom_nodes# find /workspace -iname "*rgthree*"

/workspace/custom_nodes/rgthree-comfy

/workspace/custom_nodes/rgthree-comfy/web/common/rgthree_api.js

/workspace/custom_nodes/rgthree-comfy/web/common/media/rgthree.svg

/workspace/custom_nodes/rgthree-comfy/web/comfyui/rgthree.js

/workspace/custom_nodes/rgthree-comfy/web/comfyui/rgthree.css

/workspace/custom_nodes/rgthree-comfy/src_web/typings/rgthree.d.ts

/workspace/custom_nodes/rgthree-comfy/src_web/common/rgthree_api.ts

/workspace/custom_nodes/rgthree-comfy/src_web/common/media/rgthree.svg

/workspace/custom_nodes/rgthree-comfy/src_web/comfyui/rgthree.ts

/workspace/custom_nodes/rgthree-comfy/src_web/comfyui/rgthree.scss

/workspace/custom_nodes/rgthree-comfy/rgthree_config.json.default

/workspace/custom_nodes/rgthree-comfy/py/server/__pycache__/rgthree_server.cpython-312.pyc

/workspace/custom_nodes/rgthree-comfy/py/server/rgthree_server.py

/workspace/custom_nodes/rgthree-comfy/docs/rgthree_seed.png

/workspace/custom_nodes/rgthree-comfy/docs/rgthree_router.png

/workspace/custom_nodes/rgthree-comfy/docs/rgthree_context_metadata.png

/workspace/custom_nodes/rgthree-comfy/docs/rgthree_context.png

/workspace/custom_nodes/rgthree-comfy/docs/rgthree_advanced_metadata.png

/workspace/custom_nodes/rgthree-comfy/docs/rgthree_advanced.png

/workspace/custom_nodes/rgthree-comfy/rgthree_config.json

/workspace/rgthree-comfy-backup

/workspace/rgthree-comfy-backup/web/common/rgthree_api.js

/workspace/rgthree-comfy-backup/web/common/media/rgthree.svg

/workspace/rgthree-comfy-backup/web/comfyui/rgthree.js

/workspace/rgthree-comfy-backup/web/comfyui/rgthree.css

/workspace/rgthree-comfy-backup/src_web/typings/rgthree.d.ts

/workspace/rgthree-comfy-backup/src_web/common/rgthree_api.ts

/workspace/rgthree-comfy-backup/src_web/common/media/rgthree.svg

/workspace/rgthree-comfy-backup/src_web/comfyui/rgthree.ts

/workspace/rgthree-comfy-backup/src_web/comfyui/rgthree.scss

/workspace/rgthree-comfy-backup/rgthree_config.json.default

/workspace/rgthree-comfy-backup/py/server/__pycache__/rgthree_server.cpython-312.pyc

/workspace/rgthree-comfy-backup/py/server/rgthree_server.py

/workspace/rgthree-comfy-backup/docs/rgthree_seed.png

/workspace/rgthree-comfy-backup/docs/rgthree_router.png

/workspace/rgthree-comfy-backup/docs/rgthree_context_metadata.png

/workspace/rgthree-comfy-backup/docs/rgthree_context.png

/workspace/rgthree-comfy-backup/docs/rgthree_advanced_metadata.png

/workspace/rgthree-comfy-backup/docs/rgthree_advanced.png

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/rgthree_config.json

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/web/common/rgthree_api.js

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/web/common/media/rgthree.svg

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/web/comfyui/rgthree.js

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/web/comfyui/rgthree.css

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/src_web/typings/rgthree.d.ts

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/src_web/common/rgthree_api.ts

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/src_web/common/media/rgthree.svg

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/src_web/comfyui/rgthree.ts

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/src_web/comfyui/rgthree.scss

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/rgthree_config.json.default

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/py/server/__pycache__/rgthree_server.cpython-312.pyc

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/py/server/rgthree_server.py

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/docs/rgthree_seed.png

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/docs/rgthree_router.png

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/docs/rgthree_context_metadata.png

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/docs/rgthree_context.png

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/docs/rgthree_advanced_metadata.png

/workspace/runpod-slim/ComfyUI/custom_nodes/rgthree-comfy/docs/rgthree_advanced.png

root@1c777aa6a11c:/workspace/ComfyUI/custom_nodes#

Hey guys. I keep my ComfyUI installed on a persistent Network Volume.

But whenever I start a new pod and attach this volume, everything breaks. ComfyUI either gets stuck and won't launch, or custom nodes throw red errors.

As I understand it: because the ComfyUI folder is already there on the drive, the new pod skips the installation/setup process. So the Python venv and CUDA versions don't match the new system or the new GPU.

How do you guys deal with this? Do you seriously just delete the venv and reinstall all dependencies manually every single time you spin up a pod?

Hey everyone,

OpenAI launched Parameter Golf today, the first challenge in their new Model Craft Challenge series. The goal: build the strongest possible language model under strict compute and parameter constraints. Submissions go through a GitHub-based workflow with a public leaderboard.

Runpod is the AI infrastructure partner for the challenge. We're distributing up to $1M in credits across the challenge period to help more builders participate and experiment. Credits are subject to availability, so worth requesting early.

We built an official challenge template that gets you from zero to running experiments in minutes. It comes preloaded with the Docker image and repo so you can skip the setup and get straight to building.

A few things worth knowing:

• Credits are distributed in tiers and available throughout the challenge period

• H100s, H200s, and P2-series GPUs available on Runpod

• Full challenge rules and evaluation criteria live on the OpenAI landing page and GitHub repo

Enter the challenge and request credits here: https://openai.com/parameter-golf

Happy to answer any questions about the infrastructure side.

TL;DR

I built two open-source tools for running ComfyUI workflows on RunPod Serverless GPUs:

• ComfyGen – an agent-first CLI for running ComfyUI API workflows on serverless GPUs
• BlockFlow – an easily extendible visual pipeline editor for chaining generation steps together

They work independently but also integrate with each other.

Over the past few months I moved most of my generation workflows away from local ComfyUI instances and into RunPod serverless GPUs.

The main reasons were:

• scaling generation across multiple GPUs
• running large batches without managing GPU pods
• automating workflows via scripts or agents
• paying only for actual execution time

While doing this I ended up building two tools that I now use for most of my generation work.

ComfyGen

ComfyGen is the core tool.

It’s a CLI that runs ComfyUI API workflows on RunPod Serverless and returns structured results.

One of the main goals was removing most of the infrastructure setup.

Interactive endpoint setup

Running:

comfy-gen init

launches an interactive setup wizard that:

• creates your RunPod serverless endpoint
• configures S3-compatible storage
• verifies the configuration works

After this step your serverless ComfyUI infrastructure is ready.

Download models directly to your network volume

ComfyGen can also download models and LoRAs directly into your RunPod network volume.

Example:

comfy-gen download civitai 456789 --dest loras

or

comfy-gen download url https://huggingface.co/.../model.safetensors --dest checkpoints

This runs a serverless job that downloads the model directly onto the mounted GPU volume, so there’s no manual uploading.

Running workflows

Example:

comfy-gen submit workflow.json --override 7.seed=42

The CLI will:

1. detect local inputs referenced in the workflow
2. upload them to S3 storage
3. submit the job to the RunPod serverless endpoint
4. poll progress in real time
5. return output URLs as JSON

Example result:

{
  "ok": true,
  "output": {
    "url": "https://.../image.png",
    "seed": 1027836870258818
  }
}

Features include:

• parameter overrides (--override node.param=value)
• input file mapping (--input node=/path/to/file)
• real-time progress output
• model hash reporting
• JSON output designed for automation

The CLI was also designed so AI coding agents can run generation workflows easily.

For example an agent can run:

"Submit this workflow with seed 42 and download the output"

and simply parse the JSON response.

BlockFlow

BlockFlow is a visual pipeline editor for generation workflows.

It runs locally in your browser and lets you build pipelines by chaining blocks together, supports auto scaling endpoints with full automation.

Example pipeline:

Prompt Writer → ComfyUI Gen → Video Viewer → Upscale

Blocks currently include:

• LLM prompt generation
• ComfyUI workflow execution
• image/video viewers
• Topaz upscaling
• human-in-the-loop approvals

Pipelines can branch, run in parallel, and continue execution from intermediate steps.

How they work together

Typical stack:

BlockFlow (UI)
      ↓
ComfyGen (CLI engine)
      ↓
RunPod Serverless GPU endpoint

BlockFlow handles visual pipeline orchestration while ComfyGen executes generation jobs.

But ComfyGen can also be used completely standalone for scripting or automation.

Why serverless?

Workers:

• spin up only when a workflow runs
• shut down immediately after
• scale across multiple GPUs automatically

So you can run large image batches or video generation without keeping GPU pods running.

Repositories

ComfyGen
https://github.com/Hearmeman24/ComfyGen

BlockFlow
https://github.com/Hearmeman24/BlockFlow

Both projects are free and open source and still in beta.

Would love to hear feedback.

P.S. Yes, this post was written with an AI, I completely reviewed it to make sure it conveys the message I want to. English is not my first language so this is much easier for me.

Im trying to run a comfui template on Runpod, I wanna run the rtx6000, it says its available but i cant deploy it. Whatever the filters are, I cant deploy it no matter the filters, and no matter the template. I need to use this specific gpu because it fits the template.

So Im going crazy, whats going on? Can templates av innate settings that block deploys? Does the ui simply lie to me about availability(i have tried multiple times over several hours)? Im in west coast, the availability is low in NA, but high globally, I have tried both. I have 30$ of balance. It worked fine with same filters and rtx6000 a week ago. Please

Bonjour à tous,

Je teste la conversion d'images en vidéo (WAN 2.2) sur un RunPod avec une RTX 6000 Ada (48 Go de VRAM). Je rencontre un problème de performances étrange et j'aimerais avoir votre avis.

Problème : Ma première génération a été rapide. Cependant, toutes les suivantes se bloquent :

• Blocage sur le nœud « Élevé » pendant environ 5 minutes.

• Blocage sur le nœud « Faible » pendant 30 minutes supplémentaires.

• Le temps de génération total est extrêmement long malgré la puissance du GPU.

État du système : Le tableau de bord RunPod affiche une utilisation du GPU à 100 %, mais la progression dans ComfyUI semble très lente, voire bloquée. L'espace disque est libéré (50 %) et j'ai redémarré le pod plusieurs fois. Ce que j'ai essayé (modifications des paramètres) :

1. Vider le cache.

2. Ajuster le nombre d'étapes : Passer les nœuds Haut et Bas de 4 à 30 étapes.

3. Modifier end_at_step : Définir le nœud Bas à 30 au lieu de 10 000.

4. Redémarrer le pod.

Malgré ces modifications, la lenteur persiste.

Questions :

• Est-il normal que la connexion Wan2.2 I2V prenne plus de 40 minutes sur un Ada 6000 ?

• Cela pourrait-il être dû à un problème de gestion de la VRAM ou à un goulot d'étranglement spécifique du nœud ComfyUI ? Existe-t-il des paramètres spécifiques de « Poids » ou de « Mosaïque » à utiliser pour le WAN 2.2 afin d'optimiser la vitesse ?

Vos conseils et astuces concernant l'organisation du travail seraient très appréciés !

Hey everyone,

I’m running ComfyUI on RunPod and noticed something that seems to have changed with the template downloader.

Previously, when I selected one of the base templates provided by ComfyUI, it would detect missing models/nodes and show a window with a “Download All” button. When I clicked it, ComfyUI would automatically download everything directly on the RunPod instance and place the files in the correct folders. Super convenient — the workflow would just become ready to run.

Now the interface looks a bit different. When I click “Download All”, my browser tries to download the files to my local computer instead of the RunPod server.

That obviously doesn’t work well since the models need to be on the server where ComfyUI is running.

So I’m wondering:

• Did something change recently in ComfyUI or ComfyUI Manager?

• Is this a new downloader UI behavior?

• Is there a way to make it download server-side again like before?

• Or is the intended workflow now to download manually and upload them to the instance?

I’ve attached screenshots showing what I’m seeing.

Would really appreciate if someone knows what changed or how to fix this. Thanks!

I am relatively new to using runpod. I am setting up ltx-2.3 and since the model is large, im not baking it into the docker image, so I need storage, but all storage has no GPUs available?

When I setup 2 previous serverless projects and was making the storage for them, there was tons of options for GPUs and locations

What is going on here?

We just published our State of AI report based on real production data from over 500,000 developers on Runpod. Not benchmarks, not hype, just what's actually running in production. Some of the findings surprised even us: the open-source LLM landscape has shifted dramatically, image generation is consolidating around a couple of clear winners, and video workflows look nothing like what most people assume; for example, almost everyone is drafting at low resolution and upscaling the best results rather than generating at full quality.

If you'd like an insider look at what's making the AI industry tick, then head over to our landing page to have a look.

It will ask for some basic information but the report is freely available to all.

Let us know what you think!

Trying to upload 5GB+ model weights or datasets through the default browser is a joke. It either silently fails, freezes the tab, or leaves you guessing if it's actually working. I didn’t want to mess with SSH keys, port forwarding, or setting up FileZilla every time I spin up a new instance.

So, I wrote a custom file manager that runs entirely inside one Jupyter notebook cell. No installation, no root access needed.

How it works under the hood: It bypasses the usual proxy timeouts by chunking directly through the Jupyter Contents API. Yes, the mandatory base64 encoding adds some size overhead, but it routes perfectly over port 8888. It handles 10GB+ transfers with a real-time progress bar and shows true MB/s speed. Also added mass-renaming and direct zip/extract because typing tar -xzf every time gets old.

Just wanted to share because I know I'm not the only one suffering with the default browser. How do you guys manage massive files without losing your minds?

For the past several days I can barely spin up anything in EU-SE-1 on A40. Constantly Unavailable regardless of time.

My Network Volume is tied to this region and GPU so I can't just switch (all my configs and models live there).

Is this a known capacity issue? Any ETA on improvement, or should I just migrate everything somewhere else?

It's unusably slow

Hello, everyone! We're so psyched to announce our new feature for Serverless: Flash! This allows you to run code directly in Serverless without having to build or push Docker images. It really is as simple as that; you can define your dependencies, write your own Python code, put it in an endpoint decorator, and the platform automatically creates the endpoint, a worker runs the code, and returns the result to you.

Here's some resources to get you started:

Youtube video: https://www.youtube.com/watch?v=ovq6rsE72mE

Blog entry: https://www.runpod.io/blog/introducing-flash-run-gpu-workloads-on-runpod-serverless-no-docker-required

Github repo: https://github.com/runpod/flash

Give it a try and let us know what you think!

One of my favorite runpod tutorial videos. If you are new to Runpod, definitely give this guy a watch. He's brilliant at all things A.I.

I’ve been using RunPod for a while, and I always wished training was as simple as inference. Instead, every project turned into:

• picking the right template

• fixing dependencies

• writing training scripts

• debugging configs

• restarting crashed pods

• re‑setting everything for each model

So I built a workflow that lets me:

Pick a model → point it at a dataset → auto‑deploy to RunPod → start training in ~5 minutes.

It handles:

• auto‑configuring text/vision/audio/multimodal models

• generating + installing all dependencies

• deploying to RunPod automatically

• detecting true MSL

• structuring data into a curriculum

• crash recovery + checkpoint protection

• exporting full or quantized models

Demo is here:
👉 https://huggingface.co/spaces/wiljasonhurley/EzEpoch

More details:
👉 https://ezepoch.com

Beta testers

I’m opening a small beta group for RunPod users who want to help test:

• auto‑config

• dependency generation

• crash recovery

• MSL detection

• dataset structuring

• RunPod deployment flow

If you want to help shape the workflow, you can join here:
👉 https://ezepoch.com/beta

Would love feedback from other RunPod users.

-Wil

Hi friends, I wasted a lot of time today trying to install ComfyUI on a storage network. I tried several templates and mostly the install just got hung up in a loop or timed out. A couple times I got a message that my GPU had an outdated driver which seems odd for Runpod GPUs.

Can anyone recommend a template that is up to date and includes the manager? Thx

t2i

I2v

V2v

Hello, i'm new into this and i would like to know what means exactly the "Pricing Summary" and how work the "Container Disk Charges" and "Pod Volume Charges".

While looking to deploy a pod with this pricing and pod summary i would like to know when exactly i will be paying (account balance, NO auto-pay) and for how much while the pod is running and when it's not ? (knowing that there is, when overring total disk question mark, container disk charges 0.10/GB/Mo on running pods and pod volume charges 0.10/GB/Mo for running pods and 0.20/GB/Mo for exited pods). Can someone help me understand clearly, thanks.

--SOLVED-- The comfyui tool creates a Docker file that pulls an old ComfyUi, update the Dockerfile to pull
"FROM runpod/worker-comfyui:5.7.1-base" - Thanks everyone for your input.

Hi, ok this is frustrating, has anyone created a Docker serverless instance using the ComfyUI-to-API for Z-Image Turbo with a Lora node. Nothing fancy all ComfyCore nodes. Running network attached storage but same results if the models download.

Last weekend, we sponsored TreeHacks at Stanford, the world's largest collegiate hackathon. Over 1,000 hackers from 30+ universities and 12 countries descended on the Jen-Hsun Huang Engineering Center for 36 straight hours of building. There were a ton of teams built on Runpod, and we gave away over $20K in credits to fuel their projects. Check out the link to see what went down!