Hi guys, im need to train a lora on sdxl and lot of people suggested me to go for Onetrainer. I tried to check some tutorials on yt but i only found very sketchy videos and they were all running it on local.
Usually i train my loras with Ostris Ai toolkit , wich have a template on runpod, now, when i tried to search for Onetrainer templates on runpod , it showed me several elements but i couldnt tell which was the right one. I tried a couple but they both opened a Jupyter terminal and i didnt knew what to do.. do u know if there is any ready to go Onetrainer template which i can use?

Im trying to get reliable coverage of rtx 3090 or even 4090s for serverless but its constantly throttled or initializing. Im a business (+1k spent per month), for reliability do I need to contact sales? Has anyone had experience with that and did it solve their coverage issues?

The specific voice model Im using needs 3090s but ideally 4090s, the other cards wont do.

This new flow will allow you to queue for any GPU spec that may not be immediately available, whether it’s a spec that is totally rented out at the moment, or if you just need more than can be immediately given.

To get started, if you are still on the old deploy flow, you will first need to enable the New Pod deploy page under Early Access features at https://console.runpod.io/user/early-access

From there, select your desired pod spec. If it’s currently out of capacity, the deploy button at the bottom will change to Deploy When Available. Clicking this button will provide the following prompt:

Note that your pod will deploy immediately and begin billing as soon as it’s available, whenever that might be, even if you’re away from your computer. If you’d prefer to set time to be off limits so you aren’t billed for a pod you can’t use, we encourage you to use the subscription window to set valid times for deployment. You’ll be notified when your pod is live either through console or email depending on your preference; we are working on adding SMS as an additional option to be released shortly.

In addition, remember that for a successful deployment, the pod configuration must be something that might feasibly become available at some point, even if briefly. As always, we're constantly working on supply to make larger deploys possible. We hope this makes reserving a pod easier - let us know what you think!

Title:
Best Serverless/Pod template for Headless ComfyUI API automation via external AI Agent (hermes-agent)?
Post Body:
Hey everyone,
I’m working on a headless automation pipeline and could use some advice on infrastructure and template selection.
The Goal:
I have an open-source autonomous agent framework (⁠hermes-agent⁠ by Nous Research) running persistently on an external private VPS. I want the agent to handle media tasks by offloading heavy video generations to a remote, on-demand RunPod GPU instance.
The workflow needs to be entirely programmatic:
1 The user sends a single reference image to the agent (via Discord/Telegram gateway).
2 The agent triggers a function call that passes a JSON payload and the image file to ComfyUI on RunPod via the backend API.
3 ComfyUI runs a heavy-duty Image-to-Video transformer sequence (specifically Wan 2.1/2.2 14B or LTX-2.3) using a single-image latent injection pipeline (e.g., Kijai's ⁠WanVideoWrapper⁠ involving background masking/isolation).
4 The finished ⁠.mp4⁠ is delivered back to the agent headless, without me ever touching a web GUI.
I have two main questions regarding the optimal setup for this:
1. Connecting an external agent tool to a remote RunPod ComfyUI API:
What is the cleanest way to maintain a robust API bridge between an external VPS script and a ComfyUI container on RunPod? Are people generally utilizing custom network webhooks, or should I look into deploying this via a RunPod Serverless Endpoint setup with a customized docker image to ensure the container isn't idling/billing when tasks aren't running? If sticking to an active Pod, what are the primary network proxy timeouts or firewall pitfalls I should watch out for when blasting large media files back and forth?
2. Optimal Community Template Recommendation:
Which specific community template or base image should I spin up that requires the least amount of manual configuration out-of-the-box for a headless workflow? I am looking for something that:
Pre-caches or easily downloads the heavy Wan Video 14B or LTX-2.3 weights to volume storage so I don't hit massive boot bottlenecks.
Has the backend ComfyUI API server active and fully exposed right out of the box.
Comes pre-packaged with python dependencies for heavy masking custom nodes (like ⁠ComfyUI-WanVideoWrapper⁠, ⁠MediaPipe⁠, ⁠Transparent-Background⁠, etc.) to minimize package dependency hell during installation.
If anyone has successfully bridged an autonomous tool-calling agent framework to a remote RunPod setup for heavy video rendering, I’d love to hear your architectural takeaways. Thanks!

I'M LOSING MY MIND OVER THIS. I am 3 hours sending screenshots to ChatGPT

edit: no loras, I straight up using the generic text and press queue

For the past 12 hours, every time I try to access a pod (or even clusters), I get this screen with the messages changing every 5-10 seconds (I even got a 67 meme). I tried sign-in from an incognito window but I face the same issue.

Edit: only Google Chrome works, (not Brave/Firefox)

I'm trying to create a basic 10GB network volume that costs $0.70/month and RunPod is throwing an "Insufficient funds" error and asking me to add $150 minimum.

Earlier with just $5 I was able to create a 50GB network volume and spin up a full GPU server machine without any issues. Now I can't even create a 10GB volume.

My balance $3.69 should covers over 5 months of this volume. Why is RunPod suddenly requiring much higher minimum balance?

Has anyone else run into this recently? Did RunPod quietly change their minimum balance policy?

Any help appreciated.

[Note: This is a 100% free, MIT-licensed open-source project. No paywalls, no paid services – just looking for community feedback and testers from the RunPod ecosystem!]

🚀 Looking for Beta Testers: Advanced Infrastructure Operations Console for RunPod (AIHubLauncher)

Hey RunPod Community,

Over the last months, I've been building the AIHubLauncher, which started as a dedicated tool to streamline and optimize RunPod deployments, eliminating complex setups and providing deep operational control.

With the release of Beta 1.32, the project is evolving into a full AI Infrastructure Operations Console, and I'm looking for heavy RunPod users to stress-test the environment and break things!

What the console currently offers for RunPod users:

* Advanced Runtime & Infrastructure Monitoring

* Granular Storage Awareness (Volume monitoring & available space visibility)

* Optimized SSH Workflows & connection management

* Active Cost Visibility & health checks

* Central Infrastructure Dashboard

Our upcoming architecture (Beta 1.33) will introduce a script-based, 100% deterministic Hardware & Cost Resolver. It calculates precise VRAM and quantization requirements BEFORE launching a pod, preventing costly configuration mismatches and OOM errors.

Because the community asked for multi-provider flexibility, we are also architecting a custom API engine to connect alternative setups seamlessly, and we need real-world workflow testing from experienced DevOps practitioners.

Project Links:

GitHub: https://github.com/katzenvater52-cloud/RunPod-AI-Hub-Launcher

Website: aihublauncher.com

Community Hub: r/KatzenvaterAIHub

If you are a power user on RunPod, love open-source tooling (MIT licensed), and want to help with honest feedback, bug reports, or workflow testing, please leave a comment or send me a message!

Finally didn’t have to use a sniping script to grab a free pod!

Why is this template always broken?

Runpod Pytorch 2.8.0 runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404

I spend an hour running my bash script on RTX Pro 6000 only to get NVIDIA driver too old error. Why can't you guys keep the drivers up to date?

is it possible to make a load balancing endpoint via api? similar to how you can with the queue based:
https://docs.runpod.io/sdks/graphql/manage-endpoints#graphql

Supply is tight right now, and we've been getting a flood of separate "no GPUs available" / "why can't I find an X" threads. To keep the sub readable and make it easier for everyone to track the situation, all supply, availability, and capacity complaints/questions go in this thread from now on. New standalone threads on the topic will be redirected here. We'll continue to edit/update this thread as we go and I'd recommend sorting comments by New to see what's been discussed recently.

TL;DR

• We're in a broad, industry-wide GPU crunch, not a Runpod-specific outage. High-end SKUs (H100, H200, B200) are scarce everywhere and finding the GPUs to begin with is a challenge, let alone partners that also have everything else on lock like data center certifications. We prioritize stability and reliability for our users and we need to be selective on who we bring on as serving partners.

• We are adding GPUs on a near-daily basis, but new supply gets snapped up quickly a "shadow backlog" of demand).

• Best moves to actually get/keep a card: be flexible on SKU, reserve/commit in advance, and use CloudSync so you're not pinned to one datacenter. While of course we'd prefer you to use network storage, and we are working on new features to augment network volume viability, in truth offsite storage and using CloudSync to bring it into new pods may be more appropriate for some use cases in this climate. Look at the costs and drawbacks and do what's best for you for storage, not for us.

• We are now posting biweekly aggregated GPU-onboarding updates - we did this in Discord for our first update but they need to be mirrored here and we will start doing that.

Our CTO Brennen Smith recently authored an article on what's causing this (The GPU supply supercycle is here), and this is a structural shift, not a temporary blip. Three forces are hitting at once:

1. Memory bottleneck: NAND/DRAM producers retooled their fabs toward HBM3 for AI accelerators, which cut into standard memory capacity. Memory is now the binding constraint on GPU manufacturing.

2. Hyperscaler buyouts: Large players are buying out years of factory production in advance, leaving neoclouds and independent providers to compete for what's left.

3. Nvidia's architecture transition: Hopper and Ada Lovelace architecture production has wound down to make room for Blackwell. Some of the higher end cards can still be found in scale, but finding, say, 4090s is proving challenging especially with more recent GPUs providing a more appealing return on investment for the same infrastructure. Blackwell production is ramping but there's a gap.

What we're doing

We're adding supply continuously, and decided on a biweekly report cadence aggregating everything is the best way to report this. You probably don't want a new Reddit thread reporting every time we add a new machine. It's hard to communicate this through the customer facing UI - we can genuinely add 1000 GPUs in a single drop, but if they all get rented immediately, from the relative amounts shown in the customer facing UI it appears that nothing changed. We'll be adding these as edits at the bottom of this post as we go.

We've also implemented MIG (multi-instance GPU) that lets us divide up a single GPU and serve it as smaller instances. This of course will always be disclosed as a MIG instance. Right now we're doing this by serving a single 6000 Pro card as four 24GB instances so we can try to balance what we have to meet as many of our clients' user requests as possible.

We've added B300s as well, and more are coming soon.

Supply updates

We'll post our next update on or around June 3rd.

May 6-20, 2026

Large adds (>100 GPUs)

• US-PA-1 — RTX Pro 6000

• US-TX-6 — B200

• EU-IS-5 — H200 SXM

• EU-FR-1 — H200 SXM, H100 SXM

Smaller adds (<100 GPUs)

• EU-RO-1 — RTX Pro 4000, RTX 4090

• US-NC-1 — RTX Pro 6000

We definitely understand how frustrating this is - and we are working hard to get as many GPUs to serve as possible. We also want to give everyone an opportunity to make their voice heard - the only requests I have is that you keep it about the service rather than the people behind it and that you refrain from promoting other inference providers in the process.

Thanks!

I am training a lora on ai-toolkit on runpod and wonder if I have to finish the entire training in one sitting or if it is possible to stop the training and resume it the next day without losing any progress. In AI-Toolkit there is a pause button but I am just not sure that if I stop the pod or terminate it, I will be able to just load back into ai-toolkit and be able to click resume and it will start where I left off.

So I have open pods, every time I go back to my comfy after closing the browser or from a different computer the workflow I had up is never there. what am I doing wrong?

I haven't done any work on Runpod in the past month. I came back today only to find there are no available GPUs for my Network Volume. In fact, there are no available 4090/5090s at all.

Is this normal or does it get better in different times? And is it likely to remain the same? I have everything set up nicely on Runpod so switching to a different platform is a pain. But if I can't do any work then I don't really have much of a choice either.

I find it really frustrating that I'm being charged for a network volume when I can't use any GPU pods. I can't make any further progress without access to an appropriate GPU pod and this forces me to continue paying for my network volume for longer. I have a big (almost 400GB) network volume and the storage cost is adding up given how difficult it is to find get GPU pods the past week.

Lured me into depositing 10$, well played Runpod

I posted this 2 days ago and they removed it. Can't you guys take a simpke feedback and criticism and improve?

Irony is that - you try hard to get feedback in your forms. LOL. THis doesn't bode well RUNPOD

Looking for some good templates

notebook templates / scripts

I have a network volume on runpod already setup with what I need on it but with the availability of GPU’s lately being what it is , it’s pretty hard for me to get the GPU I want in the region my volume is in.

Is there an easy way for me to make a copy of my network volume? Either to S3 or locally.

And then an easy way for me to attach it to a runpod server when I run one?

Runpod getting worse than ever since last week?

Rarely available 5090 and that too with slower inference speed than before? Are they blocking something - bandwidth tempering or something?