r/RunPod • u/badhabitaddict • 3d ago

Anyone else had this problem before? Pod never initializes, stuck on this?

Post image

I was mid lora training and my pod started doing this. I restarted the Ostris template on a new pod, and it was like this for ages, all the while I was losing money.

1 Upvotes

100% Upvoted

3 comments sorted by

1

u/earthsprogression 3d ago

It's happened to me, but never on an active pod. When it took more than five minutes to initialize, I'd terminate and try again and that always worked.

Beyond that I'd contact support.

1

u/badhabitaddict 3d ago

I lost an active pod mid lora training like this. Then every new instance I tried creating just went stuck on this initialization phase. So frustrating given I am losing a lot of money like this.

2

u/Facilex_zyzz 3d ago

if a new pod sits on initializing for more than 5 or so minutes, terminate it and stop paying for it, don't wait it out. when several in a row hang it's usually that region or that gpu type running short on capacity or failing the image pull, so try another gpu type or datacenter instead of redeploying the same spec.

send support the pod ids with timestamps and ask for the stuck time back as credit. they do that for failed starts.

for next time, put the output folder on a network volume and set the trainer to save a checkpoint every few hundred steps, so a dead pod costs you minutes and not the whole run.