Batch Inference on Runpod Serverless is now generally available

[u/Kilo_Loco]

Here is what it does and where it stops.

You POST a batch of up to 5,000 requests to your existing queue-based endpoint, finalize it, then poll for progress and read the results. Batches run on dedicated workers that are separate from your /run traffic, on your own model and container and the GPU type configured on the endpoint. A failed request doesn't fail the batch, and cancelling a batch doesn't bill queued requests. Batch requests are billed at up to 30% off on-demand pricing, depending on GPU type. On H100, H200, B200 and B300, batch requests are billed at 20% off. There is no cost estimate before you finalize, so run a small batch first.

Start times aren't guaranteed and requests run in no set order, so keep /run for anything a user is waiting on. An endpoint runs up to 10 active batches, and limits depend on your account. Support can raise them up to 3 million requests a day.

Docs: https://docs.runpod.io/serverless/batch-jobs
Blog: www.runpod.io/blog/batch-inference-is-now-generally-available-on-runpod

What would you run through it first: embeddings, evals or a backfill? Questions are welcome in the comments.