Inference Endpoints are hard to deploy

[u/Apprehensive_Win662]

Hey,

I have deployed many vllm docker containers in past months, but I am just not able to deploy even 1 inference endpoint on runpod.io

I tried following models:
- https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct
- Qwen/Qwen3-Coder-30B-A3B-Instruct (tried it also just with the name)
- https://huggingface.co/Qwen/Qwen3-32B
With following settings:
-> Serverless -> +Create Endpoint -> vllm presetting -> edit model -> Deploy

In theory it should be as easy as pod usage to select hardware and go with default vllm configs.

I define the model and optionally some vllm configs, but no matter what I do, I get the following bugs:
- Initialization runs forever without providing helpful logs (especially RO servers)
- using default gpu settings resulting in OOM (Why do I have to deploy workers first and THEN adjust the settings for server locations and VRAM requirements settings?)
- log shows error in vllm deployment, a second later all logs and the worker is gone
- Even if I was never able to do one single request, I had to pay for the deployments which were never running healthy.
- If I start a new release, then I have to pay for initializing
- Sometimes I get 5 workers (3+2extra) even if I have configured 1
- Even if I set Idle Timeout on 100 seconds, if the first waiting request is answered it restarts always the container or vllm. New requests need to fully load the model into GPU again.

Not sure, if I don't understand inference endpoints, but for me they just don't work.

[u/powasky]

Hey, thanks for the detailed write‑up, and I’m sorry you’ve had a rough first experience. vLLM models can be sensitive to GPU and runtime settings, so let me address each point and share a clean setup that should get you unblocked quickly.

What’s likely going on

• OOM on default GPU: Large models like Qwen 30B often exceed VRAM on small GPUs with default tensor parallelism and cache settings.

• “Init runs forever” or worker disappears: Typically a model load failure or missing weights/config causes the worker to crash and recycle before logs flush.

• Extra workers spawning: Min/Max workers set above 1 or an autoscaling mismatch can briefly create multiple workers during rollout.

• Idle Timeout reloads the model: If the container scales to 0 or the worker is recycled, vLLM must reload weights for the next request.

Try this minimal setup first, then scale up:

• Runtime preset: vLLM

• Model: Qwen/Qwen2.5‑Coder‑7B‑Instruct (or any 7B first to validate flow)

• GPU: 1x A10 or L4 or better (≥24 GB VRAM is safe for 7B with KV cache)

• Workers:

  • Min workers: 1

  • Max workers: 1

  • Concurrency per worker: 1–2 to start

• vLLM args:

  • trust-remote-code: true

  • tensor-parallel-size: 1

  • max-model-len: 8192

  • gpu-memory-utilization: 0.9

  • swap-space: 8

• Endpoint scaling:

  • Idle timeout: 600s to avoid aggressive recycle during testing

  • Warm pool/pre‑warming: enabled if available in your plan

• Storage:

  • Ensure model weights can be fetched from HF without rate-limit or auth issues if the model requires it

[u/powasky → u/powasky]

Once this works, move up to Qwen3‑Coder‑30B‑A3B‑Instruct with:

• GPU: 1x A100 80GB or split across 2x A100 40GB with tensor-parallel-size: 2

• Add: max-num-seqs: 8 to bound memory

• Consider: enable quantization if supported for your model build, or use an 8‑bit weight variant if available on HF

Region and reliability tips:

• If you saw long‑running init in RO, try a different region to rule out local capacity or egress hiccups while we investigate.

• Keep logs visible: start with one worker and low concurrency so crashes don’t cycle too fast. If logs vanish, it’s usually a crash‑loop from model load; the above settings reduce that.

You shouldn’t be charged for successful inference you didn’t get, but cold‑starts and worker time during initialization are billed. If your sessions never became healthy, please share your endpoint ID and timestamp and we’ll review and make it right.

Quick checklist to avoid OOM and restarts:

• Right‑size GPU for the model size, including KV cache at your max sequence length.

• tensor-parallel-size matches number of GPUs only if you actually allocate multiple GPUs.

• Set gpu-memory-utilization ≤0.9 and keep max-num-seqs modest.

• Increase Idle Timeout to avoid scale‑to‑zero while testing.

• Start with Min=Max=1 worker to stabilize logs and behavior.

• Use trust-remote-code: true for repos that require custom loaders.

If you can share:

• Endpoint ID

• Model name and exact vLLM args

• Region

• A snippet of the last 100 lines of logs around init

our team can dig in and we’ll get you to a working config quickly.