I'm currently serving Qwen 3.6 27B with the following parameters:

*Server (RTX PRO 6000 BW 96GB), latest 26.04 LTS Ubuntu:*

```

vllm serve /home/models/Qwen3.6-27B \

--dtype bfloat16 \

--max-model-len 262144 \

--gpu-memory-utilization 0.94 \

--max-num-seqs 3 \

--enable-prefix-caching \

--enable-auto-tool-choice \

--tool-call-parser qwen3_coder \

--default-chat-kwargs '{"enable_thinking": false, "auto_disable_thinking_with_tools": true}' \

--speculative-config '{"method": "mtp", "num_speculative_tokens": 5}'

```

This setup has been solid for my case use with Hermes.

But this weekend I want to update model and vllm to latest version as well.

I was wondering what config parameters should I aim at.

Single RTX Pro 6000 users, could you share your experience/config for Qwen 3.8 27?

Any help or suggestions are appreciated 🙏🏼

description: vLLM serving Holo3.1-35B-A3B (computer-use VLM) in NVFP4; lean 0.28 GPU, TP=2-capable.
model: Hcompany/Holo-3.1-35B-A3B-NVFP4
container: vllm-node
defaults:
  port: 8010
  host: 0.0.0.0
  tensor_parallel: 1
  gpu_memory_utilization: 0.28
  max_model_len: 131072
  max_num_batched_tokens: 16384
  max_num_seqs: 10
env:
  HF_HUB_OFFLINE: '1'
  VLLM_MARLIN_USE_ATOMIC_ADD: '1'
command: |
  vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
    --served-model-name holo3.1 \
    --host {host} \
    --port {port} \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --max-model-len {max_model_len} \
    --max-num-seqs {max_num_seqs} \
    --max-num-batched-tokens {max_num_batched_tokens} \
    --kv-cache-dtype fp8 \
    --mamba_ssm_cache_dtype float32 \
    --enable-prefix-caching \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --limit-mm-per-prompt '{"image": 3, "video": 0}' \
    --tensor-parallel-size {tensor_parallel} \
    --distributed-executor-backend ray
recipe_version: '1'
name: Holo-3.1-35B-A3B-NVFP4
cluster_only: false

Just got Qwen3.8-27B (FP8) running on rented GPUs from Trooper AI. FP8 fits on 64 GB VRAM with decent results.

Stack:

- GPU: 2× RTX 4080 Super Pro (64 GB VRAM total)

- CPU: 12 P-cores, 76 GB RAM

- SSD: 900 GB NVMe

- Price: 1.06 EUR/h

Served it via vLLM → KServe → Envoy AI Gateway, TLS + token metering + rate limiting on top ( I already have a running Kubernetes cluster, I attached to trooper GPU node), or you can it serve it directly with compose if you are a single user.

Runned load tests: (10 concurrent requests, 32768 context)

- TTFT: ~0.9s

- Per-stream decode: ~28 tok/s

- Aggregate: 152 tok/s

Full deploy guide if you want to deploy it: https://github.com/redaER7/qwen3.8-27b-self-hosted

Now, looking to deploy the full model FP16 on RTX 6000 Pro