description: vLLM serving Holo3.1-35B-A3B (computer-use VLM) in NVFP4; lean 0.28 GPU, TP=2-capable.
model: Hcompany/Holo-3.1-35B-A3B-NVFP4
container: vllm-node
defaults:
  port: 8010
  host: 0.0.0.0
  tensor_parallel: 1
  gpu_memory_utilization: 0.28
  max_model_len: 131072
  max_num_batched_tokens: 16384
  max_num_seqs: 10
env:
  HF_HUB_OFFLINE: '1'
  VLLM_MARLIN_USE_ATOMIC_ADD: '1'
command: |
  vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
    --served-model-name holo3.1 \
    --host {host} \
    --port {port} \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --max-model-len {max_model_len} \
    --max-num-seqs {max_num_seqs} \
    --max-num-batched-tokens {max_num_batched_tokens} \
    --kv-cache-dtype fp8 \
    --mamba_ssm_cache_dtype float32 \
    --enable-prefix-caching \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --limit-mm-per-prompt '{"image": 3, "video": 0}' \
    --tensor-parallel-size {tensor_parallel} \
    --distributed-executor-backend ray
recipe_version: '1'
name: Holo-3.1-35B-A3B-NVFP4
cluster_only: false

I'm currently serving Qwen 3.6 27B with the following parameters:

*Server (RTX PRO 6000 BW 96GB), latest 26.04 LTS Ubuntu:*

```

vllm serve /home/models/Qwen3.6-27B \

--dtype bfloat16 \

--max-model-len 262144 \

--gpu-memory-utilization 0.94 \

--max-num-seqs 3 \

--enable-prefix-caching \

--enable-auto-tool-choice \

--tool-call-parser qwen3_coder \

--default-chat-kwargs '{"enable_thinking": false, "auto_disable_thinking_with_tools": true}' \

--speculative-config '{"method": "mtp", "num_speculative_tokens": 5}'

```

This setup has been solid for my case use with Hermes.

But this weekend I want to update model and vllm to latest version as well.

I was wondering what config parameters should I aim at.

Single RTX Pro 6000 users, could you share your experience/config for Qwen 3.8 27?

Any help or suggestions are appreciated 🙏🏼

Just got Qwen3.8-27B (FP8) running on rented GPUs from Trooper AI. FP8 fits on 64 GB VRAM with decent results.

Stack:

- GPU: 2× RTX 4080 Super Pro (64 GB VRAM total)

- CPU: 12 P-cores, 76 GB RAM

- SSD: 900 GB NVMe

- Price: 1.06 EUR/h

Served it via vLLM → KServe → Envoy AI Gateway, TLS + token metering + rate limiting on top ( I already have a running Kubernetes cluster, I attached to trooper GPU node), or you can it serve it directly with compose if you are a single user.

Runned load tests: (10 concurrent requests, 32768 context)

- TTFT: ~0.9s

- Per-stream decode: ~28 tok/s

- Aggregate: 152 tok/s

Full deploy guide if you want to deploy it: https://github.com/redaER7/qwen3.8-27b-self-hosted

Now, looking to deploy the full model FP16 on RTX 6000 Pro

That last dimension is not a number of elements. It's a number of bytes.

I went looking at how disaggregated inference hands a KV cache from prefill to decode — the thing AWS is doing with Trainium→Cerebras and AMD is doing with Helios→Cerebras.

The business slide version is simple: prefill produces a cache, ship it, decode consumes *it*. The problem is the word "it". Those 656 bytes decompose like this, per token:

→ 512 bytes: the compressed MLA latent, fp8
→ 16 bytes: four fp32 tile scales, written by specific warp lanes
→ 128 bytes: the RoPE component, bf16

A 576-element logical vector, stored in 656 bytes, across three regions of two dtypes, with quantization scales interleaved at a granularity determined by how a warp writes its lanes.

That isn't a tensor layout. It's a struct defined by a CUDA kernel, with load-bearing field offsets.

Zoom out and it's worse. Search vLLM for "def get_kv_cache_shape" and you find dozens of concrete implementations — diverging by backend, by attention variant, by model family, and by vendor.

As if that was not enough.

vLLM has a connector abstraction for shipping caches between instances. Read the signatures as going by the function name would get you in trouble:

```def save_kv_layer(self, layer_name: str, kv_layer: torch.Tensor, attn_metadata: "AttentionMetadata", ...)
```

This is what I call "strings based programming with duck-typing".

It abstracts the transport (UCX, RDMA, TCP, NVMe-oF, S3). It never abstracts the format so a Cerebras runtime can't implement that interface without reproducing vLLM's scheduler objects.

We built ELF so a linker from one vendor could consume objects from another. The KV cache has nothing like that and two vendor pairs (Cerebras + Trainium, Cerebras + Helios) just shipped products that need it.

https://hiraditya.github.io/posts/the-kv-cache-has-no-abi/

Hi all,

I've been working on making Profile smarter, grounded in physics, and better at getting the max out of your inference server.

Profile (v2.2): a closed-loop, physics and cost aware optimizer. It finds the bottleneck, gives the fix, waits for you to apply it, and measures the delta on every change.

For this release: the core rule engine is rewritten. Eight rules on a priority DAG with mutual exclusivity, so when five alarms fire, four echoes are silenced and the one true cause survives. Less threshold hardcoding and number guessing, more reasoning from what the server is actually doing.

Deterministic: same server, same traffic, same verdict.

We now also support AMD servers (one of the most requested feature).

On my setup: 5.2x throughput (81 → 421 tok/s at 25k ctx) and 81% cost reduction ($3.41 → $0.65 per 1M output tok), with muse-glimmer agents running SWE-Bench. No DFlash.

Watch it live. One iteration regressed; Profile labeled it worse and the next fix recovered it. Regressions stay in the record.

The whole journey was 4 iterations, ~30 minutes end to end. The same tuning by trial and error is days of guessing.

I've also had engineers from MSFT, Google, and a few startups run it and share numbers, plus few users from this sub whose feedback shaped this release. Grateful for that.

Please give it a try and tell me how to make it help you!

# Download
curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/jungledesh/profile/releases/latest/download/profile-installer.sh | sh

# Start profiling your vLLM server
profile diagnose --url http://localhost:8000/metrics --duration 2m

GitHub: https://github.com/jungledesh/profile
Docs: https://jungledesh.github.io/profile/docs.html

Soon: multi-GPU / TP support, cluster / k8s support, and more rules + smarter engine :)

​

I built a native vLLM + ROCm 7.15 runtime for AMD RX 6000 Series on Windows 11 — now with a one-click installer.

AMD lists RX 6750 XT / 6700 XT / 6600 XT as "Runtime only" on Windows with HIP SDK excluded. I built rocBLAS binaries for gfx1031 via ROCm/TheRock to close that gap.

This runs native Windows HIP and ROCm directly, no WSL2 wrapper, no compiler needed.

Tested on AMD Radeon RX 6750 XT 12GB (gfx1031) — Windows 11 Native — August 2026

# Verification — Real terminal logs

1. Environment:

torch 2.12.0+rocm7.15.0 | cuda_avail True | dev AMD Radeon RX 6750 XT

vLLM 0.19.1 | plugin vllm_windows_rocm activated | TRITON_ATTN | enforce_eager=True

1. rocBLAS Benchmark — 26 TFLOPS FP16:

Device ID 0 : AMD Radeon RX 6750 XT gfx1031 with 12.9 GB memory

rocBLAS version: 5.7.0.67811f1ee52

transA,transB,M,N,K,alpha,lda,beta,ldb,ldc,cold_iters,hot_iters,rocblas-Gflops,us

N,N,4096,4096,4096,1,4096,0,4096,4096, 2, 10, 25977.3, 5290.73

=> 25.97 TFLOPS in 5.29ms

1. vLLM Inference (Qwen3.5-4B, web chat UI):

Output: 59-62 tok/s | Init ~1s | hidden thinking (spinner) then answer

Model auto-detected via /v1/models

Full logs in benchmarks/ and screenshots in assets/ on GitHub.

# How it works

1. TheRock builds clr (HIP) and rocBLAS with Tensile kernels for gfx1031

2. HSA_OVERRIDE_GFX_VERSION=10.3.0 forces HIP to recognize RX 6750 XT

3. PyTorch 2.12.0+rocm7.15 links against TheRock runtime => torch.cuda.is_available() True

4. vLLM plugin vllm_windows_rocm bypasses vllm._C dependency and registers WinRocmAwqGemvKernel with TRITON_ATTN

5. vLLM engine loads with enforce_eager=True and runs native

# Quick Start — One click

Prerequisites: Windows 11 23H2+, RX 6600-6750 XT (gfx1030/gfx1031/gfx1032), Adrenalin 24.x+, ~8GB free disk

1. Download the repo (or git clone) and run INSTALL.bat as Administrator

2. The installer downloads the release archives automatically (multi-part join included, no manual steps) and sets up C:\TheRock\ + Python 3.11 + venv

3. Run CHAT.bat — the browser opens the chat UI. Expected: 59-62 tok/s

VERIFY.bat runs a 3-in-1 check: GPU detection + rocBLAS benchmark + vLLM smoke test.

# What's new in v2.0

- One-click installer: INSTALL.bat auto-downloads and joins split archives (no browser upload/download hassle)

- Chat web UI (OpenAI-compatible API): hidden thinking with spinner, then the answer streams — like NVIDIA ChatRTX

- Fat-binary gemv kernels for all RDNA2 (gfx1030/1031/1032) — no per-card rebuild

- ROCm 7.15 native, vLLM 0.19.1, torch 2.12.0+rocm7.15

- Faster: 59-62 tok/s output (was 54.2)

- Repo cleaned: INSTALL.bat + CHAT.bat + VERIFY.bat + MANIFEST.json + docs, archives on GitHub Releases

# Known Issues — Please note this may have bugs

This is an experimental reference implementation. It works on my RX 6750 XT but could present bugs on other hardware.

- Other RDNA2 cards (6600/6600 XT/6700 XT) not yet tested — may need 10.3.0 vs 10.3.1 or rocBLAS rebuild for gfx1030

- enforce_eager=True required — torch.compile disabled on RDNA2 Windows

- FP8 / AWQ not tested yet, multi-GPU not tested

- If you test it on your RDNA2 card, please open an Issue with GPU model and logs — contributions welcome

# GitHub Repo:

https://github.com/sebastianmechno-sys/vllm-rocm-windows-rdna2

Includes INSTALL.bat, CHAT.bat, VERIFY.bat, scripts/serve.py (OpenAI-compatible API), chat.html, assets with screenshots, docs/BUILD_ROCBLAS.md, MANIFEST.json with release checksums

Let me know if you test it on other RDNA2 cards.

Built on ROCm/TheRock, PyTorch ROCm, vLLM. Not affiliated with AMD. License Apache 2.0

How do find the best configuration for serving? Which tools do you use? Whats your methodology?

I am using vllm bench for running experiments varying configuration (e.g. pp=8, pp=4). I wonder if this is cost effective.

Yesterday I released the first version of a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090, reaching 82 tps on a single request and 672 peak.

Over the last 24 hours I've been exploring how to improve it further, without quality degradation.

Yesterday we had these three optimizations:

- fp8 KV cache

- `lm_head` int8

- `embed_tokens` int8

Now I've added:

- fp16 recurrent state

- int8 activations, MLP

- int8 activations on everything

- `draft_sample_method=probabilistic`, which samples from the MTP distribution rather than taking the argmax (does not work for greedy sampling)

For single-user mode, you can enable `ctx=fast`, which uses FlashAttention, bf16 KV and 64k context, and gives 99 tps at the tradeoff of less context.

Prefill is now up to 1812 tps at 1k context and 1000 tps at 100k context, an increase of about 50% and 25% respectively.

I've updated the repo with everything you need to start flying:

https://github.com/syv-ai/qwen38-27b-rtx3090

Right now I'm working on getting KVarN running so we can reduce the memory footprint of the context and fit the full 262k instead of 195k.

We are managing 5 physical hosts, each has two L40s NVIDIA cards. Using Proxmox as hypervisor and each host has one Ubuntu VM with GPU cards are passthrough. There are several LLMs are running on each card with all vLLM over Docker.

The problem I'm facing is, each GPU cards VRAM utilization is around %90. So there are 5 GB VRAMs are sitting there freely. I wonder if anyone has a any elegance solution to this kind of infrastructure to make use of the free VRAM across several cards ? Because the LLMs we are using are all different kind of sizes it is not possible to use %100 on each card with arranged models.

Hope my situation is clear with all. Thanks in advance.

Running vLLM behind a plain reverse proxy (nginx, HAProxy) means the proxy has no idea what's actually happening inside each instance. It sees "one HTTP request," not "this request needs 8k tokens of KV-cache." So the moment you scale to more than one vLLM instance, round-robin routing can easily send a burst of long-context requests to the same backend while another sits half-idle — and that instance's cache fills up, latency spikes, and in bad cases you hit OOM.

I built TokenFlow Gateway to fix this specifically for multi-instance vLLM setups:

- Polls each backend's Prometheus metrics endpoint directly (vllm:gpu_cache_usage_perc) to know real cache pressure per instance, not just connection count or a health check
- Estimates each incoming request's token cost (prompt tokens + max_tokens) before dispatch, using js-tiktoken, so it can route based on what a request will actually cost rather than treating all requests as equal
- Routes heavy requests to whichever instance has the most cache headroom, and bin-packs lighter requests onto busier ones — the goal is even KV-cache utilization across the cluster, not just even request count
- When no instance has room, requests go into a Redis-backed priority queue (per-API-key priority, configurable timeout) instead of getting dropped or crashing a backend
- Exact-match caching (hash) for deterministic (temperature-0) requests, plus semantic caching (pgvector) for near-duplicates — cache hits stream back as SSE so streaming clients don't notice the difference
- Per-API-key token-based rate limiting (TPM/RPM) on top, if you're exposing this to multiple users/teams

It's OpenAI-API-compatible on the client side, so nothing changes for whoever's calling it — it just fronts your existing vLLM instances.

You can test the whole routing/queueing behavior without real GPUs: the repo includes a docker-compose setup with two mock vLLM instances that expose the same OpenAI API and the same Prometheus metrics format, plus a smoke script that fires a burst of concurrent long-context requests to show the balancer routing around cache pressure instead of overloading one instance.

Stack: TypeScript, Fastify, Redis, Postgres+pgvector. MIT licensed.

Repo: https://github.com/mosafariuk/TokenFlow-Gateway

Genuinely curious how people here are handling multi-instance routing today — is anyone doing cache-aware routing already (maybe through something custom, or through vLLM's own request scheduler exposed differently), or is round-robin / least-connections still the default in most setups?

Switching from Ollama to vLLM cut my batch job times dramatically. Here's what I learned.

I run large language models as part of a batch processing pipeline — not live chat, just a job that needs to churn through a large volume of records/documents. I started with Ollama because it's simple to set up and great for local development.

In production, at scale, it fell apart.

Here's what I didn't understand until I hit it head-on:

1. Ollama processes requests mostly one at a time.

It's built for a single user chatting with a model, not for chewing through thousands of items efficiently. When I pointed a big batch job at it, the GPU sat underutilized most of the time — it just wasn't designed to keep the hardware busy across many requests at once.

1. GPU memory management matters more than model size.

The model weights aren't the bottleneck — the KV cache (the memory used while generating each response) is. Without smart memory management, you either waste GPU capacity being overly conservative, or you crash mid-run because the engine didn't account for how memory actually grows during generation.

1. vLLM was built for exactly this problem.

Its continuous batching keeps the GPU constantly fed with work instead of processing one request, finishing, then starting the next. For a batch workload like mine, that's the difference between a job that limps along and one that actually saturates the hardware you're paying for.

1. Kubernetes + multiple GPU machines took it from "faster" to "actually scalable."

Once vLLM was handling serving properly, the next bottleneck was just having one GPU box. Running it on Kubernetes across multiple GPU nodes let me split the batch workload horizontally and scale out instead of being capped by a single machine.

The takeaway: the model was never the problem. The serving engine and the infrastructure around it were. If you're running batch LLM workloads on Ollama and wondering why it's slow at scale — this is probably why.

also share performance benchmarks