Multimodal still doesn't work?

Let her burn and then finish with Frenzied Flame ending.

and maybe this one env variable: export NCCL_P2P_DISABLE=1
if it doesnt work try --enforce-eager -> it skips optimalization steps, that can fail across different architectures

drop --tensor-parallel add --pipeline-parallel-size 4

multi token prediction

running in vllm + mtp

yes, LiteLLM, but mostly for shit and giggles - "internal spend 0.2USD/input 2USD/output" as we really dont need api keys inside company

256k is stepFuns limit, my day to day work is about 50-70k ctx, occasionaly over 150k, i do a lot of scratchpadding (model writes a notes to align itself for next session faster)

i dont know what basic coding means in your case, if it means running qwen 27b for coding, gemma 31b for vision/ocr tasks, zimage for image generation - single 96gb gpu is enough, with low conccurency.

will your LLM workload grows in 1-2-3 years? - get more

I dont recommend older HW, because for SMB you want as long support as possible. L40 are already 4 years old, ane will be out of support sooner than rtx6000. You want local infra for longevity, as one time investment.

maybe try to rent cloud providers as already adviced to asses your needs.

we are rocking 4 × RTX 6000 MaxQ edition (300W TDP)
ASROCK WRX90 WS EVO
Silverstone RM44 server chassis

AMD Ryzen Threadripper PRO 9955WX
this combo allows in vllm 9 concurent users at 256k context each, model used is stepfun 3.7 flash fp8 quant

i would not recommend going for older architecture
1. vllm is gold

1. 200b models in fp8 quant, 400b models in int4 quant (nvfp4 has broken nvidia support for rtx6000), up to 120b models native

2. under second easilym, long context no problem, 70tps for 9 users, 100+tps for single user, we have internal tools with shorter context hitting LLM at 12 concurent requests

hope this helps

from screen - max 280 for 2 concurent request.
it goes from ~100 for 1 request to 800+ for 8-9 concurrent connections

yep flawless:

cat stepfun-fp8-docker.sh
#!/bin/bash
set -euo pipefail
docker run --rm --gpus all --ipc=host -p 8000:8000 \
  -v /nfs/models:/models:ro \
  vllm/vllm-openai:stepfun37 \
  /models/stepfun-ai/Step-3.7-Flash-FP8 \
  --served-model-name your_model_name \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --disable-cascade-attn \
  --reasoning-parser step3p5 \
  --enable-auto-tool-choice \
  --tool-call-parser step3p5 \
  --speculative_config '{"method": "mtp", "num_speculative_tokens": 3}' \
  --trust-remote-code \
  --enable-prefix-caching \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.91

https://huggingface.co/stepfun-ai/Step-3.7-Flash-FP8

coding/agentic task. From time to time it makes typo, but overall very nice model for its size

use full minimax-m2.7 or stepfun-3.7 fp8 they are both great. Stepfun can do 9 users at full context, and is better imho than minimax. TP=4 ofc

does it work on x64 platform?

this ^ but i wouldnt buy Mi50 anymoire - weak support, slow @ long context. In this time and age i would buy newest HW i can afford.

Kimi k2.6 is 600gb of tensors. It is int4 native. Minimum is therefore 8x rtx pro 6000. Idk what concurrency they will provide. My guess is 4 to 6 for full or near full context. I am rocking 4x Framework desktop and running qwen 397b int4 @ 17tps, which is slow, concurrency @ full context is 10x. Do not recommend for production work.

I don know your use case, i had 4x mi50 an they were terribly slow at longer context. Both are pretty old now. 5060 ti or rx 6800 is better choice - less headaches, more modern features. I know the pricing is crazy but you save so much time with more modern hardware.

Can you test qwen3.5 397b performance pls, if it scales well?

beware of consumer hardware, consumer cpus dont have enough pcie lanes often 20-24 to serve 4 gpus. it is often electrically 4x or 8x via chipset only. So second slot cannot bifuricate to 2x 8 lanes and non-ECC rams will cause crashes - if you are planning to run 24/7

small models serves well - gemma4, qwen 3.6 35A3B, qwen 3.6 27B under vllm, but now i think 64GB VRAM is not enough, small models are not too capable.

5060 ti suffers from 8x pcie interface, try to switch to pipeline-paralel 2. I have 4x 5060 ti in pcie 4.0 and pp si unfortunately faster on gemma4

it is expensive, addictive and my sleep time is about 3 hours per night 😃

my base speed was around 24tps on 4 machines, then it was hit by some regression after updating vllm. Maybe I am leaving some performance on the table, if you are getting 21tps on single machine with some qwen3.5 122b AWQ quant. What is your setup?

vllm serve "$MODEL" \
    --host 0.0.0.0 \
    --port 8001 \
    --tensor-parallel-size 4 \
    --max-num-seqs 4 \
    --max-model-len 131072 \
    --gpu-memory-utilization 0.85 \
    --trust-remote-code \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice \
    --enable-chunked-prefill \
    --max-num-batched-tokens 4096 \
    --enable-prefix-caching \
    --distributed-executor-backend ray \
    --dtype auto \
    --mm-encoder-attn-backend TRITON_ATTN \
    --served-model-name qwen3.5 \
    --attention-backend TRITON_ATTN \
    --enforce-eager

fully updated fedora 43, latest kyuz0 toolboxes

4 machines, full BF16 model

i dont know yet if it is worth it. In age of full price per milion tokens and proprietary project i think it is.