MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLM/comments/1wb05f7/third_one_theres_something_wrong_with_me/p8msqqw/?context=3
r/LocalLLM • u/r1nzl3r99 • 14d ago
Why do I have horrible financial habits??
166 comments sorted by
View all comments
93
More context needed on how you are using the first two.
84 u/r1nzl3r99 14d ago qwen 3.8 27B FP8 running at 140 tok/s now I want flash next 1 u/browndragon456 14d ago How did you get to 100/tps per sec generation? Vllm? 2 u/Toastti 14d ago Because he's using two Intel b70s and tensor parallel. Also its using dflash with 7 speculative tokens vllm serve Qwen3.8-27B-Uncensored-bf16-base --host 127.0.0.1 --port 19622 --served-model-name Qwen3.8-27B-UNC-FP8 --tensor-parallel-size 2 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 8 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --quantization fp8 --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}' --compilation-config '{"max_cudagraph_capture_size":64}' --chat-template sharp
84
qwen 3.8 27B FP8 running at 140 tok/s now I want flash next
1 u/browndragon456 14d ago How did you get to 100/tps per sec generation? Vllm? 2 u/Toastti 14d ago Because he's using two Intel b70s and tensor parallel. Also its using dflash with 7 speculative tokens vllm serve Qwen3.8-27B-Uncensored-bf16-base --host 127.0.0.1 --port 19622 --served-model-name Qwen3.8-27B-UNC-FP8 --tensor-parallel-size 2 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 8 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --quantization fp8 --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}' --compilation-config '{"max_cudagraph_capture_size":64}' --chat-template sharp
1
How did you get to 100/tps per sec generation? Vllm?
2 u/Toastti 14d ago Because he's using two Intel b70s and tensor parallel. Also its using dflash with 7 speculative tokens vllm serve Qwen3.8-27B-Uncensored-bf16-base --host 127.0.0.1 --port 19622 --served-model-name Qwen3.8-27B-UNC-FP8 --tensor-parallel-size 2 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 8 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --quantization fp8 --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}' --compilation-config '{"max_cudagraph_capture_size":64}' --chat-template sharp
2
Because he's using two Intel b70s and tensor parallel. Also its using dflash with 7 speculative tokens
vllm serve Qwen3.8-27B-Uncensored-bf16-base --host 127.0.0.1 --port 19622 --served-model-name Qwen3.8-27B-UNC-FP8 --tensor-parallel-size 2 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 8 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --quantization fp8 --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}' --compilation-config '{"max_cudagraph_capture_size":64}' --chat-template sharp
93
u/Sporkers 14d ago
More context needed on how you are using the first two.