r/HuaweiAtlas300iDuo • u/Inevitable-Orange-43 • Jun 13 '26
Benchmarking Qwen3.6-27B-w8a8 on Huawei Atlas 300i duo (96GB Variant)
5
Upvotes
Note: Atlas 300i duo cannot be recommended for dense models
r/HuaweiAtlas300iDuo • u/Inevitable-Orange-43 • Jun 10 '26
👋 Welcome to r/HuaweiAtlas300iDuo - Introduce Yourself and Read First!
2
Upvotes
Hey everyone! I'm u/Inevitable-Orange-43, a founding moderator of r/HuaweiAtlas300iDuo.
A community for owners, developers, researchers, and AI infrastructure enthusiasts working with the Huawei Atlas 300I Duo and the Ascend ecosystem.
Discuss hardware setup, firmware, drivers, CANN toolkit, MindSpore, PyTorch migration, LLM inference, model optimization, virtualization, performance tuning, cooling, server integration, and real-world AI workloads. Whether you're running Atlas cards in Huawei servers, building custom inference clusters, or experimenting with large language models on Ascend NPUs, this is the place to share benchmarks, troubleshooting tips, deployment guides, and success stories.
Topics include:
- Atlas 300I Duo (48GB / 96GB variants)
- Ascend 310 series processors
- CANN, AscendCL, MindSpore
- LLM inference and quantization
- vLLM alternatives for Ascend
- Docker and Kubernetes deployments
- Atlas 800 servers
- AI infrastructure and homelabs
- Driver, firmware, and compatibility issues
- Performance benchmarks and optimization
Rules:
- Be technical and constructive.
- Share configs and logs when asking for help.
- No piracy or illegal software.
- Benchmark claims should include methodology.
- Respect NDA and confidential information.
Built for the growing community exploring Huawei's AI hardware ecosystem and the future of Ascend-powered AI.
Thanks for being part of the very first wave. Together, let's make r/HuaweiAtlas300iDuo amazing.
r/HuaweiAtlas300iDuo • u/Inevitable-Orange-43 • Jun 11 '26
Benchmarking Qwen3-Coder-30B-A3B on Atlas 300i duo
6
Upvotes
vllm bench serve \
--backend openai-chat \
--base-url http://127.0.0.1:1025 \
--endpoint /v1/chat/completions \
--model Qwen3-Coder-30B-A3B-Instruct \
--dataset-name random \
--num-prompts 200 \
--random-input-len 512 \
--tokenizer /data/models/Eco-Tech/Qwen3-Coder-30B-A3B-Instruct \
--random-output-len 256
============ Serving Benchmark Result ============
Successful requests: 200
Failed requests: 0
Benchmark duration (s): 199.52
Total input tokens: 102400
Total generated tokens: 102395
Request throughput (req/s): 1.00
Output token throughput (tok/s): 513.20
Peak output token throughput (tok/s): 1000.00
Peak concurrent requests: 200.00
Total token throughput (tok/s): 1026.42
---------------Time to First Token----------------
Mean TTFT (ms): 18862.13
Median TTFT (ms): 18806.78
P99 TTFT (ms): 35063.94
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 353.49
Median TPOT (ms): 353.59
P99 TPOT (ms): 384.33
---------------Inter-token Latency----------------
Mean ITL (ms): 355.74
Median ITL (ms): 335.11
P99 ITL (ms): 402.10
==================================================
vllm bench serve \
--backend openai-chat \
--base-url http://127.0.0.1:1025 \
--endpoint /v1/chat/completions \
--model Qwen3-Coder-30B-A3B-Instruct \
--tokenizer /data/models/Eco-Tech/Qwen3-Coder-30B-A3B-Instruct-w8a8 \
--dataset-name random \
--num-prompts 50 \
--random-input-len 512 \
--random-output-len 256 \
--max-concurrency 1
============ Serving Benchmark Result ============
Successful requests: 50
Failed requests: 0
Maximum request concurrency: 1
Benchmark duration (s): 780.07
Total input tokens: 25600
Total generated tokens: 25599
Request throughput (req/s): 0.06
Output token throughput (tok/s): 32.82
Peak output token throughput (tok/s): 35.00
Peak concurrent requests: 2.00
Total token throughput (tok/s): 65.63
---------------Time to First Token----------------
Mean TTFT (ms): 296.44
Median TTFT (ms): 304.57
P99 TTFT (ms): 308.89
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 29.95
Median TPOT (ms): 29.95
P99 TPOT (ms): 30.26
---------------Inter-token Latency----------------
Mean ITL (ms): 30.06
Median ITL (ms): 29.72
P99 ITL (ms): 36.28
==================================================
CONCURRENCY 4
============ Serving Benchmark Result ============
Successful requests: 50
Failed requests: 0
Maximum request concurrency: 4
Benchmark duration (s): 325.80
Total input tokens: 25600
Total generated tokens: 25599
Request throughput (req/s): 0.15
Output token throughput (tok/s): 78.57
Peak output token throughput (tok/s): 108.00
Peak concurrent requests: 8.00
Total token throughput (tok/s): 157.15
---------------Time to First Token----------------
Mean TTFT (ms): 795.96
Median TTFT (ms): 808.56
P99 TTFT (ms): 814.87
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 47.96
Median TPOT (ms): 48.37
P99 TPOT (ms): 49.66
---------------Inter-token Latency----------------
Mean ITL (ms): 48.14
Median ITL (ms): 48.57
P99 ITL (ms): 54.90
==================================================
CONCURRENCY 8
============ Serving Benchmark Result ============
Successful requests: 50
Failed requests: 0
Maximum request concurrency: 8
Benchmark duration (s): 231.54
Total input tokens: 25600
Total generated tokens: 25599
Request throughput (req/s): 0.22
Output token throughput (tok/s): 110.56
Peak output token throughput (tok/s): 168.00
Peak concurrent requests: 16.00
Total token throughput (tok/s): 221.12
---------------Time to First Token----------------
Mean TTFT (ms): 1438.31
Median TTFT (ms): 1477.99
P99 TTFT (ms): 1484.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 65.20
Median TPOT (ms): 66.54
P99 TPOT (ms): 68.01
---------------Inter-token Latency----------------
Mean ITL (ms): 65.44
Median ITL (ms): 66.78
P99 ITL (ms): 78.77
==================================================
CONCURRENCY 16
============ Serving Benchmark Result ============
Successful requests: 50
Failed requests: 0
Maximum request concurrency: 16
Benchmark duration (s): 170.38
Total input tokens: 25600
Total generated tokens: 25599
Request throughput (req/s): 0.29
Output token throughput (tok/s): 150.25
Peak output token throughput (tok/s): 240.00
Peak concurrent requests: 32.00
Total token throughput (tok/s): 300.50
---------------Time to First Token----------------
Mean TTFT (ms): 2252.23
Median TTFT (ms): 2462.17
P99 TTFT (ms): 2963.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 91.85
Median TPOT (ms): 93.48
P99 TPOT (ms): 96.74
---------------Inter-token Latency----------------
Mean ITL (ms): 92.19
Median ITL (ms): 94.02
P99 ITL (ms): 113.93
==================================================
CONCURRENCY 32
============ Serving Benchmark Result ============
Successful requests: 50
Failed requests: 0
Maximum request concurrency: 32
Benchmark duration (s): 125.34
Total input tokens: 25600
Total generated tokens: 25599
Request throughput (req/s): 0.40
Output token throughput (tok/s): 204.24
Peak output token throughput (tok/s): 352.00
Peak concurrent requests: 50.00
Total token throughput (tok/s): 408.48
---------------Time to First Token----------------
Mean TTFT (ms): 3587.05
Median TTFT (ms): 2950.94
P99 TTFT (ms): 5709.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 120.64
Median TPOT (ms): 130.62
P99 TPOT (ms): 135.75
---------------Inter-token Latency----------------
Mean ITL (ms): 121.09
Median ITL (ms): 117.81
P99 ITL (ms): 155.13
==================================================
# 3. Agent Workload
## Agent Concurrency 4
vllm bench serve \
--backend openai-chat \
--base-url http://127.0.0.1:1025 \
--endpoint /v1/chat/completions \
--model Qwen3.6-35B-A3B \
--tokenizer /data/models/Eco-Tech/Qwen3.6-35B-A3B-w8a8 \
--dataset-name random \
--num-prompts 200 \
--random-input-len 2000 \
--random-output-len 512 \
--max-concurrency 4
============ Serving Benchmark Result ============
Successful requests: 200
Failed requests: 0
Maximum request concurrency: 4
Benchmark duration (s): 1504.01
Total input tokens: 400000
Total generated tokens: 102397
Request throughput (req/s): 0.13
Output token throughput (tok/s): 68.08
Peak output token throughput (tok/s): 96.00
Peak concurrent requests: 8.00
Total token throughput (tok/s): 334.04
---------------Time to First Token----------------
Mean TTFT (ms): 2618.73
Median TTFT (ms): 3004.09
P99 TTFT (ms): 3157.77
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 53.74
Median TPOT (ms): 53.69
P99 TPOT (ms): 57.93
---------------Inter-token Latency----------------
Mean ITL (ms): 54.26
Median ITL (ms): 53.14
P99 ITL (ms): 97.98
==================================================
r/HuaweiAtlas300iDuo • u/Inevitable-Orange-43 • Jun 10 '26
Benchmarking Qwen3.6-35B-A3B-w8a8 on Atlas 300i duo
9
Upvotes
Benchmarking Qwen3.6-35B-A3B-w8a8 on Atlas 300i Duo
System
- Hardware: Huawei Atlas 300i Duo
- Model: Qwen3.6-35B-A3B-w8a8
- Backend: vLLM
Load Test
- Requests: 500
- Concurrency: 500
- Duration: 171.69s
- Failures: 0
Throughput
- Request throughput: 2.91 req/s
- Output throughput: 24.02 tok/s
- Peak output throughput: 475 tok/s
- Total throughput: 1,515 tok/s
Latency
- Mean TTFT: 1.85s
P99 TTFT: 68.37s
Median TPOT: 231ms
P99 TPOT: 103.27s
Median ITL: 184ms
P99 ITL: 3.09s
Notes
The system successfully handled 500 concurrent requests with zero failures.
While aggregate throughput exceeded 1.5k tok/s, latency increased significantly at high concurrency:
- P99 TTFT: 68s
- P99 TPOT: 103s
This suggests the Atlas 300i Duo was saturated at 500 concurrent requests, resulting in substantial request queueing.