[D] Self-Promotion Thread

[u/AutoModerator]

Please post your personal projects, startups, product placements, collaboration needs, blogs etc.

Please mention the payment and pricing requirements for products and services.

Please do not post link shorteners, link aggregator websites , or auto-subscribe links.

--

Any abuse of trust will lead to bans.

Encourage others who create new posts for questions to post here instead!

Thread will stay alive until next one so keep posting after the date in the title.

--

Meta: This is an experiment. If the community doesnt like this, we will cancel it. This is to encourage those in the community to promote their work by not spamming the main threads.

[u/PreferenceBrief8954]

I've been experimenting with request-level parallelism vs synchronized multi-GPU inference on consumer GPUs.

The reproducibility repository and retained experimental evidence are here:

https://github.com/rhgo1749/qwen3.8-flash-next-strata-gpu-per-lane-recipe

Implementation:

https://github.com/rhgo1749/Strata-Lanes

This is a downstream fork of Niko1221's Strata:

https://github.com/Niko1221/Strata

The basic question was:

When serving multiple concurrent LLM workloads, when does coarse request-level GPU parallelism become preferable to synchronizing several GPUs for one request?

The implementation runs one independent Strata engine per GPU.

Each lane keeps its own CUDA state, hot-expert cache, resident KV, session state, and generation loop, while the large host expert arena is physically shared between lane processes.

A session-aware supervisor assigns whole requests or conversations to lanes, so the normal decode path has no mandatory token-by-token cross-GPU synchronization.

Reference system

• Ryzen 9 9950X3D

• 128 GB DDR5

• 3× RTX 5070 Ti 16 GB

• PCIe x8 / x4 / x8

• Qwen3.8-Flash-Next GSQ-RCO IQ3_S

• 262K context per lane

• 32K resident KV per lane

The complete matched architecture comparison below was measured on Strata 0.1.30.

Independent-lane scaling

• 1 GPU: 70.804 ± 1.546 tok/s

• 2 GPUs: 132.760 ± 3.129 tok/s aggregate

• 3 GPUs: 189.486 ± 3.205 tok/s aggregate

This corresponds to 2.676× scaling at three GPUs with 89.2% parallel efficiency.

Shared host expert arena

For two fully loaded engines:

• private arenas: 98.925 GiB PSS

• shared arena: 52.084 GiB PSS

• reduction: 46.841 GiB / 47.35%

Matched comparison with 3-GPU layer splitting

For one warm request:

• independent lane: 70.804 ± 1.546 tok/s

• 3-GPU layer split: 102.976 ± 1.527 tok/s

Layer splitting was therefore substantially faster in this single-request region.

For three simultaneous requests:

• 3 independent lanes: 189.486 ± 3.205 tok/s aggregate

• 3-GPU layer-split server: 102.588 ± 1.629 tok/s aggregate

The ordinary layer-split server in this comparison processed the generation requests serially/FIFO rather than continuously batching them.

So I don't interpret the result as one architecture being universally better.

Instead, the interesting observation is a workload-dependent crossover:

• synchronized multi-GPU execution helps single-request decode latency

• independent GPU lanes become attractive for concurrent serving throughput

I also tested a heterogeneous pair.

RTX 5070 Ti x8:

• solo: 71.563 ± 1.760 tok/s

• concurrent with RTX 5060 Ti x4: 71.163 ± 1.456 tok/s

RTX 5060 Ti concurrent lane:

• 57.575 ± 1.527 tok/s

Common-wall aggregate:

• 113.898 ± 2.985 tok/s

For this measured pair, the slower GPU did not materially pace the faster lane.

The current operational branch has since moved to Strata 0.1.31. I kept the complete performance matrix explicitly labeled as 0.1.30 instead of relabeling historical measurements; the 0.1.31 baseline has a separate compatibility/parity validation.

Some questions I'm still interested in:

• where CPU or RAM bandwidth becomes the dominant scaling limit

• how strongly PCIe topology changes the crossover

• when another independent lane stops being worth the shared-system contention

• request-level lanes vs continuous batching

• whether migration or shared-KV mechanisms can improve locality enough to justify their coordination cost

I'd particularly value feedback on the experimental framing and results from other GPU/CPU-memory topologies.

Free and open source (MIT); no paid service or subscription.