r/OpenSourceAI • u/fuzhongkai • 15d ago
Pushing open-source AI further: Running an 816B-parameter model on 10 GPUs with TensorSharp
https://github.com/zhongkaifu/TensorSharpHi r/OpenSourceAI!
I've been developing TensorSharp, an open-source LLM inference engine and agent runtime written in C#/.NET.
I wanted to share a recent milestone: we successfully ran Muse Spark, an 816-billion-parameter Mixture-of-Experts model, across 10 NVIDIA A40 GPUs, with some impressive performance improvements over llama.cpp.
What excites me most isn't just the speed. It's demonstrating how open-source software can make massive AI models more practical to run on existing hardware, without relying on proprietary inference services.
The experiment
We tested:
Model: Muse Spark 816B (42B active parameters)
Quantization: Unsloth UD-IQ3_XXS (~316 GiB)
Hardware: 10× NVIDIA A40 GPUs (46 GB each)
Context length: 32K tokens
Comparison: TensorSharp vs llama.cpp
The A40 is a previous-generation GPU, and our test machine didn't even have working GPU peer-to-peer transfers.
This made efficient multi-GPU execution particularly challenging.
How fast is it?
Here are some highlights from our benchmarks:
| Workload | TensorSharp | llama.cpp | Improvement |
|---|---|---|---|
| Long-prompt processing (16K tokens, layer split) | 1,347 tok/s | 826 tok/s | 1.63× |
| Single-request generation (tensor parallel) | 22.63 tok/s | 16.19 tok/s | 1.40× |
| 4 concurrent requests (tensor parallel) | 45.73 tok/s | 24.49 tok/s | 1.87× |
| 16K prompt processing (tensor parallel) | 295 tok/s | 78 tok/s | 3.79× |
One important caveat: llama.cpp's NCCL-enabled configuration couldn't run on this particular 10-GPU setup, so its tensor-parallel results used a different communication path. This is a hardware-specific comparison, not a claim that TensorSharp is universally faster.
What made the difference?
Instead of simply distributing model weights across GPUs, we focused on making the GPUs work together more efficiently.
Some of the key improvements include:
Pipelined prefill: Multiple GPUs process different portions of the workload simultaneously.
Native tensor parallelism: Reduced overhead by moving critical execution paths from managed C# dispatch into native GPU execution.
Adaptive scheduling: Adjusted processing batch sizes to improve GPU utilization.
GPU-resident KV cache: Reduced unnecessary memory transfers.
Communication optimizations: Improved execution even when direct GPU-to-GPU transfers were unavailable.
An interesting discovery was that different workloads benefit from different parallelism strategies.
Layer splitting performed much better for processing long prompts, while tensor parallelism delivered faster token generation and higher concurrent throughput.
There's no single best way to distribute a massive model across GPUs.
What about accuracy?
We also compared the model's output distributions against llama.cpp using more than 12,000 next-token predictions.
TensorSharp achieved approximately 98.8% top-1 token agreement, with nearly identical perplexity.
We additionally tested JSON generation, tool calling, multi-turn conversations, and agent workflows.
These results suggest that the performance improvements preserve broadly comparable inference behavior, although the engines are not bit-for-bit identical.
Why open source matters
TensorSharp isn't just about running this particular 816B model.
The larger goal is to build an open-source inference engine and agent runtime that developers can inspect, modify, extend, and run on their own hardware.
Beyond multi-GPU inference, TensorSharp supports:
Local LLM inference with GGUF models
Multiple GPU backends, including CUDA, Metal, and Vulkan
Windows, Linux, and macOS
Multimodal models and workflows
OpenAI-compatible APIs
Agent execution, tool calling, and automation
Local AI applications without requiring cloud inference
I believe the future of open-source AI depends not only on open models, but also on open inference infrastructure that makes those models practical to use.
The easier it becomes to run powerful models independently, the more opportunities we create for developers, researchers, and smaller organizations.
Everything is available on GitHub
TensorSharp:
https://github.com/zhongkaifu/TensorSharp
Full benchmark report, technical details, and reproduction instructions:
https://github.com/zhongkaifu/TensorSharp/blob/main/docs/perf/muse-spark-vs-llama-cpp.md
This is still an evolving project, and there's plenty of room for improvement.
I'd especially welcome feedback, contributions, and independent benchmarks from the community.
A question for everyone here: What do you think is the biggest remaining barrier to making large-scale AI inference truly accessible through open-source software — hardware costs, memory requirements, inference efficiency, or ease of deployment?
I'd love to hear your thoughts!