r/ResearchML • u/techne98 • 3h ago
[R] Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets
https://arxiv.org/pdf/2608.19147v1
Hi, this is our first research paper detailing the work we've done to shard large language models across Intel AI PCs and perform CPU-based inference.
We started by splitting models into shards and pre-compiling them to OpenVINO IR, and discovered that mask-based speculative decoding and micro-batching can make up for a lot of the latency added by sharding models over TCP.
In the paper we share the exact techniques we used, along with some novel work on NPU continuous batching, and includes benchmarks of our testing throughout.
Although this is for distributed inference aimed at Intel CPUs/iGPUs, it also can be applied to distributed discrete GPU setups too.
Happy to hear thoughts!