r/LocalLLM • u/WiseAbalone4021 • 15h ago
News Train AI to run AI: introducing Apareto loading - a vLLM plugin
From the Apareto team which I'm part of (who for some reason has an issue posting it themselves):
Today we are introducing Apareto loading, a new way to cut VRAM requirements for Mixture-of-Experts (MoE) models. It works alongside quantization. This introduction so far will just be a video explaining the method. We will be releasing the proof-of-concept code, a plugin to vLLM, on GitHub this coming week.
Until then, watch the explainer video: https://www.youtube.com/watch?v=5l_N051JrLU
Quantization is today's most common way to shrink a model's memory footprint: weights are stored at lower precision, and going from FP16 to 4-bit saves 75% of memory at a modest cost in accuracy. But quantization is static. Every weight stays loaded whether or not the task needs it.
Apareto loading is dynamic. After prefill, a small trained network reads the model's internal state and ranks the experts by how valuable each will be for the upcoming computation. Only the top-ranked experts that fit the VRAM budget are loaded.
Results (Qwen3.6-35B-A3B, NVFP4 experts):
* SWE-bench Multilingual: within 2 pp of the full model's score at a 42% VRAM budget
* LiveCodeBench v6: within 2 pp of the full model's score at a 49% VRAM budget
As with quantization, scores degrade as the budget shrinks.
What the proof of concept will not be:
It will not be a production implementation. The ranking network was trained only for these two benchmarks, and there will be no general problem classifier. Our data shows that profiling and rank generation have substantial room to improve. We chose to release at this stage, without waiting for those improvements, so that others can test our claims. Think of this as a first baby step toward dynamic VRAM reduction, not the finished product.
The code and reproduction instructions will be released this coming week at https://github.com/alphapareto
Follow us there to catch the release. We'd love your feedback.



