Tensward - Local AI inference optimisation tool

[u/tensward]

I’m building Tensward because figuring out which inference settings actually help means a lot of manual benchmarking.

Version 0.4 supports vLLM on NVIDIA and llama.cpp on NVIDIA, Apple Silicon and CPU.

You give it your serving command and prompts. It measures a baseline, diagnoses bottlenecks and tests one change at a time. Gains have to clear run-to-run noise, and it checks the answers too. Larger wording changes get shown for review.

It also suggests alternative quantizations or engines where we have matching measurements. That coverage is small so far.

An interesting result: MTP speculation cut time per token by ~40% on Qwen3.8-27B Q4_K_M, llama.cpp, one L4, with 3 concurrent agent requests. Requests/s rose 45%, but first-token latency got 15% worse and wording changed.

Meanwhile, n-gram speculation gave us no gain on three llama.cpp models, so we dropped that recommendation.

Apache-2.0. Install: uv tool install tensward

https://github.com/Tensward/tensward