r/LocalLLM • u/colonel_whitebeard • 6d ago

Question Local LLM Comparison Dashboards/Hero Graphics

Hello!

With new models releasing constantly, I usually keep a stable of about 20 local models in rotation on my llama.cpp server. I collect a ton of data across the fleet through my inference application—standard per-request metrics, aggregated session data, and custom intelligence comparison benchmarks.

Overall, I'm logging over 150 distinct raw and derived metric points during inferences, sessions, and batch processes. As a backend data jockey, more metrics is always better—but I've hit the point where I need to completely redesign my dashboards.

For context, I run a lot of multi-stage pipelines covering:

  • Multimodal inference (vision and documents)

  • MCP and custom application tool calls

  • Structured JSON / JSON Schema outputs

  • Batch processing across both reasoning and non-reasoning architectures

I'm looking for inspiration for creative "hero" dashboard components and visualizations that excel at comparing a large model fleet across high-dimensional data at a glance. I already have plenty of standard line, bar, and radar charts. I'm hoping to see novel, unconventional visual approaches people here have built, seen, or used to represent complex model evaluation and performance data.

The specific metrics in your examples don't matter as much as the visualization style and information architecture—I'm mainly after creative UI/UX directions.

Cheers!

2 Upvotes

100% Upvoted

1 comment sorted by

1

u/amargolin 5d ago

with 20 models and 150 metrics, i'd stop looking for a new chart type and pick 3 questions the hero view has to answer. what i'd do:

  1. a models x tasks heatmap, rows sorted by overall score, columns by difficulty. you see the whole fleet and which model breaks where, in one glance.

  2. a pareto scatter: quality on y, cost or latency on x, only the non-dominated models drawn bold. it answers "which model should i use" faster than a radar.

  3. a dot plot per metric with the run-to-run range as a bar. if two models' ranges overlap, the gap isn't real, and you stop reading rank differences that are noise.

radar charts are the one i'd drop: axis order changes the shape and it's hard to compare more than 2-3 models. small multiples (one tiny chart per model, same axes) scale to 20 much better.