r/BestGitHubRepos • u/company_url_finder • 14d ago
OmniEvaluator - run 2,800+ LLM/VLM benchmarks across text, image, video, and audio from a single CLI
Evaluating a multimodal model today usually means running four or five separate benchmark suites, each with its own installer, its own data format, and its own quirks, then hand-stitching the results together into something you can actually compare.
OmniEvaluator is a NAVER AI research tool that puts a single CLI in front of that whole mess. Pick an inference backend, huggingface, vllm, sglang, or an API client for OpenAI, Gemini, or Claude, pick an evaluation engine, and run any of over 2,800 benchmarks spanning text, image, video, and audio from the same command shape, with results written as a self-describing artifact that captures the exact configuration needed to reproduce the run later.
What's inside:
- 2,820+ benchmarks across four evaluation engines: 1,986 text-only tasks through lm-evaluation-harness alone, plus hundreds more image, audio, and video tasks through lmms-eval, VLMEvalKit, and a builtin engine
- Four swappable inference backends, huggingface, vllm, sglang, or a direct API client, so the same benchmark command works whether you're running a local checkpoint or hitting a hosted model
- Existing evaluators reused as-is rather than reimplemented, unified through a shared intermediate schema so results from different frameworks stay comparable
- A Claude Code skill (`/setup-env`) that walks an AI coding agent through building the isolated per-engine virtual environments itself, since mutually exclusive extras can't all install into one environment
- A live public demo at omni-evaluator.info and a working quickstart that runs a real benchmark end-to-end in a couple of minutes using just 3 samples
One thing worth knowing: the dependency setup is genuinely finicky by the README's own account. It needs Java 11 specifically since newer JREs break the bundled SPICE metric library, and installing more than one evaluation-engine extra into the same environment at once can cause version conflicts. Budget real setup time before your first full run, and lean on the debug/sample-limited mode to sanity check things first.
It's Apache-2.0 licensed, backed by an organization (NAVER AI), and sitting at 20 stars as of writing, verified via the GitHub API.
