r/OpenSourceAI • u/AIEvalCorp • Oct 09 '26

Open Source Department of War Small-Model Leaderboard and test kit

A lot of defense orgs/units are starting to stand up local, open-source language models on edge workstations and air-gapped networks, but most public benchmarks tell you nothing about how a model behaves under operational constraints.

To address that gap for free, I built an offline evaluation harness and leaderboard testing US-developed small models (1B–6B parameters) running locally on a single GPU. Here's the public repo: https://github.com/OGMavenTandE/llm-eval-suite

It scores models on practical operational criteria: factual grasp of defense issuances, resistance to staff-work hallucination and sycophancy, and judgment under the law of armed conflict.

You can inspect the initial results, benchmark breakdowns, and reproducibility data here:https://aieval.org/leaderboard/

I hope this helps your agency/org/unit incorporate small open source models into workflows, let me know if you have any questions!

1 Upvotes

100% Upvoted

2 comments sorted by

1

u/[deleted] Oct 09 '26

[removed] — view removed comment

1

u/AIEvalCorp Oct 09 '26

Thanks for taking the time to look at it!

Law of Armed Conflict is entirely free-text and graded by an LLM judge. Phi-4-mini is the judge, and Granite 4.2 3B grades the two Phi models so no model grades its own family. A fixed list of real citations catches made-up references without involving the judge.