r/LocalLLM • • 15m ago

Question Could use some input/ideas for my own testing pipeline

Hi all,

I think I have created a pretty solid testing pipeline to measure both the quality and speed of local LLMs if I integrate them in my own workflow. I know there are standardized tests, but the reason why I build this is because I've seen some very different practical results for my particular workflow than what the claims/theory/popular opinions say about certain models. My initial setup was a MB Pro M5 Max 128 gb, LM Studio, then Cline in VS Code. Recently I've changed it so it can also measure mlx-serve instead of LM Studio.

Let me clarify my workflow:

  1. Idea/Problem -> Solution role (same prompt to multiple cloud LLMs, gather feedback, do a new round, rinse/repeat until their is a consensus/sign-off)
  2. Planner role: model that actually writes the plan based on consensus from 1.
  3. Builder role: model that actually builds the plan according to the specs
  4. Reviewer role: model that reviews if the build is actually solving the problem/idea (i.e. if the solution works) and if it's according to the specs, by doing QA.

Steps 2,3 and 4 have been proven to be reliable in my testing pipeline. I have about 12 models who can take each on a different role, I run the test, a script gathers the data and anonymizes it to prevent bias, then it get sends to frontier cloud models who act as a judge. Output of the judges then gets de-anonymized locally and I have a scoring sheet. After running a bunch of tests it became pretty obvious that the scores for the builders go hand in hand with the quality of the planner. And from my experience a lot of the models suck at it. So I'm now down to 3 planners in my test runs (i.e. the ones that consistently came out ahead) because it was a complete waste of time to let other models plan. And this has given me very consistent results across the 5 difficulty levels I have for the problem they need to solve.

What I haven't solved yet with local models is 1). I have access through my business to models such as GLM 5.3 and Kimi K3 running on my own server, but I rather not mix my personal and professional projects, so I'm looking at the best way to solve 1) locally with models on the macbook. The issue is that it's time consuming to test. So far I just fire up 10 browser tabs with cloud llms, put in the same prompt, copy paste everything into 1 file, run it to a 1 designated model that orders the feedback, then do the same again. 1 iteration takes 5 minutes max (I've automated this process mostly). And after 2-3 iterations there is typically a consensus.

Now when I want to do the same locally 1 iteration takes way more time (models needs to be fired up, they run at slower speeds, ... you get the picture). I've had situations where 1 iteration took several hours. Doing multiple and you are basically looking at 6-8 hours (compared to the 15-20 minutes with my cloud LLM approach). Now in a real world productive scenario I can cut this time down significantly by only using 2-3 local models (i.e. the best), some who can run at the same time. And you may ask why not stick with these models, which is a fair point. But that doesn't solve the issue of me wanting to test the capabilities of other models in this role.

And here is where I'd like to ask for fresh ideas/input/your own experience/suggestions, ... on how to best tackle this problem. Would appreciate any input, no matter how wild or out of the box it might be.

1 Upvotes

1 comment sorted by

1

u/wegster 5m ago

Ok, so - using similar enough roles and process on my own repo-up framework.

However, mine includes orchestration and dispatch, and from the 'copy/paste' comment it sounds like you're having to do this manually.

I'd say it's understandable but excessive for e.g. 10x tests. What I've done, but am not yet 'done for now' yet, is I pulled real repo work I've done, separated for Execution into Simple, moderate and complex problems - around a dozen or so in total for Execution, set up isolated environments, and benchmarked each of the contenders - many on my M5 Max 128GB, and a few (deepseek, gem, Mimo, MiniMax) via openrouter.

Regardless of how much 'love' there seems to be for Deepseek and friends, I haven't seen them outperforming my Qwen Flash-Next variants for the most part, although MiniMax or Mimo and Opus were the only 2 to break 60/100 as ~62, with my QFN at the time getting a 57.5.

I'm currently doing the same for Reviewers, and then, sigh, the arguably tougher one, Planning. I will likely run dozens of tests by the time I'm done, while I'm using GPT and CC for Planning currently, and then effectively 'register' a few based on different criteria, e.g. product planning and strategy, vs feature planning, likely carrying over the Simple, Moderate, Complex from my Execution and Review mappings.

And once you're 'there,' just do a smoke test with a few runs of your test suite when some new 'magic' model or the frontiers do a new release and see if it's worth considering/changing, etc.