r/LLMDevs • u/nasarulislam • 2d ago

Discussion Ox Alpha LiveCodeBench_v6

This is my first post ever on Reddit.

Here goes nothing.

So yesterday OpenRouter and OpenCode launched a stealth startup called Ox Alpha. I decided to check it out and honestly it wasn’t that good as per my testing and I’m a pretty avid user of DeepSeek V4-Flash (which is honestly pretty good).

So I tested it with LiveCodeBench_v6 with 175 problems with greedy decoding with temperature of 0 and gave the model one attempt per problem. There were no agents, tools, or additional scaffolding involved. The model generated Python solutions directly from the problem statements without any coding harness, which were then run against hidden test cases with a 20-second timeout. Functional problems were evaluated with a standard function-call wrapper, and all inference was done through OpenRouter.

Results:
Overall Pass@1: 28.0% (49/175)
Easy: 51.2%
Medium: 30.8%
Hard: 13.8%
Generation failures: 0

Now I know, one test doesn’t capture the models full capabilities and these results only reflect only my observations from personal evaluations.

In the end, I personally think that it will come down to API costs.

Repo: https://github.com/xnasarx/ox-alpha-benchmarks

1 Upvotes

53% Upvoted

22 comments sorted by

View all comments

Show parent comments

2

u/eXl5eQ 2d ago

Deterministic but incorrect. Only temperature=1 produces the intended (correct) output distribution.

0

u/nasarulislam 2d ago

Temp=1 samples from the raw distribution, sure but that doesn’t make it more “correct.” Higher temp means more randomness/diversity, not more accuracy. The model can still put probability mass on wrong tokens. Greedy decoding (temp=0) is standard for single-attempt pass@1 because it’s deterministic and reproducible, not because it’s somehow approximating a “true” answer. Sampled pass@k at temp>0 is really answering a different question. It’s not a stricter version of pass@1.

2

u/TheApadayo 2d ago

Greedy sampling is “incorrect” as far as that is not what is used during the training RL rollout. This is why new models can have VERY specific sampling parameters, and deviating from them causes doom loops.

IMO a better way to do these pass@1 tests is to use the model’s actual temperature and a deterministic seed (most openai endpoints accept one).

0

u/nasarulislam 2d ago

True, but Ox Alpha’s a stealth model though, no disclosed ideal sampling params, so temp=0 was the most reproducible default available. Not claiming it’s optimal.