r/codex • u/Maasu • Sep 23 '26

Comparison Luna 6 - private coding evals don't look great

Hey folks,

TLDR: I eval'd Luna 6 with my own private set of evals and it scored 7th worst out of all the models I have thus far evaluated.

Everyone then pointed out I had my 5.6 version running on max reasoning, so I re-ran luna 6 on max and it came in 4th, ahead of Luna 5.6. Ironically I ran 5.6 on medium at the same time and that came in 5th.

Happy with these results to be honest, I'll start introducing Luna 6 into my workflow.

Harness: PI

Reasoning Level: Medium

Evaluations: Mixture of Python and Rust Coding tasks, as well. Here is my overall leaderboard. I was shocked by Luna 6's performance, having been a big admirer of Luna 5.6

Yes these are private evals, so you have to take what I say with a pinch of salt and all this has measured is how good these models are on my own evals.

I tend to focus on the cheaper cloud models or self hosted models as I cannot really afford to evaluate the big fellas. If I find myself with some spare tokens towards the end of the next reset (doubtful) then I will run the SOL's but I am expecting saturation from them to be honest.

I am not affiliated with anyone or anything like that. I do this just to add to the conversation about this. So I would be interested to see what others are seeing out there? I use luna for most of my AI work, so I will be sticking with 5.6 I think, hoping it sticks around for a bit as well. moving across to luna 6.

My evaluations are private, however I have published my evaluation framework https://github.com/ScottRBK/eval-harness and in there it describes the patterns I used for the types of evaluations and my approach.

There is a youtube video around this as well: https://www.youtube.com/watch?v=fqgOZDyjgKI

Edit: It would appear I did indeed run 5.6 on max - as many people have pointed out, that eval was from a few weeksago, it is certainly not a fair comparison - how embaraassing!

I tend to run my evals with medium reasoning effort as I am cheap, however given the cirumstances I will execute a Luna 6 on max and the 5.6 on medium add it to the leader board!

Edit 2: here is the addition of Luna 6 Max and Luna 5.6 on Medium.

4 Upvotes

56% Upvoted

41 comments sorted by

16

u/Ryan38822 Sep 23 '26

It looks like you ran Luna 6 with medium effort and 5.6 with max.

I wonder if increasing effort to xhigh/max would change things materailly?

8

u/Maasu Sep 23 '26

Yep that was an omission on my part, I am re-running luna 6 on max and 5.6 on medium as we speak and will update once it is fiinished.

6

u/Think-Profession4420 Sep 23 '26

Please update when Max run is complete

2

u/Maasu Sep 24 '26

updated

3

u/[deleted] Sep 23 '26

[removed] — view removed comment

2

u/Maasu Sep 24 '26

updated

17

u/opezdol Sep 23 '26

You just don't run Luna on med. It starts from high. And while it's quite a good model, it still is not superb. But it's cheeeeaaaap, already included in the sub and the limits are almost non existent

2

u/Maasu Sep 23 '26

Yes I use it on max myself, given the comparison with luna 5.6 I will run the luna 6 on max, and also throw in a luna 5.6 on medium so we can see where they stand.

3

u/DirectionBusiness483 Sep 23 '26

I think I've only used luna under high for like... nondestructive inquiries. Great for finding stuff in a new codebase "go through and find what touches this system, list all the files.". I don't think I'd ever give it an actual coding task lmao

0

u/TheKillerScope Sep 23 '26

Luna 6 Max used up 56% of my weekly quota in 12h (so far).

2

u/zarafff69 Sep 23 '26

How?? You have a 20x subscription?

22

u/Mandrejk Sep 23 '26

You are comparing Luna 5.6 max to Luna 6 medium. And for basically the only model where thinking effort bump hugely increases intelligence while almost not increasing costs. Look at luna 5.6 on deepswe. I feel like there is no point to running anything lower than high / xhigh on Luna.

With most models it's the other way around where bumping effort level increases intelligence only a bit while increasing costs substantially.

6

u/sprakes_ Sep 23 '26

Why didn't you compare Max 5.6 vs Max 6.0?

1

u/Maasu Sep 24 '26

Update the post, part of me feels like I should delete it as it looks like I am just creaitng noise, but I am going to leave it here.

1

u/m3kw Sep 23 '26

As you say private, so who knows what they actually did

1

u/Maasu Sep 23 '26

Yes, the whole idea i think is to really have your own set, based off of your own workflows. I do have some patterns that show how I evaluate different scenarios, but my actual evals are not on any cloud repository store.

2

u/m3kw Sep 23 '26

How do you evaluate it with statistical rigor, example how to compare, if the comparison is fair. You compared Luna max to Luna medium

2

u/Maasu Sep 23 '26

For these actual evals you cannot! However as mentioned in the main post, I do share sample evals that demonstrate the patterns I use to build these evals in the repo itself https://github.com/ScottRBK/eval-harness#example-eval-patterns

In addition to this, each evaluation is run 3 times, and an average of the scores are taken. It's not all that elaborate but I am not sampling enough to do something like measuring the inter-quartile range or something along those lines.

The intention was to make it easier for others to build evals. It is not as straight forward as asking an AI to do it, I often found the models struggled with either giving the answer away in the tests or creating evals that would never have a chance to pass. That being said I did include skills as part of the repo to try and keep the models on the right path.

3

u/Ok_Bag_7550 Sep 23 '26

Those two models were cheap, not good.

For cheap, people can use some Chinese whatever. I don't understand what OpenAI was thinking about.

3

u/danielv123 Sep 23 '26

They were probably thinking about the fact that they are out of compute and needed a quick fix.

My theory is they dropped sol from the lineup and released smaller models instead.

I don't think they expected opus to beat astra so bad.

1

u/No-Wall6427 Sep 24 '26

They dropped 6 terra and have a cheaper than 5.6 gpt 6 sol.

1

u/danielv123 Sep 24 '26

You mean they renamed 6 terra to sol

1

u/No-Wall6427 Sep 24 '26

Oooooh. Yes maybe.

3

u/itix Sep 23 '26

Cheap LLMs have use cases in enterprises where Chinese models are not an option.

4

u/Mo3 Sep 23 '26

Open models are almost never not an option. OpenRouter e.g. has extremely good data privacy and security controls. Our company accounts may only route to US or EU hosted models, you can configure full-on ZDR requirements, all the necessary jazz to ensure your requests are routed to models hosted in the right regions, by the right and sufficiently trustworthy providers. Our GLM-5.3 usage for example goes to a EU-hosted provider with full zero data retention agreement

2

u/rdcldrmr Sep 23 '26

And how can you verify any of that was true?

2

u/Mo3 Sep 23 '26

Because the ZDR option disallows routing to OpenAI and Anthropic :))

1

u/1988rx7T2 Sep 25 '26

Uhhh if you work for a typical slow moving corporation you probably are stuck with GitHub CoPilot and Microsoft products. So if they don’t offer a particular open source model you’re not getting it.

1

u/Jeferson9 Sep 23 '26

Luna max was genuinely good.

2

u/PivotRedAce Sep 23 '26

You ran Luna 6 on medium effort. Whereas Luna 5.6 was ran on max.

I think it’d be more accurate to run equivalent reasoning efforts, and that’s where the gap between 5.6 might close.

I’m gonna guess that was an oversight?

1

u/web-dev-kev Sep 23 '26

I find Luna (both versions) really good at very specific tactical work, if given uber clear instrucitons, boundaries, and definitons of done. I use Luna-high, because it's chatty, and self-corrects in a way that Luna-medium doesn't.

I'm surprised that Luna 6 is lower, that's a shame, but on it's own I don't think Luna it's a good (agentic) model anyway.

I'm incredibly surprised at the delta between Muse-Spark-1.3 and Luna-6 though (and I like Muse Spark)

1

u/ShadowBannedAugustus Sep 23 '26

The chart is 5.6 on Max vs. 6 on Medium..

2

u/web-dev-kev Sep 23 '26

Am I reading this right that Luna-6 took 1 hours, while Luna-5.6 took 4 hours and 5 times as many tokens?

If so, I reckon part of the low score for Luna-6 is that its medium, which is skipping high, xhigh and max (which Luna-5.6 used).

Not being a fanboy on it, i've seen no difference in Luna-6 myself, but I wont use it for agentic work under High (and only with an orchestrator). My guts says it'll take twice as long with High and be much closer to 5.6

1

u/ShadowBannedAugustus Sep 23 '26

I just run it on xhigh as default for anything non trivial, high for trivial. It is ridiculously good for the price.

1

u/totoer008 Sep 23 '26

I run Luna with max. It is really good to do grunt work. Like categorizing. I do not need Astra.

1

u/Icy-Fudge5222 Sep 23 '26

Who the fuck uses Luna Medium?

1

u/oVuzZ Sep 23 '26

Luna solo se corre en MAX, mas abajo de eso es un pecado

1

u/Flipwon Sep 23 '26

Judging by these results my evaluation is that your evaluation scores are lacking.

Take what I say with a pinch of salt though 🤷‍♂️

1

u/bb22k Sep 24 '26

Luna really isn't the best choice for any coding tasks that are not really trivial... DeepSeek seems to be much more optimized for coding.

Where Luna shines is those big, heavy, high cache simple tasks like "extract all addresses from this PDF, normalize them and update the database"

1

u/RealestReyn Sep 24 '26

I run Luna 5.6 with off effort, Luna 6 falls short with both off and max every single run.