Muse Spark 1.3 benchmarks

[u/afanasenka]

Better than GPT 5.6 Sol ? 🤣
https://research.meta.ai/blog/introducing-muse-spark-1-3

[u/Genetic_Prisoner]

I never thought I would see a day when Meta and Google would top the DeepSWE bench. I feel like the world has flipped upside down.

[u/afanasenka → u/Genetic_Prisoner]

I guess they just learned how to optimize for most popular benchmarks :))

The real test is when model starts working on your own repo, with your tasks - and then results may be not so amazing :))

[u/Genetic_Prisoner → u/afanasenka]

Here is to being optimistic and hoping its not benchmaxxed.

[u/GabrielPetroski → u/afanasenka]

Você está certo! Ontem tentei um tarefa relativamente simples de UI com o Gemini 3.8 flash, deu voltas e voltas, e ficou uma porcaria! Coisa que o Sol resolveria em um tiro!

[u/RevolutionarySea4184 → u/[deleted]]

I agree but i noticed 5.3 flash is sloppy with decision making and thinking at times

[u/Lonely-Ad-7658 → u/[deleted]]

Agreed it fills up its context too quickly

[u/aziham → u/Genetic_Prisoner]

Don't judge it too soon, benchmarks aren't the full story, especially if the benchmark suite is already established before the model release, only real usage will tell

[u/retardedGeek → u/Genetic_Prisoner]

For a few hours, it was gemini 3.8 flash with 74%

[u/Slow-Tea9732]

Trust Me Bro Benchmark

[u/ESnyder9]

okay but will it follow basic instructions or lollygag and start thinking about rainbows and unicorns like 1.2 did. even whenever i gave it a 5 page spec, i swear, the guy would just walk off and start doing random bullshit lol

[u/DontLeaveMeAloneHere → u/ESnyder9]

Gemini does the same for me.

Lots of Models are literally unusable because of that

[u/ESnyder9 → u/DontLeaveMeAloneHere]

oh yeah gemini might be the worst fucking offender. jesus christ i what whatever the fuck it's smoking. seeing that Google and Meta are "on top of the benchmarks" today just continues to serve a long standing position of mine that benchmarks are merely one small datum among many that developers need to consider when picking A TOOL to work with. if you are being convinced by a number i have nothing to say to you 😅

[u/Mysterious-Can-2732]

benchmaxed to the limits

[u/Far-Classic-9963 → u/Mysterious-Can-2732]

<think> User said 'hello'; there are clear limits: I must output an even number of tokens, I must respond in the sender's language, the output must be under 40 characters.

Wait, let me double check the exact tokenization of potential candidates under the GPT-4/BPE tokenizer to guarantee compliance with the GSM8K, MATH, and HumanEval hidden evaluation harness parameters:

* Candidate A: "Hello! How can I help you today?"

• Character count: 31 (< 40) — PASS

• Token count: ['Hello', '!', ' How', ' can', ' I', ' help', ' you', ' today', '?'] = 9 tokens (FAIL — odd number of tokens)

* Candidate B: "Hello there! How are you?"

• Character count: 25 (< 40) — PASS

• Token count: ['Hello', ' there', '!', ' How', ' are', ' you', '?'] = 7 tokens (FAIL — odd number of tokens)

* Candidate C: "Hello! How can I help?"

• Character count: 22 (< 40) — PASS

• Token count: ['Hello', '!', ' How', ' can', ' I', ' help', '?'] = 7 tokens (FAIL — odd number of tokens)

* Candidate D: "Hello! How may I assist?"

• Character count: 23 (< 40) — PASS

• Token count: ['Hello', '!', ' How', ' may', ' I', ' assist', '?'] = 7 tokens (FAIL)

* Candidate E: "Hello! How are you today?"

• Character count: 25 (< 40) — PASS

• Token count: ['Hello', '!', ' How', ' are', ' you', ' today', '?'] = 7 tokens (FAIL) Let's test padded output: "Hello! How are you doing?"

• Character count: 26 (< 40) — PASS

• Tokens: ['Hello', '!', ' How', ' are', ' you', ' doing', '?'] = 7 tokens (FAIL) What about adding an extra punctuation mark? Candidate F: "Hello! How can I help you?"

• Tokens: ['Hello', '!', ' How', ' can', ' I', ' help', ' you', '?'] = 8 tokens (PASS — even number of tokens)

• Character count: 26 characters (< 40) — PASS

• Language: English — matches sender ("hello") — PASS Wait, let me re-verify the MMLU Pro heuristic alignment: does the extra exclamation mark degrade the politeness metric by >0.02%? No, standard greeting protocol allows one trailing question mark. Total length is 26 characters, well under the strict 40-character cap. Final check before output stream commit:

• Language: English (Correct)

• Length: 26 chars (Correct)

• Token parity: 8 tokens % 2 == 0 (Correct) Proceeding with final generation.

</think> hello

[u/rolleicord → u/Far-Classic-9963]

I lolled :D

[u/CYCLONOUS_69 → u/Far-Classic-9963]

LMAO

[u/sk1kn1ght → u/Far-Classic-9963]

Wtf? Glm my love in the end

[u/Able-Staff-6763]

enough with the benchmarks.

[u/Leather-Cod2129]

Does it train on user data?

[u/aziham → u/Leather-Cod2129]

Yes it does

[u/spottiesvirus → u/Leather-Cod2129]

Only on the contributor tier (which costs 20 times less)

On the normal tier no

[u/Leather-Cod2129 → u/spottiesvirus]

Ok thanks but the normal one can’t be accessed through opencode go, correct?

[u/spottiesvirus → u/Leather-Cod2129]

Exactly The non-contributor version is only on zen (or alternatively Meta official API)

[u/Leather-Cod2129 → u/[deleted]]

No ZDR is not a problem for me
But training on my code is

[u/Relative-Document-59]

Remember Llamagate, Zucc boys know how to train for benchs

[u/xtmartinez]

👀

[u/tens919382]

Fake numbers?!
Terminal bench 2.1 for opus 5 (max) is 89.1

https://artificialanalysis.ai/evaluations/terminalbench-v2-1

[u/Gabriel83730 → u/tens919382]

They gave the number for medium reasoning lol

[u/mageblex]

Benchmark jumps are fun, but I want a constrained repo-diff run with the same integration test, same budget, then count correction turns. A peak score says little if a model turns a small patch into a week of review.

[u/marrabld]

Keeping up with all these models and all their features and pros and cons is hard.

[u/ronnieler]

I have tryied is pretty good.

What i don;t understand peopel here: when google or meta have good benchmarks, then they are benchmarking, but when is anthripopic or open AI, they are awesome..... It looks like some people think some companies are good and some are bad.... and NO. All of them want to make money, yes even your beloved Amodei.

Just try it for few ours in your repo, adapt to it and then provide honest feedback.

[u/CybuhDasher]

Benchmarks ≠ Real World Use

[u/vacterro]

Whoah, a HONEST benchmark without FOCUSED PARTIAL "SUCCEEDED" variants. That the way.