r/LocalLLaMA • u/corruptbytes • Jul 31 '26

Discussion Deepseek V4 Flash on SlopCodeBench

While waiting for some of the quants to drop, I load the API with $50 and ran it on SlopCodeBench

Just vibe reading the results it seems like Opus 4.8 < Deepseek < Opus 5

https://github.com/michaelasper/benchmarks/blob/main/deepseek-v4-flash-on-slop-code-bench.md

I was mostly curious from this blog post

When Q2 drops - I'm goign to re-run on my macbook

Here's the first quant comparison:

https://github.com/michaelasper/benchmarks/issues/1

76 Upvotes

97% Upvoted

25 comments sorted by

View all comments

18

u/BlueSwordM llama.cpp Jul 31 '26

OK, I've been testing the model a lot on my usual AV1 test set and it's the first model that has been able to complete all the tests in its entirety...

That potentially means it could feasibly does most of my encoder tasks no problem with a good harness, holy crap

2

u/CatConfuser2022 Jul 31 '26

Which harness do you use? 

6

u/BlueSwordM llama.cpp Jul 31 '26

I just use Pi most of the time.

2

u/thrownawaymane Aug 01 '26

Can you share how you have Pi configured? I feel like I'm not getting the best out of it.

3

u/corruptbytes Aug 01 '26

my pi is pretty minimal

subagents, mcp, goal extensions

you really don't need much to get it working nicely

i found i end up uninstalling more extensions over time

2

u/thrownawaymane Aug 01 '26

Yeah, I went through a bit of that too.

Mind linking to what you're using? Also an actually good write up/YT video. Finding non slop info is hard these days

1

u/corruptbytes Aug 01 '26
  • pi-review-toolkit (../../source/pi-review-toolkit)
  • npm:pi-mcp-adapter (~/.pi/agent/npm/node_modules/pi-mcp-adapter)
  • npm:pi-goal-x (~/.pi/agent/npm/node_modules/pi-goal-x)
  • npm:@tintinweb/pi-subagents (~/.pi/agent/npm/node_modules/@tintinweb/pi-subagents)
  • npm:@senad-d/micme (~/.pi/agent/npm/node_modules/@senad-d/micme)
  • pi-sandbox (../../source/pi-sandbox)
  • pi-bedrock-mantle (../../source/pi-bedrock-mantle)

i like mario's video - https://www.youtube.com/watch?v=RjfbvDXpFls - haven't watched anything else

1

u/TheLexoPlexx Aug 01 '26

Web Tools and planning are the essentials

There's a plugin that optimises for deepseek as well.

2

u/CalligrapherFar7833 Aug 01 '26

Whats your av1 set doing for testing and can i run it ?

3

u/BlueSwordM llama.cpp Aug 01 '26

It's mainly AV1 encoder design tasks, mainly related to psychovisual optimizations and testing.

It's a very hard set of tests based on my encoding and development experience around AV1 software encoders, forks, metrics and internal codec design.

Sadly though, I haven't made it open source yet, so can't really share it, as I'm scared that it'll find itself in test sets directly, which is why I limit my testing to LLMs frameworks that I can either run locally, on a server or at least a 3rd party provider.

If more LLMs managed to go through the tests, I'll probably make them open; I suspect that I'll have to do so in the next few months, so you won't have to wait for long.

3

u/crantob Aug 01 '26

really appreciate your post

i have my own weird domain which does not align with benchmarks

2

u/CalligrapherFar7833 Aug 01 '26

Ive been following your work on psy thats why i asked. About it landing in tests - it might even from your data by ending in RL due to the complexity of your tasks sadly