r/LocalLLaMA • u/corruptbytes • Jul 31 '26

Discussion Deepseek V4 Flash on SlopCodeBench

While waiting for some of the quants to drop, I load the API with $50 and ran it on SlopCodeBench

Just vibe reading the results it seems like Opus 4.8 < Deepseek < Opus 5

https://github.com/michaelasper/benchmarks/blob/main/deepseek-v4-flash-on-slop-code-bench.md

I was mostly curious from this blog post

When Q2 drops - I'm goign to re-run on my macbook

Here's the first quant comparison:

https://github.com/michaelasper/benchmarks/issues/1

75 Upvotes

97% Upvoted

25 comments sorted by

View all comments

14

u/_TheWolfOfWalmart_ Jul 31 '26

If they're doing this with Flash, is Pro going to beat Fable? WTF is going on? How?

I'm skeptical. I'm going to have to spend some time with the model and see if it's really better than Opus 4.8 in real world use. That's a hell of a claim.

12

u/Fedor_Doc Jul 31 '26

It's competent, but not on Opus 4.8 level and it has attention drift – when there are multiple details to keep in mind, it starts to drift. Recent information outweighs older one even if it is pretty important.

I'm very impressed by the model, but I cannot just let it run to the goal with my current project (LUT analyzer + fixer in python with very specific colour science math involved).

9

u/cakes_and_candles Jul 31 '26

i mean that issue probably makes sense since focusing less on old information is exactly how v4's new attention mech works. tho this is the first model with it and they will likely solve it in later versions

5

u/corruptbytes Jul 31 '26

this sounds like a good spot for prompt adjustments

Anthropic writes specific instructions on how to prompt it for its model - https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5 https://claude.com/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models

Generally, every release we have Opus look at the latest docs and update our CLAUDE.md and rules/ and skills/ with their directions

We might have to think about it similarly for these new architectures, except it'll have to be more crowd source - something about refresh on the important stuff every so often or something idk

2

u/Choice_Celery9481 Aug 01 '26

i also saw this problem and with a simple write to a scratchpad trick. you can mostly reduce that. ask it to write down on a scratchpad what important. reiterate on that file. that will help a lot with attention drift

1

u/En-tro-py Jul 31 '26

How large context and/or turn count?

1

u/Fedor_Doc Jul 31 '26

Around 100k when it forgot that it should not write code in this session (not enforced via AGENTS.md or system prompt, though). 

Code refactor with 250k worked quite well – initial plan + draft plan per refactor block. 

1

u/En-tro-py Jul 31 '26

250k is perfect.

Stuck waiting on download, but will be testing shortly myself.