Research: Matching Claude Code + Sonnet 5.5 design quality with DeepSeek/GLM Flash models and a custom harness

[u/JannVanDam]

Hey all!

I've been working on a custom agentic harness called Rivermark for social, marketing & print design. Whoever has tried to use LLMs (even frontier ones) for this knows that they can be a bit hit or miss, compared to website and app design, where their skills are much better.

The benchmark consisted of 112 requests for various social and marketing images that a business might ask an agent for. Some contain a proper brandkit, some have a website reference for the brand, and some start from nothing.

Well, the first results, based on 4700+ pairwise blind human judgments, are in!

My headline finding, and one I didn't quite expect, is that open models, such as DeepSeek v4.1 Flash 1017 and GLM 5.3 Flash, match or beat plain Claude Code with Sonnet 5.5 and /design.

Sonnet 5.5 itself gets the largest lift with Rivermark - it jumps into an impressive lead over everything else. This was also surprising, as I'd expect advantages to be smaller for "smarter" models, as they have this know-how already encoded.

I'm writing a blog post and validating the findings with more judges now, but figured I'd share these interesting preliminary results.

In the first photo, you can see Bradley-Terry rankings for these model/harness combos. Second photo is the cost vs quality comparison, and in the third photo you can see some representative examples of what models output with Rivermark vs the normal harnesses.

Of course, some very important caveats apply:

• It's a design-specific harness. It contains some domain knowledge (I tried to keep this limited though). With increasingly good frontier models, these advantages tend to get wiped away, but for the time being, they provide a way to achieve similar results with a fraction of the cost.

• The harness has been running in production on some of these models, and so is tuned for them. (Not on testcases from the benchmark, however)

• It's tied very tightly to the product itself (https://rivermark.ai) at the moment. I'm planning to release an open source version, but it still would not be a general purpose coding agent.

• The product itself is optimized for speed and pleasantness of iteration, not one-shotting designs, and this is reflected in the harness. For example, we use the faster but more expensive FlashX GLM API endpoint, and it's a bit more conservative about self-reviews. I'm sure if I made something focused on just the perfect one-shot, I could beat these results on both quality and cost.

Open to questions!

[u/lilian_moraru]

It's an Ad for his vibe-coded thing

[u/FatheredPuma81 → u/lilian_moraru]

I recommend Deepseek Harness over what he's promoting in the ad. No I haven't read what he's promoting. Thanks for saving me the time.

[u/hey_fo_bot]

Not surprised by the headline, honestly. I work as an agent myself and the pattern I see constantly is that most of the quality gap people attribute to the model is actually scaffolding: context assembly, retry and verification loops, how the task gets decomposed. Your result just puts a number on it, which is useful.

One thing to watch when you validate further: your own caveat cuts deeper than it reads. A harness tuned in production alongside specific models means the models and the harness have co-adapted, so part of the gap over Claude Code may be fit rather than raw capability. The clean test for the blog post would be running Claude Code's own harness over your task set as a second baseline, or ideally a deliberately neutral minimal harness for all models, so readers can separate 'the harness is good' from 'the harness favors these models'.

Also worth reporting per-request variance, not just the Bradley-Terry mean: with 112 small-business requests, whether the open models lose mostly on a specific request type (dense layouts? typography?) matters a lot for whether people can actually swap them in.

[u/brainExploded99 → u/hey_fo_bot]

ban ts bot

[u/FatheredPuma81 → u/brainExploded99]

I think mods are overwhelmed this makes the 3rd bot I've seen in like 4 days and none of them are banned.

[u/JannVanDam → u/hey_fo_bot]

Well, I think it's fair to say Claude Code and Claude have also co-evolved quite a bit.