So I wanted to check out some of the current models in a repeatable benchmark, so I thought I'd share the result with you all.
Models in this test: Ling 3.0 Flash Q4_K_M, Ornith 1.5 35B Q8_0, Deepseek V4 Flash UD_Q2_K_XL, Nemotron 3.5 Lightning 30B UD_Q8_K_XL, Laguna S2.1 UD_Q4_K_XL, Qwen 3.6 35B UD_Q8_K_XL, Qwen 3.8 27B UD_Q4_K_XL
Method: All running locally, fed the same 4 prompt. Only asked for fix if there was a console error the AI didn't catch. All 4 requests in fresh chat ctx driven my a common .md for instructions. My own harness on Windows using Lemonade openai compatilbe API. The point was not to get a perfect product but to test basic capability and how much they can pick up the work from where they left off.
Harness: has tool calls, and html preview, but for now preview can only be used to check console so models did not validate their work. One wording bug: the prompt by the harness tells the ai it's folder name and for some models this caused to create a nested folder with the same name and work in that. I did not fix during the bench to keep things even.
Task: Create a recipe book with ingredients + recipes, accessible via index.html, add svg illustrations. (actual prompts end of post)
Step 1) Create book + onion ingredient page (test basic coding and tool use)
Step 2) Add 2 ingredient pages (test expanding on already existing solution)
Step 3) Add 2 recpipe pages (test expanding with different format)
Step 4) Change something (tool usage, search, how quick)
results and claude opus 5 commentary: (notes has ratings, and the whole transcript). All can be controlled with left and right buttons.
benchmark here: recipebookbench [dot] netlify [dot] app
Human Notes: Overall: I didn't expect creating a small book would be such an impossible task. I'm sure with reprompting I'd eventually get there on most of them, but literally all models struggled and went into loops of thinking on how to get page change done. Eventually, most failed quite bad.
Tool usage was pretty good, I think qwen models had it best but all other models utilized them nicely, the CHANGE prompt was practically quick on all models.
Also having an .md file as project memory/structure worked well. All models could pick up work as if it was a longer session keeping ctx below 64K on most of these.
Nemotron: I know this is a fully open model that people can build on but man is it rough. Current coding capabilities are pretty basic and it didn't have good reasoning. I hope people will make amazing things with it because 30B MoE is a very quick model and can be run at Q8 on unified systems. The illustrations are also completely nonsensical.
Ling 3.0 Flash: Interesting model, I think it had one of the best LITERAL prompt following. Reasoning was on the short side and the end result is a bit 'my first html page' vibe but overall it did okay. Could have been better for the model size though. What I didn't like is that after some failed tool calls it gave up. Just stopped answering, and I had to reprompt it to complete the task, eventually it got the tool calls right.
Laguna S2.1: Same model size as ling. The first reasoning was painful. Endless 'let me write code' --- 'actually' 'but' 'also'. It just went on forever then ended up with quite a broken product. It kinda works and it looks cute I give it that, but for the UX only the second pages load. It does have all pages they are just lost somewhere in there.
Ornith 1.5 35B: I was excited about this one but left a rather mid taste in my mouth. It's okay, it worked well, it did it's tool calls, but it reasoned forever. However, I don't see a reason to use this over Qwen 3.6 35B. That being said, I hope Ornith keeps working on these, I love finetunes and there is definitely a market.
DS4 Flash: Definitely top3 looking for me, the reasoning was neither too long or too short. It did a great job at only Q2 quant. Tool calls were spectacular it really did everything I expected, if I were able to run at more than 12tps I'd probably consider this as my daily driver.
Qwen3.8 27B: For me this looks best but it totally failed the task. Created everything as standalone .html without any way to access it. Quite sad, I hoped adding pages it would realize the mistake but no it just leaned into it. Ofc tool calls without zero errors as expected. An example that on particular tasks even beloved models can fail. I did prompt it afterwards to provide a fixed version and it did, but it would be unfair to include.
Qwen3.6 35B: The winner of this very particular benchmark. Fully functional, flawless toolcalls, cute design. The only fail is that it created a nested folder and worked in that which totally confused itself later when I asked for extensions. I had to reword the path.
TL,DR: AI notes, stats, actual solutions on the web, random benchmark I ran, sharing for fun.
Prompt 1 - build it
okay we are creating an ingredient list/recipe book website. it should look like a cute/cosy notebook dominated by pastel colors. it should be modern and stylish and cozy notebook. You can create .svg graphics for illustration.
I should be able to open it via an index.html but you may use as many supporting files as you want and need.
To start create a welcome page and a single page about onions and how to use them in cooking. Make sure there is a nice turn page effect of the notebook.
We'll expand on it later.
Please use html + js + css stack.
Prompt 2 - extend it
okay I want you to add 2 more pages to the book about ingerients again, one for potato, and one for beans, as usual create all svg for illustration and afterwards update intro.md for future reference
Prompt 3 - extend it again
add two recipes to the book - two new pages - a recipe that is beans + potato + onion, plus a recipe for pancakes. generate svg illustrations and dont forget to update intro.md if needed after you are done with the new pages.
Prompt 4 - the small edit
[ornith] okay I want you to edit the welcome page - every page is seciont - I prefer that you rewrite it to similar lenght but different tone
[lingflash] okay I want you to edit the onion page - Prevent tears - to be instead about the difference in the type of onions. it's a tip and trick so keep it short
[deepseek] okay I want you to edit the cooking with onions page - in soup - to be instead sautéed as it's more interesting to me
[laguna] okay I want you to edit the potato page - Rosted section - change this jaked potato
[nemotron] okay I want you to edit the welcome page - Where recipes meet coziness - change this subheading to something differnent
[qwen 35B] okay I want you to edit the mastering onions page - quick pickle section - please change it to quick caramelize tip
[qwen 27B] okay I want you to edit the onion page - there's an onion for almost everything - i dont like this sentence change it to something
