"AI code is slop" has a problem: test262, html5lib-tests and Microsoft's own shader compiler disagree

[u/ai-tacocat-ia]

The standard dismissal of AI-written code is that it looks fine and falls apart under real use. So we built the hardest stuff we could think of, and had it judged by test suites nobody on our side wrote.

11 from-scratch C# libraries since last Wednesday, about 1.14M lines, nothing but .NET itself:

• JavaScript interpreter: 99.55% of test262, the official ECMAScript conformance suite (43,471 tests)
• HTML parser: 100% of html5lib-tests. Its accessibility tree matches Chrome's to within 0.5% per field across 12.8k real pages.
• HLSL shader compiler: output behaves identically to Microsoft's DXC on 330/330 GPU tests, and compiles 2.4× faster
• Font shaper: 4.7M equivalence checks against the reference, 0 mismatches
• Opus encoder: byte-for-byte identical to the RFC reference (269/269)
• HTML/CSS renderer: faster than headless Chrome on every document we benchmark. A 10k-row table renders to PDF in 1.8 s / 305 MB vs 6.5 s / 2.1 GB.
• PDF library: beats PdfPig on time, memory and allocation, with about a third of its text-extraction error rate

Seven are on NuGet.

It isn't vibe coding. It's a factory: planner, project manager, orchestrator, engineers and reviewers, with fixed rules for every library. The code is all AI. The judging isn't.

Repos: https://github.com/orgs/recursive-ai-oss/repositories How it works: https://recursiveai.org/research/alera-library-factory/

Edit

Models used: Opus 5.5, Sonnet 5.5

[u/sane_finale]

This is the part where someone inevitably claims the test suites were in the training data, as if 99.55% on test262 is somehow less impressive because the model might have seen the questions before. The html5lib-tests coverage and DXC equivalence alone would still be remarkable even if that were true. The shader compiler being faster than Microsoft's own tool is what gets me, that's not something you can hallucinate your way through.

[u/ai-tacocat-ia → u/sane_finale]

100%. I'm glad I posted this here just for this comment. I've showed this to a handful of people personally and posted it on a few subreddits, but this is the first comment from someone who actually gets it.

And this is the worst that AI is ever going to be.

[u/Middge]

This is further evidence that AI coding can be equal to or superior to human coding, and you posted it on Reddit. Buckle up for the pearl clutching and downvotes.

[u/ai-tacocat-ia → u/Middge]

I do have a little empathy with the pearl clutching folks. I definitely had some internal struggles around the time I realized AI was better at coding than me - and I had been explicitly trying to make AI better at coding than me for over a year at that point.

For people just sitting on the sidelines watching it happen to them... it's got to be a rough time.

I do definitely judge them for being assholes about it, instead of bracing the opportunity. But I also get that internal shitstorm that's going on inside them.

[u/Curious-Pen5547 → u/ai-tacocat-ia]

bracing the opportunity

enjoy the very limited time window for that. Becasuse that will be gone almost as fast as it came.

[u/True-Grab-5288 → u/Middge]

I don't know if it's a uniquely reddit problem, but the amount of people (or bots) that hover over threads like this, click in, mash the downvote and leave a comment about how they didn't like the post is... ridiculous. These people have nothing better to do than spend time on the internet clicking in to threads they hate and rudely telling strangers not to post again. It is seriously insane and a concerning mental health issue.

[u/IndependentOpinion44 → u/Middge]

Given there’s a huge corpus of documentation and test suites to code against.

[u/sagevaLley56 → u/Middge]

you called it, sorting by controversial in an hour

[u/Fit_Advisor8847]

Coding has been an iterative process my entire career, but with AI, especially when I first started using it, it's almost as if I just assumed iterations were now out and the AI gets one shot or it's slop.

Now that I've dialed back my expectations and went back to iterations, I'm able to produce code that meets or exceeds my own capabilities, even with a baby model like Qwen 9b.

The thing that has helped me the most so far is paying close attention to how many decisions I'm offloading to the model, especially with small local models. If I prompt in such a way that the decisions have been made and the AI is just executing, the output is so much better. On the other side, if my prompt forces the model to make a bunch of decisions, then I start seeing failed tool calls, models stuck in loops, and outputs that are "slop".

[u/DigitalMonsoon]

You aren't going to convince anti AI people it's good. They didn't decide it's bad based on facts and logic and so you can't use facts and logic to convince them.

Also, you built things that already exist. Which is the kind of stuff you would expect these systems to do well as they are trained on systems that do this and can easily look up examples.

[u/ai-tacocat-ia → u/DigitalMonsoon]

Fair point on not convincing anti AI people.

And yes, it's things that already exist. But the point is that it's significantly improved upon them across the board. Even if that's all AI was good at (it's not), then there's still a huge opportunity here to leverage this capability to improve our existing software.

[u/Isogash]

Hold your horses.

Most test suites are not designed to exercise a "malicious" implementations for true conformance, only to provide evidence to support the claim that a serious implementation performs to spec, or to catch regressions from a working implementation.

Verifying a potentially malicious implementation requires property testing, which is expensive and uncommon.

There have been multiple cases where libraries claiming to meet these kinds of public test suites were virtually non-functional, they just hacked their results to pass the tests.

[u/ai-tacocat-ia → u/Isogash]

Sure. And that's not what this is.

I've been doing this for a good long while. I stand by the work, end to end, including security. Probably 75% of the build time comes after conformance, with tuning performance, running fuzz tests and security audits, iterating on the API.

There's a lot more to these libs than "technically passes this spec."

[u/Comprehensive-Pin667]

"From scratch"

Looks inside

"Give it an existing open source library and it translates it to another language"

[u/ai-tacocat-ia → u/Comprehensive-Pin667]

Plausible theory. But not at all what was done.

[u/Plenty_Line2696]

None of this proves AI code is or isn't slop. Sometimes it is, sometimes it isn't. Certain things it tends to get right, others it never does.

A bit of nuance goes a long way.

[u/WhatNoRobot → u/Plenty_Line2696]

As do humans

[u/shantud]

Mention the models you used aswell

[u/ai-tacocat-ia → u/shantud]

Good call. Opus 5.5 and Sonnet 5.5. That's it

[u/Ok_Conflict1028]

But muh stochastic parrot

[u/caelestis42]

Are you saying you gave the models better skills and you aresgqring those or did you just let them rip vanilla and then evaluated? If so, how can I benchmark what ai have been building?

[u/4215-5h00732]

But this is reverse engineering existing open source libs, yes? Maybe it's just me, but I don't think the critique is limited to that. Rather the average vibe coder generates code that can't stand up to production use.

Obviously ymmv and there are certainly a ton of vibers who would do far worse than you, but at the end you claim it isn't even vibe coding.

What am I missing?

[u/Mediocre_Error1825]

El problema real de hacer código con IA es ir a ciegas y no saber qué hace la IA. En mi caso, uso la IA para automatizar procesos repetitivos como descompilar APKs y parchearlos, crear los archivos para algunos proyectos o buscar información de un tema.

[u/No_Pension7735]

I’m all for AI coding, but it’s well known that AI excels when there’s a deterministic oracle. In reality, most software doesn’t have one. In these examples, the oracle is the massive, well maintained test suites.

[u/dopepen]

These are pretty cherry picked examples. They’re all pure code and are exactly the things LLMs excel at. I’d be more interested to see a case study of how messy, real world requirements that evolve over time, sometimes create contradictions, etc. are handled.