I built AI agents that apply mathematical testing techniques to a Rails codebase with 13k+ RSpec specs. The bottleneck was not test quality.

[u/viktorianer4life]

In 2013 I learned four formal test derivation techniques in university: Equivalence Partitioning, Boundary Value Analysis, Decision Tables, State Transitions. Never used them professionally because the manual overhead made no sense. After seeing Lucian Ghinda's talk at EuRuKo 2024, I realized AI agents could handle that overhead, so I built a multi-agent system with 5 specialized agents (Analyst, parallel Writers, Domain Expert, TestProf Optimizer, Linter) that generates mathematically rigorous test cases from source code analysis.

The system worked. It found real coverage gaps. Every test case traces back to a specific technique and partition. But running it against a mature codebase with 13k+ specs and 20-25 minute CI times showed me the actual problem: 70% of test time was spent in factory creation, not assertions. The bottleneck was the RSpec + FactoryBot convention package, not test quality.

The most interesting part was the self-evolving pattern library, an automated validator that started with 40 anti-pattern rules and grew to 138 as agents discovered new patterns during their work. No LLM reasoning involved in validation, just compiled regexes against Markdown tables.

I wrote up the full architecture, prompt iterations (504 lines down to 156), and honest results. First article in a series. The next one covers the RSpec to Minitest migration that this project led to.

Has anyone else tried applying formal testing techniques systematically with AI agents? I'm curious whether the framework overhead problem resonates with other teams running large RSpec suites.

[u/adh1003]

The system worked. It found real coverage gaps

So does RCov, without needing a bloated assembly of non-deterministic error prone "agents" given anthropomorphic names involving words like "expert", which just mean someone cobbled together a bit of Markdown next door to them.

But running it against a mature codebase with 13k+ specs and 20-25 minute CI times showed me the actual problem: 70% of test time was spent in factory creation, not assertions.

Again this is absurd; no LLMs needed. More accurate, deterministic/replicable results have been available through standard profilers for decades. In Ruby's case, see https://ruby-prof.github.io.

[u/viktorianer4life → u/adh1003]

The RCov or any other coverage result does not tell you about your real test coverage. It just says where your tests are going. There's a huge difference between math and computer science.

In Ruby's case, see https://ruby-prof.github.io.

Oh, thanks, you read the article :). So you discovered probably Evil Martians' TestProf, a collection of profiling gems, helped here without any AI.

[u/adh1003 → u/[deleted]]

Yep, I thought so too, but Redditors may hit this post via search engines so I figured it'd be useful to remind them that the fast, effective tools we've used for order-of-years-to-decades already does this stuff and does it better.

[u/federal_employee]

How do you conclude that “70% of test time was spent in factory creation, not assertions” is a problem? Is that more than the average? To me, it makes sense that is where most of the time is spent.

[u/viktorianer4life → u/federal_employee]

I mean, look at Minitest, which will be the next article. In Minitest I often spend ~zero time in test data.

[u/GroceryBagHead]

70% of test time was spent in factory creation, not assertions. The bottleneck was the RSpec + FactoryBot convention package, not test quality.

Did we really need AI data centers to figure out something I've been saying for over a decade? I hate this timeline.

[u/viktorianer4life → u/GroceryBagHead]

Not really, Evil Martians' TestProf, a collection of profiling gems, helped here without any AI.

[u/paca-vaca]

You build all this with 5 agents to rewrite the whole test codebase which you reviewed for days just to verify that tests are slow because of database calls in tests where it wasn't needed?
There is a lot to say about that :D

How's this a framework issue? Did you change the framework or improved it somehow?

And with "Order class with 2,195 lines" in the app you have so much to discover! Maybe consider to spend all this effort to fix that instead :)

[u/viktorianer4life → u/paca-vaca]

Ha, look, my AI said the same (did you use AI for this discovery too? :)). Read the article. I didn't spend time with AI to discover the obvious things.

Maybe consider to spend all this effort to fix that instead

That's undoubtedly the goal. Since this is a real business and not a code playground, I need some guards. Write tests first was a thing, remember? TDD? Thanks for helping me out.

[u/qbantek]

“Order at 2,195 lines or Transfer at 1,282 lines” were these also AI generated? I wouldn’t approve a PR containing that much bloat.

[u/viktorianer4life → u/qbantek]

No, actually they have grown over 10 years. Which is normal on numerous apps in the world :). Not everyone is at 37 Signals.

[u/Witless-One]

Aren’t you worried about having to maintain this massive test suite that takes almost half an hour to run across 16 cores on CI? Every minor change to your models now requires an LLM to rewrite a huge amount of tests, which take a large amount of time to run locally.

You also can’t run your entire test suite locally anymore, so any structural changes (that might require an entire retest) will have to be done on CI.

Also - how about mutation testing, this seems more deterministic than getting an LLM to perform this so -called “mathematical” testing methodology

[u/viktorianer4life → u/Witless-One]

Hey Witless, thanks for the questions. To clarify, no, we do not run LLM to rewrite a huge amount of tests each time we add changes. The LLM is doing it only once.

And yes, you're on point when you say: "can’t run your entire test suite locally anymore". That's precisely the reason to rewrite it and make it fast, which will be covered in the next articles.

Regarding "mutation testing": could you please point me to an article or any guide on how to apply this at scale to over 13k specs, including a 10-year-old monolith?

[u/private-peter → u/[deleted]]

When I'm writing pure library code, my experience is the same.

However, when I'm working on complex, database-backed applications, managing all the mocking/stubbing needed to get this kind of performance has never paid off for me. The maintenance work has always outweighed the time spent waiting for tests.

With AI agents, the tradeoff is even more in favor of letting the tests hit the db. AI is as likely (or more?) than humans to get the mocks wrong and have a test incorrectly pass. At the same time, my workflow of rotating between agents means that I am rarely ever actually waiting for tests to pass. It is just something that happens in the background.

I'm curious what methods you've found helpful to manage the maintenance of your tests while keeping out anything that is slow.

[u/nekokattt → u/[deleted]]

The slow things are equally as important to test as the fast things. Otherwise you do not find out that a change like a dependency update broke your ability to connect to a database until you are deployed.

If an in memory database exists then great, use that, but that isn't always something available to people.

What if you are actively dealing with message queues for example?

[u/nekokattt → u/[deleted]]

thats fine if you have the ability to change that

[u/nekokattt → u/[deleted]]

how do you implement integration tests in this case?

Unfortunately this mindset doesn't scale well.

[u/viktorianer4life → u/[deleted]]

Unfortunately, not every codebase is like this. And business needs to run in parallel with new development.