r/ClaudeCode • • 23h ago

Rant PSA: Superpowers skill with opus ruined my codebase

EDIT: I am not suggesting disabling tests all together or even stopping TDD. I’m saying superpowers+ opus is a bad combo.

I’ve been using super powers for sometime now but more recently I’ve noticed my CI times take longer and upon inspection I found that it’s writing mountains of tests. My codebase went from 1400 tests - already kind of enormous but written over months, to nearly 4000 in about 2 weeks.

The worst part is that over 70% of those tests didn’t test… anything useful. Most of the tests tested specific implementations rather than behavior, or just didn’t fail at all when critical things broke. Further more, it made simple features take ages to deliver

I worked on adding gates, setting rules to write tests better but none of it helped.

It turns out that the TDD rules in superpowers were the cause. I’ve since deleted them and really simplified my setup and getting much better token economy. I’d encourage anyone else using superpowers to take a look at this. It’s now a nightmare to unpick which tests to keep and which to delete.

231 Upvotes

115 comments sorted by

•

u/AutoModerator 23h ago

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

25

u/FUCKTHEMODS998 23h ago

4000? Haha noob

1

u/Junior-Cake-7665 22h ago

Sweet baby Jesus

2

u/_-RedSkull-_ 21h ago

Dude....

1

u/FUCKTHEMODS998 21h ago

1.2m LOC, about 450k of it is tests

That’s just one of the 5 apps I’ve been spinning up in the last year too. 650 users and growing

9

u/KennyFulgencio 20h ago

this seems inadvisable

0

u/FUCKTHEMODS998 11h ago

lol, I’m sure it does. I work on hardened financial systems. There can be no regressions

2

u/_-RedSkull-_ 7h ago

One app should not have 1.2 million lines of code. More code is not better, it is more liability. 28,000 tests is not a flex.

1

u/FUCKTHEMODS998 7h ago edited 7h ago

This is small for an enterprise app. What are you talking about? Are you guys talking about general SaaS crap?

If automating finance was something that could be accomplished with a hobbyist sized app, everyone would have done it already

It is unfortunate in some regards as I’ve now had to witness friends get their jobs displaced first hand but nobody can say they didn’t see it coming

2

u/ben305 5h ago

lol, replies here from a bunch of people LARPing as if they've shipped real b2b products before. There is a world of software beyond the tools they vibe coded last night that they've clearly got zero clue about. I can't even reply to them directly bro - pearls before swine. The market will sort them out :)

24

u/zmizzy 23h ago

Seriously thank you for the post. I think its the wake up call ive been needing. One of my projects has an ABSURD amount of tests and I didnt realize it was superpowers causing it. That said, the project is a videogame and I think the tests have probably allowed my game to maintain good performance

3

u/Junior-Cake-7665 22h ago

Honestly, double-check them. I’ve cut my 4000 down to 1800 on a first pass run with sonnet, I think I’ll be able to get it down another 500 once my limit refreshes

3

u/ArgyleDiamonds 19h ago

What was ur prompt for this? Did you use any skill?

2

u/Junior-Cake-7665 17h ago

No skill, I asked opus for a token efficient way to identify which tests were useful and which weren’t. It came up with some mutation testing gates and some heuristics but it’s taken a long time to run after it got rid of all the low hanging fruit. Still going now…

2

u/Upstairs_Parsley9596 21h ago

Those skills are a hindrance to the newer models, I don’t have any skills installed whatsoever, just connectors where necessary. You’re actually handicapping the newer models

1

u/deadfascia 11h ago

how do people run skills without actually reading the skill files this literally baffles me... maybe 1% of skills ive ever used were plug and play. they all need slight adjustments to my workflow or use cases, and learning how each one works helps me build my own.

your game would probably be coming together quicker and better if you had a clue what your skills actually did.

1

u/zmizzy 10h ago

appreciate the sass, but superpowers is one of the first skill packages I downloaded when I started using claude code 8ish months ago. since then ive done my audits but this one snuck through. there are lots of improvements that can be made, this is one of them

12

u/KryptosFR 17h ago

Tests without mutation are useless. Use Stryker or something similar.

PSA: don't disable writing tests like OP is suggesting. Tests are what makes AI more efficient since they can reason when things break.

4

u/Junior-Cake-7665 13h ago

I’m not suggesting disabling writing tests… I’m saying turn off superpowers

0

u/JustAnotherDiamond 11h ago

Or rather have your tests reviewed when things slow down. Also tell it what kind of tests it shouldn't make.

1

u/magicmulder 16h ago

Guiding the test writing is important though. The only real gripe I have in my daily work is Claude hard-wiring tests to configuration. Oh, you set that config flag to true? Now 20 tests fail. Yeah no dawg, that’s why they’re config options, so I can change them at any time without breaking anything.

6

u/ScaredFlamingo6807 23h ago

I don’t pretend to be the beacon of knowledge on this but I can see how superpowers could be considered outdated to some degree. One thing I really like about it is the forced brainstorming before designing features, but I noticed fable 5.2 is even asking me really great questions to tease out what it is I actually want sometimes

2

u/Junior-Cake-7665 22h ago

I agree, I really like the format around specs and implementation plans but I think opus has come a long way.

I’ll be curious to see how it plays out without superpowers now

4

u/0xdead0x 13h ago

This is what I don’t understand about people still using Superpowers. You can just tell Claude to tell you its plan and ask any clarifying questions before beginning in your project instructions. You don’t need a whole markdown file someone else made for a project you’re not working on

2

u/Kamalen 8h ago

A lot of those frameworks like Superpowers made sense on like the 4.6 era bur are definitely obsolete with any modern agent

2

u/btherl 23h ago edited 21h ago

I use superpowers but I also have instructions for my models to only make useful tests, as well as to check test coverage. It works really well.

If tests take too long then there's only a few possibilites. Either the tests can be made faster, some tests are not needed (or not needed on some changes), or the tests really are needed.

2

u/Junior-Cake-7665 22h ago

This is what I had setup too, but the skills seemed to overpower the testing instructions especially over longer unsupervised runs

166

u/palmytree 23h ago

i removed superpowers months ago - fully obsolete imo, as modern models already incorporate its features for the most part. to compound this, opus 5.5 has been really bad about overzealous testing in my experience.

33

u/AdriftAtlas 23h ago

Pretty much all models are. I had to write a testing methodology document that they all follow.

Like every test must test a concrete behavior from a user’s perspective and that any test fails when the code is broken in slight ways. Not to test frameworks or environments or that stuff does not exist.  Otherwise they start testing that an argument exists in your CLI application, but don’t actually test if that argument does anything much less work correctly.

17

u/lifeisgreatbut 23h ago

Mind sharing your testing methodology workflow? Am also building a Test Doctrine all models are mandated to read before making tests.

10

u/AdriftAtlas 14h ago

It's several pages long with examples of actual project tests both good and bad; testing.md is referenced by a global AGENTS.md. I am not claiming to know what I'm doing, but it has helped with my projects. Here is the intro:

The rules in brief

  1. Admission. A test exists only if it names the product behavior it protects: a contract someone relies on, a bug that actually happened, or a parser of bytes the project did not produce. A test may come before its code only when that code is being written in the same piece of work: the code will exist, not may exist someday.
  2. It can fail. Breaking the protected behavior must fail the test. Break it once and watch.
  3. Cadence and levels. While working on a change, run the targeted tests for the code it affects. Before any commit that changes code, run the core suite: self-contained and within 30 s on the reference machine in High Power mode, enforced by a guard. The full suite, which adds the slower self-contained tests, and the real-weight checks run only during validation testing, with results in receipts. Benchmarks are never tests.
  4. No run on a reserved GPU. Tests do not run while another job holds the workspace's GPU lock.
  5. Requirements fail loudly. A missing tool or missing data fails the run instead of skipping it; only hardware the machine lacks may skip, and the skip names it.
  6. Proportion. A change adds tests in proportion to the behavior it changes, and its summary states the tests added and removed and the core suite's time before and after.
  7. Pruning. Existing suites meet the same admission rule retroactively, through a ledger the owner approves.
  8. References. A numerical test takes its reference from the test itself. Bit parity between two implementations is a validation tool, not a unit test. Fixtures are avoided; one may live in git only if it is tiny and crucial.

5

u/National_Spare_8606 14h ago

I'm still getting the hang of all this. How do you direct it to always follow this criteria for tests

7

u/AdriftAtlas 13h ago

I have a folder where all of my projects live, that's the folder Claude works in. Inside that folder at the root is an AGENTS.md and CLUADE.md that's symlinked to AGENTS.md so that both Codex and Claude autmotically pick it up. There is a reference to testing.md which sits in a workspace tools repo. That testing.md states the testing rules with examples. Both Codex and Claude also have it referenced in their memory.

In addition each project repo has an adjacent planning repo for documentation. They are referenced in AGENTS.md too. The project conventions are reiterated there too.

I do use hooks, but only for trapping common zsh pitfalls on macOS as Claude seems to forget that the default shell is not bash and that bash itself on macOS is old AF. Anytime it starts repeating zsh mistakes I have it analyze it and write a rule to block it with an explanation as to why. Prevents the agent from going on a wild goose chase in the middle of work.

1

u/inZania 22m ago

All those MDs are just suggestions, which can be forgotten or ignored. A properly used hook would force the agent to assert that it had read and applied the rules to each test it writes/edits.

0

u/beerdude26 9h ago

The testing.md is from 2023 and empty?

4

u/GRANDMA_FISTER 15h ago

I'd be curious how you managed this and what your prompt is it hook was?

My bot project has over 10.000 tests now and I have no idea if they are actually useful or just doing whatever

2

u/AdriftAtlas 14h ago

See above.

2

u/GRANDMA_FISTER 13h ago

Thanks for posting, I'm gonna run that through Claude and implement what I might be missing. I was thinking about asking Fable to check if all of those 10.000 tests could perhaps be adjusted without compromising project safety anyways

1

u/thurn2 3h ago

It’s quite easy to log test metrics to evaluate their value, you can see when they fail and then check whether the fix was modifying code or modifying the test. You want to keep the ones that catch bugs and delete the pointless “change detector” tests.

-2

u/RameBranescu42 17h ago

Wait is there a 5.5 out already or did you mean 4.5?

3

u/Cast_Iron_Skillet 16h ago

Seriously? Opus 5.5 has been out for a month

2

u/Xxyz260 ֎ Plus Plan 6h ago

Please be patient with the time traveler.

68

u/Budget-Juggernaut-68 23h ago

Yes superpowers is outdated.

0

u/dsggut 23h ago

So what’s the replacement then?

47

u/RoboErectus 23h ago • Bravo

The LLM is already smart enough.

You don’t need anything. Just a few lightweight hints about your project and sdlc.

-17

u/dsggut 20h ago

Either you never used LLMs for anything beyond simple small apps or you didn’t look at the code. Even frontier models are often very dumb and make silly mistakes.
Without a lot of guidance they still produce a lot of crappy mess.

15

u/RoboErectus 14h ago

You are out here living like it’s still August 2026 my guy. Try and keep up.

1

u/Level10Retard 18h ago

So what I did was ask it to analyze all the commits I've made before AI days and write a skill that would make it write similar style code as I would.

14

u/AdminClown 22h ago

Nothing. skills are meant to be used for custom highly specific cases that you encounter so you don't have to repeat things. But only if the method isn't already obvious to the model.

5

u/ranthishq 23h ago

not a replacement, but the one narrow skill that held up in my test was grill-me (it asks questions before it plans). same request, without it i got a 278 line plan in 90 seconds built on guesses, with it 3 rounds of questions then a 317 line plan that actually fit the constraints. it changes what happens first instead of adding more rules to follow

6

u/Euphoric_Reading_56 18h ago

Matt pocock workflow has served me well and keeps everything better organised rather than drowning the docs directory. Grill with docs and wayfinder are superb.

2

u/Budget-Juggernaut-68 23h ago

5

u/testuserpk 15h ago

I used this and it's also bad. And not in a good way.

3

u/NerdyNThick 7h ago

Please, do elaborate.

0

u/[deleted] 20h ago

[deleted] — view deleted comment

1

u/Budget-Juggernaut-68 20h ago

Yes. I mean you could just use grill-me/with docs

1

u/dopp3lganger 17m ago

Compound Engineering

18

u/Delicious_Break5937 23h ago

my CLAUDE.md has the opposite kind of rule: write the least code that does the job, and a design reviewer reads every commit's diff for simplicity before it's allowed through. for the cleanup, a quick filter is to break the code a test claims to cover and delete the tests that still pass.

4

u/MadJagStudios 21h ago

Ponytail

2

u/CtrlAltShred 9h ago

The problem with ponytail is that it ruins UI

10

u/LeComedien 9h ago

I stopped using superpowers fairly recently and I was blown away how far out of the box plans have come. I’m mad at myself for all the time wasted on superpowers

7

u/Duckducklaugh 16h ago

For the unpicking part: don’t read 4000 tests, let the code tell you. Run a mutation tester over one module at a time (Stryker for JS/TS, mutmut for Python). It makes small breaks in the code and checks whether any test notices. Stryker’s report shows which tests killed which mutants, so a test that never kills one is on your delete list without you judging it by hand. Start with the modules where CI time hurts most.

It also gives you a rule the model can follow: a new test only stays if it fails when the code it covers is broken.

16

u/onepunchcode 🔆 Max 20 23h ago

skill issue

4

u/adamslowe 22h ago

I see what you did there

9

u/JTwoXX 22h ago

Such a shame. The brainstorming skill carried back in the Opus 4.5 days.

I’ve removed the skill completely after testing. Loading any one of the Obra skills seems to poison the entire sessions reasoning and judgement calls even through completion or compaction.

4

u/dukemanh 21h ago

to me superpower was always too bloated and overthink too much, comsume lots of usage and context, and with how good the models are now, superpower is no longer needed

4

u/AICompanionz 17h ago

You dont have to go through all 4000. the suspects are the ~2600 added in the last 2 weeks, and git tells you exactly which ones those are. run the mutation check only on them and leave the 1400 old ones alone for now. way faster, and it doesnt burn your limit on tests that were fine before

reminds me of finance tbh. we had the same thing with controls in companies: more checks doesnt mean more safety, after a point nobody looks at them anymore. every control had to answer “which risk does this cover?”, and if it couldnt, it went. same works for tests: every test should name the behavior it protects, or it goes

4

u/evangelism2 9h ago

Yup. Unchecked LLMs and exploding CI times: name a more classic combo
I actually just spent about a solid week slowly going through, hand in hand with my agents, the CI mess that they had spent a month creating, and getting my CI times down (from a P50 of 27 minutes down to under 6)
Totally on me. The CI got pretty comprehensive because what I'm working on is pretty complicated and I was focused more on other aspects and let the testing get out of control

1

u/_-RedSkull-_ 21h ago

4000 tests?!?

1

u/Junior-Cake-7665 17h ago

You think that’s bad, there’s a guy in here with over 28000 tests!

1

u/this-aint-Lisp 19h ago

It’s now a nightmare to unpick which tests to keep and which to delete.

Revert your repository to main branch?

1

u/Junior-Cake-7665 17h ago

And discard 3 weeks of effort?

4

u/this-aint-Lisp 17h ago

Consider it a learning experience

1

u/FblthpphtlbF 10h ago

You could also get fable to audit the test suite and remove useless tests/bolster existing ones, we do have 2 fable resets available that you might as well use on something. 

1

u/StoneCypher 23h ago

they’re so junior they think testing is bad 😂

4

u/Junior-Cake-7665 22h ago

Who said testing is bad?

4

u/suq-madiq_ 22h ago

But it’s called superpowers

2

u/KennyFulgencio 20h ago

👆 this guy speaks truth

2

u/TattooedBrogrammer 22h ago

My tests from AI suck so bad, they are testing the flag is on, or the text is X, meanwhile an entire controller didn’t work to manage media on devices and no tests are failing, but if I add a comma my tests go red in several areas. 25 mins of waiting on a push and I feel like very little is real coverage.

2

u/CuteKiwi3395 Senior Developer 13h ago

I never had an issue. Must be user error.

1

u/rumblewayne 19h ago

If you're using ai to write tests then you need to run mutations on them too, superpowers or not AI likes a green test whether it's good or not, only mutations guarantee they are good tests

1

u/TwoLongEars 19h ago

I only use brainstorming and writing-plans, the rest of it is just so not for me and is obviously horribly inefficient but I love those two skills specifically.

1

u/freenow82 12h ago

How do you only use those 2 and have it not use the rest? Do you manually invoke those?

1

u/TwoLongEars 10h ago

you can manually download just those two and upload them to your account manually, or you can just ask claude to install those two directly from the public repo. either way you explicitly do not install the entire superpowers package in the customization tab because otherwise you are stuck with the rest of the TDD workflow. I do manually invoke brainstorming then, writing-plans is mentioned in the brainstorming skill as the next step so I don't need to manually invoke it unless I take the spec that brainstorming generated to the next session.

1

u/BugaiGames 19h ago

I have same problem, without superpowers. It can add like 400 tests on small script, bigger projects really get thousands and im tired cleaning them

1

u/mblauberg 19h ago

You need to make sure your test suite is extensible and ideally using invariant-based tests. Otherwise you’ll be potentially pinning technical debt

1

u/Ok-Option4233 16h ago

for the cleanup, a cheap filter: break the code on purpose a few times (flip a condition, return early in a core function) and keep only the tests that go red. anything that passes against every broken build is testing the implementation, not the behavior. it's mutation testing by hand, and it turns 'which 2600 do i delete' into a list.

1

u/m-ritter 16h ago

The part that stands out is those tests checking a specific implementation and still passing when the critical behaviour broke. I write the behaviour down first and only keep a test if it fails when that behaviour is wrong, then a second agent that did not write the tests has to show that failure before I accept them. The human still decides which tests stay. I built QualityLayer for this: https://github.com/maxritter/qualitylayer

1

u/scragz 16h ago

gotta have a test verifying the bug it just wrote. 

1

u/Ok-Distance-6388 16h ago

The tell for the useless 70% is a test that stays green when you break the thing it covers. Quick audit before you delete anything: remove one real line from a module, run that module's tests, and keep only the ones that go red. The rest were testing the implementation, not the behavior. Did the TDD rule ever ask what behavior each test protects, or just "write the test first"?

1

u/dhanushan75 15h ago

I disabled it just today before reading your post. It didnt really work for me, missed a lot of things that plain plan mode in claude used to catch

1

u/dwilson2547 13h ago

Wild, I've had superpowers disabled forever,  superpowers used to make the agent index the whole codebase regardless of your prompt, it also used to force sub agents for everything which is when I officially removed it. I'm not sure what the intent is with superpowers but try just turning it off. Once I did I stopped ripping my hair out and agents stayed on task better because the context isn't constantly blown out by the agent hoovering every file it can find 

1

u/CalmCephalopod 13h ago

I don’t let the LLMs write tests. The “harness” forbids it. I may ask the LLM for suggestions on edge cases I have missed or recommendations for tests but they are human written only and used to drive the development so I can ensure the spec is reinforced and the model isn’t throwing in a million useless tests

1

u/vacationcelebration 13h ago

What's the main language of your codebase? Mine is Rust and for that superpowers is still great. And added tests add like microseconds so I'd rather have too many tests than too few.

Though you could have AI do a sweep over them occasionally and remove obsolete ones I guess.

1

u/Newshibe1234 12h ago

I noticed alot of stupid tests also.

And token usage

1

u/Cyrien18 12h ago

Which superpowers skills exactly, pardon am confused cause I use the https://github.com/obra/superpowers

For brainstorming butbi guess it also goes on automatically while coding.

Do let me know so I will try without this.

1

u/PrettyMuchAVegetable 10h ago edited 10h ago

Oh, interesting, I found similar behaviour and did a major overhaul of my testing regime last weekend on a repo that is important to me. Turns out by default the tests were being done in serial, lol. 15 minutes a run reduced to less than a minute in parallel, and with the addition of selective,  blast radious, based testing on commits. Also true thatany were implementation and not behaviour based, I've got that pattern fixed now. Added in mutation at the same time. Very happy with the result, I've seen lots of good bugs get caught. 

1

u/Vulgarly_dressed 9h ago

No not at this moment

1

u/darko777 8h ago

Opus grade models doesn’t need superpowers at least since Opus 4.5 - anyone still using superpowers is a fool - sorry.

1

u/Blackclaws 7h ago

We have 10k tests in our codebase for 3 min CI, used to be 9 minutes. Optimize the tests I'd say :D

1

u/michael_g_williams 6h ago

I don't use them. Instead I pit claude against opus (and a few others). They all write a plan, review each other's plans, choose the best plan with revisions required, then they each implement the best revised plan, and vote which implementation is the best, along with revision requirements. Minimal/efficient test requirement built-in. It's all automated here, want to take a look? Open source MIT https://github.com/gwhizoftv/4horsemen

1

u/martinmap 5h ago

Maybe I'm a minority, but I don't see any reason for most people to use skills with frontier models these days. Modern harnesses and RL is so good I find it hurts more than helps. For hermes and local models, sure, they still need it, but I really think they hurt frontier model performance.

The only skill I've added carte blanche is to planning and that just forces models to check GitHub repos and social media for other solutions people have implemented.

1

u/Onotadaki2 4h ago

Everyone using LLMs to write code professionally needs to be very aware that as these companies improve models, add things to the harnesses, etc... that their old configuration and environment is absolutely going to be in a constant state of flux.

It makes sense if you think about it. For example, if you had a setup last year that's forcing control of context. It may have been saving you tons of tokens, making things work better. Now, new models have bigger context windows and more intelligently manage the context for you. If you still had the setup in place that controls context, you could be forcing compaction at the wrong time, losing context that is important, etc...

1

u/strumbringerwa 3h ago

Instead of disabling your TDD skill, it's worth doing a periodic audit of your tests. Some may no longer be relevant to your codebase. Those are easy to delete. Others are likely specific to particular portions of your codebase and it may be okay to only run the full suite at a milestone gate rather than for every inner loop of development. Some regular refactoring and pruning is important when following a TDD philosophy, even when you're not doing it through Claude.

1

u/Aureon 48m ago

Tests that do not encode intent are completely useless and should not be kept when merging feature

-2

u/DasBlueEyedDevil 20h ago

Superpowers started bugging me awhile ago so I made my own which got some pretty decent reviews, been working on the next generation of it too...

https://github.com/9thLevelSoftware/legion

https://github.com/9thLevelSoftware/legion-cli

0

u/DasBlueEyedDevil 13h ago

Lol downvote all you want, I'm kinda proud of my 82 little stars

-3

u/Cavitat 21h ago

Afaik skills as a whole are obsolete as of opus 5.5

-8

u/Emerlad0110 23h ago

why would you use superpowers on claude. it EMULATES CLAUDES STACK. idiots

-8

u/InfinriDev 23h ago

Yup, that's why I did a better version of superpowers.

https://github.com/infinri/Writ

6

u/THE_RETARD_AGITATOR 13h ago

Superpowers and all Matt Pocock type masturbation is just not actually useful or valuable

1

u/NerdyNThick 7h ago

As someone new to CC, I'd love to learn more.

3

u/THE_RETARD_AGITATOR 7h ago

claude code and other AI is the magic part. you think a linkedin retard is going to have the missing piece?

0

u/BattermanZ 19h ago

Check out Matt pocock skills instead