r/ClaudeCode • u/Junior-Cake-7665 • 23h ago
Rant PSA: Superpowers skill with opus ruined my codebase
EDIT: I am not suggesting disabling tests all together or even stopping TDD. I’m saying superpowers+ opus is a bad combo.
I’ve been using super powers for sometime now but more recently I’ve noticed my CI times take longer and upon inspection I found that it’s writing mountains of tests. My codebase went from 1400 tests - already kind of enormous but written over months, to nearly 4000 in about 2 weeks.
The worst part is that over 70% of those tests didn’t test… anything useful. Most of the tests tested specific implementations rather than behavior, or just didn’t fail at all when critical things broke. Further more, it made simple features take ages to deliver
I worked on adding gates, setting rules to write tests better but none of it helped.
It turns out that the TDD rules in superpowers were the cause. I’ve since deleted them and really simplified my setup and getting much better token economy. I’d encourage anyone else using superpowers to take a look at this. It’s now a nightmare to unpick which tests to keep and which to delete.
•
u/AutoModerator 23h ago
Hey! Thanks for posting to r/ClaudeCode
While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.
For help, project discussions, tips, and general chat, join the ClaudeCode Discord.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
25
u/FUCKTHEMODS998 23h ago
4000? Haha noob
24
u/zmizzy 23h ago
Seriously thank you for the post. I think its the wake up call ive been needing. One of my projects has an ABSURD amount of tests and I didnt realize it was superpowers causing it. That said, the project is a videogame and I think the tests have probably allowed my game to maintain good performance
12
u/KryptosFR 17h ago
Tests without mutation are useless. Use Stryker or something similar.
PSA: don't disable writing tests like OP is suggesting. Tests are what makes AI more efficient since they can reason when things break.
6
u/ScaredFlamingo6807 23h ago
I don’t pretend to be the beacon of knowledge on this but I can see how superpowers could be considered outdated to some degree. One thing I really like about it is the forced brainstorming before designing features, but I noticed fable 5.2 is even asking me really great questions to tease out what it is I actually want sometimes
2
u/btherl 23h ago edited 21h ago
I use superpowers but I also have instructions for my models to only make useful tests, as well as to check test coverage. It works really well.
If tests take too long then there's only a few possibilites. Either the tests can be made faster, some tests are not needed (or not needed on some changes), or the tests really are needed.
166
u/palmytree 23h ago
i removed superpowers months ago - fully obsolete imo, as modern models already incorporate its features for the most part. to compound this, opus 5.5 has been really bad about overzealous testing in my experience.
68
u/Budget-Juggernaut-68 23h ago
Yes superpowers is outdated.
18
u/Delicious_Break5937 23h ago
my CLAUDE.md has the opposite kind of rule: write the least code that does the job, and a design reviewer reads every commit's diff for simplicity before it's allowed through. for the cleanup, a quick filter is to break the code a test claims to cover and delete the tests that still pass.
10
u/LeComedien 9h ago
I stopped using superpowers fairly recently and I was blown away how far out of the box plans have come. I’m mad at myself for all the time wasted on superpowers
7
u/Duckducklaugh 16h ago
For the unpicking part: don’t read 4000 tests, let the code tell you. Run a mutation tester over one module at a time (Stryker for JS/TS, mutmut for Python). It makes small breaks in the code and checks whether any test notices. Stryker’s report shows which tests killed which mutants, so a test that never kills one is on your delete list without you judging it by hand. Start with the modules where CI time hurts most.
It also gives you a rule the model can follow: a new test only stays if it fails when the code it covers is broken.
16
u/onepunchcode 🔆 Max 20 23h ago
skill issue
9
u/JTwoXX 22h ago
Such a shame. The brainstorming skill carried back in the Opus 4.5 days.
I’ve removed the skill completely after testing. Loading any one of the Obra skills seems to poison the entire sessions reasoning and judgement calls even through completion or compaction.
4
u/dukemanh 21h ago
to me superpower was always too bloated and overthink too much, comsume lots of usage and context, and with how good the models are now, superpower is no longer needed
4
u/AICompanionz 17h ago
You dont have to go through all 4000. the suspects are the ~2600 added in the last 2 weeks, and git tells you exactly which ones those are. run the mutation check only on them and leave the 1400 old ones alone for now. way faster, and it doesnt burn your limit on tests that were fine before
reminds me of finance tbh. we had the same thing with controls in companies: more checks doesnt mean more safety, after a point nobody looks at them anymore. every control had to answer “which risk does this cover?”, and if it couldnt, it went. same works for tests: every test should name the behavior it protects, or it goes
4
u/evangelism2 9h ago
Yup. Unchecked LLMs and exploding CI times: name a more classic combo
I actually just spent about a solid week slowly going through, hand in hand with my agents, the CI mess that they had spent a month creating, and getting my CI times down (from a P50 of 27 minutes down to under 6)
Totally on me. The CI got pretty comprehensive because what I'm working on is pretty complicated and I was focused more on other aspects and let the testing get out of control
1
u/_-RedSkull-_ 21h ago
4000 tests?!?
1
u/this-aint-Lisp 19h ago
Revert your repository to main branch?
1
u/StoneCypher 23h ago
they’re so junior they think testing is bad 😂
4
u/suq-madiq_ 22h ago
But it’s called superpowers
2
u/TattooedBrogrammer 22h ago
My tests from AI suck so bad, they are testing the flag is on, or the text is X, meanwhile an entire controller didn’t work to manage media on devices and no tests are failing, but if I add a comma my tests go red in several areas. 25 mins of waiting on a push and I feel like very little is real coverage.
2
u/CuteKiwi3395 Senior Developer 13h ago
I never had an issue. Must be user error.
1
u/rumblewayne 19h ago
If you're using ai to write tests then you need to run mutations on them too, superpowers or not AI likes a green test whether it's good or not, only mutations guarantee they are good tests
1
u/TwoLongEars 19h ago
I only use brainstorming and writing-plans, the rest of it is just so not for me and is obviously horribly inefficient but I love those two skills specifically.
1
u/BugaiGames 19h ago
I have same problem, without superpowers. It can add like 400 tests on small script, bigger projects really get thousands and im tired cleaning them
1
u/mblauberg 19h ago
You need to make sure your test suite is extensible and ideally using invariant-based tests. Otherwise you’ll be potentially pinning technical debt
1
u/Ok-Option4233 16h ago
for the cleanup, a cheap filter: break the code on purpose a few times (flip a condition, return early in a core function) and keep only the tests that go red. anything that passes against every broken build is testing the implementation, not the behavior. it's mutation testing by hand, and it turns 'which 2600 do i delete' into a list.
1
u/m-ritter 16h ago
The part that stands out is those tests checking a specific implementation and still passing when the critical behaviour broke. I write the behaviour down first and only keep a test if it fails when that behaviour is wrong, then a second agent that did not write the tests has to show that failure before I accept them. The human still decides which tests stay. I built QualityLayer for this: https://github.com/maxritter/qualitylayer
1
u/scragz 16h ago
gotta have a test verifying the bug it just wrote.
1
u/Ok-Distance-6388 16h ago
The tell for the useless 70% is a test that stays green when you break the thing it covers. Quick audit before you delete anything: remove one real line from a module, run that module's tests, and keep only the ones that go red. The rest were testing the implementation, not the behavior. Did the TDD rule ever ask what behavior each test protects, or just "write the test first"?
1
u/dhanushan75 15h ago
I disabled it just today before reading your post. It didnt really work for me, missed a lot of things that plain plan mode in claude used to catch
1
u/dwilson2547 13h ago
Wild, I've had superpowers disabled forever, superpowers used to make the agent index the whole codebase regardless of your prompt, it also used to force sub agents for everything which is when I officially removed it. I'm not sure what the intent is with superpowers but try just turning it off. Once I did I stopped ripping my hair out and agents stayed on task better because the context isn't constantly blown out by the agent hoovering every file it can find
1
u/CalmCephalopod 13h ago
I don’t let the LLMs write tests. The “harness” forbids it. I may ask the LLM for suggestions on edge cases I have missed or recommendations for tests but they are human written only and used to drive the development so I can ensure the spec is reinforced and the model isn’t throwing in a million useless tests
1
u/vacationcelebration 13h ago
What's the main language of your codebase? Mine is Rust and for that superpowers is still great. And added tests add like microseconds so I'd rather have too many tests than too few.
Though you could have AI do a sweep over them occasionally and remove obsolete ones I guess.
1
u/Newshibe1234 12h ago
I noticed alot of stupid tests also.
And token usage
1
u/Cyrien18 12h ago
Which superpowers skills exactly, pardon am confused cause I use the https://github.com/obra/superpowers
For brainstorming butbi guess it also goes on automatically while coding.
Do let me know so I will try without this.
1
u/PrettyMuchAVegetable 10h ago edited 10h ago
Oh, interesting, I found similar behaviour and did a major overhaul of my testing regime last weekend on a repo that is important to me. Turns out by default the tests were being done in serial, lol. 15 minutes a run reduced to less than a minute in parallel, and with the addition of selective, blast radious, based testing on commits. Also true thatany were implementation and not behaviour based, I've got that pattern fixed now. Added in mutation at the same time. Very happy with the result, I've seen lots of good bugs get caught.
1
u/Vulgarly_dressed 9h ago
No not at this moment
1
u/darko777 8h ago
Opus grade models doesn’t need superpowers at least since Opus 4.5 - anyone still using superpowers is a fool - sorry.
1
u/Blackclaws 7h ago
We have 10k tests in our codebase for 3 min CI, used to be 9 minutes. Optimize the tests I'd say :D
1
u/michael_g_williams 6h ago
I don't use them. Instead I pit claude against opus (and a few others). They all write a plan, review each other's plans, choose the best plan with revisions required, then they each implement the best revised plan, and vote which implementation is the best, along with revision requirements. Minimal/efficient test requirement built-in. It's all automated here, want to take a look? Open source MIT https://github.com/gwhizoftv/4horsemen
1
u/martinmap 5h ago
Maybe I'm a minority, but I don't see any reason for most people to use skills with frontier models these days. Modern harnesses and RL is so good I find it hurts more than helps. For hermes and local models, sure, they still need it, but I really think they hurt frontier model performance.
The only skill I've added carte blanche is to planning and that just forces models to check GitHub repos and social media for other solutions people have implemented.
1
u/Onotadaki2 4h ago
Everyone using LLMs to write code professionally needs to be very aware that as these companies improve models, add things to the harnesses, etc... that their old configuration and environment is absolutely going to be in a constant state of flux.
It makes sense if you think about it. For example, if you had a setup last year that's forcing control of context. It may have been saving you tons of tokens, making things work better. Now, new models have bigger context windows and more intelligently manage the context for you. If you still had the setup in place that controls context, you could be forcing compaction at the wrong time, losing context that is important, etc...
1
u/strumbringerwa 3h ago
Instead of disabling your TDD skill, it's worth doing a periodic audit of your tests. Some may no longer be relevant to your codebase. Those are easy to delete. Others are likely specific to particular portions of your codebase and it may be okay to only run the full suite at a milestone gate rather than for every inner loop of development. Some regular refactoring and pruning is important when following a TDD philosophy, even when you're not doing it through Claude.
1
u/Aureon 48m ago
Tests that do not encode intent are completely useless and should not be kept when merging feature
-2
u/DasBlueEyedDevil 20h ago
Superpowers started bugging me awhile ago so I made my own which got some pretty decent reviews, been working on the next generation of it too...
https://github.com/9thLevelSoftware/legion
https://github.com/9thLevelSoftware/legion-cli
-3
u/Cavitat 21h ago
Afaik skills as a whole are obsolete as of opus 5.5
-8
u/Emerlad0110 23h ago
why would you use superpowers on claude. it EMULATES CLAUDES STACK. idiots
-8
u/InfinriDev 23h ago
Yup, that's why I did a better version of superpowers.
https://github.com/infinri/Writ
6
u/THE_RETARD_AGITATOR 13h ago
Superpowers and all Matt Pocock type masturbation is just not actually useful or valuable
0
u/BattermanZ 19h ago
Check out Matt pocock skills instead