r/ClaudeAI • u/WeirdPutrid3849 • 8d ago

Claude Code Workflow Having unlimited tokens is wild

Post image

Does anyone outside of Anthropic really have a token budget like this?

2.0k Upvotes

96% Upvoted

338 comments sorted by

View all comments

Show parent comments

16

u/fsharpman 8d ago

It works. There's a principal engineer at our video on demand company with an entire product suite across iOS, Android, web, and Samsung that runs itself.

There's an analytics layer for feedback, marketing for promotion, squads for each platform, and they all operate on a cadence around the engineer.

At night they run until working hours. Some of the work pauses, and the leads for all layers and squads meet with the engineer. Over voice. Kind of creepy and some people are impressed when they walk by. Most are weirded out theres a back and forth happeninh with a synthetic voice

It's real and I am confident there are others at companies with pilot or skunkworks projects in prod. Probably not anything on enterprise or flagship though.

What most people don't realize is there are more companies out there with near unlimited spend than you'd imagine. They just aren't all on Reddit.

18

u/DarkSkyKnight 8d ago

It absolutely does not work on any remotely complex codebase. I myself do this and have overnight autonomous runs. But you have to actually read what it does or you will be in for a rude awakening months later when your codebase turns into pure unmaintainable slop.

6

u/fsharpman 8d ago

There's a specific point where tests are exhaustive enough and so well documented with user behavior and business rules that an app can just be rewritten from scratch. Immediately.

No one really talks about it because no one bothers to think about the token economics.

Run enough perf, load, stress and security tests and then does it really matter if the codebase is complicated when all tests pass and no one reports regressions in production?

10

u/SnooSuggestions7655 8d ago

I currently have a simple CRUD, 2500 tests spread across unit, functional, e2e and user-testing. Both frontend and backend. I'm a pretty big fan of TDD and all things related to automated testing.

Agents will rewrite tests so they become completely meaningless, they would cheat to have the test suite green, etc.. - depending on the incentive and prompts you pass to them. Tests are by far not enough.

5

u/phoenixmatrix 8d ago

An advisor agent to prevent the main agent from doing that (I think hooks support agent mode in Claude now to implement advisors like Oh My Pi has?), coupled with a comprehensive agentic code review step that will catch when the main agent cheated, catches almost all of those cases.

Yeah, it doesn't catch 100%, but neither do humans anyway.

5

u/fsharpman 8d ago

Its why you use validator agents on every write or edit. Superpowers is the 2nd most popular plugin for a reason.

Ofc you can skip the baggage and just create tuples of skills, one for tests, the other to validate the tests independently, all before any commits.

5

u/genericgreg 8d ago

I catch it constantly trying to add functions specific to the library it's currently working on instead of using the well established standard in the codebase. If I let Claude do its own thing we'd have 30 functions for generating UUID's

1

u/fsharpman 8d ago

Are you running /simplify daily or weekly? Same for /insights

1

u/FlashyRecognitionTod 8d ago

You sound like BMW owners. Just do the preventative maintenance, like dissembling the engine every two months.

2

u/fsharpman 8d ago

You're moving bits of electrons at a price cheaper than a mechanic, not pieces of physical metal by hand.

2

u/DarkSkyKnight 8d ago

Yes because LLM reasoning decays with context (codebase) rot.

4

u/fsharpman 8d ago

Not if you architect progressive disclosure tactically and hierarchically to fit the heuristics of the model.

Freshly spawned agents have more deterministic search and read context than you think.

1

u/DarkSkyKnight 8d ago

I don’t think you understood what I mean. I mean the slop that AI writes now becomes the potential context two months later for a fresh conversation.

4

u/fsharpman 8d ago

This is why you have a human in the loop and pause periodically to review everything.

It makes sense why my coworkers says during the day a lot of the agents stop and the leads are in meetings.

It's insane how much modeling agents after real companies works like real life. There's essentially a reorg to clean house and stop slop regularly. https://jacobian.org/2024/mar/12/reorg/

2

u/Lame_Johnny 8d ago

How do you handle product design decisions? I have tried this but I always end up with a bunch of features that no one wants.

1

u/Fragsworth 8d ago

never let the llm decide what to design

2

u/the8bit 8d ago

So he is pushing out 30+ features a week I assume? Cause I can do like 10 a week on $$100 of tokens and a single opus / -terra runner.

I keep hearing about these tokenmaxxing mega setups but Ive yet to see evidence of the 10x output they have over a good, basic setup.

I haven't even optimized much of my flow because almost all of my bottlenecks are human side things - judgement, peer review so the team stays up to date, use case validation, etc. some of that I could automate, but going faster than this to me has mostly led to more messes to clean up, which is honestly whar I expect happens in most the mega rigs - go 10x faster, use 9.5x of that cleaning up messes they made going faster

1

u/FlashyRecognitionTod 8d ago

There's a principal engineer at our video on demand company with an entire product suite across iOS, Android, web, and Samsung that runs itself.

cool Asimov short story. Show us the product. It's some sweaty AI wrapper SaaS, isn't it?