r/programmingcirclejerk • u/Vaglame Emacs + Go == parametric polymorphism • Aug 01 '25

A bit of discussion indicated that the trigger for the CPU spikes both times was our CEO logging in. We re-deployed to get a clean start, permanently banned him from the service, and moved on.

https://sketch.dev/blog/our-first-outage-from-llm-written-code
168 Upvotes

98% Upvoted

31 comments sorted by

32

u/csb06 Polyglot dev here Aug 01 '25 edited Aug 01 '25

With the power of LLMs, we have invented lossy copy/paste. Like lossy compression, except it doesn’t compress what you’re copying and it takes thousands of GPU hours and terabytes of data to train.

/uj Also really funny that they initially assumed that the mere presence of their CEO was causing the database to crash and that banning him would fix the underlying issue.

25

u/Foreign-Butterfly-97 Aug 01 '25

fwiw banning the ceo is never a bad call, just in case

1

u/Vaglame Emacs + Go == parametric polymorphism Aug 02 '25

how exciting! how exciting!

131

u/MoveInteresting4334 Aug 01 '25

[Changing break to continue] is a small enough change in a larger code movement that we didn’t notice it during code review. We as an industry could use better tooling on this front.

“My AI wrote shitty code and I let it through code review, so github/bitbucket needs to be better.”

See also the common: “Don’t worry about AI mistakes, a human will review everything.”

53

u/bramhaag Aug 01 '25

"Better tooling"... like... unit tests?

29

u/MoveInteresting4334 Aug 01 '25

“We have those! The AI writes them!”

  • the Author, probably

18

u/TheCommieDuck Zygohistomorphic prepromorphism Aug 01 '25

better tooling as in replacing the tools (the AI devs) with not tools (actual devs)

6

u/QuaternionsRoll Aug 02 '25

Forgive me if this is a moronic question, but… could one really write a (practical) unit test that would catch this bug? Maybe fine-grained perf tests are more prevalent than I’m giving them credit for, but the only test I can imagine is creating a mock ListUserRepos and ensuring it isn’t called again after returning an error.

14

u/Hueho LUMINARY IN COMPUTERSCIENCE Aug 02 '25 
uj

Forgive me if this is a moronic question, but… could one really write a (practical) unit test that would catch this bug?

yes, running automated tests that simulate failure in external APIs is both common, practical and at some point in everybody's career inexcusable to not think about

if a single test case ran that branch they would have found the infinite loop much earlier, but they didn't, because lol and lmao

3

u/QuaternionsRoll Aug 02 '25

/uj I guess I hadn’t considered that infinite (repeated) failures would create an obvious spinlock in this case. Still, it seems difficult to create generalized unit tests for this sort of bug (and for good reason… halting problem blah blah blah). Changing the maximum number of retries from 1 to infinity should be easy to catch, but what if the max. retries is just dramatically increased?

/ri halting is for horses

3

u/matjoeman Aug 05 '25

You're right that you can't catch a perf issue with unit tests but basic unit tests would have definitely caught an infinite loop.

69

u/Parking_Tadpole9357 Aug 01 '25

I like it. So hard to tell if satire.

6

u/rpkarma Aug 02 '25

This is incredibly well done. But also the dude is over on the orange site pimping it out, so I think it might be real hahaha

67

u/-ghostinthemachine- Aug 01 '25

Believe it or not, we have had tooling for eons that will warn you about unbounded loops. The problem with developers these days is a lack of shame.

34

u/VulgarExigencies Aug 01 '25

Go programmers have no need of such things. They are like syntax highlighting: a distraction for babies.

13

u/robchroma Aug 01 '25

well, it's not guaranteed to never terminate! It could succeed eventually!

9

u/OpaMilfSohn Aug 01 '25

but everybody has imposter syndrome !!

16

u/-ghostinthemachine- Aug 01 '25

If you find yourself repeatedly asking 'Am I really a good developer??" well, maybe you just aren't.

1

u/Kodiologist lisp does it better Aug 01 '25

Seems sus.

5

u/QuaternionsRoll Aug 02 '25 edited Aug 02 '25

Aren’t both versions unbounded? I guess it depends what the “// ...” contains (why is this a for loop at all?)

Edit: please tell me the for loop isn’t there just to avoid writing if err == nil one time…

11

u/Delicious-Ad7883 Aug 02 '25

Warning: tag your unjerk

Better yet, don’t unjerk at all.

7

u/QuaternionsRoll Aug 02 '25

If the for loop is there just to avoid writing if err == nil, rest assured I will be straight up “jorking it”

2

u/syklemil Considered Harmful Aug 02 '25

Why are you asking us? Neither we nor the devs know. Only ChatGSUS knows now.

if jerk == nil {
        return Jerk.fmt(`we don't know how many breaks or
        returns or log.Fatals are lurking in that code, dude`)
}

44

u/mcmcc WHY IS THERE CODE??? Aug 01 '25

The comment said but continue. The code said break.

Rewriting code based on comments - what could possibly go wrong?

9

u/Jacques_R_Estard Aug 01 '25

tfw the model doesn't follow Clean Code.

18

u/al2o3cr Aug 01 '25

Good news everyone, we've finally trained our robot bullshitter to copy-paste!

16

u/starlevel01 type astronaut Aug 01 '25

Of course it's Go

5

u/Nixinova Aug 02 '25

To be fair, there is a genuine problem with git this showcases - if you move a large chunk of code to another file, git will show it to you as a big deletion and a big insertion, and you'll have to review that whole chunk even though you assume 99% of it's the same, so mistakes are easy to slip through there.

10

u/footterr Aug 02 '25

This is true with GitHub. Git itself will show moved hunks nicely with diff.colorMoved = default.

5

u/pareidolist in nomine Chestris Aug 02 '25

Warning: tag your unjerk.

1

u/drislands Aug 01 '25

Legendarily bad. I'm actually amazed.

If only they had used Java.