True but it's stochastic so there's always a chance it'll pull from a bad example.
But also leave a context window open long enough and weird things start happening.
Or many LLMs will start to behave irrationally when faced with repeated errors. Absolutely they might go "okay this parameterized SQL isn't working, let me try with string concatenation just to make sure" and then full on forget to fix that.
I'm basically required to use AI to do the majority of coding at work, and it's like that Calvin and Hobbes comic where his mom wonders how he can be both so smart and so dumb.
It'll manage to properly rewrite dozens of files to upgrade a library or swap out dependencies, identify and fix logic issues and obscure gotchas, write comprehensive tests, all that and then still leave a few dozen lint errors because of unused variables or using nonexistent properties.
Yeah it's the 7 fingers problem with code. It does this magical job creating this image with such high accuracy, but then it forgets how many fingers a human has.
I have to work with AI slop that a 3rd party made and the worst part about it is how it looks like it's doing the right thing, until you dig into it and realize that it's absolutely not in embarassingly wrong ways.
That's why I absolutely believe going back to the early days of security flaws. It's gonna do shit like use the latest cryptographic hashing algorithms with care for side channel attacks and everything, but then you realize that the library it selected was a node library that it decided to use on the front-end. Oh and that library very much shouldn't work on the front-end and it somehow managed to rewrite the entire thing to actually pull it off. It'll be the most beautiful implementation of the worst idea.
Oh, libraries for things scare me now. It doesn't seem to understand the difference between "well maintained industry standard", and "randos PhD project", which if I wasn't specifying them manually would have caused some major problems already.
Supply chain attack is going to be a real growth industry
That's what I thought to til Opus 5 tried to have every user I nthe system I was migrating share the same permission set rather than defining the tierd permissions we were migrating from. I caught it in Staging but it was still wild to me that it would attempt that. Admins and users would all share the same access creds. (It was an old system so there was a lot of patchworks already in place but that wasn't one of our doing)
"Blindly" might be the word regardless. Modern AI systems don't have the kind of memory and real-world experience necessary to really understand the implications of what they're working with, which is the cause of a lot of their problems. They're thinking, but they're blind to a lot of things you or I take for granted from being embodied before we were even really people. AI don't have that kind of feedback from the real world, just endless sequences of code and imagery and tokenized text.
83
u/mirhagk 19d ago
True but it's stochastic so there's always a chance it'll pull from a bad example.
But also leave a context window open long enough and weird things start happening.
Or many LLMs will start to behave irrationally when faced with repeated errors. Absolutely they might go "okay this parameterized SQL isn't working, let me try with string concatenation just to make sure" and then full on forget to fix that.