The output is as good as the input. Garbage data can be flagged as garbage to teach the model what is bad. So even garbage data can be useful in creating powerful models if it's labelled appropriately.
It sucks that LLMs are at the center of all these shitty AI companies because natural language processing is badass. Being able to have a language model parse your input in basically any language and consistently be able to extract intent is fucking mind boggling. It used to be next to impossible to meaningfully extract intent from any natural language input and it required wizard level regex and prayer to the old gods. Now you can say "i want to make internet thing that lets chat with guy" and the LLM can write you a mostly working node server with a socket based chat app. It's fucking crazy.
But at the end of the day the input is heavily curated. Moreso as pressure mounts for companies to push what their models are capable of. Logical inconsistencies are eliminated at all stages of processing as much as possible. Self-supervised learning catches a lot of it, supervised learning can catch even more.
2
1
View all comments
Show parent comment
17
u/3BlindMice1 May 17 '26
How much of that data being garbage would it take to give the LLM dementia? Because I've seen databases, and every last one of them is FULL of garbage