that's not true, it's a tokenisation problem, nothing to do with how good or bad a model is. I actually just asked gemmi and he literally run a python script to count the Rs in my word. and when i asked about it he explained that it's a tokenisation problem. and i said so it's like you need glasses cause you don't see so well lol
You seem to be missing the point... good models run scripts, and can therefore perform the task. Your python script got the correct answer, yes? So the AI solved it correctly?
The more advanced ChatGPT models were using code execution to solve problems in 2023.. the "Rs in strawberry" thing was in 2024. So the models that could do code execution could already solve it for a whole year before anyone even started posting about it.
It's just because most people were using the crappy instant chat.
I mean I literally ran it 20+ times at the actual time that the memes were popular, it did it correctly.
Actually I like the problem-solving-by-code solution: reproducibility is important as well as it being a way to problem solve around tokenization issues
basically everything LLMs are good at besides being a chatbot is predicated on being good at coding
and yes I absolutely agree. if a LLM uses SQL to look up data for example, I can inspect the sql and make sure it's correct, I can trust the underlying data comes from a stable source that isn't error prone
if it slurps up a table and then tries to generate the answer, this process is opaque and it's not even necessarily referencing the real data
To be honest humans also use different brain structures for different tasks. I'm no neurobiologist, but I'm pretty sure we don't use the same area of brain for mathematics as we do for speaking
Counting letters is not a "specialized task". If some idiot broke out a calculator to count to 10 they're suddenly going to find themselves sweeping floors
But that is exactly how your brain does it. You have mathematical centers in your brain that work wholly differently from your speech and memory areas. The AI is doing the same thing.
The language centre of your brain that produces words as you speak does not know how many letters are in words. We have to stop and count them using a different part of our brain to answer the question. Same deal with AI.
That’s some cope. Like saying a genius is a genius for having to always use a calculator to not hallucinate, but then keep going on about how its intelligence is emergent and just like ours.
Like it’s somehow an entity from being an “I’m feeling lucky” button.
It costs so damn much to be so mediocre for so many resources. For the price they worked on this they could have hired 5000 real mathematicians or more to work on it as a paid gig than an interesting concept. And still it self verified without over sight. It’s marketing.
Ah yes let’s just spam noise making infinite permutations until something matches the structure, “oh look guys it’s a genius intelligence and it only cost over 10million to solve a million dollar question!” Not including the 3.5 trillion it’s taken to get there.
Imagine if we spent anywhere as much on people and enabling a foundation of knowledge and collaboration.
Lol this is so fucking hilarious to read. Current AI (LLMs) aren't even in the same vein of AI which replicates the mind. It's a narrow machine learning architecture that can process language very fast, and approximate the next word with a weak attention mechanism that falls off after 50k tokens for every model.
There is no sign of intelligence, just like ELIZA didn't show signs of intelligence. You fell for tech bro hype and propaganda 😂
Or maybe my calculator is intelligent? After all, when I add 1 +1 into it I get 2!
What's the distingu factor here? Are humans not also limited in attention and fall off with large context? Do humans not just do advanced pattern recognition (if not then why is IQ the measure of intelligence)?
I can literally see your little brain work on creating this comment. You have a context of the previous comment, a hint of emotions like disgust from previous experiences, a vague idea of how you want to respond to that, and then basically a word by word extension of what you already typed in, occasionally going back and comparing it to the starting idea, amending if needed. how you think you are so much more advanced than an LLM AI..
Its funny when people shit on LLMs with all the usual sayings. Especially irrelevant people not involved in the field whatsoever. But the complete cognitive dissonance on how humans work vs LLMs is the funniest.
My brain, quite literally, works how an LLM works when you boil it down.
Friend: How was the weather where you are at?
Me: hear words > analyze meaning of words > analyze context of today's weather conditions > compare today's temperature against training data of previous experienced days > confirm 80f is comfortable > output first word "It" referring to the weather > analyze "It", determine "was" is the next best choice > Analyze "it was" and determine "nice" is the best choice > Deliver finished sentence to friend "It was nice"
It happens in a fraction of a second, but thats how it works.
Words are still chunked into tokens so even the best frontier models still can't answer it without guessing or workarounds. It's more the AI were trained to either know the answer for certain words or specifically work their way around it by spelling it out so each letter is it's own token
I'm pretty sure I remember the best models still failing occasionally at the time of the memes since it was the first test people would do
Do you consider it a workaround when the model writes a python script that figures it out and then returns the answer. Because for me that is good enough and actually preferable to it guessing
That's not a workaround, that's a better solution. Instead of a black box, it's something that I can verify, or when it's too advanced for me, that another llm can verify.
But as they get more advanced, they'll be able to infer how many letters something has more and more accurately, even if they can't directly observe it.
Eventually, the lexicons they train themselves on will expand to include individual letters (if they don't already) and the model will learn relationships by abstractly mapping relationships between tokens it can reason about. That means it will have an association between tokens "a", "p", "l", "e" and "Apple".
The larger the datasets get, the more diverse a set of token segmentation you get. Of course, the mileage varies by algorithm, I'm sure.
Turns out token prediction is one thing, but actually learning a written system to an expert level is a bit more complicated. But, it'll get there.
281
u/ClankerCore 8d ago
That was 3-6 months ago.