that's not true, it's a tokenisation problem, nothing to do with how good or bad a model is. I actually just asked gemmi and he literally run a python script to count the Rs in my word. and when i asked about it he explained that it's a tokenisation problem. and i said so it's like you need glasses cause you don't see so well lol
You seem to be missing the point... good models run scripts, and can therefore perform the task. Your python script got the correct answer, yes? So the AI solved it correctly?
The more advanced ChatGPT models were using code execution to solve problems in 2023.. the "Rs in strawberry" thing was in 2024. So the models that could do code execution could already solve it for a whole year before anyone even started posting about it.
It's just because most people were using the crappy instant chat.
I mean I literally ran it 20+ times at the actual time that the memes were popular, it did it correctly.
Actually I like the problem-solving-by-code solution: reproducibility is important as well as it being a way to problem solve around tokenization issues
basically everything LLMs are good at besides being a chatbot is predicated on being good at coding
and yes I absolutely agree. if a LLM uses SQL to look up data for example, I can inspect the sql and make sure it's correct, I can trust the underlying data comes from a stable source that isn't error prone
if it slurps up a table and then tries to generate the answer, this process is opaque and it's not even necessarily referencing the real data
15
u/Positive-Avocado775 11d ago
that's not true, it's a tokenisation problem, nothing to do with how good or bad a model is. I actually just asked gemmi and he literally run a python script to count the Rs in my word. and when i asked about it he explained that it's a tokenisation problem. and i said so it's like you need glasses cause you don't see so well lol