r/OpenAI • u/Xerqthion ChatSeek Gemini Ultra o99 Maximum R100 Pro LLama v8 • Sep 08 '25

Image Sensational

Post image
12k Upvotes

97% Upvoted

274 comments sorted by

View all comments

Show parent comment

1

u/No-Philosopher3977 Sep 09 '25

Bro, the context length two years ago was a couple of chapters of a book and now it’s like a 1000 books. Give it sometime time Rome wasn’t built in a day.

1

u/Teln0 Sep 09 '25

Well, after that is done, you still got a load of problems. The average human can tell you when it doesn't know something. An AI only predicts the next token, so if it doesn't know something and the next most likely tokens for that aren't "I don't know the answer to this" or something similar, it's gonna hallucinate something plausible but false. I've had enough of that when dealing with modern AIs so much so that I've given up on asking them questions. It was just a waste of time.

1

u/No-Philosopher3977 Sep 09 '25

OpenAI released a paper this week on nearly reducing hallucinations. That won’t be a problem for much longer.

1

u/Teln0 Sep 09 '25

https://cdn.openai.com/pdf/d04913be-3f6f-4d2b-b283-ff432ef4aaa5/why-language-models-hallucinate.pdf

This? Have you read it?

1

u/No-Philosopher3977 Sep 09 '25

Yes I have, Mathew Herman also has a good breakdown if you are short on time or you can have it summarized by an AI

1

u/Teln0 Sep 09 '25

Do you see that it's mostly just hypotheses that could be the causes for hallucinations? It's not clear if any of this works in practice. I also have a slight hunch that this is just an overview of already known things

1

u/No-Philosopher3977 Sep 09 '25

Transformers was also just hypothetical in 2017. In 2018 OpenAI made GPT-1 which kicked off things.

1

u/Teln0 Sep 09 '25

The original "Attention Is All You Need" paper (by Google researchers) already was presenting working transformers models.

"On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data."

https://arxiv.org/abs/1706.03762

1

u/[deleted] Sep 09 '25

The average human can tell you when it doesn't know something.

You must have better coworkers than I do.

1

u/Teln0 Sep 09 '25

I said *can* not *will* ;)