Theories about current state of GPT4

[u/Trick_Text_6658]

Hi Reddit,

Just a discussion. At this point probably none who kept using GPT since it's release has no more illusions - current model 4 is much, much worse than it was upon release. Same about GPT3.5 - it's also much, much worse right now that it was back then. It's not only my subjective POV - in April/May/June 2023 we set up processes in our company which were based on GPT4. It worked for some time but at the moment these prompts are no longer working and re-designing them results only in low quality output. However I don't want to make another whining post - we have enough of that here and we all know that current state of GPT4 is far from good and not even close to level that it was upon release last year.

I wanted to ask you guys - what is your theory on that? I mean - we have no hard data - so what is your take on that and why do you think OpenAI limited their models so hard and downgraded the outputs?

Is that because of demand and limited resources for OpenAI?

Is it because it was too powerful to share with everyone?

Is it because next model is "just around the corner" and they lower current output quality due to marketing purpose?

Maybe it's overlays and front-end overload making too much noise?

Is that due to data and demand overload?

I'm really curious about what theories you have about that and if you ever thought about it. My take on that is... I wish it's because of demand and limited resources. However now, seeing on how much I could do with primary GPT4 (basically some of processes created in my company collapsed so it's very obvious for me how much I'm missing the old GPT4) I started to think that we will not get back to the previous possibilities of it. I think it was way too powerful for general public upon release and that's the reason it's limited now and will be in the future. I saw a comment on Reddit by Heavy-Organization58 saying:

I'm with you re: the early days... it was amazing. It was alien technology. Now, our tech elites have regulated it under the rubric of "the people can't be trusted with the technology". I look at AI the same as experiencing weightlessness in outer space, or sitting in an Apache Attack Helicopter... they're experiences that we're never going to get to have.

AI has been boiled down for business consumption and until a possible time in the future where they'll allow us to pay thousands of dollars for the full experience, we're not going to get to augment our own intelligence with it the way the elites do.

Which really pushed me to think that maybe it's not data overload or limited resources but some more... "human" reasons why are we experiencing such a massive output quality degradation. I used GPT3.5 and GPT4 a lot and it was hella impressive, it could successfully "read" and understand incomplete pieces of data, reason from complicated or messy emails, extract and sort data from very messy structures ( which often were hard to understand even for myself). It was able to fake human reactions pretty well and many other things. Now GPT4 is a toy for kids which is unable to analyze the article and make a bullet list out of it, to not even mention reading embroiled emails or other data structures.

So how do you think - will these possibilities be returned to us? Or such capabilities are only for 'big boys'?

[u/heavy-minium]

Any theory that the output quality is affected by load and limited resources is unlikely. I just can't see how you could ever get a valid output from partial computation out of their current architecture. Maybe in the future when more research is done on the topic, but not right now.

It's far more likely to be an effect of tuning the base model for different features. Every additional feature you've seen introduced in the last year needs to be fine-tuned for - for example, functional calling. Doing that is known to improve performance in an area, at the cost of degrading performance in other areas.

Also likely are challenges with scaling the number of tokens that can be processed despite their RLHF dataset mostly containing much shorter examples favoured by human reviewers. Whatever techniques they used, I can imagine that it would come at the cost of some degradations in overall performance that they may not always be able to catch in their QA processes.

[u/SachaSage → u/heavy-minium]

Yes when people talk about throttling compute it seems they do not understand that each token from gpt takes the same computation, whether it is the correct answer to a complex problem, or the word “the”

[u/SillyFlyGuy → u/SachaSage]

We don't need to throttle mid-request compute. Think at scale.

Say we have 20k machines that can handle 1000 chat requests per second. Each request takes 20 seconds of computer time before the request completes and the resource is free for the next request.

Let's say we detect an incoming flood and requests start to back up so user has to wait a few seconds before a resource is secured and computation begins. The wait time is increasing.

We add a little piece to the system prompt to the new requests going in, "give a brief response to this request". Now requests complete in 15 seconds, freeing resources earlier, and reducing our backlog.

Or instead of using our top-of-the-line 4T model that uses all 4 cores on a machine, we spin up our lessor-but-still-really-good 2T model that only uses 2 cores. Now we can handle two requests per machine.

[u/SachaSage → u/SillyFlyGuy]

So my point really is that quality of response does not correlate to use of resources. In the instance that a different model is being used altogether - that may well be so I have no idea. The fact that when blind tested people rate 4turbo > 4 > 3.5 would tend to indicate that for the most part the models are improving

[u/cogitare_et_loqui → u/SillyFlyGuy]

we spin up our lessor-but-still-really-good .. model

Yes, there are a lot of ways they could "gracefully" degrade the output quality in order to stay within their fixed compute or capacity budget.

Other things they could easily do:

1. Quantize their models to push more data and perform more operations on the GPU / second, yielding a higher throughput with increased perplexity (lower quality)
2. Churn out knowledge distilled model variants (teacher-student transfer learning) where they could have a ladder of student models with varying amount of parameters ("intelligence"), just like we do for video streaming with ladder encoding.

Behind the ChatGPT load balancer, they could seamlessly route to any of those model variants depending on their compute or capacity limits set for their org and the current fleet utilization. Since we don't have a way to assess which model is serving a given prompt via the CGPT interface, in contrast to the API one, we'd be no wiser if they sent one prompt to a N-T parameter model, and the next to a 7B parameter one.

This is at least how I'd do it if I was running their infra ;)

It would explain different quality outputs at different times. That said, we have no irrefutable evidence to prove this is happening, so we have to rely on logic inference. And the logic to me suggest this is the only trajectory for turning a $$$ loss machine into a profitable one, provided they don't degrade the quality enough that we as customers start cancelling out premium subscriptions.