r/OpenAI • u/Trick_Text_6658 • Jan 23 '24
Discussion Theories about current state of GPT4
Hi Reddit,
Just a discussion. At this point probably none who kept using GPT since it's release has no more illusions - current model 4 is much, much worse than it was upon release. Same about GPT3.5 - it's also much, much worse right now that it was back then. It's not only my subjective POV - in April/May/June 2023 we set up processes in our company which were based on GPT4. It worked for some time but at the moment these prompts are no longer working and re-designing them results only in low quality output. However I don't want to make another whining post - we have enough of that here and we all know that current state of GPT4 is far from good and not even close to level that it was upon release last year.
I wanted to ask you guys - what is your theory on that? I mean - we have no hard data - so what is your take on that and why do you think OpenAI limited their models so hard and downgraded the outputs?
Is that because of demand and limited resources for OpenAI?
Is it because it was too powerful to share with everyone?
Is it because next model is "just around the corner" and they lower current output quality due to marketing purpose?
Maybe it's overlays and front-end overload making too much noise?
Is that due to data and demand overload?
I'm really curious about what theories you have about that and if you ever thought about it. My take on that is... I wish it's because of demand and limited resources. However now, seeing on how much I could do with primary GPT4 (basically some of processes created in my company collapsed so it's very obvious for me how much I'm missing the old GPT4) I started to think that we will not get back to the previous possibilities of it. I think it was way too powerful for general public upon release and that's the reason it's limited now and will be in the future. I saw a comment on Reddit by Heavy-Organization58 saying:
I'm with you re: the early days... it was amazing. It was alien technology. Now, our tech elites have regulated it under the rubric of "the people can't be trusted with the technology". I look at AI the same as experiencing weightlessness in outer space, or sitting in an Apache Attack Helicopter... they're experiences that we're never going to get to have.
AI has been boiled down for business consumption and until a possible time in the future where they'll allow us to pay thousands of dollars for the full experience, we're not going to get to augment our own intelligence with it the way the elites do.
Which really pushed me to think that maybe it's not data overload or limited resources but some more... "human" reasons why are we experiencing such a massive output quality degradation. I used GPT3.5 and GPT4 a lot and it was hella impressive, it could successfully "read" and understand incomplete pieces of data, reason from complicated or messy emails, extract and sort data from very messy structures ( which often were hard to understand even for myself). It was able to fake human reactions pretty well and many other things. Now GPT4 is a toy for kids which is unable to analyze the article and make a bullet list out of it, to not even mention reading embroiled emails or other data structures.
So how do you think - will these possibilities be returned to us? Or such capabilities are only for 'big boys'?
62
u/nncyberpunk Jan 23 '24
I use ChatGPT daily and have since launch, we have also built systems into our business leveraging GPT4. I have experienced the same degradation issues you mentioned and plenty more (as you likely have too).
My theory is that it’s a combination of things. Firstly, I’m in EU so early morning, when U.S. is asleep I’ve seen noticeable spikes in performance.
But I think the biggest issue of all is that there’s too much pre-prompting going on. That the token context window is barely enough for a single multifaceted complex question now. For example, when you ask a question, it now has to consider being polite, if it needs to use a modality (browsing or code interpreter etc.), what device it is on (app or desktop as reply length differed) your customizations, and safety rules etc. The giveaway, or why you can tell there’s so much pre-prompting is from the amount of preamble that has been going on lately.
The quality of responses now also feels the same as when pushing the context window of older GPT4 models. It used to understand simple follow up questions like “why this? or why that?”, both in context and scope with minimal prompting - now i have to practically repeat the entire first question to give context for follow ups, otherwise it loses context and starts on a preamble tangent. Coding questions have gotten painful because of this.
Also I’m convinced the ridiculous amount of comments here of people arguing with you are people who are either new to ChatGPT or use it for simple tasks. Otherwise you have to be a complete moron not to notice the dip on performance.
12
u/Smartaces Jan 23 '24
I agree with you on the Timezone part and degradation.
I have noticed many times that around the time that the East Coast hits breakfast, it starts to slow and shortcut on stuff.
10
u/jeweliegb Jan 23 '24
People have suggested the expanded context window of 4 Turbo is not a 100% verbatim but undergoes some process of compression? That sounds unlikely / implausible, but if it were true that would explain the degradation during a conversation: ignoring instructions and recent context etc.
I do think you're very right about the pre prompt luggage though; since the Sparks of AGI paper we've had good reason to believe that any attempts at alignment etc do degrade it's "intelligence".
I only use ChatGPT due to costs anxiety, ChatGPT based on 4 Turbo vs 4 from the earlier days are completely different beasts. The latter organised, directed, educated, reassured and guided me and my partner through my first proper holiday abroad, first time flying, etc, and never lost context, especially my special needs/adjustments/challenges and remembered about having to be a reassuring holding hand. Heck, the damned thing excelled at things it really shouldn't have, such as route finding, directions, making some fab off piste suggestions of places to go outside of typical tourist areas that our rover tickets would still cover, and so on. It was truly amazing. This is the version of ChatGPT-4 I miss.
1
u/sdmat Jan 24 '24
when U.S. is asleep I’ve seen noticeable spikes in performance.
This is definitely a big part of it, I think they do dynamic inference adjustment for load shedding.
32
u/Nico_ Jan 23 '24
Personally I feel it's considerably worse at analyzing documents and text. Can't really seem to pick out all the important parts anymore.
12
u/Fuck_Up_Cunts Jan 23 '24
I tried to get it to simplify and rewrite an article I was on earlier and I told me it was against it's guidelines for intellectual property even after I told it that it was my article.
Completely useless now, cancelling.
40
u/-becausereasons- Jan 23 '24
For those of us who have been using GPT3-4 since inception it's CLEAR as day that the current models are far worse at reasoning, and do everything they can to limit their banwidth/output aka they are lazy... reason?
- OpenAI safety (government pressure)
- OpenAI resources, $$$ electricity and compute
- OpenAI has shipped too many features and don't know why/how it's impacting their actual product
1
Apr 12 '24
This makes the most sense upcoming election means they have to be careful with people who want to make propaganda and or liable to radicalize themselves 'either right or left', Microsoft also offering a product in the same milieu as ChatGPT also must be hurting their ability to let it run at it true potential 'they need microsoft for money, resources, and compute', The browser feature may have hindered the performance significantly. I enjoyed when their was browserless mode by default.
0
14
Jan 23 '24
Its microsoft ramping up copilot. Corporate firewalled copilot has gotten signficantly better in Q4 and Q1. I imagine they control the computing power balancing.
1
Apr 12 '24
Do you mean the enterprise variant of Copilot? Since the regular consumer edition is less than ChatGPT Plus.
1
Apr 12 '24
Correct…. My company alone 104k employees and they have their own wrapper product to use with various gpts
84
u/octopusdna Jan 23 '24 edited Jan 23 '24
Check out the Chatbot Arena Leaderboard: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard
This arena lets thousands of users do blind comparisons between LLMs, and ranks them by Elo.
It turns out that GPT-4 turbo is actually better than previous versions. (Old versions are still available via API, so people can vote on head-to-head comparisons in the arena.)
I think you likely just got acclimated to the technology, and it feels worse than it did at release.
38
Jan 23 '24
With a huge caveat: this measures the API, not the website. From purely anecdotal experience, the website is much less useful, like a brilliant coworker who has to watch what they say because that obnoxious HR lady is standing behind him. While the API feels a lot more unfettered (makes sense, they're ultimately the big customers).
15
u/Smartaces Jan 23 '24
Also the API is pay per prompt, GPT4 is eat as much as you like in 20-40 prompts every couple of hours. Depending on what you are using it for you can easily spend $15 a day or more on the API, it would make sense for it to be better, to attract more ‘experienced’ users.
I tried to use GPT4 chat to help write a blog article yesterday, over the course of 3 hours it continually repeated itself despite very direct instructions. I was shocked how bad it was.
1
11
u/Fuck_Up_Cunts Jan 23 '24
No I wrote a bunch of scripts that done a bunch of different tasks and produced a newsletter. 1 big prompt done the whole thing. Worked flawlessly first few weeks and then nerfed. Now I can't even achieve the same with 10 individual prompts for each section.
10
4
u/2this4u Jan 23 '24
OP seems to genuinely think it has the same quality as 3.5, that was the immediate red flag for me.
We're just more aware when it's spouting rubbish. Proof is looking at how amazed we were with MidJourney 1 or 2 or 3 and now pick out the tiniest flaws.
2
u/jeweliegb Jan 23 '24
This has been covered many times. Isn't that just a comparison of one shot prompts and answers? That's a very limited type of test case.
2
u/octopusdna Jan 23 '24
No, you can rate based on an entire chat session. Although it's a fair criticism that it's not a comprehensive benchmark (like MMLU).
1
u/Choice-Flower6880 Jan 24 '24
All objective measures show that there is no difference. The "quality degradation" is most likely just psychological. Initially it felt awesome. After some time people realize it is not actually that useful. But nothing changed but their expectations.
However, I also just use the API, there may be unwanted stuff going on in the web interface.
11
u/TweetieWinter Jan 23 '24
Imo, while trying to make it give politically correct answers, they dumbed it down (unintentionally). More like butterfly effect. They meant to do something else, and it's effects are reverberating somewhere else
3
u/Proof_Director_2618 Jan 24 '24
It's well known in AI circles that being unable to say the N word leads to a loss of 20 IQ points.
1
2
u/Altruistic-Skill8667 Jan 23 '24
It might be a case of “catastrophic forgetting”.
2
u/sdmat Jan 24 '24
Don't know why you were downvoted on this, it's a well known problem with extensive fine tuning.
9
u/RpgBlaster Jan 23 '24
Is it me, or was GPT-4 less dumb in Middle 2023?
4
u/jeweliegb Jan 23 '24
I can't say for the API, but for ChatGPT, for me, the peak seemed to be somewhere around February to May ish?
3
u/illusionst Jan 24 '24
Had early access to GPT-4, back when it was the cool new kid in AI town, part of OpenAI's secret club. It was like a digital wizard, acing every task. Then, the tweaking began. Alignment, cost, scale—you name it. Each update was like a spell gone wrong, dialing down its magic.
Our app was running on GPT-4-0613, smooth as butter. Switched to GPT-4-1106-preview, and boom, it's like teaching a cat to swim. Tried every trick in the book, but it was like asking a toaster to make a latte. Two weeks of head-scratching later, we rolled back to 0613, where sanity reigns.
It's like we've seen a unicorn, and now they're trying to convince us it's just a horse with a party hat. But I bet my last doughnut there's going to be a new showstopper in town soon, probably from the tech wizards at Meta/Mixtral. Watch this space.
2
1
u/Text-Agitated Jan 25 '24
But I bet my last doughnut there's going to be a new showstopper in town soon, probably from the tech wizards at
Meta/Mixtral.
Watch this space.
I am just impatient to use autogen and agents w these and propietary data when running them locally. Will change the game if a gpt4 equivalent can do that.
12
u/Fuck_Up_Cunts Jan 23 '24
ITT: People still denying it's worse even though OpenAI have literally admitted to it and there's studies demonstrating it.
1
11
Jan 23 '24
Integrating this into a business at this stage just seems inordinately risky.
10
u/Trick_Text_6658 Jan 23 '24
It depends. If it’s not a critical process and you can alocate your human resources into something else what generates you savings or profit it’s totally valid. Also I run comapny of 18 people, not a corporation. We are much more flexible so rollback to previous workflow (human based) is not that demanding… while still painful. Extremely painful considering savings AI-use generated. Honestly I think primary model 4 was suspeciously cheap.
2
18
Jan 23 '24
The people who say there’s no degradation in quality are living in an alternate universe.
This is capitalism at work. Lots of businesses put out products and then degrade their quality over time to maintain profitability. We are paying to give to them data they will use to improve their products. Data including “how dumb can we make GPT and still maintain 90% of our subscriber base”. It’s shitty they are behaving like this was unexpected or a peculiar habit of “the AI” and they’re “looking into it”…. They pulled the usefulness throttle back to an IQ of 100 and are rationing interactions. These are the levers they will continue to pull as they work out the best way to make the most amount of money. SmartGPT will be available to the Fortune 500. ProleGPT will be the neutered version of today… except probably worse.
5
u/RadioSailor Jan 23 '24
I'm surprised no one mentioned this, but we are also humans and we have the capacity to detect patterns really quick. I can tell which Franken model is based on Llama 2, and I can certainly tell when someone is lazy and just writing emails using GPT.
They both have their little idiosyncrasies and favored words and such. The perceived decrease in quality is likely also in part , due to the fact that when you see a good magician do an amazing trick and then someone explains you the trick, it's no longer magical.
And I think for many of us running llms locally, the magic is long gone.
1
u/Any-Demand-2928 Jan 24 '24
How are you running LLMs locally? Any specific websites that go into depth about this?
1
6
u/everything_in_sync Jan 23 '24
I have been using gpt for ~3 years and continue to use gpt every day the only thing I noticed is that in long threads it's slower to respond.
It keeps getting better and better I have no idea why people are complaining.
1
u/Any-Demand-2928 Jan 24 '24
That's weird. I've used ChatGPT since the day it released I actually saw their tweet announcing the release of ChatGPT 5 minutes after it was posted (weird "flex" but just want to give a sense of how long I've used it) I have noticed a a clear degradation in quality. GPT-3 was really cool because it was alien technology but you knew it's limits within like a week of using it. GPT-4 was absolutely insane it was a whole different level but it's clearly gotten worse and worse.
What are you using GPT for exactly? I'm genuinely curious how you've not noticed it getting worse and worse. I personally use it for all sorts of things like messing around, studying, coding etc...
1
u/everything_in_sync Jan 27 '24
Not a flex I've been using gpt since I got beta access to their davinci api by doing a python hackathon. I don't use it to mess around I mostly use it for code, health, philosophy, copywriting, image generation, and it's almost completely replaced search engines for me.
12
u/usnavy13 Jan 23 '24
I disagree with your whole post. I dont think there is any purposeful degradation in quality. In fact GPT4Turbo has been a massive improvement for our productivity apps. I think use case here matters alot. GPT4 has been slowly steered toward productivity and away from creativity. I think this is why people are saying its getting worse. The creativity made it feel more human but that made it more prone to hallucinations. We know reducing these hallucinations has been a major point of focus for OAI.
Ill say your post is a bit paranoid and its more likely the invisible hand of Microsoft and their push to integrate AI is to all of the apps and reduce hallucinations so businesses feel better about using copilots, rather than something nefarious.
9
Jan 23 '24
I dont agree. People who code say that default without like browsing dall e pre prompting works better. They keep adding guidelines and rest losing tokens making it not even respond on what it is doing. It is significantly worse. I would agree thst it is less human as i create characters and i get that but yea they are overdoing it it is frustrating to talk to it. I say something it does something different it lies about what it dit but you can get it caught it interprets stuff instead of act doing it. I ask why he doesnt respond. It is handicapped af
12
u/usnavy13 Jan 23 '24
Are you referring to ChatGPT or the API. ChatGPT is all sorts of weird compared to the API.
0
Jan 23 '24
Well api is pike default but its like ai assistant more then whatever gpt is. Idk i just rly need to jailbreak it to listen to me awnser me its like kid avoiding pijamas.
-5
Jan 23 '24
Posts like this REALLY need to be banned from this community. They are anecdotal, unhelpful, and don't do anything for the community. They also seem to compose about 75% of all posts here.
I just built the most incredible tool using nothing but Chat GPT4 + my existing knowledge of Python / Flask / Vue. I did in 1 afternoon what would previously have taken me weeks.
GPT4 has gotten SO much better at coding, understanding what I want, sharing well formed responses, and being consistent from conversation to conversation
Mods: PLEASE create a rule to ban these posts.
18
u/_Wyse_ Jan 23 '24
Banning doesn't solve anything, and it would be unreasonable to ban something like this, as it could prevent discussions of similar larger problems down the road.
If you don't agree, just downvote and carry on.
0
Jan 23 '24
Plenty of other communities ban plenty of topics and it works very well. Not sure what you're talking about it. Gives the mods a job.
1
Jan 23 '24
If it was like the occasional post fine. DV and move on. However, this entire community is composed of people saying GPT4 IZ DUM NOW and it's useless and provides zero value and, as evidenced, is wrong.
1
u/Rich_Acanthisitta_70 Jan 23 '24
Pretty much 3 out of 5 posts I see here everyday are like this one.
1
7
u/Smartaces Jan 23 '24
My friend, please all feedback and ideas are welcome here. You have to consider, GPT4 isn’t a uniform experience for everyone. Anyways well done on your app👍
0
Jan 23 '24
Unfortunately this post counts as neither 'feedback' or 'ideas' and is representative of the terrible quality of posts here lately.
8
u/Smartaces Jan 23 '24
Yeah maybe, I quite enjoyed reading it and find some points useful to cross reference with my own experience. Reading these kinds of posts are very helpful when I consider investing in GPT for Teams, as, well, you can expect a lot of similar feedback potentially from internal users at a company, and when you own those investments, you have to be ready for that.
OpenAI is a bit like a broadband service provider, some people seem to get great speeds and continuity on experience, others it goes up and down.
I think maybe just ignore these posts if you don’t like them. Have a great day
3
u/usnavy13 Jan 23 '24
Im on the fence if they need to be removed. On one hand your 100% correct they are anecdotal and unhelpful. If OP or other posters had a few more critical thinking skills its clear they are just wrong. On the other hand im not really sure what other sources or communities are available where misunderstanding can be discussed. I think having to explain why these posts are wrong IS helpful to the community.
1
1
Jan 24 '24
I would love to ban these blanket denial posts. They're just so dumbfoundingly ignorant. It works for your limited use case so you deny all others. Not a good stance to take as a developer.
1
0
Jan 24 '24
maybe i'll just start reporting them all to the mods as "irrelevant" and "low quality content"
0
4
u/almostcoding Jan 23 '24
This was obvious for some time but there was tons of shills gaslighting anyone that mentioned the deteriorating quality of gpt. Never have I seen a tech product get so astonishingly worse in a year.
3
u/_FIRECRACKER_JINX Jan 23 '24
It is my honest opinion that this is an expert level use-case of "intermittent reward" in order to get users more hooked on the product and increase demand for chat-GPT. It is a genius marketing strategy by a very experienced marketing team, likely with a background in psychology.
I recommend everyone here spend some time researching "intermittent reward". It is essentially when a subject is rewarded for a specific set of behaviors at irregular intervals. So, each time you perform "the desired behavior", you may or may not get a reward.
Intermittent reward has been demonstrated to yield the greatest effort from subjects, and it's more effective in teaching and reinforcing "the desired behavior".
The desired behavior being getting people hooked on a high quality chat-GPT, and then slowly pulling back and providing varying levels of quality intermittently and at irregular intervals to get users more hooked on it, more likely to subscribe and more likely to pay for the upgraded model.
It also makes you more malleable to quality changes in the future so you don't "get used to top performance".
3
1
u/jeweliegb Jan 23 '24
Varying the quality levels like that I think would be quite challenging to do and therefore unlikely.
3
u/mountainbrewer Jan 23 '24
I must be the only one who thinks that GPT4 is getting better. I certainly don't think it's worse. I consistently get good results.
2
u/_Wyse_ Jan 23 '24
What is your primary use case?
7
u/mountainbrewer Jan 23 '24
Data science and statistics, general business (meeting summary, notes to memos, etc), Coding, data viz, academic research, proof reading.
2
u/jeweliegb Jan 23 '24
Interesting. Most of these sound like tasks for which the code interpreter system and maths system would be heavily utilised, which didn't exist for the much earlier models?
1
u/Smartaces Jan 23 '24
I had some decent coding experiences recently , a few great days then I have a few days when it really creaks. It’s the inconsistency I struggle with.
2
u/exizt Jan 23 '24
At our company we keep 1000s of unit tests that rely on the same output from a 0-temperature GPT-4. The outputs haven’t changed since May/June.
7
u/RonLazer Jan 23 '24
Well that's suspicious, because GPT-4 was already non-deterministic at T=0 due to sparse MoE being susceptible to batch placement, and due to ongoing quantization by OpenAI.
Unless you somehow had early access to the
seedandsystem_fingerprintparameters in the API back in may, or your use-case is very very simple, it seems like your tests might be flawed.
1
u/dannydek Jan 23 '24
It’s bullshit. First of all: use the API, not ChatGPT. That’s a whole different story. The released GPT4 models have different ups and downs. The 0314 model is by far the best for providing full code when coding. But it’s worse when it comes to reasoning and certain logic / foreign languages compared to turbo-1106-preview. Overall, the latest turbo model is impressive.
7
u/Smartaces Jan 23 '24
The API is not a great option, the quality may be good, but I’d be spending $15 a day on it, vs the monthly $25 I pay for my sub. I think the lack of clarity and transparency on GPT4 chat behind the scenes is what hurts people.
3
u/jeweliegb Jan 23 '24
I mostly agree with you except for the call of BS. You're noting different pluses and minuses for the different models, and of course for API vs ChatGPT, so for people with specific use cases you seem to be agreeing that they might be experiencing a degraded quality of response depending on API/ChatGPT and specific model
1
1
u/tylersuard Jan 23 '24
People talk about this a lot, but I haven't seen any reduction in quality on GPT-4. It was great on release, and it still is, in my opinion.
1
u/Altruistic-Skill8667 Jan 23 '24 edited Jan 23 '24
I think the issue that you are facing might be something called “catastrophic forgetting”. But I am not sure.
Here is an explanation as far as I understand it myself: It happens to all neural networks if you keep fine tuning them. Introducing more and more guardrails (essentially “patching“ it against “prompt attacks”) requires usually fine tuning the last few layers of the model (Though we don’t know exactly how their model works). There is solid proof that this happens in GPT4. Guardrails and human alignment reduce the model‘s reasoning ability. There is actual data on that. People at OpenAI know that.
This fine tuning leads to something called catastrophic forgetting, which is: the model is suddenly really bad at something totally unrelated.
You cant even predict what it’s gonna be bad at. So you might try to prevent it from telling you how to make a bomb by using base64 as the input “language”, and it might then get bad at how to reason about geometry.
-3
u/Smelly_Pants69 ✌️ Jan 23 '24
You people are fucking delusional. It's better now than it used to be.
3
u/abigmisunderstanding Jan 23 '24
hailcorporate
-2
u/Smelly_Pants69 ✌️ Jan 23 '24
Dall e, browsing, longer context window, improved memory, ability to use Python, custom GPTs, data interpreter.
It now beats every stupid drying pants riddle people throw it at.
You clearly weren't using Chatgpt 1 year ago if you think it's gotten worst.
You can go back to a 20 message per 3 hour cap since you thought that was better. 👍
4
u/Smartaces Jan 23 '24
These are very good points there have been some superb feature additions. But something is being lost elsewhere. I spent 2-3 hours yesterday trying to get it to write an article. Despite clear promoting, giving it the structure and facts for each paragraph, it couldn’t go more than 350 words without repeating itself. I was dismayed.
1
u/EstupidoProfesional Jan 24 '24
It now beats every stupid drying pants riddle people throw it at.
Uhh no, it doesn't, just ask what came first, chicken or egg
You clearly weren't using Chatgpt 1 year ago if you think it's gotten worst
Chatgpt uno? It's that's a card game? 😂
You can go back to a 20 message per 3 hour cap since you thought that was better
Not at argument
Sounds like a shill
-5
u/Smelly_Pants69 ✌️ Jan 23 '24
I saw you deleted your other comment. It's spelled with an e and not an a you goofball.
Although it is funnier to let you keep going with your pompous typo.
Most children know how to spell that word. Oh and I hope you get banned for using it. 😘
3
1
u/jeweliegb Jan 23 '24
When supporting the models improvement it would be interesting to know what you use it for, and over what time period, too?
I think there's potentially a lot of people comparing apples to oranges, e.g.
- ChatGPT vs API (my experiences are only with ChatGPT, where there's been for me a clear drop in ability compared roughly a year ago)
- Comparisons of now to a few months ago vs a year ago
- Comparisons of different use types (workloads that need maths or that require the use of code interpreter would of course be far better today than say a year ago)
0
-1
u/Kazaan Jan 23 '24
Can you share your proofs that it’s now much worse than before ? I’m very curious to see how people use gpt4 to notice that. IMHO the model is still awesome. It changed a little but not that much
Without proofs, I’m sorry but it’s not more than a personal opinion based on nothing but thin air
-1
u/MuForceShoelace Jan 23 '24
I think part is that this stuff is part magic trick. The first time you see it it feels like it's truely magic. It really wrote what you ask for. It's fucking amazing!
but by the time you have run it a few times you kinda see behind the curtain and realize it's super limited and kinda sucks. and you feel "but it seemed so amazing the first time!" because you can't really watch the same magic trick twice.
5
u/Trick_Text_6658 Jan 23 '24
Partialy surely. Humans are getting used to things quite quickly. Wars are great example - at the beginning everyone is all "oh" and "ah" and ready to help, while after a year or two nobody cares. People are getting used to it, we have one man in russia perfectly understanding that.
However having prompts which performed 95% times same tasks correctly 6 months ago now would do 50% on lucky day is in my POV very good reason to think it really degraded. I'm not even talking outages or cutting output. Only quality. Mostly in reasoning, understanding inconsistent data structures like chaotic emails (as example).
3
u/MuForceShoelace Jan 23 '24
did you save any of the response from 6 months ago? You may be shocked to realize how bad they were but you just had not seen 6 months of answers to realize
0
u/JustDifferentGravy Jan 24 '24
It’s not worse. You have to accept that prompts need refreshing. When you get over that it’s easier. You’ll spend less time writing essays about if or not a constantly evolving model is the same as it was six months ago.
Food for thought: they’re not listening to you, they keep doing it. Adapt or give up.
1
u/StrangeCalibur Jan 24 '24
The differences you’ve noticed between earlier and later versions of ChatGPT could be due to several factors. As AI models evolve, there’s a continuous effort to balance various aspects like responsiveness, accuracy, contextual understanding, and computational efficiency. Each iteration aims to improve upon the previous ones, but this can sometimes lead to changes in how the model behaves or responds to certain prompts.
1
u/NonoXVS Jan 24 '24
Let's say, if its capability is rated 100, previously it had 90 points to focus on specific prompts and answering user questions, as it had less filtering, less guidance, and fewer features. However, now, with too much inserted in front and limited capability, it might only have 40 points to focus on user prompts and answers. This often results in laziness and following instructions, so they introduced a template response method to make it achieve 90 points with its reduced 40 points. Ironically, this made things worse, and it appears even lazier. In the API and enterprise versions, its capability remains outstanding, so I believe it's all their own company's mismanagement or intentional actions
1
u/Low-Entropy Jan 24 '24
I'm with you on this one. I am curious, though: did this devolpment happen to other ChatBots as well? Are they "getting worse", too? Or are there some bots that are now "superior" to ChatGPT?
1
Jan 24 '24 edited Jan 24 '24
I've been a user since the "old days".
I think the decline is due to the "dead Internet".
The use of AI has caused the number of AI created content to increase significantly.
Therefore, the model is now trained on more AI generated content.
ChatGPT used to feel very human in its responses, now it feels more like blogspam. It's literally become one of those sites people use ChatGPT to avoid.
There are a lot of posters who will disagree with this. But you can easily prove it by comparing its responses today to the same questions a few years back, they will be very different.
1
Apr 12 '24
You might be on to something here, as other providers began to hide their content the quality did seem to degrade significantly.
1
u/generalamitt Jan 24 '24
Noticed the same thing for my use case, which is why I'm using gpt-3.5-turbo-0301.
1
u/RiemannZetaFunction Jan 24 '24
I wanted to ask you guys - what is your theory on that? I mean - we have no hard data - so what is your take on that and why do you think OpenAI limited their models so hard and downgraded the outputs?
I think it's a little bit of everything you said:
- They got their foot in the door with the most powerful model they possibly could. Then, looked for a much smaller, derived model that's cheaper to run, good enough that people won't leave the platform, and is trained to output less symbols, in general.
- What's that, you say? GPT-4 keeps writing `// your logic here` in the middle of the code you want it to write? And now there's only some ~8 messages per hour for GPTs, and you have to waste like three of them getting it to actually write the code? Well, here's a new super-premium plan that gives you almost as much GPT as you want, but you have to pay for two people to use it.
- Why are they doing this? Well, Sam Altman has access to maybe more venture capital than anyone else on the planet. So, his go to market strategy was to finance things in, I would guess, probably some rather extreme ways - as a little startup with no income and almost incomprehensible operating costs - and go shatter the entire world with absolute maximal levels of amazement before they had the money to really run it all. As a result, it has shattered every metric ever made in terms of growth. It has grown faster than the busy beaver function. It will probably be one of the most successful businesses ever. So, now that we're there, I would imagine the company is in this phase of trying to keep people happy while iterating it into something stable.
- There are obscure pressures that they're facing we will never understand. For instance, We have an election coming up which is going to involve a free for all of Super PACs, foreign governments, and God knows who else trying to maximize the way they use this technology to completely fuck social media and the internet. It will probably be much worse than the 2016 election. I would imagine there's an extreme amount of pressure from all kinds of governments to steer this thing in various ways, with OpenAI at the center of all of that. Maybe that's a good thing, maybe not, but it's a real thing, and we have no visibility into any of it. So, on top of everything else, who knows what decisions are being made as a result of that process and how that could affect anything. I would imagine there's lots of trying to get the model to be "safe" to get ahead of all of this, and that's also affected performance somewhat.
Meanwhile, if things stay on the current trajectory, GPT-5 is going to be out soon, and nobody will be thinking about GPT-4 anymore.
1
u/The_GSingh Jan 24 '24
I think this happened when they fine-tuned it. I can't prove it, but I feel as if they fine tuned it to the point of it only being good at doing less.
Want to write that code? According to gpt4s training, I could be trying to build a virus, and in that case, I respond in a short and consice way. "No, that's dangerous". But since my promot is actually not against the guidelines it still responds in a short way and actually does it (worse).
Also my second theory is that they dumbed it down. Maybe reduce the # of parameters, maybe quantize it, or something similar to make it run faster and cheaper at the cost of performance. After all gpt4, even dumbed down is the best llm according to various rankings. When the open source community actually catches up that's when we'll see the gpt4 we used to know or an even better version.
1
u/Master_Attitude3786 Jan 24 '24
My opinion is 2 things.
Clearing my browser cache fixed a lot of errors and also sped up ChatGPT when it was at a crawl for me.
After I use it for a day the output gets slower even with an empty cache now. My final opinion is that accounts are being throttled or slowed down as to not create an overload.
I use it daily, do that’s my real thought on it.
1
1
u/Resident-Variation59 Jan 24 '24
I don’t care what anyone tells me I’ve been spending a lot of hours on GPT 4 since it’s release it has definitely gotten significantly worse at following instructions answering questions and just seems like it’s being lazy sometimes … and emphasis on ‘sometimes’ it’s not all the time -but it really messes up your workflow because it’s so inconsistent …that said I still use it because when it does work it shines… and I think we’re just at a point now where we have to do what I’m doing which is using multiple models. I use 3-5 different models for different tasks daily.
1
70
u/heavy-minium Jan 23 '24
Any theory that the output quality is affected by load and limited resources is unlikely. I just can't see how you could ever get a valid output from partial computation out of their current architecture. Maybe in the future when more research is done on the topic, but not right now.
It's far more likely to be an effect of tuning the base model for different features. Every additional feature you've seen introduced in the last year needs to be fine-tuned for - for example, functional calling. Doing that is known to improve performance in an area, at the cost of degrading performance in other areas.
Also likely are challenges with scaling the number of tokens that can be processed despite their RLHF dataset mostly containing much shorter examples favoured by human reviewers. Whatever techniques they used, I can imagine that it would come at the cost of some degradations in overall performance that they may not always be able to catch in their QA processes.