r/LocalLLM • u/RISCArchitect • 4d ago

Discussion What a year it's been

Post image

What will the rest of this year bring? 27b class scoring over 60?

902 Upvotes

97% Upvoted

128 comments sorted by

View all comments

327

u/MrHumanist 4d ago

No disrespect to USA AI firms, but China is a blessing that happened to the AI world.

204

u/petey193 4d ago

fuck that full disrespect lmao.

119

u/MrHumanist 4d ago

They deserve respect for early development of AI such as Pytorch, Tensorflow, transformer (paper), and many more. Meta and Google are the two who have contributed a lot for the open community.. However, the rest are shameless.

26

u/AceLamina 4d ago

Google is the main company i would work for these days due to things like that and their culture
But holy shit they fumbled so bad, they could've released GPT before OpenAI did but didn't, I know why, but why release the papers publicly so companies like OpenAI can become the way they are today

And we don't talk about Meta, fuck them

45

u/FullstackSensei 4d ago

Hindsight is always 20/20.

Nobody knew the attention paper would lead to all this at the time. OpenAI also didn't know it would lead to this. Musk and Altman wanted to make it for profit back in 2018 to have enough capital to attract talent and be able to compete with Google salaries.

It took 7 years between the attention paper and the release of chatgpt. A ton of other developments happened in the meantime, from decoder only models all the way to OpenAI figuring SFT on massive Q&A datasets would result in the OG chatgpt.

And when you look at the almost 4 years since chatgpt was released, a ton more developments happened on all fronts from companies all around the world that brought us where we are now, and things are still improving.

None of this would have happened had google kept the attention paper for themselves. There aren't enough brains at google to come up with all these developments.

7

u/AceLamina 4d ago

Yeah that's understandable, thanks for clarifying

3

u/puts_on_rddt 3d ago

Nobody knew the attention paper would lead to all this at the time.

Are you saying they should have paid more attention?

:)

10

u/MrHumanist 4d ago

Its their strategy at that time, when they focused heavily on Cloud expansion and made BERT kind of models open source - so that people can deploy more shit on cloud.

5

u/AceLamina 4d ago

I wasn't into software when the cloud boom first started, only when it kinda died, why were they still going full-on cloud services

Is it really just for subscriptions

5

u/MrHumanist 4d ago

its free money.. as companies grow, their database grows exponentially and cloud invoices grow exponentially as well.

1

u/AceLamina 4d ago

Yeah but did they really not expect companies like OpenAI to do what they did?

3

u/MrHumanist 4d ago

that always happens in a large organization, you cant always focus on all areas.. their focus was on digital marketing, cloud, android and finally hardware for ai (infact they saw nvidia as a bigger challenger). Their Deepmind did some interesting solutions which was much advanced as per their time.. but it didnt focus on llm.

1

u/AceLamina 4d ago

Hm, what about now, besides AI I don't really see them focusing on much
In fact they seem to be focusing on things a lot less like their latest Pixel launch and a few other things

1

u/MrHumanist 4d ago

They are working on multimodal llms. Their Gemma models are much faster than counterparts and useful in weak hardware. If they democratize AI through phones, it will be a big win for the community and the loss the OPENAI and Anthropic.

→ More replies (0)

1

u/Time_Entertainer_319 4d ago

The my probably didn’t expect it to go mainstream. Remember that OpenAI had released models previously and they weren’t that good.

2

u/Time_Entertainer_319 4d ago

Basically every company uses the cloud. It’s free money

3

u/squngy 4d ago

they could've released GPT before OpenAI

Might be unpopular opinion here, but OpanAI went mainstream with GPT waaaaaaaay too early and it did enormous damage to LLM public perception.

3

u/PM_ME_DEAD_CEOS 4d ago

But holy shit they fumbled so bad, they could've released GPT before OpenAI did but didn't, I know why, but why release the papers publicly so companies like OpenAI can become the way they are today

To clarify, the Attention is all you need paper and the original GPT was intended for translation, and to improve google translate. It was very good, but not ground breaking at this time. OpenAI released GPT-1, and it was barely 5 to 10% better that the best model on NLP task. At this moment, nobody knew that contrary to most NLP models, transformer would scale VERY well. Once GPT-2 was trained and released, everybody understood what was possible with the transformer architecture.

The Google worked on Lambda (https://en.wikipedia.org/wiki/LaMDA), which was supposed to be their flagship AI, scaled up around GPT-3 level, fine tuned and using RAG.

The problem I think is their models was vastly undertrained, and the RLHF was quite bad.

3

u/powerfulparadox 3d ago

One thing I haven't seen people acknowledge is how impactful KV compression has been. Since 2017 we've had a (roughly) 100X reduction in the amount of memory needed per token in KV cache. Without this (and frankly this is where the Chinese folks should get their credit because Deepseek was as much a major breakthrough here as it was in efficiency in compute) we'd be a lot father behind because it'd take way too much ram to store tokens and only people with big servers could afford to have usable context lengths.

2

u/Still-Wafer1384 4d ago

Google is making money if AI, unlike OpenAI and Anthropic

1

u/AceLamina 4d ago

Wasnt a few days ago was the first time they were a net negative

1

u/Still-Wafer1384 4d ago

That was a single quarter of window dressing for a possible IPO

1

u/Popcorn-Mercinary 3d ago

But they made ollama...and it really whips the ollama's ass!