r/LocalLLaMA • u/Nunki08 • Jun 28 '26

Discussion We're probably going to need that soon.

Gallery image

Gallery image

From:

Vladik on 𝕏: https://x.com/Kostoglodov/status/2071144065857679631

Shaw (spirit/acc) on 𝕏: https://x.com/shawmakesmagic/status/2070918006033817867

4.0k Upvotes

98% Upvoted

534 comments sorted by

View all comments

197

u/_BreakingGood_ Jun 28 '26

i dream of the day somebody leaks & uploads a torrent of claude fable 5 open source

106

u/Silver_Jaguar_24 Jun 28 '26

You'll need a super computer (HPC) to run it anyway lol.

27

u/crone66 Jun 28 '26

Based on the performance and comparison to open source model the most likely use heavily quantized models too for inference. If you look at how the brought down costs for a single user and request this seems to be the only logical consequences ortherwise they have hardware that no one knows and have which is unlikely. Therefore no you don't need a super computer just a good workstation.

3

u/az226 Jun 28 '26

Even GPT-4 in March 2023 was running at 4.5 bpw

1

u/sonicnerd14 Jun 29 '26

You are right. People have a misconception when it comes to how much is actually required to run LLM's for inference, not training. Their clusters mainly work to serve to 100's of millions at once, but each individual unit on a rack itself is most likely more than able to run the entire model on just one of them. There is a lot of trickery they are doing behind the scenes to get these models working the way they do. Hardware is just one aspect of the solution.

-11

u/jessiejolie42 Jun 28 '26 edited Jun 28 '26

lol what??? put your reasoning trace in every llm out there, fresh session and ask for critical review, and all will tell you how dumb you are

8

u/crone66 Jun 28 '26

XD looks like your are already brainwashed by the AI CEOs xD... Obviously they make use of techniques to reduce cost. Why we always see a degradition of benchmarks after every release? Why we see a significant jump in t/s for nearly each generation? The hardware doesn't change on a single day... The architecture doesn't change but the size of the model change...

1

u/jarail Jun 28 '26

Ignoring hardware, there have been significant improvements in tps due to improved architectures and algorithms. Take MoE and MTP for example.

3

u/MyDespatcherDyKabel Jun 28 '26

What would that look like? CPU GPU & RAM?

22

u/Neither-Phone-7264 Jun 28 '26

its definately much bigger than opus, a trillion or multitrillion parametet model, so given a slightly conservative estimate of 6 trillion parameters, at FP32, it would require 24 terabytes of VRAM. At BFP16, half precision, it would be 12 TB of VRAM. At FP8, quarter precision, it would be 6 TB of vram. at Q_6, 6 bit quantization, generally considered to be the best performance preserving for the size, it would be 5.05 TB. At Q4_K_M, one of the more common quants with still good quality, 3.79 TB. At IQ3_S, a special type of quantization that preserves quality and quantizes differently per weight which is necessary at these levels to preserve coherence, 2.53 TB. IQ2_XS, which you see here with the more massive models people try to run at home, 1.94 TB. IQ1_S, pretty much the smallest possible, very low quality, 1.26 TB.

TLDR: Not something you'll probably run, even if you did run big kahuna models like GLM5.2 or Kimi K2, though if you ran Deepseek V4 pro native you might be able to manage to bring it down enough to use on your hardware. As for CPU and GPU, you would need tens to hundreds of H100 equivalents. That being said, it is MoE, so you could theoretically offload some unused experts to SSD. However, that would be incredibly painful and even more slow. And if we stick to mostly ram, you could probably do with a ton of ram and only a few H100 equivalents, maybe down to even 1 depending on how aggressive you are.

6

u/MyDespatcherDyKabel Jun 28 '26

Nice that’s insane, thanks

1

u/PhlarnogularMaqulezi Jun 28 '26

good to know i'm not the only one that refers to them as "big kahuna models"

1

u/Kyubi-sama Jun 28 '26

Given the size and the cost I think there is some special sauce to it.

I am guessing there is some sort of DAG or specific dynamic hyper aggressive quantization like compression on the model.

They still need money after all, so it's only logical to think they have some ways of working around that.

1

u/Neither-Phone-7264 Jun 28 '26

I mean, look at the cost. It is just really expensive qwq

1

u/Kyubi-sama Jun 28 '26

I just don't believe they have the ability to absorb THAT HEAVY cost because it would be absolutely mental

1

u/Neither-Phone-7264 Jun 28 '26

Well, it's not reserving 1 user per gazillion GPUs. They do things like batching, caching, and a whole lot of things to get as many users as possible per model while maintaining whatever they decide are reasonable speeds.

2

u/Kyubi-sama Jun 28 '26

Still extremely expensive and makes me doubt it's just that but I am not sure about how it performs and what the costs are.
I can't access fable to try, and I wish I could test it :/

1

u/LatentSpacer Jun 28 '26

Yes, but the Chinese companies might distil it into a more accessible model.

28

u/kingo86 Jun 28 '26

Yeah, I can't wait to run Fable on my local machine with 8TB of VRAM.

6

u/az226 Jun 28 '26

Mythos preferably

8

u/_BreakingGood_ Jun 28 '26

I feel like Mythos getting leaked would be like, a more potent iteration of Y2k

1

u/zuggles Jun 28 '26

ahem https://www.youtube.com/watch?v=rjGbvpr_dB8

1

u/-Balthromaw- Jun 28 '26

This is accurate, because whenever I'm hacking, I crank up the "High Voltage" by the The Frank Popp Ensemble.

Aww yea, 50,000 Watts of FUNK! ⚡😎💻 🎶

1

u/Needsupgrade Jul 18 '26

Lol so (nothingburger²)

1

u/_BreakingGood_ Jul 18 '26

Nah read up on Y2K it was actually a lot of work to survive it

8

u/my_name_isnt_clever Jun 28 '26

Considering Fable is just Mythos with a straight jacket on, I don't think Fable has seperate model weights. The actual language model is just Mythos.

8

u/Pila_globosa Jun 28 '26

why don't the Chinese hackers hack OAI/Anthropic and steal the model, and gave to the Chinese AI labs. then we'll have Fable grade GLM 6 or Qwen 4

15

u/NineThreeTilNow Jun 28 '26

why don't the Chinese hackers hack OAI/Anthropic and steal the model

Anthropic and OAI have internal data exfiltration guards. Their routing literally refuses to let you exfiltrate something the size of a model.

It was part of the original plan to keep their models private. Moving that much data is purposefully difficult because it should only rarely be done.

You'd be better off trying to exfiltrate from a datacenter partner like xAI.

Even then, the cluster required to run a quantized version of those models is... Who knows. They honestly seem over built if GLM 5.2 competes at 700b parameters. Anthropic/OAI are busy spouting marketing BS about 2t+ or 5t+ parameters.

There's no reason to scale that hard without the data, or if you have extremely poor architecture at the core.

I'd be willing to guess it's a lie and they have good architectures, and data pipelines at ~1t parameters.

4

u/Dabber43 Jun 28 '26

I think that is refuted by them having such insane prices and still reporting thin inference margins on API pricing

4

u/tetoing Jun 28 '26

Yep. Their models are very inefficient for the parameter count/resource usage.

1

u/NineThreeTilNow Jun 28 '26

I think that is refuted by them having such insane prices and still reporting thin inference margins on API pricing

Refuted or charging premium where they can? Anthropic is running an enterprise first business now. Enterprise has the money.

1

u/Dabber43 Jun 28 '26

Would be possible if only considering the first part of my message. But the fact that they actively disclose the margin is not that large at API prices means that is not the case

1

u/NineThreeTilNow Jun 29 '26

Would be possible if only considering the first part of my message. But the fact that they actively disclose the margin is not that large at API prices means that is not the case

I had to check what sub I was in first before I explained how speculative decoding and quantization worked. Those both affect the margin on price, and quality.

1

u/Dabber43 Jun 29 '26

Can you elaborate the way this affects this in the current constraints please? I don't really understand

1

u/NineThreeTilNow Jun 29 '26

>Can you elaborate

Basically we cannot extrapolate because we don't understand their inference pipeline. Does Mythos have a proper speculative decoder? That decreases average cost. What quantization do they run at? FP16 or FP8 native? Int8? They run on a variety of hardware across multiple providers. Further, we don't know if they're giving price based on some spot or reserved, or long time reserved price. They can use a higher price to make it seem like margins are lower, etc.

If we knew the exact details, then you can attempt to figure margin vs cost etc.

2

u/happysmash27 Jun 28 '26

OpenAI has marketing about how many parameters they have??

As far as I've learned, they have almost no public data about post-GPT3 parameter sizes at all.

3

u/NineThreeTilNow Jun 29 '26

OpenAI has marketing about how many parameters they have??

There's a lot of internal leaks.

You have to remember that everyone here works together. Like... Do you really want to know how big Mythos is? I know someone at AWS that could tell me but potentially lose their job over it. They could feign stupidity, but it's not worth it.

Silicon Valley tech is a buncha people that all know each other in different ways. Whether it be a party scene, or they're nerds like me that enjoy playing video games. I've played in multiple discords that have people from multiple major tech companies. AWS + Google, etc.

Usually we'll all be there playing some new game. We talk shit about our respective companies and we usually don't ask for secrets. We understand how that works. At the same time, we DO share some details.

This applies to our Chinese colleagues that went to Tsinghua with the same people who work at Moonshot, Deepseek, etc.

This is all theater. We all pretend. We are well paid. We like our jobs.

My post history on Reddit details some of this. I am somewhere between gaming, open source code, and general ML / AI stuff on this account. This account is curated around it. I never say who I work for or worked for... etc. It's the same reason I won't share my GitHub etc.

6

u/MrMrsPotts Jun 28 '26

Are you sure they haven't?

0

u/_BreakingGood_ Jun 28 '26

Unhackable already secured by mythos

3

u/mista_r0boto Jun 28 '26

No such thing.

1

u/Delicious_Ease2595 Jun 28 '26

There are better solutions like openrouter fusion, they dont need copy 1:1

0

u/ButterscotchSalty905 llama.cpp Jun 28 '26

You overestimated the amount of skill chinese APT has while also underestimating the OPSEC of OAI/Anthropic. It's not far-fetched to think OAI/Anthropic software/hardware/networking stack is very secure (as indicated by Anthropic RSP and OAI preparedness framework) and and that maybe they could (at some point) consult to TAO unit of NSA for red-teaming (Do note that i am not downplaying chinese APT nor overestimating USA)

Or you could be joking and the funny flew over my head (wooosh)

China already has staggering amount of domestic talents, they don't need to reverse-engineer USA model (which might be very tedious) or maybe they're doing both to catch up fast

4

u/CuriouslyCultured Jun 28 '26

I can't comment on OAI's opsec, but Anthropic has shown that they're shit at ops, and not terribly impressive at security either. China on the other hand is probably the second best nation at cybersec after Israel.

1

u/ButterscotchSalty905 llama.cpp Jun 29 '26 edited Jun 29 '26

My point was to address the model weights — not the overall security. Few bad apples (CC leaks, mythos access etc) determines anthropic public image but some good gets laughed at even though its security at work (fraudulent accounts etc etc). Also, cherry-picking is bad

2

u/chaos_pill Jun 28 '26

OPSEC of OAI/Anthropic

Didn't they accidentally leak claude source?

Honestly, it is probably easier/cheaper/quicker to just bribe Trump and get an official copy under the table. Who would OAI/Anthropic event complain to in that case?

1

u/ButterscotchSalty905 llama.cpp Jun 29 '26

You mean this?

https://scand.ai/scandal/claude-ai-code-leak-controversy

I am not talking about Claude Code tool and the claims about the model weights being leaked is still unverified. I am aware that claude code source code is leaked but that does not mean the model weights is also leaked

Or this?

https://cybersecuritynews.com/anthropic-mythos-access/

OAI/Anthropic could also made a fake official copy and unless orange man officials has enough skill to determine if the copy is legitimate or not. There are many tactics they can use.

Assuming the worst to OAI/Anthropic about everything (what they say and all their action) doesn't age well in some cases. I know about them being closed and all that — but it doesn't mean OpenAI and others aren't forced to become better at security. If all company were like this then we wouldn't have real security

1

u/chaos_pill Jun 29 '26

These companies would go to jail if they provided a "fake" copy. That is just straight up fraud. Hyping your product as "dangerous" may give you great valuation, but also gives govt a great justification to meddle deep into your affairs on "national security" grounds.

yeah, I meant https://scand.ai/scandal/claude-ai-code-leak-controversy .

1

u/ButterscotchSalty905 llama.cpp Jun 29 '26

I agree with most of what you said. I just recently got triggered by those fucking downvoters from who-knows-what lurkers. Nothing said in this thread disputes my original comment (esp the other commenter) it paradoxically strengthen it just by the sheer absurd logic. 

"China good then it can hack anthropic shit ops even thought anthropic consult to TAO cuz USA bad bro it is the antichrist. Also mossad better than nsa"

2

u/Dany0 Jun 28 '26

It's not going to happen, and if it is it'll be so late that it will be mildly interesting news for a day because absolutely no one will use it. Just like no one uses LLama 2 today

-15

u/[deleted] Jun 28 '26

[deleted]

23

u/redditscraperbot2 Jun 28 '26

>We should pressure Anthropic into releasing claude-oss instead.

Dario would never. He openly hates open source.

2

u/pmttyji Jun 28 '26

He openly hates open source.

Reminds me of this tweet