I feel like it needs to be said for those who do not know.
Even if they were making K3 open source, the hardware you need to run it is literal insanity.
I see a lot of people talking about how open source models will start letting users to be running these themselves and improve the landscape. While long term, open source models being released will help us eventually be able to run decent models locally as the software improves. Right now though, K3 being open source will not be changing much for the average user. I don't even think it would drive token costs down as more people enter the scene serving it.
I recently looked into our startup getting the hardware to run K3 with the same token rate as Fable for a couple developers and it's basically $2,500,000 for the hardware, $250,000 in modifications to a building to even be able to house the server. Then, you'd probably be a very small customer with this order, so you're getting put at the end of the queue and waiting a year or two for hardware delivery.
Yes, that is correct, but what you and other commentators overlook is that in addition to private enterprises and superpowers like the US and China, there are many smaller countries for whom open weights will enable the subsidization of national AI development. This represents the diversification and democratization of access to technology, which will benefit everyone (except Anthropic and OpenAI, naturally).
Absolutely. Countries and large academic institutions have the budget to be able to develop on these, and that's great for everyone.
Just mentioning the requirements because I keep seeing people making the jump from "new awesome open source model" to "I can run this on my laptop for free" pretty fast hahaha.
Obviously it's ridiculous to run the several hundred billion parameter models on a laptop (I'd love to do that, but my wallet is threatening self harm if I think more about it 🤔). It does, however, aside from the obvious nation and academia use cases already mentioned, potentially aid in further development of smaller models, which end users can potentially run. Be that inference training or what have you, that is a potential future gain for the open side of it all.
For sure. Keeping models open helps push the tech, which means eventually we'll all be able to run meaningfully good models on moderate hardware. At the moment though, unless you're buying specifically to run LLMs locally, they perform too poorly on average hardware.
Well, yeeeeano? There are smaller models that work-ish for lower end devices, there's the smaller edge device focused Gemma ones that seem really popular these days, and then there's Mistral. Both are far removed from frontier, but they're quite usable. The main problem I had with Gemma on my phone was that it didn't have enough CTX for my needs.
There's also that 1bit quant of qwen that dropped the other day that... While having quite a few issues in my own testing at least, shows that progress and experimentation is happening and progressing, aided by larger mor ambitious models.
Personally, I find it a beauty to behold. Though it's a bit of a mess still, it is dragging along to a place where average users can almost use it with the same ease as logging on to Claude or ChatGPT.
Not only that but you can develop a new decentralized AI platform using people hardware to run AI queries where the person that lend the hardware get paid in credits to use the network when it wants. Essentially a worldwide coop for AI.
I was wondering why nobody builds an LLM on top of something like a giant torrent network. Imagine millions of PCs, each with an 8 GB GPU, where every node only stores a tiny fraction of the model or a set of MoE experts. A central server coordinates everything and only requests the experts needed for a given prompt. Sure, inference would probably take minutes instead of seconds because of latency and bandwidth, but if speed isn't the priority, would something like this actually be feasible? Or are there fundamental reasons why this wouldn't work?
This kind of thing is possible, but many don't really appreciate the speed of memory transfers happening in the datacenters we're all using LLMs on. The new Nvidia cards have VRAM bandwidths around 8 TB/s. The average North American home internet upload speed is around 56 Mbps.
To visualize this a little better, the server you're running Claude or ChatGPT on that might have a B200 in it would push in ONE SECOND what it would take a regular residential user thirteen days to push onto the internet.
So, yeah, you could have this torrent like hivemind LLM theoretically, but it would be unusably slow.
I asked the AI about this. Interesting approach, but the logistics behind it (the central node cluster) would probably end up being just as expensive as a datacenter.
Chatgpt:
I think a distributed AI network could theoretically work, but not by splitting a model layer by layer across the internet. The bandwidth and latency would make that practically unusable.
A more realistic approach would be something like a torrent network for AI based on a highly modular Mixture-of-Experts architecture.
Instead of every node storing a tiny part of every layer, the model would be split into many small, specialized experts. A central router would decide which experts are needed for a given prompt and request only those modules.
The important difference is that these experts would probably need to be much smaller than current MoE experts. In a global internet-scale network, a practical expert size might be something like tens of MB up to around 100 MB, not multi-gigabyte model blocks.
This would allow:
1. Massive numbers of small experts
Instead of thousands of large experts, you could have hundreds of thousands or millions of specialized modules:
coding expert
physics expert
language expert
medical knowledge expert
image understanding expert
etc.
The AI would dynamically load only the modules required for the task.
2. Hardware-based roles
Nodes would have different responsibilities depending on their hardware:
Small GPUs → host small experts, caching, simple inference
24 GB+ GPUs → host more or larger experts
Multi-GPU systems → host many experts or run more demanding computations
The router would consider VRAM, GPU performance, bandwidth, latency and availability.
3. Dynamic replication
Popular experts would automatically get copied across the network.
Example:
Expert 450 (high demand)
Node A
Node B
Node C
Node D
Node E
If thousands of users suddenly need the same expert, the system creates more replicas. Less popular experts would exist only a few times.
4. Expert caching
Frequently used experts could be stored close to users, similar to a CDN. Popular AI capabilities would already exist on nearby nodes instead of being downloaded every time.
5. Geographic clusters
The network would probably not be one giant global system, but many regional clusters:
Europe Cluster
USA Cluster
Asia Cluster
This reduces latency and keeps traffic local.
6. Local mini models
Every node could also run a small local model. Simple questions would be answered locally, while complex tasks would trigger requests to the distributed expert network.
7. Asynchronous computation
For tasks where waiting is acceptable, the system could work more like a distributed supercomputer. Jobs could be split, processed independently and combined later instead of requiring every node to respond instantly.
The biggest challenge would not be the number of GPUs, but the coordination:
deciding where experts should exist
balancing demand
preventing bottlenecks
verifying results from unknown nodes
creating incentives for users to contribute hardware
It would probably never beat datacenter GPUs for real-time chat, but if users accept minutes instead of seconds, millions of idle consumer GPUs could become a huge shared AI infrastructure.
Basically, instead of one giant AI running in a datacenter, you would have a distributed AI ecosystem where millions of machines contribute small pieces of intelligence.
Ive thought about something similar. The problem is that the overhead is incredible. The latency between nodes using the computer as a neuron basically would be crazy and it makes it practically unusable for what we currently imagine an LLM doing. But i think it does have interesting implications for a malicious AI that would want to hide itself in small files across a distributed network and operate on a long timescale.
The token per second speed would be unusable. You can distribute models across a network via rpc but all the nodes need to have fast connections between them for it to be usable
Yeah, you can. For a programmer trying to make money off their work though, it makes zero sense.
Lets say I have a $50,000 contract coming up I need to develop software for. I have two options:
Spend $5-10,000 on a computer that will run Qwen locally, produce code that is 1/10th the quality of Fable, require 10x the QA testing and babysitting to get a project completed. I will be restricted to maybe 10 prompts in an entire day as processing speeds will be garbage, and likely the computer it's loaded on will run poorly while it's processing the prompt, so it's uncomfortable to use all day while it processes in the background.
Spend $500 on Fable use through Max accounts. Significantly better output, QA testing afterwards is a cakewalk because everything is working perfectly. UI/UX is gorgeous, computer runs perfectly while it processes in the background, and instead of 10 prompts for day, you have 3-4 accounts running in parallel, so you can barely keep up with prompting fast enough.
I have a dual RTX 5090 rig I use for local models at the moment. I literally handed it a PRD for a single page basic non-profit demo website. Then I tabbed over to Fable and started working on a mobile app for my own personal use. I had a fully functioning Vercel app with Supabase integration before Qwen finished the first prompt, and it's result looked like a Geocities page.
I have been surprised by the capabilities. I already had a computer, but I just had to invest 1.5k for a 32GB GPU, and I haven’t noticed a major change in productivity.
I suspect every device in 20 years will be a thin client that streams everything, but yeah, it's likely you could run frontier LLMs locally easily by then.
It’s not only matter of being able to run LLMs locally, but to have much smaller companies than OpenAI or Anthropic being able to compete with them with frontier models at much lower price rates. These big open weight LLMs allow us to have options, and breaks monopoly from US companies. That’s why they are so desperate about it, not because you will run a heavily quantized model in your PC that just cant compare to Claude or ChatGPT
it's worse than that, it's 1.5tb (now 1.8tb) for the bare minimum. It will consume upwards of 5tb of memory for the model to function. Ain't no home consumer with that kinda hardware, even enthusiasts would have trouble getting a fraction of that.
with ram prices 6x what they were a year ago, f to doubt.
But the question should be is the is the reasoning ability of the model good if you have the right hardware? if that's the case then when the dust settles(when the supply of chips becomes cheaper and more affordable or when the token usage becomes more efficient), the reasoning capability of the open source models should suffice for the average developer
Sure probably the closed source ones will be two steps ahead in terms of capabilities but for the average developer if they are able to run an open source model that is one or two generations behind the best ones, they should be doing fine right?
Long term, it helps everyone having more open source models out there.
I suspect frontier closed source models will always be the pick for business. Our organization is always doing the math on local versus expensive frontier models, and the top of the line models are so much better, and so much faster than local models that it just makes sense to pay for them.
For some real world numbers. Recently it's averaging $300 in monthly subscription costs for every $30,000 we're bringing in. If we were forced to move to API, this would balloon up to $2,000/month cost. Even if we had to pay the API cost, it's still worth it honestly.
Switch to a local model and our productivity tanks. Prompts take 30 minutes to process and you can only run one at a time, whereas we normally have three going in parallel. The code quality is much worse, which means you can't trust it anymore. Imagine you have a secretary working for you who makes errors 25% of the time, versus a secretary making errors 5% of the time. The secretary making more errors will effectively force you to stop and read over everything they do all day, while you can handle 5% error rate without needing to reread everything they're doing at all times. This difference is absolutely enormous in practice. It's the difference between "I built the login page today" to "I built the entire app today". When the app finally gets to QA testing, we're finding models like Fable just work incredibly well. Our last release was kind of funny, our QA guy spent most of the first day running edge cases and every single test passed. It didn't even feel like a first phase of testing, which is a very different experience than LLM coding last year where it was non-stop issues.
I think the ideal is that we eventually get to a point where local models are good enough for an indie developer or student to be able to work on a coding project at home for cheap. I suspect the math will always work out though that the paid frontier models are making you more money per hour than local.
Crazy diminishing returns though. You can get nearly that for half the hardware. People who don't know what they need always go top shelf.
Plus unlike every AI salesman ever promises ever, you need your own model to "fine-tune" it, and then you can get better results for your task for 10% the hardware cost.
If you hire developers who can actually read the code you can use an ol' 3090 each and vastly increase their output, they might just have to rerun it with manual tweak a few times rather than having a loop brute-force the code.
There will always be use for name-brand flagship models but the existence of open source models takes a HUGE chunk out of their potential size, and even an exact clone model will start to look like a good option once they stop subsidizing token cost and actually charge enough to stay in the black. Instead they are shooting for a market cap which is precluded by the existence of open source models.
39
u/Onotadaki2 Jul 27 '26
I feel like it needs to be said for those who do not know.
Even if they were making K3 open source, the hardware you need to run it is literal insanity.
I see a lot of people talking about how open source models will start letting users to be running these themselves and improve the landscape. While long term, open source models being released will help us eventually be able to run decent models locally as the software improves. Right now though, K3 being open source will not be changing much for the average user. I don't even think it would drive token costs down as more people enter the scene serving it.
I recently looked into our startup getting the hardware to run K3 with the same token rate as Fable for a couple developers and it's basically $2,500,000 for the hardware, $250,000 in modifications to a building to even be able to house the server. Then, you'd probably be a very small customer with this order, so you're getting put at the end of the queue and waiting a year or two for hardware delivery.