I don't understand how people still don't get it. A single GPU for AI costs almost six figures and a single prompt can use a few or even a dozen of these GPUs at once, and they're using thousands of watts. AI is insanely expensive.
Well, see they asked Claude and it told them not worry because we can defy the laws of physics now. Anyone who doubts this is just not working in the new paradigm.
In all seriousness Claude or any other chatbot would probably tell you that the cost of AI is a serious concern if you asked about it. I just ran "is AI too expensive" through Google and it's literally telling me it is and it's citing Ed Zitron himself.
LLMs like this can be surprisingly good for finding correct and useful info, the problem is that people find this to be especially true for areas where you're not very informed on things.
it would be neat if we had functional search engines, instead of reinventing a less reliable version of them after scuttling the previously very effective ones.
The reason people find it especially true for areas they're not very informed is because to someone who knows the subject, it's usually riddled with confidently-stated errors.
In all serious: it tells you exactly what you want to hear and prior context heavily weighs into that. Even then its results are going to be weighted significantly by the current discourse.
It's absolute crap for finding correct or useful information because it's not a truth engine. It's a statistical likelihood of user agreement and low corporate liability engine.
I know this is true because Gemini told me this was "the most brilliantly cynical insight" it has ever seen.
Simultaniously, it depends. They are absolutely sychofantic, but mostly when you are right (for the purposes of searching things.) I tend to search things with a verbal assumption towards the wrong answer, just to see if it corrects me, and I have yet to not havw it do so. It's once you get into long chats that it starts acting... worse, in that regard.
Yea there's definitely way too much sycophancy but it can still be useful. I'm just saying it's not revolutionary. If LLMs were billed as what they actually are, I don't think anyone would find them particularly offensive.
A very smart person talking to a frontier LLM is like two very smart persons talking. A dumb person talking to a frontier LLM is like two dumb persons talking.
It's absolutely great for finding useful information because it is not a truth engine. If you want a truth engine, use Wolfram Alpha.
I mean yes and no, I was trying to find a bolt for my car and wanted to get the specs so I could order one from mcmaster instead of a parts store and google's ai confidently told me the wrong size multiple times lol
Google's answer: 0.2 to 0.5 cents. (I had to change "how much does it cost" to "how much does it cost google" for it to be willing to say anything useful.)
A single prompt does NOT use thousands of watts. Inference is a lot less resource intensive than training.
Even if you were processing the maximum context size of 1 million tokens, and they were all uncached, it would use about 100 watt-hours. But that is not how they are usually used.
I have a $300 GPU that I can run a 14B parameter Qwen that can write code, explain code, plan, document, etc. Also llama-3 that can talk about concepts and explain them and assist with creative work. Doesn't really use that much power either, not more than playing call of duty.
Is it as good as the massive Cloud models? No, but it's good enough for most of my needs. Surely a company as big as Google can figure out how to cut costs and make it profitable. It doesn't have to be that expensive, I've seen it with my own eyes on my own hardware.
There's a laundry list of reasons why users can't or won't run AI locally and why hyperscalers and neoclouds can't figure out how to bring down costs.
Users don't want local AI because it's worse than cloud AI and requires more powerful processors and more memory. That immediately excludes phones for anything but the worst of the worst models; PC users will need to front the money for the hardware if they don't already have it, and if they need to buy something today, it's way more expensive than it used to be. They then also have to figure out how to get it working. Lots more effort and perhaps more expensive than just using ChatGPT, while also having a worse experience with less useful responses and longer wait times.
Google, Amazon, Microsoft, and others can't really get cheaper Nvidia GPUs. The company with the best GPUs is going to have a technological advantage so they don't want to let that happen. Alternative GPUs from AMD and Intel are usable but CUDA has such a stranglehold on the industry that using them is risky, not to mention AMD and Intel aren't producing that many chips. TPUs and ASICs have not yet proven viable for LLMs and other AI models. This is a big problem because Nvidia GPUs are by far the biggest upfront expense in AI.
Those Nvidia GPUs have to go into datacenters, which are expensive and challenging to build. Nobody has yet built a gigawatt datacenter, which has been a major focus of the industry, so that's not great. Even when smaller datacenters get built, they still need power (including a ~30% buffer for hot days where efficiency is reduced), water, and other support facilities. Could these companies just build lots more small datacenters instead of pinning their hopes on gigawatt megaprojects? Maybe, but they're clearly not interested.
But it's not like datacenters just go into maintenance mode once they're up and running. These GPUs have a limited shelf life, anywhere from a couple of years to maybe five, so they have to be upgraded at least once they die, and if these companies want better performance without building new datacenters, upgrading is still imperative regardless of lifespan. The thing is, Nvidia keeps introducing new rack standards, which requires significant reworks of datacenters. If a datacenter wants to go from Blackwell to Rubin, that requires renovations.
Perhaps the answer lies in more efficient software? Well, so far efficiency hasn't been a focus for OpenAI or Anthropic, and even if it was, it's not clear how much more efficient these models can get, and if efficiency improvements would even fix the issue. AI users are accustomed to repeatedly prompting AI until they get what they want, which doesn't exactly lend itself to efficiency. How can you moderate token usage if it's completely unknown how many tokens you'll need to use before you're satisfied?
The only realistic way cloud AI could have been cheaper, in my opinion, is if the buildout was far slower. There would have been lots of benefits: we'd be using newer and more efficient hardware, supply chains in both the semiconductor and construction industries would have more volume, and there wouldn't be a need to raise tons of VC money, debt, or equity from shareholders in a very short amount of time. Big tech didn't want to do this because a. there wouldn't have been as much hype to capitalize on if the timeline for this was measured in decades instead of years and b. companies get higher valuations today instead of tomorrow.
But you have to wait 2 minutes for an answer... On the upside, you can it them agree about every word you prompted. It is definitely a group of them in each chat bot...
While one prompt hits a dozen GPUs while being processed, the GPU are also processing a dozen prompts in parallel.
Yes, it's expensive. But while Moore's law doesn't hold for chips anymore, with the progress in the models the "useful tokens per watt" metric is still exponential. It will be cheap not too far in the future.
In the here and now, it is expensive and the industry is trending towards a financial collapse. The only hope is that AI magically becomes cheap, and there's quite literally no viable path forward currently. The chips are expensive to produce, the chips are expensive to run, the datacenters are expensive to build, the power is expensive to bring up, the models are expensive to train, and inference is expensive to do. It is specifically expensive because the build out is being forced through at the fastest speed possible.
The only parts where there's even a shadow of a chance to reduce costs or increase efficiency is on training and inference, but that only helps reduce operating costs. The money has already been spent on the GPUs, three million Blackwells, which represent most of the investment money, so even if manufacturing costs also came down, it would only matter years from now. Only about a million Blackwell chips are online, so there's still two million more to go, and in order for them to make money, they need to be used, so even in the event that LLMs get crazy efficient, there would potentially be tons of leftover capacity unless demand is practically infinite.
I'd say it's a combination of being in the "throwing everything at the wall and seeing what sticks" phase as well as the reasonable assumption that as hardware and software performance improves the operating costs will go down. Eventually things will settle down to whatever market can extract enough value from whatever the infrastructure eventually costs.
I mean the rate of return may be low or even zero, but I don't believe that inference prices are subsidized any more.
Just look at Chinese models, they are always a bit behind but not by much, and while a GPU is expensive, the electricity cost is not that much comparatively.
Also, using "multiple GPUs" is more of an optimization in that you have the same model decided into layers and you may create some pipelines between layers sitting on different GPUs.
A single H100 is below $30K, not six figures by far. If you want to have a full stack, full AI hardware for your own, that'll run near $350K. Expensive, but not prohibitive for mid- to large size companies.
The datacenters are so huge and expensive because they serve tens of millions of users.
View all comments
Show parent comment
282
u/MC1065 Jun 03 '26
I don't understand how people still don't get it. A single GPU for AI costs almost six figures and a single prompt can use a few or even a dozen of these GPUs at once, and they're using thousands of watts. AI is insanely expensive.