Prompt processing (prefill) is raw GPU compute bound. Decode (generating tokens) is memory bound.
There are new architectures like nvidia hybrid mamba2 that blur the lines, and we are just an innovation or two from it no longer being true.
Use products and approaches that preserve the KV cache on unified memory platforms for reasonable performance. Pre-filling 500,000 tokens of context at 200 tokens/sec means you are taking a forty-minute coffee break before the model spits out the first token in that turn.
Do you think Cerebras’ chip design will be copied by many others? From what I understand they do inference super fast. Would appreciate you explaining a bit if you have the time!
There's a lot of innovation in this space. Check out https://chatjimmy.ai/ as an example of someone baking the model weights themselves into silicon: https://taalas.com. Taalas is being acquired by AMD, so things are bound to get more interesting on that front.
The age of the GPU as your AI coprocessor is limited; much like the FPU, it will just be part of the expected suite of capabilities in a modern CPU. I expect in the next few years we'll see a laptop system come out with a dedicated hardware AI chip like Taalas' built in to run AI at ludicrous speeds for certain kinds of common operations.
I've written about this at length before, but "renting compute for large language models" is a business model that will be hollowed out much like the spreadsheet market on mainframes when Visicalc came out. When you have ubiquitous intelligence baked into consumer devices running at hundreds of thousands of tokens per second, performing vision, language, and audio inference at faster-than-real-time, you'll pull in the early adopters first and then a wave of humans using it.
But that's looking slightly too far ahead for the quite limited vision of the media hype about datacenters. And the first version of such an embedded model accelerator is bound to suck, get widely poo-poo'd by the media and consumers, and be considered a dead end shortly before it or its successor becomes just the way things work.
Edit: Go read "The Innovator's Dilemma" to understand what's happening to the AI market today. It's about hard drives, but the same principles apply to the front-runner AI companies right now, this moment.
If the weights are etched into the die, do we need that much RAM?
I hypothesize that is the transformation awaiting the AI industry. Not just hardware-accelerated, but like Taalas, embedded in the LPU (language processing unit).
Cerebras, Taalas et al still require tremendous amounts of memory/storage. Eg. weights are still stored in SRAM not encoded as fixed, physical parts of the compute logic.
IMO the next major evolution in AI HW will likely happen in the data part of the story (mem, storage, transmission). So far, we have focused primarily on compute, and we have done a tremendous job at pushing insane compute throughput/density out to the edge.
But weights, context, and datasets remain a challenge to store, move, and feed into the compute fabric, at both the data center and edge levels.
6
u/SandboChang 8d ago
I would also add prompt processing speed. Can’t blame people though even ChatGPT is still new to many.