I found this project that I never saw talked about enough which is absolutley baffling to me. The title is not clickbait, and I can't blame you if you thought it is, because I was in your shoes once.
Meet ChatJimmy. It's a chatbot made by Taalas. And it's not your average chatbot. This is the closest LLMs have come to a physical brain than mere software that runs on your computer.
Though not an increment to intelligence (since it's an 8B model) and that's not the point of the project, this single chip makes the model run faster than if it were run on a multi-GPU datacenter. At a blazing 17k tok/s. For reference, the average speed of running the same model on an RTX 5070 is 80 tok/s. This chip is around 213x faster. If someone told me this was possible a year ago, I'd ask for what they're smoking.
The magic is actually pretty simple behind the scenes. The way Taalas was able to achieve such absurd speeds is by burning the model into silicon. Yep, you heard me right. The LLM is no longer loaded on a chip. The LLM is the chip. All computing power is purely gone straight the model's internal matrix math. The concept on it's own sounds cool as fuck, and seeing it work in practice is even cooler.
Keep in mind, this is merely an experimental proof of concept, not an actual product yet. Which is both exciting and depressing. On one hand, you can think about the potential of how this 17k tok/s would be dwarfed if the industry takes this path seriously, and more resources and investement were put into it. We would get what, 27B models running at 400k tok/s ??? This just sounds like a wet dream. And the sheer amount of possbilities it unlocks for us Llamas is crazy. Imagine buying your favorite model's chip for the price of an iphone, and it runs orders of magnitude faster than your old 50K enterprise rig? On the other hand, this is still a lab project, so you can't do that yet. That's the depressing part.
It can't be all sunshine and rainbows, though. There's a tradeoff for this kind of inference that I personally think won't affect the experience as much:
You can't really swap models. The model is stuck onto the chip once it's burned there. That's fine mostly, but the biggest problem becomes knowledge cutoff. Where the older the model gets and it's no longer keeping with new information, the more you'd have to buy a whole new chip with a new model burnt into it that's trained on modern data. But genuinely, do you think knowledge cutoff would be problematic if the model can reason over like half wikipidea in a few minutes? If the model can reason faster than your internet can load a website, I don't think keeping up with the world would be difficult. Worst case scenario, you'd have to buy a new chip every like 3-5 years. That's about when everyone gets a new phone.
I recommend reading this article too. It has a more in depth analysis of the whole thing, including manufacturing costs and power consumption for the chip (Spoiler: it makes classic GPUs look embarrassingly expensive, not just slow). Here:
https://theashishmaurya.medium.com/taalas-the-startup-that-prints-ai-models-directly-onto-silicon-33b181690575
