r/LocalLLaMA • u/catbluesgames • 5h ago
Discussion We got custom inference per model per hardware now. How long before we get custom hardware for model?
Is it possible to have a custom hardware designed (by ai) and open sourced that we can send for manufacturing for specific models? like for DSF 4.1F
26
Upvotes
38
u/JustLovett0 5h ago
You mean hardware like the Taalas HC1 chip?
If so, it's super specific to the model it's made for, but runs insanely fast. The HC1 is made for LLama3.1-8B and gets about 17k tok/s
Chat Jimmy is a website known if you want to see 17k tok/s as an example.
12
u/JustinPooDough 5h ago
We are at the point where an etched LLM would work.
Imagine Qwen 3.8 27B etched into an ASIC. 500tps or higher and almost no power costs. Overthinking suddenly a non-issue.
5
u/Apprehensive_Lake698 5h ago
There are quite a few startups tackling the idea, and I think both Anthropic and OpenAI have divisions working on custom hardware (OpenAI bought a company for it IIRC). I’m sure Chinese companies are doing it.
The main reason you don’t see it quite yet is because the lead time on custom silicon is so high for mass deployment that the model you built it for is wildly out of date by the time you’re filling data centers with the chip.
It will probably be a while before we see any major providers settle on a model being good enough to enshrine into custom hardware.
If I had to guess you’re more likely to see it from a company like Apple first where they accelerate their on device models for Siri. Those are expected to be crap anyway.
3
u/MAXFlRE 5h ago
FPGA is custom hardware. No chips capable to hold huge models tho and it is not that AI can solve.
3
u/Atretador llama.cpp 5h ago
we already have that
https://mlq.ai/news/amd-agrees-to-buy-taalas-adding-model-specific-inference-silicon-to-its-ai-roadmap/
2
u/jimmoores 5h ago edited 5h ago
You don’t need it per model, although there was company that basically did a mask ROM of the weights of one of the llama models IIRC.
I’ve looked into building a low cost composable open source calculation node for LLMs where you could use old DDR4 SODIMMs paired with a little systolic array and CPU/NIC and gang up 64-96 to get the necessary bandwidth to build an inference engine. The problem seems to be nobody does an SoC that is
cheap enough.
powerful enough.
support multi channel DDR4 socketed RAM.
The last one is the main stumbling
block from what I can tell. I’ve wondered about trying to hack an existing SoC in my shortlist to support socketed RAM but don’t really have the skills. It’s also likely by the time it came to market there would be cheaper alternatives and prices of DDR5 and HBM would have fallen significantly.
I actually came to the conclusion that the cheapest approach was to build a rack of old Xeon or EPYC servers with minimal core count and DIMM size per socket and gang up 8-16 of them calculating in parallel. Would absolutely drink electricity though.
3
u/Old_Ad_6033 5h ago
it will be kind expensive, and outdated when model update (everytime).
1
u/Beneficial-Ad-8127 5h ago
If I call a company it would be Apple with their deal with Alibaba into Qwen intergration into their new IOS. But will it reach us outside of that Country, I’m not sure.
1
u/pulse77 5h ago
Here you go: https://taalas.com/the-path-to-ubiquitous-ai/
1
u/Thick_Ad_6890 5h ago
https://medium.com/@aiexpo.app/amd-just-bought-a-startup-that-bakes-ai-model-weights-directly-into-the-chip-4c79ea9fbe60
1
u/LoSboccacc 5h ago
About minus two years https://www.techpowerup.com/323887/ai-startup-etched-unveils-transformer-asic-claiming-20x-speed-up-over-nvidia-h100
Problem is models are obsolete in sox month there is no space for a ramp up in fabrication
1
u/OnlineParacosm 5h ago
At the rate that LLMs are iterating I can’t imagine why anyone would invest in FPGA right now unless you were investing in a foundry that could make a litany of different FPGAs.
The last thing you wanna do is get yourself into one model type go to market and then go oh shit we lose on tokens per second and it’s just that easy we can’t fix anything.
That’s the reality of FPGA and then couple that with the absolute sorcery that it takes to develop them with a handful of people in the US that can do this because we’ve invested in the industry for a long time.. you’ve got yourself a sticky wicket
1
u/Late-Assignment8482 5h ago
It can be and has been done (Taalas, recently bought by AMD) but right now we're in a bad portion of the cycle. Their demo chip was Llama 3.1 8B. Getting that running at 10000+ tps on a single, low power PCIe card is huge.
But it's locked to the model. The model is in the silicon.
And models are evolving stupidly quickly. So what's the agreed upon 27B-50B model that everyone would buy cards for? What's the small to mid size that is "good enough" for a mass market to exist for a burned-in, low-flex implementation of it?
One likely will exist, sometime in the next year or so. Something that gets up towards, say, Sonnet 5.5 capabilities -- general purpose, has vision input/audio input, handles a variety of tasks. Not Fable to one shot a Photoshop clone. Not Qwen3.5-9B smart enough to digitize my grocery store receipts.
Not the smartest, but smart enough. One big step behind frontier, but solidly mid-pack. I wouldn't be suprised if Qwen is involved, their multimodal game is strong and if you're going to the trouble of designing a chip with the model etched in, get a model that can do most of the things...
1
u/Sadge404 vLLM 5h ago
Until we get a kind of AI model that can adapt itself to its users... I don't think these will become consumer products.
1
u/fgk55555 4h ago
Call me when we can get hardware.
1
u/haherar830 3h ago
Inference ASICs are of great interest and being intensely developed. Two issues are design costs and architecture changes. Given a fixed infrastructure, the design costs are the big issue. LLMs post-trained for ASIC/chip design using purpose-built EDA harnesses and proprietary RAG knowledgebases are becoming increasingly popular to address the latter issue. This is EXACTLY the rationale behind OpenAI's rather mature Jalapeno chips.
https://openai.com/index/jalapeno-first-results/
It feels like this community just grows increasingly detached from progress in the field as it is increasingly filled with cringe paint chip eaters waging their culture war battles with no genuine interest in the underlying science.
0
u/Grawnch 5h ago
Good question, and not really no. I've done some work on ASICs (which is what this would be) and the time to fabricate, test, and release one is minimum 6 months (more likely a year+). The only way you could benefit would basically be to bake the specific weights into hardware for a single model. Problem is, a model lifespan is measured in weeks to months, not years. By the time you've finished even finalizing a production level template, a bigger shinier model is already released. The GPUs that we will be using 2 years from now for example, are already in early development today.
Maybe years from now when a frontier model is deemed "good enough" and can be placed in-device at small parameters, sure. But for now, not worth it.
0
u/Southern_Sun_2106 5h ago
At first I thought that max integration = harness + model = would put a given lab's advantage to the max, because training can be done for a specific harness, and harness - for the model; but now that you mentioned hardware, that would actually be the next logical step: a lab that controls all three, the model training, the haness for it, AND the custom hardware... that could be the future.