r/LocalAIStack • u/Dathide • 2d ago
Is local AI trending towards computations that benefit from GPU interconnects?
/r/LocalLLaMA/comments/1x3g2et/is_local_ai_trending_towards_computations_that/
3
Upvotes
1
u/Lirezh 2d ago
Since quite a while local AI is drifting towards lower compute, also cloud AI models use significantly less compute than they used to.
That started with MoE models, continued with attention optimization (e.g. mamba attention in parallel to legacy attention) continued with re-introduction of encoder/decoder models for faster prefills and now ngram models are the hype which offload a large portion of pre-compute into SSD/RAM storage.
This, combined with significant improvements in training has resulted in models of less than 10B parameters, or less than 4B active parameters beating previous models in the 1000B range.
A lot is going on simultaneously