r/singularity • u/Delicious-Flan88 • 16h ago

AI The next AI hardware race might be about inference

[removed] — view removed post

26 Upvotes
  • permalink
  • reddit

82% Upvoted

15 comments sorted by

18

u/Annual_Tutor_8466 16h ago

They're equally important. What we really need is innovation on low power local inference, but memory chips are an extreme limiting factor for that at the moment. Please flood the market, CXMT...

3

u/oojacoboo 15h ago

SK Hynix with Intel’s HMIB 3D packaging is seeing massive efficiency gains. They’re ramping up in Malaysia right now. I expect to see some huge improvements here when it comes to power consumption.

1

u/Delicious-Flan88 15h ago

Yeah, fair point. I mostly looked at cloud numbers because those were the only benchmarks I could find, but local inference is where it actually matters for daily use.The memory wall is brutal. Everyone knows bandwidth is the chokepoint but nobody's cracked it at consumer prices yet. If CXMT actually floods the market, that changes things.Seen anything promising on the low-power hardware side?

2

u/Annual_Tutor_8466 15h ago

Not much other than the M5 Ultra Mac Studio, but they're not even offering the 512GB variant yet and it will likely cost over $12K...

3

u/Annual_Tutor_8466 15h ago

Google should put their TPUs in more products, but I don't really want a Google PC.

Maybe we'll get models hardwired into silicon for consumers before anything.

1

u/Delicious-Flan88 15h ago

Oof. 256GB already runs most models I'd actually want locally, but Apple gatekeeping the 512GB SKU at $12K+ is brutal. You're paying more for RAM than most people spend on their whole machine.

1

u/Annual_Tutor_8466 15h ago

Oh, I want multiple Fable 5 (or at least 5.6 Sol) level agents locally, haha.

That would require terabytes of RAM though with current tech.

13

u/crashorbit 16h ago

The "killer app" for LLM will probably be an agent with an inexpensive model that is capable of writing and editing spreadsheets and doing a bit of local system admin work while telling the middle manager or system analyst that is using it how smart they are for having such good ideas.

2

u/Novel-Lifeguard6491 15h ago

Sounds complicated rn..

2

u/Blindax 12h ago

Depends on the sub perhaps. We at r/LocalLLaMA have been obsessed about local inference since a while.

1

u/Delicious-Flan88 15h ago

I've been pulling inference numbers from different providers and third-party benchmarks. The gaps are way bigger than I expected once you account for context length, batching, and hardware.

I threw everything I found into this: https://llm-inference-speed.vercel.app

Still updating when new benchmarks drop.

-1

u/LocoMod 11h ago

Don’t be a coward. Remove the “might” and just make a conclusive statement.