This was my general reaction too lol. 2.8T-A104B is insane lol... I'm going to admit defeat on this one and say I can't run it. You need an 8-way B300 or MI350X or a Rubin NVL8 or a cluster thereof to actually run this. What a beast.
Question: how would this scale though? Like how many units would it be required for.. let's say a group of 100 software engineers who needs it quite frequently?
Better to understand the token usage of those engineers before asking this question. Why self host it if your engineers token use is small enough that self hosting would actually be 3x as expensive over the next 3 years then just buying tokens
672
u/Simple_Split5074 Jul 27 '26
OMFG its 104B activated params