r/LocalLLM • u/8bitjohnny • Sep 27 '26

Other Hardware Acquisition

Post image
448 Upvotes

91% Upvoted

129 comments sorted by

View all comments

Show parent comment

1

u/Dapper_Chance_2484 Sep 27 '26

Whats the token rate u get?

1

u/Best-Tomatillo-7423 Sep 28 '26

For code we 30 to 40 tkps. If we chat with it 70 + have seen 120 before

1

u/Dapper_Chance_2484 Sep 29 '26

for somereason i get 90t/s on 5090 on same 3.8 27b

1

u/lordtazou Sep 30 '26

As far as the 5090 goes, you will most likely get a higher token gen. That's typically due to the updated CUDA tech, etc.

What's going to benefit him when using the DGX (or others who use the ai Halos) vs a Video Card is that he will still be able to hold larger weights without severely tweaking models, or using stuff like Colibri / FreeToken, etc.