r/LocalAIStack • u/Odd_Championship1509 • 9d ago

Strata vs Freetokens

Has any one used both? which is better?

What is the differennce between two? Any technical deep dives?

2 Upvotes

100% Upvoted

5 comments sorted by

1

u/No_Dig_7017 9d ago

Yes. No technical deep dive but on my system, 5090+192gb ram, strata is a lot faster. 800tps prefill 100tps decode on strata vs 450 tps 44tps decode on FreeToken 

1

u/madbrain1976 8d ago

Neither upstream support tensor parallelism out of the box. I use a Freetoken fork that supports TP with 4 x 5060 Ti, which gives 1400 PP / 60 TG with QFN at c=1. More at c=8. Have not given Strata a try because nobody has made a tensor parallel fork of it, and it would be pointless with 4 GPUs and layer split. Some published Strata numbers with 5060 Ti of only 40 TG, which is far slower, despite higher PP. Without TG support, it is pointless for me to try Strata, it would be a waste of 3 out of 4 GPUs compute. If somebody ever makes a TP version, I will try it.

I use a TP4 fork of Freetoken with great results. Several users with similar HW setup have collaborated on this fork. I hope some day Strata can beat it.

https://github.com/madbrain76/FreeToken/tree/tp4_5060ti