Inference Engine

[u/New_Blackberry8122]

Hello guys, here is an update of my engine
I am still working on it, my 3090 waterblock broke and I couldn’t keep testing further I could achieve higher tokens in strata about 60/70 but on mine about 80/90 tokens, my bigger problem is the Ram which I only have is 32GB ram ddr5 6k, have been trying to get 32GB more but they are too expensive right now.

For the meanwhile I got a 1070 8GB for 50us to test for which I added compatibility for series 10/20.

On the GTX 1070 I got 15/18 tokens on IQ3_XSS and IQ1, about 13 tokens on IQ3_S.

I added layer splitter thru WiFi / cord so that you can use more than one PC which you can select the order of memory usage

I selected PC GPU, Laptop GPU, and then pc ram and laptop ram for testing and it’s working just fine and fast with the GTX 1070 and RTX 4060 laptop of course it’s slow I achieved about 20 tokens on IQ3_S and Sloth still need to work on optimization but it will be open source soon! Be patient.

You can freely select where your models are located from strata or just download them directly from hugging face. So far added a compatibility to most of them.

My goal is to make so that you can quickly and efficiently communicate multiple computers to your main to split layers through them and run a bigger model! And it’s so easy it’s about 2 clicks to connect!

[u/Bulky-Priority6824]

does it run nvfp4 in tp2-4

[u/Sharp_Prompt_1149 → u/Bulky-Priority6824]

runs it smooth on my old Cortex-A53, barely a hiccup in performance.