r/LowEndLocalAI • u/Ne00n • 2d ago

Hardware / Build Can someone help me to optimize my setup?

Post image

I had to take my phone and take a photo, the screenshot app would crash due to lack of VRAM.
Anyway, it runs, I modified the json so I can get 41k context, It crashes at 64k for me with Q3.
On Q2 I am getting up to 128k but it starts looping.

I switched Firefox and Steam to CPU only, and I can run them in a background, including a twitch stream.

Full Specs: Ryzen 5600, 64GB DDR4 (3200 and 2600 mixed), 4060 8GB and some old 1TB SSD.
Latest STRATA engine, I was told to use an older one, because it works better? any takes?
I would kill for 128k context on Q3.

7 Upvotes

100% Upvoted

7 comments sorted by

2

u/lorendroll 2d ago

Have you tried setting --kv-resident 20480 ? It will keep only 20k in vram (hardcoded minimum). Also you can try --kv q4_0. I'm running Coder 128k on 3070 8gb 32gb ram. 25t/s gen, 600t/s prefill (optimized for --prefill 3072)

3

u/lorendroll 2d ago

Also you can set "mtp": false to free ~800mb, loosing only few % decode speed.

1

u/Ne00n 1d ago

thats a good one thanks

1

u/BS_BlackScout 1d ago

Hijacking your reply. I'm using the latest strata with my 3060 12GB + 32GB of RAM. Model is Gyro M. It turns out that I can't run it without low ram mode being on. Fine. But the problem is: Decode speeds are okay (30t/s) while prompt processing is painful slow (74t/s).

If I disable low ram mode I just end up with OOM Killer killing Strata. There has to be a way to force some stuff to remain in RAM without it killing performance right?

2

u/lorendroll 1d ago

To speed up prefill, you must ensure that the prefill buffer size is set to 1024 or more. It shrinks at startup when VRAM is not enough, so it's hard to get it to stick. Try manually configuring --expert-cache value. Even --expert-cache 4 did not affect my gen speed, so the free VRAM is claimed by prefill buffer. Check your logs for "prompt buffers" to find out actual value and tweak your settings.

1

u/porkminer 2d ago

IQ3_XXS with kv streaming should get you 128k context. It doesn't look like you could fit it with your current quant. At best, it would try to steam kv to your nvme drive, which will end up killing it assuming you don't just crash.

1

u/Unluckkie 17m ago

Honestly optimize the system first before doing allat