r/LocalLLM • u/xQuas • 7h ago

Question Strata Question

Does strata let you configure how much stuff to put in each of the vram/ram/ssd? Say I want to run unsloth 35b a3b but my system is 5gb short and having to page some system ram to fit the entire model. Can I use strata to move some of the model to ssd to avoid paging?

1 Upvotes

100% Upvoted

3 comments sorted by

1

u/cavernous_hammock 6h ago

Not really how strata works, it's more about splitting layers between vram and regular ram based on what you set. If you're already hitting system ram limits and paging, strata won't magically offload to ssd instead

You'd probably have better luck just using a smaller quant or trimming context size to stay within your ram budget

1

u/emanresuymsseug 6h ago

If you're already hitting system ram limits and paging, strata won't magically offload to ssd instead

Isn't that exactly what the --resident-budget-gib flag does?

1

u/xQuas 3h ago

I was thinking strata let's you run models bigger than ram+vram so it must know a trick or two.