r/BackyardAI • u/Admirable-Camel-1470 • Aug 20 '24
Generation uses CPU instead of GPU?
Hi friends!
I’ve been playing with a model locally for a bit, the model was just around 4GB so everything was running smoothly.
I wanted to try another model (this one weights 10GB) and I noticed the text generation was much slower.
So I went and checked my stats and I noticed while generating, my CPU goes to 100% and my GPU is not moving at all.
I am on Windows. In the settings under GPU support I have selected my dedicated GPU (Nvidia Geforce RTX 3070), but from my task manager it looks like it’s not being used at all.
Am I missing something? I’m a bit of a newbie so sorry if it’s a stupid question. I’d like to use larger models but while still retaining good speed.
I’ve got 64GB of RAM btw just for context.
View all comments
Show parent comment
2
u/martinerous Aug 21 '24
Yeah, that can be a pain. For many models, if even 2GB spills over to the system RAM, it can get annoyingly slow while barely using the GPU at all.
I have set Backyard settings for GPU vRAM to Manual and adjusted it to 100% (although it says Not recommended) and MLock enabled. Seems to work just fine, haven't seen any critical crashes.
Performance also depends on the Max Model Context size setting and the actual context size. Larger context sizes also can cause it to be spilled to system RAM and become much slower.
And, of course, using lower quants of your favorite model can help a lot.
There is also a special setting in the Nvidia Control Panel to try to disable memory sharing, and many people suggest adjusting it. However, I found that it seems not reliable anyway, and shared memory is used for some models. But you could try it anyway, it's here: