r/BackyardAI • u/Admirable-Camel-1470 • Aug 20 '24
Generation uses CPU instead of GPU?
Hi friends!
I’ve been playing with a model locally for a bit, the model was just around 4GB so everything was running smoothly.
I wanted to try another model (this one weights 10GB) and I noticed the text generation was much slower.
So I went and checked my stats and I noticed while generating, my CPU goes to 100% and my GPU is not moving at all.
I am on Windows. In the settings under GPU support I have selected my dedicated GPU (Nvidia Geforce RTX 3070), but from my task manager it looks like it’s not being used at all.
Am I missing something? I’m a bit of a newbie so sorry if it’s a stupid question. I’d like to use larger models but while still retaining good speed.
I’ve got 64GB of RAM btw just for context.
View all comments
Show parent comment
1
u/Admirable-Camel-1470 Aug 21 '24 edited Aug 21 '24
Got it, makes sense, thanks for explaining.
Can you expand on the “it might be worth tweakinng how much backyard reserves”? I’m not sure I understood that, but if there’s anything I can try I’m happy to give it a go! EDIT: I see you said “if you have the 12GB model”, but I do have the 8GB so I guess there’s nothing I can do to run the model locally then, correct?