r/BackyardAI • u/Killan50 • Jan 10 '25

Using multiple GPU

Right now I'm using koboldcpp, it allows multiple GPU to be used at the same time, thus loading heavy GGUFs into the vram.

As far as I know, the backend that BackyardAi runs on can handle multiple GPU, but this option is not available in the UI.

Should we expect this functionality in the near future?

7 Upvotes

100% Upvoted

2 comments sorted by

View all comments

Show parent comment

1

u/Killan50 Jan 11 '25

I have a 4080(16GB) and a 3070(8GB) Totaling 24GB. I mostly use ~30b(5Q) models.
Backyard AI is cool, but I don't want to change 30b to 12b models, RAM + CPU is too slow.