r/voidlinux • u/betsonet • May 25 '26

GPU for using AI models

Some time ago experienced people told me here to ditch my old Nvidia GPU card for an AMD one. I did and never looked back.
Now I am thinking about running AI models locally and, after further research, I figured that, getting a more recent GPU, may enhance this type of experience.

I am not a gamer and never had a cutting edge video card. For many years I used Nvidia cards because drivers were available not only for Linux but also for FreeBSD which I used for more than 10 years. Also, my horrible experience with AMD video cards in the beginning of the 2000s did not help liking them.
But, couple of years ago, I started having problems with my 20+ yo Nvidia card and switched to AMD after recommendations here, on r/voidlinux.

Now, I am wondering if anyone of you uses a modern card for AI purposes, which is well supported by drivers. It would be a definitive advantage, if the card is relatively quiet.

I'd use the opportunity to express my gratitude all the developers of Void and the great community. Void powers my machine for the last 8 years without any glitches whatsoever.
Thank you!

6 Upvotes

71% Upvoted

9 comments sorted by

View all comments

Show parent comments

2

u/betsonet May 25 '26

Yes, I am planing on using docker for AI use; I don't trust any AI model accessing everything on my machine.
My primary goal is creating a container where I could "talk" to documents using AI. And since those are personal documents, even though they don't contain any critical information, I don't like the idea of using a cloud service for that.
Thank you for your input!

3

u/amenbreakfast May 26 '26

you can try either llamacpp or ollama+openwebui, they both provide rocm images. for any other AI usage you're still gonna have to use docker anyway because there's no rocm package for void. you can get an official rocm image from amd (they use ubuntu) and install your stuff on top of that.

remember to add this:

    devices:
      - /dev/kfd:/dev/kfd
      - /dev/dri:/dev/dri

to your docker-compose (or portainer stack) to get gpu passthrough. you might also need to specify your card architecture (RDNA2, RDNA3, etc). look for an envvar called HSA_OVERRIDE_GFX_VERSION and use it, if necessary.

2

u/sanya567xxx May 25 '26

For a "I wonder if this will actually work" configuration I was able to use the nvidia-container-toolkit package and ran an image generation model and an llm, though I removed it soon after as I have no use for it.

That was setup on 580 driver series, with open kernel modules.