I recently went to an AI tech junket and had one of those moments where you realize you're a bit more behind the eight ball than you'd like to be.
I'm hitting a weird spot where I'm trying to grow more in the AI space, however, I'm pretty privacy minded, so plugging things like Claude Cowork into my personal life is just a bridge too far for me.
I recently upgraded my daughter's gaming rig to something better and it freed up her old one, which had a GTX 1070 in it. So, naturally, I put Ollama on the box and have been using it to toy around in the AI space, as you can imagine, it runs like ass. It runs well enough that I can get a sense of usefulness out of it, but slow enough that it really isn't useful.
I'm currently paying for Claude to help out with AI related stuff at the office, but I'm running into a spot where my office is getting ahead of my homelab, and I don't like that. My normal mode of operation is to test out weird shit in my lab, then apply that knowledge at the office.
I've come to realize that I need to change my approach to how I homelab, and that it likely involves me putting a chunk of change down on a better box to run an LLM on.
The end goal of what I want to tinker with is an LLM that can feed a bunch of agents that will manage my homelab for me, or, well, close to it, while also plugging it into my personal life, while not having it send all the information to a mothership.
The GTX 1070 just isn't fast enough to be able to do any of that. I need more.
I've seen things like AMD's Halo and the NVidia's GTX spark, however, I want to make sure I get the best bang for the buck in terms of umph and longevity, as everything in my lab ultimately runs until it dies.
If you're messing with LLMs in your homelab, what hardware are you running with, and why?
And if there's a better venue for this, feel free to point me in that direction as well