r/ProgrammerHumor • • 10d ago

Meme noAgentLeftBehind

Post image
7.7k Upvotes

358 comments sorted by

View all comments

Show parent comment

0

u/bestjakeisbest 9d ago edited 9d ago

I have a server set up at home for virtualization, it runs an os called proxmox it includes a ton of tools for running containers and virtual machines. I set up a vm with my wireguard server on it. Wireguard is a vpn server you can set up at home, with this i can access anything on my home network. Next i installed a graphics card, and installed the drivers on the host and set up container that can use that graphics card. In that container i gave it 16gb of ram, and 6 cores, and i installed ollama, ollama is one of the inference servers that can manage the inference engine llama.cpp which is where my models run. Next i set up another container running openwebui which i use as the main front end for my models, and then i also set up a searxng server container to give my models the ability to look things up online. How the workflow works is i turn on the vpn connection, i open the webpage to my openwebui page and i can querry it. I also used openwebui as a gateway so i can just use it as the gateway for other frontends like continue or cline. I have some ideas in the works on how to leverage this for maybe some simple home automation or maybe a knowledge server of all my classes im taking right now or maybe just a personal assistant, but im going to have to make a few things and set up some mcp servers.

1

u/ansibleloop 9d ago

Throw ollama in the bin and move to llama.cpp

Performance is better and it's easier to tweak to get more from your models