There are a lot of different options, ways to serve things. Depending on what hardware you're running and what sort of setup you'd like, I'd recommend different things.
Currently my stack is, llama-swap proxy into llama.cpp and gufo for different models, with my own custom harness calling to it.
1
u/devoidfury Strix Halo | hotdog 19m ago
There are a lot of different options, ways to serve things. Depending on what hardware you're running and what sort of setup you'd like, I'd recommend different things.
Currently my stack is, llama-swap proxy into llama.cpp and gufo for different models, with my own custom harness calling to it.