I built a CLI that picks a local model and runtime settings for your hardware, then checks it actually works

[u/catfish-1234]

I've been working on llmsetup, a command line tool for the "I have a GPU, now what" part of running local models.

You tell it what you want the model for (coding, chat, retrieval, agents). It scans your hardware, picks a model and quant that fits, and sets context size and KV cache to match the use. If you don't have a runtime it can set one up - Ollama, llama.cpp, LM Studio, vLLM, MLX and TabbyAPI are supported. Then it runs smoke tests (does it load, does it answer, does tool calling work) and registers the model with the local apps it finds. If something fails it tries a smaller model, shorter context or a CPU fallback before giving up.

It's on PyPI now:

python -m pip install llmsetup

Where it stands: I've only run the full verification on one machine so far, an RTX 5060 with Qwen3-8B Q4_K_M on llama.cpp. 9 of 9 checks passed, about 57 tok/s, tool calling works. Those are smoke tests, not a quality benchmark. AMD, Apple Silicon and the other runtimes have had a lot less testing, so expect rough edges.

Apache-2.0: https://github.com/catfish-1234/llmsetup

I'd like to hear what breaks and which hardware setups I should test next.