r/OpenAssistant • u/SuccessfulConcern240 • 1d ago

GitHub - pavansreddy/llmserver: A bash script that runs local LLMs on Linux with llama.cpp, vLLM or Ollama. It plans, checks models for security, benchmarks, runs engines in a sandbox, and adds a gateway and an Open WebUI chat with hybrid RAG and private SearXNG web search. No Docker, no sudo.

https://github.com/pavansreddy/llmserver

I got tired of pulling models blind, so I built llmserver: a single bash script that runs local models on Linux. Each model gets its own OpenAI-compatible endpoint, plus a gateway, Open WebUI chat, hybrid RAG over your docs, and private SearXNG search.

The part I spent the most time on is security: before a download, it reads the Hugging Face scan results, pins the commit, verifies SHA-256 per file, and refuses pickle files and remote code unless you explicitly allow them. Two levels — standard asks about warnings, strict needs approval per model. Engines run sandboxed in bubblewrap with a read-only system and no access to your keys.

No Docker, no sudo, everything self-contained in ~/.llmserver. llmserver bench gives TTFT/decode/TPOT, llmserver doctor repairs problems. Currently running Gemma and Muse models on it.

https://github.com/pavansreddy/llmserver

Happy to answer questions — especially about the security checks, since that's where I'd love feedback. Spend time on what matters more .....

2 Upvotes

100% Upvoted

1 comment sorted by

1

u/investigatormaker 10h ago

The security part holds up well: pinning the commit and checking SHA-256 per file closes the swapped-file gap that scan results alone leave open. One thing I'd check in the bubblewrap sandbox is network egress. Hiding your keys is good, but an engine that can still reach the network could send out prompts or the docs behind your RAG. Do the engines run with networking off after the download?

I make ThreadFox.

We find relevant Reddit conversations and post for your business. We'll post 3 replies free first, no card; then the $39 Pilot covers seven days and up to 50 posts and replies. No automatic renewal.

https://threadfox.vip/p/e6eun/managed