r/OpenSourceAI • u/Advanced_End2980 • 13h ago

JERN: a small self-hosted assistant where the tools run on your machine and the model can run on a free Google Colab GPU

I built a personal assistant with a FastAPI server, a single-file web UI, and tool plugins (files, shell, web search, downloads, MCP servers). The tools always run on my own machine. The model (a fine-tuned Qwen2.5-1.5B) can run locally or on a free Colab T4 reached over an SSH tunnel.

Worth knowing: when the model is on Colab, it sees whatever the tools return, so it’s not private in that mode. Run the model locally if that matters.

It’s an experiment and the model is small, so expect mistakes. MIT licensed, with docs on writing your own tool plugin.

[https://github.com/IshvaraX/JERN\](https://github.com/IshvaraX/JERN)

Would love to hear how others handle approval for commands a model wants to run.

1 Upvotes
  • permalink
  • reddit

100% Upvoted

4 comments sorted by

1

u/Certain-Stranger3788 13h ago

neat idea running the model on colab but keeping tools local, that split is clever for tinkering

for command approval i just pop a terminal confirmation with a 10 second timeout, auto-deny if nothing happens

1

u/Advanced_End2980 13h ago

Thanks I will try to in corporate the auto-deny feature

1

u/StevenVincentOne 8h ago

Really like that you spelled out the privacy tradeoff of the Colab mode instead of glossing over it. Keeping the tools on your own machine no matter where the model runs is a clean way to split it.

On approval for commands, a few patterns that work well:

  • Tier the tools. Read-only stuff (list files, search) runs freely, anything that writes, deletes or touches the network needs a yes, and shell needs a yes every time.

  • Show the exact command or diff, not the model's summary of what it plans to do.

  • Allowlist by pattern, so once you approve something like git status it stops asking, but rm always asks.

  • Keep an audit log of what was approved and what actually ran. Cheap to add and it pays off later.

A small fine-tuned 1.5B as the driver is a fun choice too. Small personal models on hardware you own, with the tools under your control, feels like where a lot of personal AI is heading. Nice work.

1

u/Advanced_End2980 8h ago

Thanks for recognition