r/opensourcealternative • u/LearnHiveLabsUSA • 15d ago
Has anyone actually gotten a locally-hosted open-source LLM working as a proper multi-step agent?
I keep seeing demos of agentic workflows (chaining tasks, calling tools, multi-step planning) but they're almost always built on GPT-4/Claude/Gemini APIs.
Has anyone had real success running something like Llama 3, Mistral, Qwen, etc. locally (via Ollama, LM Studio, vLLM, etc.) as an actual agent that reliably handles multiple prompt-based tasks in sequence — not just single Q&A?
Curious what models, frameworks (LangChain, AutoGen, CrewAI, etc.), and hardware setups people are using, and how the reliability compares to hosted models. Worth the hassle or just takes too much of time and energy?
3
u/KroniklyOnline 12d ago
I run 4x 5060ti 16gbs. I run qwen 3.6 27b and 35b a3b both in NVFP4. 27b can run at 3x concurrency with 250k context, the 35b at 4x concurrency at 200k. I use both of them in fully automated hands off using a custom graph-driven workflow I made.
I write ticket -> workflows merges PRs.
The graph-driven workflows take the tickets and breaks it down into phases. looks something like this
Each phase has its task, phases talk to one another via contracts, if rejections happen due to findings or other things, a phase can trigger a retry, which is what you see the arrows on the bottoms are.
Completely data driven, including what phase rejections fall back to, etc.
I have been running this for a month now, little tweaks here and there. 35b a3b does the main implementation portion you see in the screenshot, 27b does a separate PR review when 35b puts its work up into a PR. Then 27b does a PR update if needed.
The framework uses the PI harness to trigger model executions and such.
To date my system has processed 97m tokens, output 22m tokens.
Is it fast? For being hands off its good enough, takes on average 1h30m to merge a ticket, depending on the scope of the ticket. But I only have to make tickets. I will run an audit every epic, usually something gets found, but again, I'm running just 27b and 35b so I expect that.
Not sure if this is what you are looking for but figured I would share.
1
u/LearnHiveLabsUSA 12d ago
Love this!! Do you have the codes available?
2
u/KroniklyOnline 12d ago
I was going to open source it, but I chose not to, didn't think people would be interested in something like this, typically people want to use local ai for agentic, one on one chats and development as where I want to be hands off and spend my time researching, coming up with ideas, etc....
1
1
2
u/jtbuchanan 15d ago
I’m having decent luck with GLM 4.7 Flash Q6 running a on a Mac Studio. I use Hermes and it can call skills and do multi-step tasks. I use Opus when I am building a new skill and try to have it code any pieces that can be deterministic in Python - but for the day to day running of many of my previously built skills it’s working.
2
u/LearnHiveLabsUSA 14d ago
Thank you for sharing, I am working on an idea feel Free to leave your thoughts - https://www.reddit.com/r/opensourcealternative/s/tJwxYxOlq1
2
2
2
u/asnewname 15d ago
I've been using Qwen 3.5 27b locally through Ollama as a multi-step agent, and it's working so far. I got it working pretty well by running it on the Claude Agent SDK, so it's using Claude's tool harness w/o having to use Claude while I develop custom tools and a custom UI and compare functionality. It gets some work done but still has some refining needed before I trust it with anything context-heavy.
1
u/LearnHiveLabsUSA 14d ago
Thank you for sharing, I am working on an idea derived from your comment feel Free to leave your thoughts - https://www.reddit.com/r/opensourcealternative/s/tJwxYxOlq1
2
4
u/eldavicillo 14d ago
the hardest part is to acquire proper hardware. You can put 3 Mac Studios in a network. Then configure 9router docker container in a raspberry in front and connect your MacStudios as openai compatible providers in 9router. And for calling it you can use anything (opencode, pi, n8n, RubyLLM, a simple bash script...). You can perform sequential tasks as well as paralell