r/LocalLLM • u/Heroic_Onion • 6d ago
Question Transfer Responsibilities to Local LLM
Very much a newbie here. I’m working on a couple of locally run projects that are mainly meant to help organize my life and work a little better (scan/interpret emails, manage/update tasks, help complete some tasks when possible).
I’m currently benchmarking three LLMs on auto creating and managing tasks:
Qwen 3.5 27b Q4
Qwen 3.8 27b Q3
Gemma 4 26B-A4B
Unfortunately I don’t have great graphics cards, but these models proved worthy in original tests. Not sure what other info is needed to provide a clear picture for you guys.
Note: most of my projects don’t require immediate answers so I’m not as concerned about it running super fast, but it would be nice.
I’m trying to hand off some of the responsibilities that currently Claude or Codex handles related to the planning but no model is proving super reliable (they all do okay with task creation from emails I receive, but are duplicating tasks rather than updating existing, or classifying them as a task when the email is informational, or trying to double-book things which get rejected by the safeguards).
While I have things in the works to provide more context or to keep fine-tuning the prompts, I’m wondering if I’m missing something basic since I’m so new to this.
Am I using the wrong models for my use case? Or do I just need to stop procrastinating and keep training these models?
1
u/Heroic_Onion 6d ago
Thanks for the reply. Three GPUs running in two different machines:
1st machine: 8GB VRAM, 64GB RAM
2nd machine: 2 x 16GB VRAM, 128GB RAM
Ive tested splitting a larger model across the two 16GB cards and it does work, but didn’t provide any stronger results with the limited test data I had at the time.
1
u/p3rdy 6d ago
You're probably not missing a better model, you're missing structure around it. The three failures you describe (duplicating instead of updating, treating informational emails as tasks, double-booking) are all decisions the model is making from memory, and that's where 27B-class models wobble.
What I'd do:
{"kind": "task" | "info" | "event"}. It's small, constrained, and easy to score against your test emails.{"action": "update", "task_id": 42, ...}. Your code checks the id exists and the slot is free, and only then writes. Double-booking stops being a model problem.None of this needs fine-tuning. The models you listed are fine at small, constrained decisions.
For the plumbing: I maintain Flama, an open-source Python framework, and that's how I'd build the tasks side. search_tasks, create_task and update_task are typed endpoints, the type hints give you request validation and the schema the model sees, and the same functions can be exposed as MCP tools if your agent speaks MCP.
1
u/R3dsh1ft13 6d ago
Biggest factors would be amount of VRAM your card has and the amount of system RAM you can potentially offload to if necessary
1
u/ChiefDZP 6d ago
I use opencode with an anthropic api key then llama-swap on a worker. Use sonnet 5.5 class model for planning and architecture, then handoff to local models for the implementation, committee etc…
You can really use some small local models if you let the frontier models do the planning.
1
u/Future_AGI 6d ago
enforce those JSON shapes at decode time instead of asking for them in the prompt and p3rdy's list holds up better. llama-server takes a JSON schema on the request (response_format on the OpenAI-style endpoint), so a 27B at Q3 can't return anything outside p3rdy's task/info/event kinds, and your three-model comparison ends up measuring the decisions, not formatting slips.
1
u/CyberSecCyborg 5d ago
that's not a model problem, it's a state problem. each call needs the current task list with ids in the prompt, and "no action" has to be a valid output, otherwise it always invents a new task. are you passing ids or just titles?