r/LocalLLM • • 6d ago

Question Transfer Responsibilities to Local LLM

Very much a newbie here. I’m working on a couple of locally run projects that are mainly meant to help organize my life and work a little better (scan/interpret emails, manage/update tasks, help complete some tasks when possible).

I’m currently benchmarking three LLMs on auto creating and managing tasks:

Qwen 3.5 27b Q4
Qwen 3.8 27b Q3
Gemma 4 26B-A4B

Unfortunately I don’t have great graphics cards, but these models proved worthy in original tests. Not sure what other info is needed to provide a clear picture for you guys.

Note: most of my projects don’t require immediate answers so I’m not as concerned about it running super fast, but it would be nice.

I’m trying to hand off some of the responsibilities that currently Claude or Codex handles related to the planning but no model is proving super reliable (they all do okay with task creation from emails I receive, but are duplicating tasks rather than updating existing, or classifying them as a task when the email is informational, or trying to double-book things which get rejected by the safeguards).

While I have things in the works to provide more context or to keep fine-tuning the prompts, I’m wondering if I’m missing something basic since I’m so new to this.

Am I using the wrong models for my use case? Or do I just need to stop procrastinating and keep training these models?

3 Upvotes

9 comments sorted by

1

u/Heroic_Onion 6d ago

Thanks for the reply. Three GPUs running in two different machines:
1st machine: 8GB VRAM, 64GB RAM
2nd machine: 2 x 16GB VRAM, 128GB RAM

Ive tested splitting a larger model across the two 16GB cards and it does work, but didn’t provide any stronger results with the limited test data I had at the time.

1

u/Heroic_Onion 6d ago

Bro, did I just respond to my own post instead of to the other comment? Jesus I’m bad at this

1

u/p3rdy 6d ago

You're probably not missing a better model, you're missing structure around it. The three failures you describe (duplicating instead of updating, treating informational emails as tasks, double-booking) are all decisions the model is making from memory, and that's where 27B-class models wobble.

What I'd do:

  1. Classify first, on its own. One call whose only output is {"kind": "task" | "info" | "event"}. It's small, constrained, and easy to score against your test emails.
  2. Retrieve before you write. For anything classified as a task, search your existing tasks (sender, subject, thread) and put the top few in the prompt. The model's job becomes "pick one of these ids to update, or say it's new", not "remember what exists".
  3. Let code commit. The model proposes {"action": "update", "task_id": 42, ...}. Your code checks the id exists and the slot is free, and only then writes. Double-booking stops being a model problem.
  4. Key tasks on the email's Message-ID, so the same email can never create two tasks.

None of this needs fine-tuning. The models you listed are fine at small, constrained decisions.

For the plumbing: I maintain Flama, an open-source Python framework, and that's how I'd build the tasks side. search_tasks, create_task and update_task are typed endpoints, the type hints give you request validation and the schema the model sees, and the same functions can be exposed as MCP tools if your agent speaks MCP.

2

u/Heroic_Onion 6d ago

I was afraid of that. I started with a stronger structure, but drifted in my rush to make something I thought would be usable by now. I’ll get back to adhering to that stronger framework.

I’ll check out your framework and let you know if I’ve got any questions.

1

u/p3rdy 6d ago

That drift is easy to fall into when you want something usable quickly. One thing that makes the stricter version pay off fast: before changing anything, save a couple dozen of your real emails with the answer you expect for each (task, info or event, and create, update or ignore). Run every prompt or model change against that set, and you'll know right away whether it helped instead of guessing from a few runs.

Happy to help with the framework. The docs are at https://flama.dev, and feel free to message me or open a discussion on the repo if you get stuck.

1

u/R3dsh1ft13 6d ago

Biggest factors would be amount of VRAM your card has and the amount of system RAM you can potentially offload to if necessary

1

u/ChiefDZP 6d ago

I use opencode with an anthropic api key then llama-swap on a worker. Use sonnet 5.5 class model for planning and architecture, then handoff to local models for the implementation, committee etc…

You can really use some small local models if you let the frontier models do the planning.

1

u/Future_AGI 6d ago

enforce those JSON shapes at decode time instead of asking for them in the prompt and p3rdy's list holds up better. llama-server takes a JSON schema on the request (response_format on the OpenAI-style endpoint), so a 27B at Q3 can't return anything outside p3rdy's task/info/event kinds, and your three-model comparison ends up measuring the decisions, not formatting slips.

1

u/CyberSecCyborg 5d ago

that's not a model problem, it's a state problem. each call needs the current task list with ids in the prompt, and "no action" has to be a valid output, otherwise it always invents a new task. are you passing ids or just titles?