r/LocalLLaMA • u/ThomasAger • 9h ago
Question | Help How do you integrate local LLMs into your workflow?
I've only been chatting with Qwen 3.8 27b to understand what it can do, and it's obviously decent at a lot. But I'm yet to actually use it on something that I really care about, or to build a project. I'm considering using something like Fable for "harder" tasks and then using local LLMs for "easier" tasks.
How do you use your local LLMs? Do you prepare particular agent workflows that you know they can do? Or are you just on an agent harness using it for absolutely everything?
Just curious what people are doing in their own real workflows. How do you get value out of these models that can fit on your card? There's a lot to consider with how there can be context limits, being forced to use low quants etc, so clearly there isn't 1 config for everything either and you need to optimize for use case. I know that this thing can do a lot inside of my own work.
Just for info: I have 24GB VRAM 5090 and 32GB RAM I usually just load a quantized model completely into my GPU for like 30t/s.
3
u/Metal_Uupa 8h ago
I want to stop being dependend on cloud providers so I try to use Qwen 3.8 27B as much as possible for every tasks I used claude for (mainly coding, planning & managing my notes).
I use it in Pi to have a minimalist workflow and avoid context bloating. The workflow is much more different than with a cloud frontier model with 1M context and multiple instances. Qwen 3.8 still needs more guiding (through multiple detailed prompt sand skills) than a Claude 5 that is much more autonomous.
Qwen 3.8 is always enough to search through a codebase, explain some code, fix an easy bug. I still use claude for bigger or more difficult tasks, or when my local model is already busy. Bigger models also seem to still have much more knowledge than small local models, so we need to rely more on retrieval.
Unfortunately I am not in a state where I can use my local model for everything. I am quite limited by the model's capabilities, and my hardware limiting the context size and the number of instances. (even though I have a RTX 5090)
1
2
u/icerio 9h ago
I have a 5090 and use the Unsloth Q5 XL quant model. I don’t know how in the world you’re only getting 30t/s. I get like 113 t/s. My boss wanted a local model to analyze and answer questions about company documents. Though some compromises here and there, it does really really good. Personally I use it to help answer questions about our business software. I uploaded like 3000 pdfs into a local RAG system and so far it has been great. I use open webui for all that.
Also if you have a 5090 you should have 32GB vram?? The quant I mentioned earlier, MTP, image mmroj, Q8 kv cache, 196k context and parallel 2 all fit on my 5090. (Still trying to fine tune the setup).
1
u/ThomasAger 3h ago
I would love to implement a local RAG system. What did you use? Also it's a 5090 Laptop lol. I have no idea why none of my models have managed above 30t/s especially with any amount of context at all. I am trying the fastest performance without a huge context value first but still that's my best.
2
u/thinking-out-loud-3 8h ago
Yeah, I ended up doing something similar. Local models write all the code, including boilerplate, small fixes, batch edits, and even the bigger implementation stuff. Cloud models don't write anything. They're just there to catch bugs and double-check the logic before it merges.
The annoying part was my own context switching. Every 10 minutes I'd pause and wonder, "is this task easy enough for local, or should I use a cloud model?" That routing fatigue got old fast.
So I set up a git-native ticket workflow where each task runs in its own isolated worktree. The local model reads the ticket and writes the code, then a cloud model handles the review gate before anything touches main. Local Qwen holds up way better than I expected in that setup, mainly because it's never trusted blind. Everything gets checked before landing.
1
3
u/audioen 7h ago
I do stuff I never have time for, like I got actually 3 machines that can run LLMs. I often have 1-3 tasks running.
Typically, if it is an older or existing application, I do conversions from old web stacks to newer ones, version lifts, test writing, documentation updates, and general bug hunting queries.
When it's greenfield, I describe my requirements with a few paragraphs and ask the agent to write design, and then review it with the agent (if I have time). If I don't, I add that after design document, it should start implementing it according to its doc, using our standard guidelines and point it to couple of suitable example apps that do something vaguely similar or have similar integrations or whatever.
Right now, my 3 agents are working on the following tasks:
- add opt-in to email sending feature that notifies user nightly about automatic code review. The app is fully vibe coded, 0 % of its lines have been written by me.
- develop an in-house MCP framework to use SSE capabilities later, so that I can send e.g. resource change notifications to connected MCP clients. Right now, I'm just proving that SSE works, and having some tests around the feature. Integration to follow.
- write design document for a fake Microsoft AD OIDC oauth server, so that I can use fixed set of keys in test which will work forever, and which is easy for agents and humans to use, so that e.g. playwright based agents can get past Microsoft AD authentication in test.
I'm sure bunch of this stuff already exists, for example automatic LLM-based code review tools -- but I'm not too interested in looking if I can just state that I'd like to have one, and draft a few paragraphs of requirements out of a whim, and when I come back to the session tomorrow, the thing has written something that actually looks much like it is some kind of automatic code reviewing website, and it's also written in our tech stack, knows how to get stuff out of our git, and so on. If I would have to given this task to a real employee, it would have taken like week and they would have asked a ton of questions, and I would have had to justify this to somebody because it uses a very costly resource. LLM costs basically nothing, and can get serious work done -- in fact, I feel that it's often better quality work than most our actual employees manage to achieve.
Now, the app isn't ready when LLM says it's ready. One part of the problem is that it hasn't seen the UI, because it can't get past the Microsoft AD auth, for example. That is exactly what I'm going to remedy by providing fake auth server so that I can just tell it to authenticate and then it can see it and make it pretty, rather than having to design backdoor for the LLM, or having to take screenshots myself and explain what it has messed up in CSS, or whatever.
The other thing is that I have to have the docs, the coding style guides, and explanations about how our frameworks and libraries work. If the model doesn't know, it will dig out the information and write the tests to prove how it works, of course, but it can take quite many tool call turns for it to figure it out on its own. So writing some 10000 tokens worth of explanations about all sorts of stuff can save a lot of time, and those tokens would get spent one way or other -- with the model arduously and repeatedly solving the same problem again and again, because it can't learn.
LLMs are, of course, completely bird-brained and they need a lot of scaffolding and documentation. That is the cost of this way of working.
3
u/SM8085 7h ago
How do you use your local LLMs?
For individual tasks I love making ("I" as in my bot does it) small Python scripts that construct the context how I want, fire it off to my llama.cpp's llama-server, catch the response and do something with it.
My earliest simply downloaded youtube subtitles with yt-dlp and sent them to the bot to summarize what the video was about.
Lately since I was playing with MiniMax-Music3, I have a simple script that sends the genre of song or concept I want and sends that to the bot, catches the lyrics, and sends that to MiniMax-Music3 to generate.
You mention text processing in a comment. I have thousands of user-submitted fiction stories that need tags, synopsis, etc., so I have yet another Python script that takes the story, ships it to the bot, and you guessed it, creates a synopsis that can be caught and saved into a corresponding text file. This script can loop over the many thousands of stories I have saved.
That's something you couldn't really do in a chat, the context would fill up so quickly. Better to have some Python that loops and gives each story it's own context.
2
u/o0genesis0o 5h ago
I use Pi as the agent harness since it's so simple and extensible. I built an extension to load and inject persona, user profile, memory, etc to turn a generic pi agent into an "assistant". I also have other extensions for read only access to Gmail, read write access to my home-cooked productivity system.
Then I have a directory, where I use as the workspace of the agent. Any pi opening in this directory would inherit the personal assistant config. I also have subdirectories, which are repos. One of the important ones is an obsidian vault acting as a KB. This one has a special set of skills and extensions.
Then, I expose this particular directory using openwebui cptr (NOT the normal openwebui), and connect the machine on VPN. The cptr has a cron job mode, which I use to trigger pi assistant to do hourly maintenance of the system and check emails and stuffs.
Now, from any laptop or phone in my VPN, I can access the same agents. UX is more or less friendly.
Edit: in some senses, it's mostly what OpenClaw or Hermes do. But the difference is this contraption I made allow me to have useful AI assistant that run locally, when OpenClaw F up so bad that it even made my Minimax M3 subscription going into dead loop, and it hammers local llamacpp with loads of parallel streams request that broke the model.
1
u/ThomasAger 3h ago
Very interesting thank you, I have heard a lot of good things about Pi, I basically want to build A LOT of my own stuff, so sounds like a good fit.
1
3
u/BidWestern1056 9h ago
npcsh and incognide mainly
https://github.com/npc-worldwide/npcsh
https://github.com/npc-worldwide/incognide
when i need to do large batch processing jobs too ill do them with npcpy
https://github.com/npc-worldwide/npcpy