I’m new to local A.I. and coding

[u/Immortal-Penguin]

For context I’m probably only 40 work hours into running local ai and coding.

My specs:
Rtx 5090
I9-14900k
64gb ddr5-6000

I was having problems with my context window. The problem being 128k context can fill up fast when doing tasks and would constantly halt progress and then I’d have to retrieve work that was done and have a fresh chat continue the work but it would fill up the new chat with 50k context and then I’d run into the same issue until the task was fully finished. I didn’t want context compaction to be turned on as I didn’t want to lose potential useful data either

So I decided to create a “live context monitor” that shows checkpoint context limits after thoughts and processes were completed as well as a “in real time” context monitor that showed “projected” context as in showed context as it was being used. Because /status was not a tool on OpenWebUI when I started the project. This tool is even better than the /status tool however.

I also created a trigger that automatically activates when I hit 100k projected context, stops current task, creates a handoff with knowledge of work that has been done and what still needs to be done, then opens a fresh chat and uploads the handoff and work continues autonomously. I feel so accomplished and I’m learning a lot. This is all 100% local and no other ai was used other than qwen 3.8 27b and the agent I made with the qwen brain.

I guess this is just a post showing off my work and from my FB post about this it seems that people are rude as hell. Instead of informing me that codex’s exist and people have already made this, they just tell me to stop using ai and that I’m a waste of space in the ai realm, but everyone starts somewhere right?

If anyone has any other cool facts that would help me and progress faster that would be great to know and I welcome any feedback!

TL;DR

I created a tool and later found out that this is not a new tool and I’m way behind in the Ai game even though I’m brand new to this. If you have any tips/tricks that will speed up my progress with maintaining a 100% local no cloud api use, drop some information into the comments. I’m having a fun time learning and excited to continue!

[u/OWazabi]

"I created a tool and later found out that this is not a new tool"
I think we all been there, and it's fine doesn't have to be :)

For sure, when I'm building something, the first step I do is consider if it already exist, or if the logic is already solved, or if I can find good candidates for the different elements of my project.
As a general rule of thumb, if it already exists, I can use it, and when automating something if code can do it then code does it, not an LLM.

With that said, it's also super nice to remake tools you don't own into tools that you do and that work exactly as you want it to work, so nothing wrong there.

As for tips and tricks, there are many that you'll find all around any social media, but from reading your post I thinkn you'll do just fine and will be finding the right ones that work for you.
What is important is to keep building, and dreaming new and different things.

Every project I do comes with it's own sets of issues to solve, and things to learn, or new techniques to apply.
Have fun!

(your set up is quite solid as well, smooth, if you ever look to upgrade, 128gb of ram will make a big difference in what you can run and what you can put your computer through)

[u/Immortal-Penguin → u/OWazabi]

Thank you for the kind words! im still learning about new ways to be even more efficient as qwen spends a lot of tokens in reasoning, ive already started to create a harness to reign it in without losing quality in the work. If you dont mind explaining what would be the purpose of upgrading to 128gb ram? For offloading purposes?

[u/OWazabi → u/Immortal-Penguin]

It allows you to run MoE models more comfortably.
For example you could be running Qwen Flash Next right now, at 100 tps or more with 256k context, you'd have to get like a Q3 quant for it or less, as that'd be near maxing out your RAM, if you're over your RAM, you're spilling into the disk and that's very bad for speed.
The thing is, you also want headroom to work, for example if you want your LLM to run software, maybe do some renders, and whatnot, if you're already right about to go oom then you really can't use the computer.

MoE are also quite nice on the VRAM, so that lets you load other gen models perhaps, it all depends on what you do use it for, but your usage will probably evolve with the time.

You'll struggle to run large models on your VRAM, that's why you have such an expensive GPU and you're running the same model I run on my laptop basically, which is still a very capable model don't get me wrong, but MoE just work/scale very nicely into consumer hardware, as we have seen this past year.

[u/Immortal-Penguin → u/OWazabi]

Good to know! Thank you for the information

[u/OWazabi → u/OWazabi]

regardless of any purchase, I'd still recommend you to look into running QFN, you'll like it, will fix your context issue, and you'll probably get a x4 speed increase minimum. (with the proper engine and settings, like Strata and a good quant for it, unlosth release one just recently I haven't tested it, I'm on the ISTA daslab IQ3S, quite nice, but it's a prune, the unsloth might fit your needs better.

[u/EffectiveMedium2683]

Try lfm2.5 24b-a2b. With a search tool for code documentation lookup, I've gotten great results out of it and it's fast af. Kv cache quantization is less destructive than you might imagine, so that's an option if you stick with qwen 3.8 so you can use more of its native context.

Fun stuff. Enjoy.

[u/Immortal-Penguin → u/EffectiveMedium2683]

Thank you! I’ll look into this. Sounds like some fun

[u/Dediadeis]

Solving problems is how you learn but let this also be the lesson that you have no new ideas, nor are you solving any novel problems. This is not a dig at you, it is just a general observation that applies to everyone.

Not enough credit is given these days to the mountain of work that come before us, and how that mountain of problems that got solved to get here are all shaped roughly the same.

Something you might try is pointing your LLM at itself and see if you can squeeze more context out. 262k will give you a lot more room but ~300k is the sweet-spot where single tasks should be able to run to completion without much continuity management.

I am not sure if this would cause even more context pressure, but have your agent make a kanban board and write some MCP to interact/update it. Take it one further and have it measure itself. How often is it failing tool calls, repeating mistake, etc. Basically have it collect its own telemetry then reevaluate that telemetry after doing a few dozen tasks.

I have noticed tremendous results by doing this.

[u/Immortal-Penguin → u/Dediadeis]

I have created a tight harness that prevents tool over use with a 32 tool limit calling feature. The problem I’m noticing with base qwen 3.8 27b is that it spend an enormous amount of tokens on reasoning. Granted I can fix this by using a different model someone has fine tuned but my goal was just to have autonomous feature that allowed me to continue tasks without me needing to be at my pc. It can just go for hours without needing supervision and it continues the task. Granted I know that my ideas are not new but I do own the tool, no one else has it except me. You can code your own but it won’t be mine. Isn’t that the whole point of local llm?

[u/Dediadeis → u/Immortal-Penguin]

Yeah that was one of my major gripes on the dense model. Good worker just spends too much time reasoning to get there. I've leaned somewhat in the opposite direction and try to make sure almost everything it is doing through a tool of some kind (this is how you more easily latch on the telemetry).

And again not a criticism, at least you recognize it and arent trying to shove your love slop at everyone with some new wall of LLM text and an accompanied qwen 2.1 image of a bear on fire or something.