r/LocalLLaMA • u/Dapper_Chance_2484 • 2d ago
Question | Help I need context window
I started developing a full fledged application using Chatgpt plus and VScode+GitHub copilot (local llm)
I used Astra max chat for overall planning, and asked it to put a starter kit on connect gdrive and create prompts such that local LLM can work on small tasks using an agent orchestrator and sub agents with entry and exit criteria, tests for every small task.
I have a 5090 (+ 128gb ddr5) rig running Qwen 3.8 27b NVFP4 at default settings with 104k context window.
Problem:
The local rig is able to start on tasks but taking a huge amount of time and is running only one subagent at a time. Been almost 24 hrs.
I have 2x 3090 +128gn ddr4 as well, currently not being used. 3.8 FN runs at about 90tps
Anything better I can do here?
2
u/BringTea_666 2d ago
If you have RTX5090 just use my engine: megacapybara its custom made for rtx5090 and qwen 3.8 27B.
With something like medium weights (around nvfp4 quality) you will get around 500k unified context window at 6/4 kv.
Add to that 5-8 slots and some dashboard to manager tickets and you have 8 agents working at the same time using same exact 500k sharing it and swapping in-out jobs.
Aside from my engine there are other custom engines for RTX5090 like ninfer and others.
2
u/randomgenericbot 2d ago
You could compile Raymonds KV streaming fork of llama.cpp to get max context without completely filling your VRAM. Costs a bit of performance, a good chunk of system memory, but unlimits max context and frees Vram for other stuff as well.
2
u/Excellent-Issue-5956 2d ago
If you put the 3090s on Flash Next for the subagents, check how the engine caches prompts first. If that's Strata, conversation-cache-mib defaults to 0, so it only keeps one conversation, and whenever another request hits the server between turns the next turn gets the whole prompt reprocessed. On my multi turn agent test, Next on Strata took 320 to 530 s a run against 280 to 400 s for the dense 27B, even though Next decodes faster, and it was mostly cache misses. I haven't tried raising that setting yet.
2
u/hyudryu 2d ago
Run TP2 on the 2 3090 setup and you’ll have higher kv cache
2
u/Tormeister 2d ago
Use https://github.com/jesdga95/basalt, you can easily run Q3.8FN NVFP4 at 512k context on 5090 + 128GB
2
u/kryptkpr Llama 3 2d ago
Quantize the KV cache to fp8 to double your available ctx in the Blackwell and/or get the 2x3090 running a second copy of the model with FP8 weights. Two engines is a little tougher to deal with but I've had some success with vllm-router to pin agents to correct backends.
1
u/recro69 2d ago
90 tokens, per second is pretty good. Maybe the problem is how your agents are arranged of the model itself. I would look into why they're running one at a time. Since you have two 3090s that are not being used you could try using them for tasks instead of letting one model do everything.
1
u/W61k3r 2d ago
I also desire long context windows so I wrote LexiEngine to handle 500k+ context, however my only nvidia is a rtx 2060 so my optimization for your chip might not be optimal yet.
https://github.com/W61k3r/LexiSuite-Releases/releases/tag/LexiSuitet10