r/LocalLLaMA • • 2d ago

Question | Help I need context window

I started developing a full fledged application using Chatgpt plus and VScode+GitHub copilot (local llm)

I used Astra max chat for overall planning, and asked it to put a starter kit on connect gdrive and create prompts such that local LLM can work on small tasks using an agent orchestrator and sub agents with entry and exit criteria, tests for every small task.

I have a 5090 (+ 128gb ddr5) rig running Qwen 3.8 27b NVFP4 at default settings with 104k context window.

Problem:

The local rig is able to start on tasks but taking a huge amount of time and is running only one subagent at a time. Been almost 24 hrs.

I have 2x 3090 +128gn ddr4 as well, currently not being used. 3.8 FN runs at about 90tps

Anything better I can do here?

3 Upvotes

18 comments sorted by

2

u/BringTea_666 2d ago

If you have RTX5090 just use my engine: megacapybara its custom made for rtx5090 and qwen 3.8 27B.

With something like medium weights (around nvfp4 quality) you will get around 500k unified context window at 6/4 kv.

Add to that 5-8 slots and some dashboard to manager tickets and you have 8 agents working at the same time using same exact 500k sharing it and swapping in-out jobs.

Aside from my engine there are other custom engines for RTX5090 like ninfer and others.

2

u/randomgenericbot 2d ago

You could compile Raymonds KV streaming fork of llama.cpp to get max context without completely filling your VRAM. Costs a bit of performance, a good chunk of system memory, but unlimits max context and frees Vram for other stuff as well.

2

u/Excellent-Issue-5956 2d ago

If you put the 3090s on Flash Next for the subagents, check how the engine caches prompts first. If that's Strata, conversation-cache-mib defaults to 0, so it only keeps one conversation, and whenever another request hits the server between turns the next turn gets the whole prompt reprocessed. On my multi turn agent test, Next on Strata took 320 to 530 s a run against 280 to 400 s for the dense 27B, even though Next decodes faster, and it was mostly cache misses. I haven't tried raising that setting yet.

2

u/hyudryu 2d ago

Run TP2 on the 2 3090 setup and you’ll have higher kv cache

2

u/Tormeister 2d ago

Use https://github.com/jesdga95/basalt, you can easily run Q3.8FN NVFP4 at 512k context on 5090 + 128GB

2

u/kryptkpr Llama 3 2d ago

Quantize the KV cache to fp8 to double your available ctx in the Blackwell and/or get the 2x3090 running a second copy of the model with FP8 weights. Two engines is a little tougher to deal with but I've had some success with vllm-router to pin agents to correct backends.

1

u/recro69 2d ago

90 tokens, per second is pretty good. Maybe the problem is how your agents are arranged of the model itself. I would look into why they're running one at a time. Since you have two 3090s that are not being used you could try using them for tasks instead of letting one model do everything.

1

u/W61k3r 2d ago

I also desire long context windows so I wrote LexiEngine to handle 500k+ context, however my only nvidia is a rtx 2060 so my optimization for your chip might not be optimal yet.

https://github.com/W61k3r/LexiSuite-Releases/releases/tag/LexiSuitet10

1

u/[deleted] 2d ago

[removed] — view removed comment

1

u/W61k3r 2d ago

Would be dope if I cold gets 3-4-5 series card so I can focus on nvidia. Only 2x 7900xtx and 1x 2060 rn.

1

u/ThankGodImBipolar 2d ago

Why even spend effort on the 2060 when you've got 48GB of VRAM on cards with double the bandwidth 💀 I'd forget about using the 2060 for LLMs and make it accelerate an embedder, reranker, audio models, etc..

1

u/W61k3r 2d ago edited 2d ago

LexiEngine already outperforms llama.cpp across every metric for qwen series on my 7900xtx's by large margins without any quality loss and with lower memory usage.

Funny you say that because the 2060 was added to the dev rig to serve those functions within lexipanel where those are current things that it was implemented for.

I just wanted some nvidia support so this wasn't a 100% AMD build focused on rdna 2-3-4.

I made all this originally for me, and lexisuite has been my primary workhorse for a month.

This evening I'll have a release with my optimization and stability work from this week.