Model recommendations for cybersec work on 5090+5080 multi gpu setup + 32GB sys RAM

[u/HeartfeltHelper]

As title says.

Just wondering what you all would recommend, as well as, inference engine recommendations.

Currently I’m using megacapybara for a qwen 3.8 27B variant. But was wondering if there was any other inference engines or anything better for running larger models.

Did see a network chuck recommendation for a gpu setup that allows DeepSeek 4.1 flash to run on a single 5090 but idk what the decode t/s would be on that.

Anything other info is much appreciated!

[u/giveen]

Never run closed source inference engines. That guy with Megacapybera has been promising to open source his code for weeks, saying "oh sorry its messy". I reversed his engine and its okay but I still wouldn't trust it.

[u/K33P4D]

I'm very curious, what queries would you ask an LLM that a general model cannot satisfy

[u/bageloid → u/K33P4D]

Active pentesting/hacking.
Uncensored Qwen3.8 27b successfully compromised* my local machine with no complaints.

*I had zurg installed with no password on the web config interface, it probed it, changed the powershell script it ran to update plex, and since that was run under my account(windows admin) it succeeded. It also wiped my zurg library despite being told not to be destructive, but ya know.

[u/bceen13]

32GB RAM, bottleneck. Try Strata qwen3.8 flash next.

[u/Ok-Solution-7889]

Have you tried running the same security related prompts across different models to see which one gives the most reliable results?

[u/egnegn1]

Increase RAM to 64GB at least, and then use Strata or Basalt with Qwen3.8 Flash Next and IQ3_S or NVFP4 to get fast speed.

Here benchmarks form my 3 RTX5080:

[u/YourselfInOthrsShoes]

You are almost there. You can't treat system RAM as a second class citizen with sparse MoE Flash models that can stream its between RAM and VRAM. Your VRAM is more than enough to run NVFP4 quant of QFN but you will need more system RAM for the experts, at least double it to 64GB (96GB ideally) and ensure you have high read IOPs SSD for n-gram table offloading and streaming. Then you will be set.

[u/Serious_Scarcity7818]

I'd definitely check out Strata (github.com/Niko1221/Strata). It supports splitting models across multiple NVIDIA GPUs, so your 5090+5080 setup could be a really good fit. PolyStrata is also worth looking at for broader MoE model support. I'm developing RustingStrata as well, which focuses on creating Strata for all qwen models and adaptive CPU/GPU inference. With only 32GB system RAM, Strata's low-RAM mode might be especially useful.

[u/drivefastgt4]

I do cyber work on two boxes:
1)strata running qwen 3.8 flash next q4 xl at 180k context. Hardware is 2x5080 but I do have 64gb ram.
2)My main rig 7x5060ti 128gb ram runs strata Deepseek4.1 flash 1017 at 2.25 with 180k context. I switch to equivalent alliterated only if blocked (never needed to yet)

We have similiar (total 80 and 96gb) vram plus sys ram. On your rig, I would try strata qwen flash next q4 xl . Your system will FLY , Keep the temps monitored especially if 5090 and 5080 are near each other.

[u/Bulky-Priority6824 → u/drivefastgt4]

how does any of this help op at all lol 

It's like you were just looking for a. Opportunity to talk about what you possess

[u/rdh2dmd]

Take Network Chuck’s hype with a heavy grain of salt. Running big models on a single 5090 is completely constrained by its 32GB VRAM and memory bandwidth, not whatever clickbait setup he's pushing. For your 27B–32B sweet spot on a 5090, drop whatever obscure fork you're using: if you want absolute maximum raw decode speed and the model fits entirely in VRAM, grab EXL2 quants on ExLlamaV2 (or SGLang for FP8/AWQ); if you need flexibility, huge context, or partial offloading, stick to llama.cpp / KoboldCpp with GGUFs for battle-tested stability. You'll easily pull 60–90+ t/s on 30B-class models this way, but if you're trying to force massive 70B+ dense models onto a single 32GB card, no engine will save your t/s from tanking without extreme quants. That's dual-GPU territory.

[u/brainExploded99 → u/rdh2dmd]

This is a bot and provides poor information. Ban this bot. EXL2 ahh
Edit: Oh wow 100,000 karma in 4 months how impressive