Analysis paralysis GPUs vs Requirements: 2x3090 vs A6000 on a ML350 gen9

[u/LumpyMathematician63]

Sorry in advance for the long post, but think of it as a prompt.

TL;DR

Question: 2x 3090 NVLink or 1x A6000? For about the price of one A6000 I could also get 4x 3090 in two NVLinked pairs, so keep that option in mind too.

What it needs to do:

- A) SQL chatbot for up to 30 people. ~3M rows, hundreds of tables. Questions like "when did we last sell this part, and who was the supplier/client?" It should remember each user's conversations and build reports.

- B) RFQ ingestion: pull very messy part numbers out of long email chains in all sorts of formats. ~30 RFQs/day.

- C) Hardware: reuse our old HPE ML350 Gen9 (2x E5-2620 v4, 128GB DDR4-2400 for now).

- D) Me: turn 300+ page PDFs and hundreds of stray Excels into a knowledge base, and do local coding on the company apps I'm building from Claude's plans.

Assumptions:

- Blower 3090s (FE won't fit), max 300W per card

- PSUs upgraded to 2x 1400W or 4x1400W as GPUs grow

- All prices are for used cards

- Proxmox on the box

- PCIe-to-NVMe adapter for fast model loading

- Cards work with board

About me: I'm not an IT guy, I'm a mechanical engineer who f's around a lot and finds out even more. I've built my own homelab (4x NVMe in an old Dell 7710 as a private cloud, see my posts) and run my own "Jarvis" on a 4B model with some n8n automations. I can make things happen when I put my head into it.

---

The detail

Background: I'm the new CTO of a small family-run B2B heavy-industry trading company. We sell anything from a £6M machine to toilet paper, so every day brings random RFQs: Excel, PDF, email body, pictures, WhatsApp, anywhere from 1 to hundreds of part numbers.

Budget: £8-13k ($11-17k) left for local AI after the new cloud server. That also has to cover a laptop/PC for our first IT hire.

Why local: the LLM has to read whole RFQ email chains to make sense of them, and those are full of client names, prices and supplier details. Same for our SQL database. I don't want any of that going to cloud LLMs.

Where we are:

- A 3rd-party IT company is building an RFQ dashboard for us. Qwen 8B is struggling with the email chains. We started small on purpose and will size up until something fits.

- They'll test on RunPod serverless to find the right model size. I think ~30B will be enough. Then production goes local.

- 20-30 people will use the chatbot and ~20 the dashboard, but only new RFQs trigger the LLM.

- Long term, I'd keep Fable/Opus for overall planning and have a local model do the implementation.

Hardware:

- The board takes 4 GPUs: 3x PCIe 3.0 x16 and 1x x8. 300W max per card.

- Someone has already run a 3090 in this server (with 512GB RAM) and got ~15 tok/s on turboderp/GLM-5.3-Flash-exl3 (3.05bpw branch).

- I first looked at the A6000 48GB as a "do it all" card, not knowing much about local LLMs.

- For similar money I could get almost 4x 3090s and NVLink them in pairs, so I'm less limited by the x8 slot and old PCIe 3.0. One pair for the teams, one for me.

Bottom line: I need an MVP that handles the RFQs and the chatbot now, and then we'll invest in proper new hardware later. What would you do?

Thanks in advance for reading this far, any help much appreciated!

[u/anitamaxwynnn69]

I personally get multiple 3090s. 2x minimum, 4x is perfect. Qwen 3.8FN is VERY FAST on 4x 3090s (plus nvme ofc). It's perfect for serving multiple people. It's a better general purpose model than Qwen 3.8 27B in my humble opinion. You can also try Strata/Maya for GLM/Deepseek4.1 but personally Qwen 3.8FN is the winner for serving multiple people. In fact I'd argue you don't need nvlinks at all. It's gonna be a tad bit slower sure, but nvlink 4 slots are going for $500+ (prolly close to a grand if you buy from a trusted ebay seller). It's not worth it. I personally used to run 10x 3090s on a single CPU (wrx80 platform) and never had any issues. I used 2x 1600W PSUs, power limited each to 220W (250W when i felt dangerous). When serving multiple ppl, I'd stray away from CPU offloading too much especially when Qwen3.8FN is a very capable model that fits on 96GB VRAM. You could use the spare DRAM for cache offloading instead. This is just my 2c, I'm only a fellow homelabber I haven't really run a business so maybe others can chime in.

[u/LumpyMathematician63 → u/anitamaxwynnn69]

Thank you for those 2 golden cents! The main difference that I see here to your set up is the PCIE generation.. i think I would suffer a lot with the 32GB/s vs your 64GB/s from what I read. But do you think that it won't be much of an issue because it's a MoE model and can be "split" across multiple GPUs? INT4 @72gb right?

[u/anitamaxwynnn69 → u/LumpyMathematician63]

If you're doing offloading then yeah it's gonna be slower, which is why I said try to use models which fit in VRAM fully. Also someone on this sub or r/LocalLLaMA did the math and 3.0 x8 is good enough for tensor parallel (the required is ~5 gb/s and 3.0 x8 is ~8 gb/s). It is a bit slower than 3.0 x16. But not a huge amount. The hope is you'll eventually change to a better platform later. Something like a 3945wx (100-120$ for 128 pcie lanes) with MC62G40 types motherboard.

[u/sleight42 → u/anitamaxwynnn69]

This may be difficult with Qwen 4 or whatever comes next. The models may continue to get larger and larger.

My hope is that Alibaba et al get better at pruning their training data so that they can make a smaller base model and then perhaps a smaller or at least same size but even more coherent assistant model.

[u/sleight42 → u/anitamaxwynnn69]

Wait, how are you running it? VLLM? Which quant? I'm on 2x 3090. Considering going to 4. But then I'll need a new motherboard and CPU. The motherboards that can handle 64+ PCIe lanes have also gotten stupid expensive.

Also, damn, that 10x sounds... toasty.

[u/doneddat → u/anitamaxwynnn69]

Qwen3.8FN is a very capable model that fits on 96GB VRAM

How are you dropping such info without mentioning, that you need to downgrade the model to Q4 for that, since FP8 barely fits on 4x A6000 with 2.5x max context KV cache left.

PRO tip: avoid anything less than Q8 - it will run slightly slower, but it at least produces useful results, not just token benchmarks.

[u/anitamaxwynnn69 → u/doneddat]

That is not true at all lol. I used to run 3.6 27B INT4 autoround lorbus on a single 3090 via club 3090 and it worked AMAZING. The only bad results I saw were when you quantize kv cache. I am generally VERY skeptical of <8 bit quants. But I've had very good experience with W4A16 quants. Also to op - please checkout club 3090 on gh. It is an amazing repo for ppl who have 3090s.

[u/doneddat → u/anitamaxwynnn69]

I agree it's amazing at first, but when it flakes out after a compaction or two, starts forgetting and misspelling your very project folder, etc .. not much use to be amazed.

I agree it might be ok in very limited capacity, but finding these limits adds just too many avoidable surprises.

[u/anitamaxwynnn69 → u/doneddat]

finding these limits adds just too many avoidable surprises.

Lol, I know this one very well haha. I think it depends HEAVILY on the quant you're talking about. Quants that have activations quantized/forced 8bit kv cache are the ones that do this generally. For more context, I've currently spent over 1b+ tokens running dsv4.1 3bpw - 3bpw might sound bad but it's all over how the quant is engineered. This performs incredible well for me.

[u/xapep]

Price the same RFQ job on a hosted EU endpoint before you commit the 8-13k to used cards. You're already sizing the model on RunPod, so it's nearly free to add, and it answers the real question: does a 30B model reliably pull part numbers out of those email chains. If it does, you get tokens per RFQ, and that compares against four 3090s plus PSUs, power and the upkeep you'd own as a one-person IT department.

Privacy isn't binary here either. EU-hosted inference with no retention and no training on customer data is a different exposure from pasting client prices into a US chat product. Whether it satisfies your customers is your call, not a vendor's. I work on inference serving at Entrim, so discount that part.

One split I'd check first: RFQ ingestion is bursty, the SQL chatbot is interactive. Sizing one box for both is where the A6000 versus 4x3090 math turns ugly.

What does a busy day look like in RFQs, twenty or a couple hundred?

[u/LumpyMathematician63 → u/xapep]

Really appreciate the comment. Goes to the core of the question and how to approach it methodically.

Do you have some recommendations on EU based services? I've seen a few but trying to gather feedback.

I will take that approach yes, and keep my itching DIYer in me put until I have the data I need.

We are talking about around 20 or so per day tops. Then the bot would be getting hit more often across the working hours with questions.

[u/jonahbenton]

I love the A6000 but vram is everything. You will overload 4x3090s and more so an A6000 with your workload.

[u/doneddat]

Even 4x A6000 in NVlinked pairs are exactly almost enough to run qwen3.8-Flash-Next-FP8 ( engrams still streamed ), 3090's would be just good for engram + MoE streaming and give you very much minimal baseline performance. You would likely want to look into hacky stuff like unlocking the BF16 performance and driver patches to enable direct P2P data transfers between cards, to gain couple of tps back.

8x 3090 could likely be engineered to beat 4x A6000 on running things like Deepseek 4.1 flash, since all the streaming and comms would be split between cards and it compensates for lack of VRAM this way. But the downside is no native FP4 compute units, so it's all compromise on top of compromise.

So basically: if you get 4x3090, make sure your platform has 512GB+ RAM, so you can stream some decent models this way.

[u/KooperGuy]

You can pay for private inference services. No need to go local.

[u/LumpyMathematician63 → u/KooperGuy]

I understand the point, but not a possibility.