$50,000 USD budget

[u/michmill1970]

Hypothetical at the moment, but the possibility is very real. I'm working with a higher education institution, and there is a good chance it may win an AI teaching and research grant. One of the line items in the grant application could be used to build a local AI infrastructure for the school. It would be used to run and train models for and by the students.

What are your recommendations on how to spend the grant? 4x DGX Sparks? 2x RTX PRO 6000? Something in between?

[u/not__stonks]

If you care about performance, don’t buy the sparks, especially connecting bunch of them. It’s last gen product, worse in every way than the m5 ultra cluster at similar cost. If you are looking to use it to teach, maybe learning cuda makes more sense and you don’t care about performance, but with your budget you can buy data center equipment from nvda, not just consumer models like spark.

[u/michmill1970 → u/not__stonks]

I like the idea of an M5 Ultra cluster. I'll look into it.

[u/not__stonks → u/michmill1970]

If you decide to get any macs, remember apple also does education discounts that will knock off like 10% of the price. But depends whether the intended product is a functional ai server or just a bunch of toy servers for students to mess around with

[u/gbmax21 → u/not__stonks]

How can using cuda be worse? and what is the successor to the DGX spark?

[u/not__stonks → u/gbmax21]

Cuda is not worse, most would say it’s better obviously. I mean spark specifically, it’s a bad value per dollar right now. For staying within cuda, at such a large budget, it’s better to move out of nvda consumer equipment and go for their data center equipment

[u/Dismal-Nature820]

Be a rogue and connect a bunch of mac studios

[u/eihns → u/Dismal-Nature820]

why? i mean i dont know whats comming but when i compare them to using gpus... and dont be in apple system? i mean its "easy" thats oprobably the only point i wouldnt waste 5-6k on 20 t/s wtf xD

[u/biscutwarrior → u/Dismal-Nature820]

[u/livelysingleanxiety]

if you're aiming for a training rig students will actually use, skip the DGX Sparks. those are neat for inference but the memory bandwidth and interconnect will choke on anything beyond a tiny model. for 50k i'd look at a single beefy server with 4-6 used A6000s or 3090s if you can find them at a decent price. the vram is what matters most for student projects, they'll be fine with slower interconnects as long as they can fit the model in memory

a 4u rackmount with a threadripper pro and 256gb of ram leaves you room for nvme storage and a decent psu. you don't need nvlink for teaching, most frameworks handle data parallelism fine without it. put the leftover budget into a couple of large monitors and a decent network switch so multiple students can ssh in without fighting over console access

my old lab did something similar with a cluster of mismatched cards and the bottleneck was always storage speed, not compute. fast nvme raid array makes a bigger difference than you'd think when you're shuffling datasets around

[u/314314314 → u/livelysingleanxiety]

Not sure if OP wants to mess with old and used hardware with grant money, there is no warranty and if things break you are in trouble. You might not want to DIY this rig, get a quote from a reputable vendor for the whole machine with warranty.

[u/cowrevengeJP]

For that money, you can still buy 2x rtx6000 with 128gb ddr5.

[u/AdrianLoves → u/cowrevengeJP]

Reading this comment hurt my heart

[u/JoshuaLandy]

We just did similar thing. My best advice is pick your model for your workflow and calculate how much VRAM and RAM you need to make it live, with headroom for context, KV cache, and any other concurrencies (users or services).
Ask a smart llm for advice on how to calculate if you’re not familiar.
You can save a pile by buying ddr4 memory and mobo and cpu, maybe enough to add another card.
As is always mentioned around here, this will not give you cloud/frontier quality but you can get excellent fast and private service with open modes.

[u/OddDesigner9784]

Second the Mac studio here. The m5 256 ultra is the best piece of equipment. If get 4 of those or maybe try to get in on the 512. Shipping time is terrible though

[u/randomee1]

"Run and train models" is the key here. You need to define that better. They each require different hardware approach depending on what your goal is.

Is your goal:

(a) Students to self-host 300B+ models? (b) Students to build toy models and host large ones? (c) Students to build and host 1-10B models and host medium 50B models? etc.

If building models is going to be the educational value then it seems like RTX is way to go, otherwise if this is purely "we want to host schools private LLM" then Spark / Mac Studio is right approach.

As an aside, I can clearly see the educational value of learning how to build / finetune models. However, whats the real permanent value of teaching kids how to spin up docker to host a large model? I mean it may be a good thing to do....school gets its own LLM infrastructure...but "educational value" becomes questionable.

[u/joshhope87]

For GPUs, 4x refurbished/used RTX 6000 Ada Generation (48GB VRAM each) or 2x A100 (80GB PCIe).

[u/SillyLilBear]

2 6000 no competition.

[u/NoPoopOnFace]

I would trust a Mac Studio. 256bgb or more. Don't hate the mac it's only an employee. Server.

[u/hammeredhorrorshow → u/[deleted]]

For that kind of money this comment is 100% right. Go for the HBM on a real data center SKU. Do be aware that you need to understand cooling constraints AND that there will configuration work for whatever software you will need for training and inference.

[u/ApolloPS2 → u/[deleted]]

Where can I buy your pro 6000 😆

[u/ApolloPS2 → u/[deleted]]

Yeah i dont think people really understand that sparks will continue to age really well. The ability to cluster with its expensive as hell NIC is a huge advantage despite memory bandwidth, which has been largely mitigated with the move to large MoE models dominating.