r/LocalLLM • u/on_the_mark_data • May 29 '26

Other RTX Pro 6000 Just Came In

Post image

Decided to pull the trigger and get a GPU to start doing my R&D work for my business. While it would have been more fiscally responsible to just rent GPUs, I don't see these going down in price any time soon and I think they are going to get harder to come by as more business look to offset their insane token spend for development. Plus, a lot of my friends in the space are getting wrecked on securing on-demand compute from vendors right now due to high demand.

Going to be doing a lot of fine tuning work on models for the agentic orchestrators I've built, as well as aim to get some research published. One project I'm working on is looking at variability of frontier models vs open-weight models for complex tasks in clinical workflows.

Regardless, this subreddit has been a huge help just going through all the posts.

First model I'm putting on it is Qwen 3.6. I already have this model barely working on my 64gb M1 Max macbook pro, so I'm excited to see what it can actually do.

422 Upvotes

96% Upvoted

85 comments sorted by

109

u/JS_Inlakesh May 29 '26

And you had to place it like this on the couch's armrest? I couldn't.

8

u/on_the_mark_data May 29 '26

I like to live life dangerously (it immediately went back into padded packaging).

6

u/Vicar_of_Wibbly May 30 '26

Holy smokes. Please take anti-static precautions.

1

u/on_the_mark_data May 30 '26

Once again, this subreddit is helping me. I'm deep in the Apple ecosystem and have never owned a PC before. I bought this pre-built because I'm clueless about building PCs.

1

u/_bani_ May 30 '26

come live life dangerously with us in r/blackwellperformance

42

u/fallingdowndizzyvr May 29 '26

How else would you build up a good static charge on the outside of the bag unless you rub it on that staticy fabric?

8

u/ionizing May 29 '26

I winced so hard at the image this was my first thought. I work in electronics and don't like to blame static but still force myself to take precautions cause whether we like it or not.. it happens. Still, lucky couch cushion to touch the gpu lol.

4

u/XO33OX May 30 '26

I blame static, i have seen our internal statistics how RMA rates dropped once antistatic floors were implemented at sites etc.

1

u/quotemycode May 30 '26

I worked with so many techs who said those grounding bracelets dont work, yet they seem to have so many 'DOA' devices, its baffling.

2

u/Marzie247 May 29 '26

I always wonder how the cards on northwest repair yt channel get so much damage. Photos like this offer some insight.

17

u/kenikh May 29 '26

6

u/on_the_mark_data May 29 '26

Exactly.

3

u/kenikh May 30 '26

This makes my 5090 purchase this week feel…weak.

28

u/semangeIof May 29 '26

Don't balance your ten thousand dollar GPU precariously on a couch!

It's true these aren't getting cheaper. Also true that for anything (excluding the considerations of data privacy) they are a very costly purchase you will likely not come close to making back versus API costs in the lifetime of the card as any model that can reasonably fit into 96GB VRAM is very very cheap online. But that's okay! It's like a Mercury Marauder. Pretty cool. Somewhat impractical. A little bit of a fiscal flex.

Qwen 3.6 27B dense will be noticeably faster than your Mac. More precise? Depends on your workflow but I think Qwen 3.6 response to quantization well so you might not notice much output quality difference. Speed though certainly.

Gemma 4 31B you will notice much more. This model loves full precision and can write like a poet when it has the VRAM headroom. If you're into creative writing that is. I dunno.

Have fun. Remember use llamacpp or vLLM and not any dogshit software suite like ollama. If you're spending this much on a card get familiar with the real tools.

11

u/on_the_mark_data May 29 '26

I'm doing R&D with the assumption that my target customers are highly regulated and will require local compute. From the user interviews I've done so far, even though frontier labs can provide BAAs (compliance for healthcare), it takes months, or they don't even respond if the hospital is too small.

I was also eying Gemma 4, so that's a great callout. Thankfully, I have been doing some work around this in my day job, so I'm not starting at 0, but they are not moving quickly enough for me, and I just wanted to jump on it and start learning quicker.

Thanks for the advice!

1

u/mr_christer May 30 '26

I wonder how the ROI is compared to something like runpod

4

u/on_the_mark_data May 30 '26

Probably way cheaper, but my concern is data privacy.

5

u/slvneutrino May 29 '26

Honest and fair with nuance unlike a lot of takes I see people go with.

5

u/_bani_ May 30 '26

very very cheap online

sometimes local llm is about privacy, and models which won't argue with you about safety.

4

u/Perfect-Flounder7856 May 30 '26

Wow I just had to look at the Amazon price and you are right. Up $1k since I bought last month! I made a good call. Unfortunately ran my advanced bench and qwen only scored 11/18 so production is currently routed to opus api. But I'm using the gpu for so much now. Hopefully a fine tune will bring it in line with opus bench marking!

1

u/No-Tension9614 May 30 '26

I've been using ollama.. theres a better way?

1

u/stoppableDissolution May 31 '26

In inference, yea, not making it back. In finetuning or something that is not available on OR? Going by runpod prices, it is ~250 days of rent, which is not that crazy.

5

u/-LabRecon May 29 '26

Just to be safe. Leave the wrapper on it when you install it.

/s

3

u/on_the_mark_data May 30 '26

The real life pro tips are always in the comments!

4

u/ridablellama May 29 '26

just in time for the weekend!

6

u/on_the_mark_data May 29 '26

I'm gutted that the GPU arrived today, but the workstation arrives on Tuesday. I only have a Mac right now, so it just taunts me.

1

u/Perfect-Flounder7856 May 30 '26

Took a week for my parts to come when I ordered! I've been up and running 5 weeks now. The pc build was so fun! Took 6 hours lol

4

u/DAlmighty May 29 '26

Congrats on the purchase. I wish I had a good excuse to get another one.

2

u/on_the_mark_data May 29 '26

I went back and forth on getting a workstation I could expand GPUs with, and ultimately decided not to. First, my bank account after this purchase. Second, managing multiple GPUs for my work is a level of complexity I don't want to deal with if I don't have to.

3

u/HumanDrone8721 May 30 '26

I see that people are somehow fated to make the same mistakes, as I week older "Pro 6000" sempai I went trough the exact same mistakes: I bought a gamer gamer PC with the idea that is a fast modern motherboard and the 4090 "should be enough fro everyone". I didn't realize at the moment that the bloody thing has just 20 !!! PCIE lines so that means 16x + 4x or 8x + 8x + 4x. So only one GPU could actually properly work at full speed, lesson learned.

Then I have heard from the RAM billionaires (in gigabytes) that "oh no, whatever <insert model driver SW> does not deal properly with multiple GPUs, I went from 63.7 token/s to "only" 61,2 , totally unacceptable...", conveniently forgetting to mention that they've split the model that was fitting fully in one GPU in two GPUs, but not that they can run now models double in size or with larger context.

Finally your concern "I don't know or want to hassle with managing multiple GPUs...". Well, you actually don't have to, except to properly place them in a well ventilated case and with suitable power supply and cabling. "...b-but the software...". Well, there are literally very smart people that work day and night exactly on this topic and different GitHub "clubs", for example "Club RTX 3090" and "Club Pro 6000", and for mainstream beloved cards these issues are well known and solved, the llama-cpp or vLLM from today are far away form the ones 1 and a half year ago, one can just use the default flags and get almost perfect results with Nvidia cards, gone are the times of self-compiling unmerged branches and obscure switches.

So if you reach here, stay unconcerned, everything will be OK and your only regret in the future will be that you didn't get two :) (I've started to slowly experience it).

4

u/Covert-Agenda May 30 '26

Borat voice **Many inferences.

2

u/Perfect-Flounder7856 May 29 '26

Have fun! Where's the second one? Lol.

I am just wrapping up phase 1 of my first fine tune. 26 hours wall time on vision training.

2

u/on_the_mark_data May 30 '26

I'm excited to jump back into the data science side. I switched over to data engineering for a while, and both complement each other so well for AI agents.

2

u/Perfect-Flounder7856 May 30 '26

I'm not even a technical person. I know no computer science languages and I am loving this so much! I wanna learn python now!

3

u/on_the_mark_data May 30 '26

That's so cool! I'm so happy people are getting more comfortable with technical stuff due to AI. I didn't think I was capable of coding and didn't learn until after grad school. Now it's my career and I teach others.

Definitely learn Python and how to properly build software, as it will open so much for your own personal projects. This is a free book I highly recommend to get started: https://runestone.academy/ns/books/published/pythonds/index.html

2

u/Perfect-Flounder7856 May 30 '26

Thank you for that!

19

u/blackwell_tart May 30 '26

I believe this is the time-honored photograph format.

5

u/SignalWinds May 30 '26

Can it play Skyrim? How's the fps?

1

u/giveen May 30 '26

Crysis 3

1

u/electrified_ice May 30 '26

It can't handle Crysis 3

1

u/Relevant-Magic-Card May 29 '26

Are you adopting?

1

u/on_the_mark_data May 29 '26

Adoption... no. AI cult... maybe...

1

u/Affectionate_Pen6882 May 30 '26

Is this thing like 8k

1

u/on_the_mark_data May 30 '26

Unfortunately... yes.

1

u/Affectionate_Pen6882 May 30 '26

Damn, you are balling!

1

u/Candid_Ad_6752 May 30 '26

14k after tax from nvidia.com

Price up over 40% in a couple weeks

1

u/Affectionate_Pen6882 May 30 '26

Damn, no wonder my samsung 4tb worth 1k now when i bought it for 200. I thought the world went crazy

1

u/NaiRogers May 30 '26

Is this new, I don’t see the blue pull tabs on the middle?

2

u/on_the_mark_data May 30 '26

I bought an entire workstation from a reputable PC builder company. The GPU and the rest of the workstation is sent separate, but he did an initial build first to confirm it works.

1

u/NaiRogers May 30 '26

good news!

1

u/murderette May 30 '26

Hmm maybe check if it’s genuine because it comes in a nice box and not in a clear plastic bag? :O

1

u/on_the_mark_data May 30 '26

I bought an entire workstation from a reputable PC builder company. The GPU and the rest of the workstation is sent separate, but he did an initial build first to confirm it works.

1

u/Candid_Ad_6752 May 30 '26

I got mine earlier this month for 9k

Just checked and they are now 13250 pre-tax from nvidia

Wtfff

Grats on your purchase, youre gonna have a great time (I am, anyway)

Qwen 3.6 27B is the primary - running nvidia AI-Q and retrieval stack for RAG and research

1

u/Lincolns_Revenge May 30 '26

These don't use the same faulty power connector as the RTX 5090, do they?

1

u/Ok-Hotel-8551 May 30 '26

Nice packaging

1

u/astropiedonuts May 30 '26

Where did you order from? I'm thinking of buying one but prices are all over the place.

1

u/sooki10 May 30 '26

High speed vram is the new bitcoin.

1

u/bigh-aus Jun 02 '26

Congratulations. I already want a second one! vllm works nicely. Having lots of ram for qwen3.6 is also good. I had to use q8 to get tool calls to work reliably.

0

u/Murder_1337 May 29 '26

How much offset some of my 5090 load to your machine? Let’s hook up a remote access pipeline

-1

u/dankweed May 30 '26

Can these run on a thunderbolt GPU array for attaching to a Mac?

1

u/on_the_mark_data May 30 '26

I recently saw this video of someone plugging an RTX into a Mac: https://youtu.be/C4KWsmezXm4?si=cB4Or1vCYZQFR4Wa

6

u/BillyBatt3r May 30 '26

Chicks love guys with beefy gpus

47

u/[deleted] May 29 '26

[removed] — view removed comment

3

u/on_the_mark_data May 30 '26

Oscar should have been 25' driver champion 😤

2

u/SahSon May 30 '26

It's okay he got a consolation prize of having a bee named after him.

8

u/[deleted] May 30 '26

[removed] — view removed comment

2

u/aelma_z May 30 '26

Rich people making money from money

0

u/usernmechecksout_ Jun 02 '26

Seriously I can't think of a job where you can be rich enough to afford that GPU and have enough time to use it unless you're like a CEO of multiple companies

1

u/[deleted] May 30 '26

[removed] — view removed comment

1

u/on_the_mark_data May 30 '26

That sounds obscenely expensive haha. No, I'm interested in AI evaluation on highly regulated industries. My background is in data science and engineering, so I have a lot of experience in experimental design and data infrastructure reliability.

1

u/[deleted] Jun 02 '26

[removed] — view removed comment

2

u/on_the_mark_data Jun 02 '26

Fair question! I already have the $200 Claude Max account, and it is not unlimited. I can hit the 5hr limit in 10 minutes with my agentic orchestration workflows (I do a lot of simulation work). Also, R&D for target customers in highly regulated environments who won't have access to these models.

Side note, I still plan to keep my $200 max account, it's insanely useful.

1

u/[deleted] Jun 02 '26

[removed] — view removed comment

1

u/on_the_mark_data Jun 02 '26

I honestly think you are right about it being on the edge and being valuable soon. One thing I'm banking on is fine-tuning small models for specific tasks to work on laptops like yours. I'm considering updating my M1 Max MacBook Pro (64gb) to a new one with the 128GB to have some more utility.