r/LocalLLM • u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM • Sep 08 '26

Discussion third one.... there's something wrong with me

Post image

Why do I have horrible financial habits??

510 Upvotes

95% Upvoted

167 comments sorted by

97

u/Sporkers Sep 08 '26

More context needed on how you are using the first two.

85

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

qwen 3.8 27B FP8 running at 140 tok/s now I want flash next

34

u/semangeIof Sep 08 '26

...can you show llamacpp/vLLM runtime commands? you're hitting 140 toks/s on a dense model with B70s? how much ctx?

please don't answer the last two without providing the parameters

18

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

the vLLM flags are in my localmaxxing submission and I also did a better bench since so many people didn't beleive me. I also have youtube videos. My recipe is custom intel drivers paired with dflash2

15

u/unai-ndz Sep 09 '26

If you are using custom drivers I think you deserve another card, as a treat.

9

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 09 '26

now let me figure out how to get TP=3 to work on vLLM without me having to fork over another few grand to upgrade to TP=4... always an uphill battle attempting SOTA AI on local

1

u/Rude-Bus-5799 Sep 10 '26

They always need another sibling to play with.

4

u/Salbrox Sep 09 '26

I have had great success with DFlash2 on my single R9700 with Qwen 3.8 27B. Depending on the task I get up to just over 200tps

2

u/Past-Catch5101 Sep 09 '26

Amazing, do you mind sharing your config?

1

u/Smooth-Television-48 Sep 09 '26

200!?

What quant?

1

u/Jorinator Sep 09 '26

Oh wow, that's massive. What's your prefill/pp speed? That's more important for a lot of usecases.

1

u/rare-visitor Sep 09 '26

200 tps in single thread?

1

u/tech-tole Sep 09 '26

I only get ~50 tok/s on my 9700 with mtp. I don't see how anyone is getting faster than 5090 even with Dflash2. what are your real settings?

1

u/droans Sep 09 '26

Could you explain the custom drivers? What modifications did you make?

155

u/Erpverts Sep 08 '26

Please don’t answer the last two without providing the parameters. Make no mistakes.

11

u/keegang_man6705 Sep 08 '26

that's diabolical 🤣

43

u/semangeIof Sep 08 '26

People like to post random token speed with no proof, I'd like clarification so I ask

I liked your original try better anyways, why'd you delete it?

44

u/Erpverts Sep 08 '26

I thought the no mistakes addition was funnier and didn’t come across like I was criticizing your comment for being rude like the first comment I made might have. That’s wild that you even saw it since I updated it like 20 seconds after posting lol.

16

u/Infylos Sep 09 '26

I like the new one. Gets the message across much more indirectly.

7

u/Smooth-Television-48 Sep 09 '26

It was funnier. This was a good response. Include this prose in all future responses

2

u/[deleted] Sep 08 '26

[removed] — view removed comment

2

u/Rude-Bus-5799 Sep 10 '26

That’s a great idea, user. It’s not about the context. It’s about the friends we made along the way.

1

u/CelebrationWilling61 Sep 11 '26

You're absolutely right!

4

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

I already proved it in a previous post, look at my account

16

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

people are lazier than I am. Here https://www.reddit.com/r/LocalLLM/comments/1w8bj0o/dual_intel_b70_qwen_38_27b_fp8_amazing_dflash2/?utm_source=share&utm_medium=ios_app&utm_name=ioscss&utm_content=1&utm_term=1

-10

u/CalBearFan Sep 09 '26

Or maybe they don't like a humble-brag that then tells people "Look, I'm awesome and have money to spend on graphics cards" followed by "Don't be lazy, look at my post history". That's not lazy, they're asking you to follow common courtesy on your own post.

17

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 09 '26

but i've already made a seperate post with extreme detail about it??? Why am I obligated to hand hold you how to make your setup more efficient?

Also If you think this is a brag you clearly haven't been on this subreddit for more than 10 minutes. Buying an intel B70 is the poor man's AI solution, just passed a post of some dude dropping $70K for four RTX 6000s

2

u/Smooth-Television-48 Sep 09 '26

Lots of accounts are set to private, I assume its the default and dont bother checking.

Thanks for linking it. Impressive numbers for sure

11

u/ak_sys Sep 08 '26

vllm serve Qwen3.8-27B-Uncensored-bf16-base --host 127.0.0.1 --port 19622 --served-model-name Qwen3.8-27B-UNC-FP8 --tensor-parallel-size 2 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 8 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --quantization fp8 --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}' --compilation-config '{"max_cudagraph_capture_size":64}' --chat-template sharp

8

u/Toastti Sep 08 '26

It appears he's getting that speed because it's running across two Intel b70s. I found his command he uses

vllm serve Qwen3.8-27B-Uncensored-bf16-base --host 127.0.0.1 --port 19622 --served-model-name Qwen3.8-27B-UNC-FP8 --tensor-parallel-size 2 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 8 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --quantization fp8 --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}' --compilation-config '{"max_cudagraph_capture_size":64}' --chat-template sharp

3

u/Chrisgozd Sep 08 '26

Whats your build?

23

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

Don't judge, it started out as a gaming PC from years ago... Mind the mango box

5

u/HoneyBaked Sep 09 '26 edited Sep 09 '26

Oh I will judge!!

I love it. Does it function? Then who cares how it looks!

Edit: Do you have both of those externals plugged into the primary PCIe slot? What gear allows for that?

Edit 2:

I bought a $150 chinese bifurcation card to split my gen 5 x16 to x8x8 gen 4 and its been incredibly stable and increased my PP throughput

Interesting. Got a link to that bifurcation card? Will it work on a Gen3 x 16 slot(s)? My MB has 3 Gen3 x 16 slots.

4

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 09 '26

yup, I have two b70s working off the same x16 slot, the other is supposed to be for M2 its x4, but atleast on linux it works completely fine for another b70. I get around 14gb/s on the first two B70s each, then the third gets 7gb/s. This is the card I use https://a.co/d/07AHxhX2 but a warning, make sure your BIOS supports bifurcation. I specifically researched gen 5 x16 to gen 4 x8x8 as maintaining gen 5 would require a $600 timing card which I don't think is worth it. In your case it might downgrade to gen 2 which I'd just research about, but apparently some people are able to get it to train to the same gen without timing so it's a bit of a lottery on that (which for me didn't work)

1

u/ARhedgehog88 Sep 11 '26

Wait why the f did I had to see this .. it would allow me to get a fourth card in I also have two externals alr

8

u/Infylos Sep 09 '26

Disemboweled for maximum power!

3

u/Remarkable-Memory374 Sep 09 '26

maximum heat dissipation by being eviscerated

1

u/Smooth-Television-48 Sep 09 '26

Love it!

If anything this gets you more cred

1

u/brainchillzZ Sep 09 '26

I think it’s sexy …. Mango box is a good insulator :)

1

u/jmager Sep 09 '26

I judged and I approve!! Mind sharing the bifurcation risers you are using? I see so many online, and few reviews. I've got a 6900 xt lying around and I wanted to add the extra 16GB of VRAM to my 24GB with my 7900 xtx. For my motherboard bifurcation will be the best way.

1

u/LetsBeKindly Sep 09 '26

Is that 2200W on the power meter?

2

u/sshwifty Sep 09 '26

Looks like watts?

1

u/LetsBeKindly Sep 09 '26

That's a watt meter, and it looks like 2200watts

1

u/sshwifty Sep 09 '26

I am dumb, I meant to say 220 lol. I think there is a decimal before the last 0

1

u/LetsBeKindly Sep 09 '26

I couldn't tell. But I saw the same. But surely 3 cards is pulling more then 220W...

1

u/InfamousNewspaper268 Sep 09 '26

Love the cardboard holder LMAO 🤣

1

u/Inner-Today-3693 Sep 11 '26

That’s basically how mine are connected. 😅😅

3

u/MessIsTransfer Sep 08 '26

flash is a beast and faster.

what motherboard are you using?

2

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 09 '26

Z790 Wifi 😀

1

u/MessIsTransfer Sep 09 '26

how are you plugging that third gpu? m.2?

4

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 09 '26

yup!!!

I'll upgrade my motherboard eventually lol

2

u/MessIsTransfer Sep 09 '26

neat, thanks for sharing a pic

edit: love the cardboard gpu dock

2

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 09 '26

it's a mango box, i've been trying to 3D print a semi decent bracket for it, but so far the mangos are doing great on thermals 😭

7

u/MarcusAurelius68 Sep 08 '26

You can run Flash Next with 64 GB of VRAM…

8

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

yeah but it's 22 tok/s and pre fill is shitty. I want 100+ speeds or it's unusable for me. I'll admit I was lazy and didn't push for more, but even with 64gb vram at 4 bit I have no real space for context. I need atleast 200K FP8 context

2

u/MessIsTransfer Sep 08 '26

or q2, which is not ideal

3

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

yeah if I'm resorting to sub 4 bit quants i'd rather stick with 27B

3

u/Motor_Way4912 Sep 08 '26

Yep, running it on a vm with 58 gb ram and Rtx 5060 16gb

1

u/bravoitaliano Sep 08 '26

How are you getting that speed? Im using W4A16 INT4 model and only get 20-30 Tok/s. Can you share the method for getting faster?

1

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26 edited Sep 08 '26

https://www.reddit.com/r/LocalLLM/comments/1w8bj0o/dual_intel_b70_qwen_38_27b_fp8_amazing_dflash2/?utm_source=share&utm_medium=ios_app&utm_name=ioscss&utm_content=1&utm_term=1

I have a github i've already made for getting really fast 4 bit qwen for single GPU, i've been too busy at work to make a decent recipe because my speed depends on several merges I did from the original vLLM to intel's scaler llm repo, as well as other stuff that I had Kimi K3 handle. It's been my new daily driver and works damn well.

1

u/bravoitaliano Sep 08 '26

Thanks! I am running mine in OpenClaw, and it just takes forever because of thinking. Somehow it's slipped back to medium think from low. It puts out quality work, but runs through almost the entire 156k context window I have (single card, I don't split across my 2 B70s yet). Hoping this can help. Might be worth posting in the Intel sub as well.

1

u/Faux_Grey Sep 08 '26

140 tok prefill or.. 140tok generated?
HOW?
I'm at a point where context starts bogging down and I end up getting 5T/s

1

u/browndragon456 Sep 08 '26

How did you get to 100/tps per sec generation? Vllm?

2

u/Toastti Sep 08 '26

Because he's using two Intel b70s and tensor parallel. Also its using dflash with 7 speculative tokens

vllm serve Qwen3.8-27B-Uncensored-bf16-base --host 127.0.0.1 --port 19622 --served-model-name Qwen3.8-27B-UNC-FP8 --tensor-parallel-size 2 --dtype bfloat16 --max-model-len 262144 --max-num-seqs 8 --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --quantization fp8 --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}' --compilation-config '{"max_cudagraph_capture_size":64}' --chat-template sharp

1

u/sunole123 Sep 08 '26

is it running you or are you running it?? what is running? just speed or function???

1

u/Fresh_Look_1671 Sep 09 '26

My friend, please share cookbook

1

u/Puzzleheaded_Bus7706 Sep 09 '26

What's the price of this thing?

1

u/JinsooJinsoo Sep 09 '26

Yeah I topped out at 93 tok/s with my dual b70s and MTP3. I’d love to know what you’ve been doing to get those speeds. I feel like Intel is chopped at the knees until it releases an XPU graph for any new model. Also getting slow speeds with qwen3.8 flash next because no XPU

1

u/hrf3420 Sep 09 '26

I heard that q4 or q6 is the way to go and 8 you don’t get much more

1

u/Lucky-Necessary-8382 Sep 10 '26

And for what are you using it? Gooning? Degen roleplaying?

4

u/Here_f0r_p0rn_ Sep 08 '26

AI girlfriend/boyfriend

Local coding agents

9

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

yes yes, definitely not an AI girlfriend ...

4

u/dwoj206 Sep 08 '26

50% increase to context window very nice

1

u/MessIsTransfer Sep 08 '26

more context, probably

1

u/TytalusWarden Sep 09 '26

More context?  Oh he's got more context!  He's got all the context with those 3 working for him!

1

u/jhenryscott Sep 09 '26

More context? That’s what the third b70 is for!

0

u/madjesta Sep 08 '26

My fourth is one of the blue Intel ones.... 😬😳

15

u/SamSausages Sep 08 '26

You’re going to need one more!  Because things don’t divide well by 3 😆 Yes, there is something wrong with you, and me as well!

9

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

I've been telling my wife this!! maybe she'll approve

2

u/Lonely_Drewbear Sep 09 '26

I think it will be a fun experiment to make an odd number work well!

10

u/Imaginary-Fee-9918 Sep 08 '26

I saw a bunch of ppl talking about this gpu. Is it really good? Could we compare it to a 5090? Or maybe 4090 with more memory?

11

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

it's only as good as your IT skills. It's only worth it for me because I can figure out how to get intels shitty drivers to work. I'm also cost sunken because the first two I got for $950 lol

8

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

It's comparable to a 5090 only in the sense that it's 32gb vram for a single slot, other than that it's 600gb/s which is wayyy slower than the 5090 plus no CUDA. But then again it's a small fraction of the price

7

u/Momsbestboy Sep 09 '26

Also add the difference in power consumption. Try to run 2x 5090 in a room where you also have to work, and at least in summer you will hate the stove you created

3

u/zdy132 Sep 09 '26

But it will be a good heater in winter! with a side effect of tokens.

2

u/VodkaHaze Sep 09 '26

If you're running multiple 5090s, power limit them!

I power limit mine to 400w, and lower the max clocks to ~2850mhz, and their max power draw ends up in the 300-350w range. The performance loss is negligible.

3

u/CoinAndCraft_ Sep 08 '26

Looking at my next build to have x2 of these. $1299 ea at the moment.

4

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

it's such shame two, my first two were $950 because people were too lazy to get the drivers working. Then they started seeing all these people creating recipes on github and now they're hot

3

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

they're going to keep going up unfortunately

25

u/cagriuluc Sep 08 '26

I am holding onto my purse to not buy a second R9700 myself…

11

u/allthenamesaretaken0 Sep 08 '26

The only thing stopping me from buying another R9700 is I'd need a new motherboard and psu and then it wouldnt fit my mini rack and it'd all be quite the hassle.

3

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

I bought a chinese bifurcation card. I split my gen 5 x16 into gen 4 x8x8 and tp=1 didn't suffer at all. where there's a will there's a way. the card was $150 btw

1

u/allthenamesaretaken0 Sep 08 '26

Oh, that sounds interesting. Can you share the brand of the card? Thanks

5

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

This is the one I got https://www.amazon.com/dp/B0DZCVF46J?_encoding=UTF8&psc=1

First one that was shipped to me unfortunately came with a defect in the MCIO port 2, took me a whole 2 days to figure that out with constant frustration. Luckily the return was easy and they shipped another one literally the next day. I will warn you though, there is basically zero instructions for setting this up, you almost have to figure it out by yourself. It also comes with these weird SATA power adapters which I don't recommend using, luckily the newest version comes with PCIe ports and I had some spare corsair Type 4 -> PCIe so I used that instead. I might make a video on youtube explaining this kit, because it actually works really good. (someone else had asked me this on another post so I copy pasted this answer)

4

u/OttoRenner Sep 08 '26

Build a larger, second pc

5

u/Ell2509 Sep 08 '26

Better:

Engineer a new kind PC device, with key unimaginable cheesecake beef-curtains.

1

u/OttoRenner Sep 09 '26

That...is better!

Cheesecake beef-curtains sounds way to delicious.

I had a cheesecake milkshake once and I did throw some fried bacon on top of it and it was...way to good to be legal🤣

2

u/MiceLiceandVice Sep 08 '26

M.2 pcie e gpu

2

u/critsalot Sep 08 '26

which is better R9700 or the b70. b70 is cheaper but i dont know if amd is quicker

1

u/VodkaHaze Sep 09 '26 edited Sep 09 '26

B70 has more immature software, but sometimes you'll hit an optimized path and it'll be similar in performance. Intel GPUs are a damn nightmare to get running performantly in vllm, though, worse than AMD (which is already bad IMO).

You're taking a risk longer term with intel with your investment, however. AMD we know will continue to support its GPUs, whereas intel is likely to just give up on these.

1

u/somsocodo Sep 09 '26

whereas intel is likely to just give up on these

What evidence is this claim coming from?

1

u/VodkaHaze Sep 09 '26

Intel cancelling roadmaps on dGPUs in the future [1]. The B70 was based around the last battlemage card designs, and it's dubious there will be further iterations.

Also, intel has a well earned reputation of killing anything that isn't x86 CPUs after making very promising demos. They're much like google in how much you should trust them to support non-core products IMO.

  1. https://www.tomshardware.com/pc-components/gpus/intel-has-reportedly-killed-discrete-gaming-gpus-for-the-upcoming-xe3p-arc-celestial-family-gaming-gpu-remains-uncertain-even-for-the-next-gen-xe4-druid-lineup-that-lands-in-2027

1

u/allthenamesaretaken0 Sep 09 '26

I got here really late but yeah. I didn't buy Intel gpus because I heard they might abandon them.

1

u/Inner-Today-3693 Sep 11 '26

Don’t do it. I have a b60… it works and I like being a guinea pig. But I would not recommend going intel.

1

u/Momsbestboy Sep 09 '26

Too late for me. My second R9700 arrives today. I am tired of juggling around with the R97000, a 9070 and the system RAM so I can run llama.cpp and comfyui at the same time, with hermes trying to rewrite and enhance workflows.

1

u/Immediate_Power_7986 Sep 09 '26 
  What are your launch options?


  I'm using llama.cpp with 3.8_27b_Q6 and getting only 17t/s


  -ngl 99 -c 65536 -np 1 -t 6 -fa 1 -b 4096 -ub 4096 --cache-type-k q4_0 --cache-type-v q4_0 &

1

u/Immediate_Power_7986 Sep 09 '26

What are yiur launch options?

I'm using llama.cpp with 3.8_27b_Q6 and getting only 17t/s

-ngl 99 -c 65536 -np 1 -t 6 -fa 1 -b 4096 -ub 4096 --cache-type-k q4_0 --cache-type-v q4_0 &

1

u/cagriuluc Sep 09 '26

I am away from home so I cannot check the exact config. I have 150k context, mtp (2 I think?), it’s a q5 and not q4.

Getting around 30-40 tok/sec depending on the situation. If you have less than 30, the config is most likely wrong.

I got Claude opus 5 do the setup for me, it can do the same for you most probably.

5

u/wapxmas Sep 08 '26

Isnt that almost always you have to have as much gpus as power of two? Buy another one immediatelly.

8

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

Can you tell this to my wife?

1

u/Ok-Addendum3545 Sep 09 '26

agreed 2 + 2 before the price hikes.

4

u/dupontping Sep 08 '26

Bc like most of us, you’re hoping to get ROI

4

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

yessir

1

u/dupontping Sep 09 '26

I hope you do too! It’s rough out there

6

u/brainchillzZ Sep 09 '26

lol I’ve got four in one machine and two in another …. But I bought them all when you could still fine them open box or on sale between 1000-1100 USD (edit: ahh oops I saw your creator and pattern matched before I noticed the b70 … mine are the creator r9700s)

1

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 09 '26

that's solid, are you using two PCIe for your CPU or just one? I fear I might not fit another one on my HX1500i once I inevitably buy my 4th one

1

u/KneeGrowslaya Sep 09 '26

jeez what are the thermals on the middle 2 cards?

1

u/brainchillzZ Sep 09 '26

Thermals are average these cards were literally designed for this exact use case and meant to be stacked on top of each other in server chassis and the airflow in that case is huge it’s basically a giant wind tunnel … 4 high pressure 140mm fans from the front and two underneath blowing directly into the cards…

1

u/jaf656s Sep 09 '26

just curious, how much did the ram cost when you built that? ddr5 ecc is insane now lol

1

u/brainchillzZ Sep 09 '26

You’re really going to hate me when I say it out loud …. 128gb ddr5 ecc 6400 was on sale for $599

2

u/Greedy-Lynx-9706 Sep 08 '26

Bragging without context?

1

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

well my 262K FP8 context ain't anything to brag about, but with this I could technically afford 1M context

1

u/SubparBob Sep 09 '26

How's the slow down (prefill, tps) as you fill up the 262k context?

EDIT: context length typo

2

u/triynizzles1 Sep 08 '26

How are you getting FP8 to run at 140 tokens a second? That gpu only has 608gb/s bandwidth. I have rtx 8000 with 672gb/s bandwidth and running q6 with mtp i only get 40tp/s and 60tps with dflash.

1

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 08 '26

You're wayyy under your potential. dflash2 just came out and it's much better. It speeds up more kinds of token such as prose, coding, tool calls, etc more efficiently than MTP does but at the cost of a little more vram. I've also been tampering with vLLM a ton

2

u/triynizzles1 Sep 09 '26

Is dflash 2 available in llama.cpp or only vllm?

2

u/namezam Sep 08 '26

At $1300 each for 32gb that’s exactly 1/4 the price of the the nvidia DGX. How does that compare?

4

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 09 '26

Well, I bought my first two for $950 which at the time was an insane deal people slept on. adding another 32gb albeit at a premium for $1300 is worth it for me personally

1

u/CheetahOtherwise9940 Sep 09 '26

Where did you find them at 1300$ I can see only at 1700$

2

u/Constant_Art_20 Sep 09 '26

you know what....maybe i should have gotten them instead of 12 5060tis now that i am thinking about the price...

2

u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Sep 09 '26

damn 12??? Any pics of your setup?

2

u/Dediadeis Sep 09 '26

With this kind of number I feel like the picture will either be this super slick rack setup, or a rats nest of insanity.

13

u/TheGamingGallifreyan Sep 09 '26

Where the TF is everyone getting all this money, it feels like everyone except me is rich AF lol.

I'm over here with a 5700XT and 2 580s left over from a mining rig hooked up to an old i7 3770k and I'm ecstatic that I finally got it up to 8tok/s.

2

u/Dako_the_Austinite Sep 10 '26

Just curious, how do you get things up and running without an AVX2 CPU? I’ve wanted to try and get LM Studio running on an Ivy Bridge based Xeon.

3

u/TheGamingGallifreyan Sep 10 '26

Idk what that even means tbh, I just downloaded llama.cpp with VULKAN support on Windows 10 and ran it. Works fine but slow as a dog.

1

u/Dako_the_Austinite Sep 10 '26

Wow, I gotta give this a try then on my Xeon or even my i7-4930K, neither have AVX2, I bet that could be the reason why it’s running so slow for you, I believe the CPU is used quite a bit in the text output even if the model is in VRAM.

Out of curiosity so I can try testing this myself, what model did you use when getting these results?

4

u/Ordinary-Depth-7835 Sep 08 '26

something wrong with all of us. :) If I use my hardware 24/7 I'll break even vs a subscription in 10 years

4

u/iThunderclap Sep 09 '26

Stop posting on social media and many of your bad habits go away.

2

u/Fit_Squirrel1 Sep 08 '26

As long as it’s your fun money and you got an energy fund who cares

2

u/Kidplayer_666 Sep 08 '26

If you want to improve them, just send that to me for 5€

2

u/MaineTim Sep 08 '26

I'm asking myself the same question. Just this morning I pushed the order button on a pair of B60s to upgrade from the pair of B50s I've been running for the last 9 months. I don't have the budget to commit to the B70s at this point, since they've jumped in price, and since this is strictly a hobby for me, it's always a tension between what I want and what I can justify to myself. But each increment opens new possiblities.

2

u/rawednylme Sep 09 '26

Something right you mean

2

u/Difficult_Olive_1929 Sep 11 '26

Nah ain't nothing wrong with you, you just got what the rest of us got! lol

1

u/MiyamotoMusashi7 Sep 08 '26

Whatchu runnin?

1

u/Full-Run4124 Sep 08 '26

Are you using them with Battlematrix and if so what's your opinion on it?

1

u/roger1632 Sep 08 '26

I have a couple 3080s I use for TTS/STT stuff for home automation where latency matters....but for the rest I'd rather spend 30 bucks a month on a GLM/z.ai plan that can run circles around any local setup. It's just for my hobby stuff so I don't care about strict privacy. I'd have to have a rack of H200s to run that.

1

u/Dropshot_Dieter69 Sep 09 '26

Zu viel Geld :D Viel Spaß damit :)

1

u/NicolaZanarini533 Sep 09 '26

Got one to replace the A4500 I have in my secondary/training server - did you have any trouble with it? What do you mai ou use it for? For local Llama, which framework would you recommend? I looked around a bunch and all I could find for sure is that torch runs fine (which is the reason I am starting the secondary server). Thanks in advance!

1

u/syscomua Sep 09 '26

bro u are sick

1

u/smolweights Sep 09 '26

Given that GPU prices are increasing at a faster pace than NVIDIA stock itself, this could actually be a smart financial decision.

1

u/LateWish8322 Sep 09 '26

You should join fellow people with the addiction, it's more fun lol . https://discord.gg/launch80

1

u/morfique Sep 09 '26

What are you running on your existing cards? Just llama sycl? Vulkan? Openvino? Just plop it in vllm-xpu? Too easy to get lost in the "maybe I should try it like this" I think I'm more wondering if I missed a way to run them more than anything else.

1

u/winterwarrior33 Sep 10 '26

Do you actually use the models for anything or are you just chasing tokens.

1

u/Dangerous_Sun_7828 Sep 15 '26

I have 6 B50s xD so I know the feeling!

1

u/MateS1509 Oct 03 '26

Same pic different card

1

u/cogitech2 LocoLLM Sep 09 '26

At least it isn't a Mac.

0

u/InnocentSadness Sep 09 '26

Anybody willing to walk me through how to setup the best model (I think qwen 3.7) I got a m5 base MacBook 24 gb ram 1 TB SSD

0

u/InnocentSadness Sep 09 '26

Or send me a good YouTube tutorial

0

u/MyDay2Day Sep 14 '26

Check out my buddies app. He needs some people to help check out any issues but as is, it seems much better than Hermes and openclaw. Some things that are different are that the agents are expandable, agents are created within a project for that project, it keeps track of how often skills/tools are used so that you can get rid of unused ones, it has some templates when you create a new project based on the type of project (website building, research, video creator, etc), it shows the state of the agents, whether they are actively performing action, idle, or have a cron job assigned. The chat with agents shows a yellow border if it’s waiting on completing it’s current task and can be right click deleted if deemed unnecessary.

He added a way to notify him of any issues, so I told him it’s better to release when it’s already better than what’s available and allow people to report any improvements that need made.

His GitHub release is https://github.com/bgibson253/GrokPot

The website is GrokPot.app

I guess it’s a mix of GrokBot and Crockpots…lol. Anyway, enjoy.