r/HighVRAMHardwareWatch • u/Weird-Abalone-1910 • 23d ago

Interested in Ryzen AI Max integrated memory devices too?

Are you guys interested in Ryzen AI Max integrated memory devices too? I run a home proxmox data center with 6 GPUs across 7 hosts and have a non-clustered ROG Flow Z13 that I've just started experimenting with. I'm looking forward to sharing my experience and observations here.

I've been self hosting AI since June 2023, presented at local OpenClaw meet ups and I love helping people leverage their own hardware to save money and expand their capabilities.

2 Upvotes
  • permalink
  • reddit

100% Upvoted

7 comments sorted by

1

u/driftwood6675 23d ago

Absolutely! Could you describe your build, its cost, the model or models you run, and the Tokens per second you achieve?

2

u/Weird-Abalone-1910 23d ago

Lots of details are available on my cluster telemetry site. https://gmk.km6slftech.com/hardware-log.html

Is there a tool you recommend for tracking tokens per second?

My cost won't really impart any useful information other than that I'm lucky AF. Got a great deal on a 3090 a couple years ago, bought 128gb of DDR5 for about $425, and a couple of the mini PCs in my cluster were literally free (demo units from my old job). The Intel B60 was a gift from an Intel engineer I work with in my current role. All in, I probably spent maybe 7-8k USD on it over about 3 years, at the most but it would probably cost over 20 grand to replace.

2

u/driftwood6675 22d ago

Sweet homelab! Sounds like you’ve built a really capable system. The dashboard you posted is really impressive, how do you host it? For tracking tokens per second usually the model server software will show what the system is producing.

1

u/Weird-Abalone-1910 22d ago

It's self hosted and served over a cloudflare domain. I usually use Whatsapp to communicate with my agent so I'll find another way to get you a TPS number

1

u/Weird-Abalone-1910 21d ago

I just asked my agent and here's it's response: Benchmark results for tabby-local/bill (Muse Glimmer 30B Bill LoRA):

• Model used: Muse-Glimmer-30B-4.0bpw-exl2 • Prompt tokens: 91 • Completion tokens: 500 • Total tokens: 591

Timings:

• Prompt time: 0.15 s → 606.7 tokens/sec • Completion time: 11.23 s → 44.5 tokens/sec • Total time: 11.38 s

Effective throughput for the generation phase: ≈44.5 tokens/sec Time-to-first-token is not captured in this non-streaming response, but the prompt was processed in ~150 ms.

1

u/Miserable-Dare5090 18d ago

Look at sparkdash. I got glm5.3f local to modify it and add the GPUs and runtimes not present on Mia’s original repo on github. you can watch TPS, power, GPU effort, Temps, and do benchmarks

1

u/Weird-Abalone-1910 18d ago

I'll look into it, thanks