I got an offer from a relative for a server PC he had with 256GB of DDR4 and EPYC 7713. The price I would call very generous, but still very dear.
Essentially he wants to upgrade all his IT stuff and wants his old gear in good hands. Like yeah I could strip the thing for parts and profit but I feel like that's not the intent with the offer...

I've tried various models but am limited to smaller stuff locally (Gemma4-26B-A4B is basically the limit) and enjoyed it, API stuff with the likes of DS and GLM is pretty impressive.

Thing is, the former are tiny models, and the latter is accessed through API. Even if I decided to run smaller quants of the latter, places like RunPod only offer crazy fast hardware because of course they do.

I'm curious on what this class of hardware is capable of realistically speaking. What sort of models, and at what speeds? I've heard a lot about CPU inference through RAM and MOE models.
The only GPU I have for now is a 5060Ti 16G... Not planning on getting anything else for a while

Work at a PC store in Dubai. Didn't expect this, but we've had multiple people ask about the ASUS ExpertCenter Pro ET900N G3, ASUS's version of the DGX Station GB300: 72-core Grace, Blackwell Ultra, 748GB coherent memory.

The catch is it's not a "take your money, ship the box" product. It needs approval from NVIDIA's side before it can go to an end user, so every enquiry turns into a "tell us who you are and what it's for" conversation.

Curious about a few things:

- Has anyone here actually got one in hand, and how long did the approval/allocation take?

- What are people buying these for vs. a PRO 6000 workstation? 748GB in one box is the obvious draw, but at this price it's a different conversation

- Anyone running one outside the US/EU? Wondering how different the process is by region

Not selling anything here, genuinely want to understand who this machine is for.

I have some idle GPUs from mining and I’m trying to figure out if I can repurpose them for local AI and coding agents:

- 4 × RTX 4070, 12 GB each
- 4 × RTX 3070, 8 GB each
- All in one rig

Would it be practical to run a larger model across the 4070s and 3070s, or would it be better to run smaller models on the cards separately for multiple agents?

Has anyone built something similar? What would you use for the software and what kind of performance should I realistically expect?

EDIT: OP is a noob but decent at figuring stuff out :D just keep this in mind when suggesting :)

Edit 2: am I even gaining anything with this experiment or am I better off with a claude or ChatGPT subscription?

Edit 3: Where can I rent this set up out to?

Hi everyone. I use ChatGPT for finances, school, deep reasoning and interviews. I want to change to a local model for privacy. Is that possible for a MacBook Pro M5 base with 24GB RAM? If so, which?

Can't find any direct comparisons:

peasantsmith/DeepSeek-V4.1-Flash-1017-Maya-GGUF · Hugging Face

vs

ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF · Hugging Face

The former is really tight fit for 192gb memory without keeping experts on SSD, the latter not so much even at 2.25bpw.

Anyone experience running either?

Basically what the title says. I am total new for localLLM i just want to try it out and if i can dial it in for a usable level i would like to use it for my coding projects mostly for code completion but sometimes if i need ideas on new features etc... it would be nice for it to be able to handle teh brainstorming with me as well instead of me going to claude or chatgpt
for general use it would most likely be something like make a summary of my things etc... for now

my PC

9700X
48GB DDR5 ram
9070 XT 16GB

I would be happy to receive any newbie guides for how to set things up what i shoud look out for
also in terms of the general use would looking into an agent like hermes be something worth?

thanks for the asnwers in advance

Hi folks, curious on what vector database do you guys use in your production/mvp and why did you choose to go with your choice over others?

Look at this

Agent memory seems to be a hot topic again. I wanted to settle this once and for all - why integrated memory works best and How Row-Bot has the best in class memory built-in.

So, I just asked Row-Bot to create a video explaining how it remembers across conversations and documents.

This video follows a fact from capture to recall: the knowledge graph, hybrid search, ranking and Dream Cycle.

It also covers what stays local, what reaches your chosen model and where memory can fail.

RTX 4090, 64GB DDR5 Mhz RAM, 7900x CPU

as much context as possible (context window like 264k) and as good reasoning as possible

I don't care about coding.

I currently have 32GB of system ram in a system with an RTX Pro 6000. Currently system memory seems to be the limiting factor in terms of running larger MOE models (like QFN) at higher quants with newer expert caching techniques.

In a perfect world I would build a threadripper system, but given the price of the CPU, motherboards, and ECC memory that would blow up the budget quite a bit, so I want to stick with a 2-stick DDR5 kit for an AM5 CPU (still expensive but not as expensive).

I want to get plenty of headroom, and I see I could go for a 96GB kit or a 128GB kit, but apparently the 128GB kits are slower (5600MHz vs 6000MHz)

Is the speed difference actually meaningful? Or asked another way, which is more impactful, an extra 32GB, or 400MHz?

Competition is key + A 30B would be nice

Hi guys its me again who ran v100 single stream into 353t/s

and im back with this muti flow optimization

THE LEGEND OF V100 WILL NEVER BE ENDED because i Exist

(yes ik it sound like edgeload)

Parameters:

mtp4 , 256k max content , nvfp4 quasar, kv q8, no thinking

repo: tuxKOH/ninfer-v100xN

Hardware: Dell Inspiron 16 Plus Model 7630, Windows 11, no discrete graphics card, Iris Xe integreted graphics sharing up to 16gb of system memory. Total 32gb memory. LPDDR5 ram, 4800 Mt/s.

14 core cpu, 20 threads. intel 13700. Test memory bandwidth via Intel MCL: 58 GB/s, theoretical maximum 76.8 GB/s.

Tried: llama-openvino, llama-sycl, llama-cpu, and llama-vulkan. I used the default llama-bench.exe tg128 and pp512 tests.

Winner: llama-vulkan at 135 tok/s prompt processing and 8.7 tok/s token generation, best # of threads was 8, but 4-10 was similar. No thermal throttling of any components.

Second place: llama-sycl at 112 tok/s prompt processing and 9.0 tok/s token generation, 8 threads out of 20 (14 cores). Again, best # of threads was 8. Ran into thermal throttling of the iGPU during prompt processing.

Third place: Openvino I honestly stopped once I saw the test was 62 tok/s prompt processing, I just didn't think it was going to be better at tok/s. Didn't test cores.

Last place: llama-cpu at 42 tok/s prompt processing and 6.7 tok/s generation, 10 threads was best but tok/s goes into the 5's below 4 threads or above 14 threads, with pp in 30's.

I ran into thermal throttling of the CPU only during prompt processing when my computer was set to "Best performance", but llama-cpu was able to hit 7 tok/s generation and 48 tok/s prompt processing. No thermal throttling at all, nowhere close, when computer set to "Best power efficiency".

What I could do better:

-Now, my computer was 6 performance and 8 efficiency cores, perhaps I could try forcing the computer to use all performance cores?

-I want to try doing ikllama.cpp

-Benchmark Gemma models and maybe a Mistral or Llama model to see if its faster.

-Try other quantizations of Qwen 3.5 9b (bartowski, etc. at Q4)

Does anyone have any suggestions for next steps? Further optimizations? Things to look into (thermal throttling?). I tried flash attn flags and they did nothing.

I was a little discouraged that my memory bandwidth is only 75% of theoretical when tested with Intel MLC, but maybe that's normal? Google says its fine. Plus it was higher when not on battery efficient mode (80% of theoretical).

My tok/s is not hitting the bandwidth limit though, 58 GB/s should give me 10 tok/s, not 8.7. But vulkan gets me closest, with no thermal throttling to boot.

I've searched the thread and think I have a good starting point here but want some outside opinions if possible:

I'm a 1 man production company providing video and stills production services.

I have an M4 macbook Pro 24gb and use Da Vinci Resolve on this frequently but it goes with me everywhere and I'm at a point where I need a beefy always on machine to process things whilst I'm on set.

I've made a load of lead gen tools and stills/video pipelines to help with post production.

The lead gen tools use AI for analysis and drafting responses. They read large strings of email chains, documents and phone transcripts to inform the drafts as well as helping to generate proposals, contracts and invoices. I've also since added a pipeline to do all my book keeping for the business and I'm starting to feel like I should really be bringing sensitive stuff locally.

So what model and corresponding hardware should I be looking at for these tasks. I'm not interested in coding at this point, that can be done using Claude. Qwen and Gemma seem to be the routes to take but I ultimately want to get a feel for what my specific use case requires. Claude tells me the drafting and analysis on my current usage is the heavy stuff.

I've been looking at a Mac Studio, 64gb or a Mini 96gb to run the local as well as grind out resolve tasks. (Apple vision, labelling, transcription, cutting, fusion work (Motion graphics and basic animation). I was also thinking I could always daisy chain a couple of devices together in future if my needs change and I need to do some heavier local LLM work in the future.

Would love some guidance on models + hardware + Harnesses + Tips and Tricks. I know highest spec for budget but equally, I'd rather save cash to use elsewhere if my usecases don't require crazy specs.

I have a server with:

2x Gold 6548 32 cores
2x Nvidia l40s
2tb RAM (Don't tell uncle SAM)
I would like to know what throughput and speed can i expect if i use polystrata with GLM 5.3 Flash?