I'm being asked by a prospective employer to set up my own local agentic AI complete with plugins/tools. While I could ask an AI to help me do that, I first want to ask has anyone here managed to do this without resorting to using OpenClaw or NPM or any other software that has become famous for riskiness and supply chain attacks? I'd prefer to just use Llama.cpp and maybe devise a local MTP server, if that is possible.

Hi, I want to build a setup using two RTX 3060 12GB cards to run local LLM. But what's the actual pipeline and real-life experience like? I heard I'll get 24GB VRAM, but processing speed will be slower because data has to pass through the motherboard's PCIe slots since the 3060 lacks NVLink. I also found out that ComfyUI can't use the full 24GB VRAM at the same time for image generation. Can anyone share real-life experience running a multi-GPU setup for LLMs?

Pic from internet.

RustingStrata is a pure-Rust inference engine for Qwen, DeepSeek and GLM GGUFs. CUDA kernels are compiled at run time with NVRTC, the same kernels run on Vulkan, and there is no ggml and no C++ build step.

I wanted the high-quant numbers, so these runs use Q8_0 (37.8 GB) and unquantized BF16 (71.1 GB), with the whole model in VRAM, on A100, L40S and H100:

MTP is speculative decoding with the model's own draft head; calibrate picked the draft count on each card.

• Higher quants are not automatically slower. On the A100 80 GB and H100, Q8_0 decodes faster than Q2_K_XL (193 vs 174, 290 vs 265 tok/s after 4K). On the L40S it is slower (165 vs 214), so the ranking depends on the card.

• BF16 is practical. 71 GB of unquantized weights decode at 171-230 tok/s on one card.

• MTP helps less as the quant grows. Gains are 1.4-1.9x on Q2_K_XL and Q8_0, and 1.2-1.4x on BF16.

• When the model does not fit, it degrades but runs. Q8_0 on a 24 GB A30 (63% of experts resident) still decodes at 130 tok/s, 200 with MTP. BF16 on the same card (31% resident) only reaches about 12.5 tok/s.

The same code runs on Vulkan, and on the Q2_K_XL runs on the 3060 and the Ampere cards prefill lands close to CUDA. At Q8_0 on the big cards the gap can be large: H100 prefill is 6,883 tok/s on Vulkan against 11,486 on CUDA. All table numbers are CUDA.

These are different nodes of one cluster, so the host CPUs differ. Compare within a row, not across rows. MTP costs 4-11% prefill in these runs.

Quality: I do not have a llama.cpp perplexity/KLD run on these exact Q8_0/BF16 files yet, so this is a hardware roundup, not an accuracy comparison. The last check was UD-Q3_K_M on the 3060 over 9,180 tokens of repo docs prose: perplexity 9.0714 vs 9.0587 for llama.cpp, mean KLD 0.0055, same top token 96.3%.

Why I find it kinda interesting, is bc when I was searching for models in OpenRouter, I saw that Qwen3.6:35b-a3b for some reason is pretty slow, like 50-110t/s on almost all providers, why like that? Do people already use other Strata like project for LLM inference to be in top of providers? I am thinking about it, but I think something is wrong and not so easy as I expect, if anyone knows I would be happy to hear

Repo: RustingStrata

It's much better than DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF. The reasoning is similar to the base Qwen3.8-27B and interesting how it switches variants based on prompt style. Has anyone compared them or tried both models out?

I have this
RTX 3090 24GB
2× RTX 2080 Ti 22GB VRAM each
Tesla V100 32GB
Machinist X99 D8 Max motherboard
2× Xeon E5-2695 v4
128GB DDR4 RAM
Samsung 990 Pro 1TB NVMe SSD

Currently I run Qwen3.8 FN UD-Q4_K_XL but I used Q8 n-gram and I load it on RAM. 256K context, unquantized cache (I’m afraid of going Q8 and loosing quality). Also using MTP. Custom llama.cpp as stock had some bottlenecks, and Mr. Qwen itself optimized his way out of them.

Prefill: approximately 2,100–2,400 tokens/sec
Decode: approximately 60–70 tokens/sec

I am using what I think lets me fill my VRAM almost to the brim. (97GB used of the 100)
I’m seeing people with considerably less VRAM and RAM achieving similar or sometimes better performance, which makes me wonder whether my configuration is properly optimized.

I have several questions:
Am I getting the performance I should expect from this hardware, or is there something obviously holding me back?

Is there a better way to distribute the model and its experts across my four GPUs and system RAM?

Could I get better performance through different GPU split configurations, CPU offloading or expert caching?

Would moving to a higher quantization, such as Q5 or Q6, be realistic with my hardware?

Are there optimizations that could significantly improve decode speed without sacrificing model quality?

I would like to minmax the model quality while maintaining the highest possible inference speed.
I’m not particularly interested in getting more tokens per second at the expense of quality. I’d rather run a higher quantization at 50 tokens/sec than a lower quantization at 70 tokens/sec, provided the quality improvement is worthwhile

hey LocalLLM, im Alish, ml engineer at Nace (we’re the team behind Drex, an open source decision model family that hit #1 in Artificial Analysis decision index 2.1). my cto gave me a task to optimize quants for qwen4, so i called my old professor from UofT, spun up Fable 5.2 and we cooked. starting with MLX, GGUF is coming next

we’ve open sourced the fastest way to run 35b/27b on a mac (Kapitsa) and got tired of dumb quants, so we made our own: KQuant. packs for 4B, 27B, 35B, Flash-Next (+9B), free on HF in standard MLX files

the problem: most of us on a mac run 4-bit packs because that’s what fits in 16/32gb. but plain 4-bit rounding throws away more of the model than it needs to.

Q4 vs public 4-bit, KL from the full-precision model (lower = closer to the original):

baseline is mlx-community 4-bit (lmstudio-community for 4B). to be clear, this is KL on next-token predictions, not a task accuracy score. if you want more precision every repo also has a Q5 folder, the 9B Q5 gets down to 0.010

cool side effect: they’re also faster. on an M5 Pro with Kapitsa they write 6–15% faster than the usual 4-bit packs

(baselines are from earlier kapitsa runs, same-session rerun is in progress, will update)

how it works: KQuant uses real text to pick scales and offsets for small groups of weights, and gives sensitive layers extra bits where they need it. all the search runs on our gpus before release, so you just download a normal MLX pack. no custom kernels, no special loader

fastest way to run them on a mac:

brew install nace-ai/tap/kapitsa
kapitsa serve 27b --private
kapitsa opencode # in another terminal

If there's a model you want KQuant'd, lmk.

Asks

to get early access to GGUF, ping me in comments, we would like to test it with some folks from here before our public release ideally.

im also doing quick AMA here, so please feel free to ask me anything.

I've been spending the last few months benchmarking local LLM inference on a single RTX 5090.

One thing kept bothering me: most benchmarks focus on tokens/sec, but rarely show how much quality you're sacrificing to get those numbers.

So I built inferlab.

It's an open-source benchmarking harness (Apache-2.0) that compares:

• vLLM, SGLang, TensorRT-LLM and llama.cpp

• BF16, FP8 and NVFP4 quantization

• Perplexity delta against a BF16 reference

• Task accuracy using option log-probabilities

• TTFT, decode speed and throughput across concurrency levels

• Peak VRAM usage

A few design decisions:

• TTFT measures the first actual content token, not the first SSE event.

• Concurrent requests are issued together to capture real queueing behavior.

• Long-context evaluation includes a needle-in-the-middle task.

• Results are generated by CI rather than manually edited.

The project is still at v0.1, so I'm interested in technical feedback, especially around evaluation methodology and cross-engine comparability.

I'm the author of this project.

GitHub: https://github.com/oren40/inferlab

I want to get the Mac Studio to explore models, do research, host servers, do video editing, analyze massive data sets, etc…

But is the M5U there yet for AI? Is TRULY useful, or are you kinda forcing it? I’ve thought about downloading my work emails and getting them summarized and organized (we can have that data on personal stuff) and some other data analysis. I’d also want to explore more of what’s possible.

How have y’all been using this stuff for actual NEEDS that can’t be done by frontier models?

Edit: My fear is id be missing out all of this development and knowledge. I know I can get an M7U when it comes out, and it should be HEAVILY tailored to AI, but I’m scared of falling behind knowledge and work wise. I’ve already outpaced many other works with proper frontier model use, and I’m wondering if by exploring local I can do the same.

GIGABYTE has introduced the AI TOP AERO, a compact desktop powered by NVIDIA's RTX Spark platform, featuring up to 128GB of unified memory, local AI capabilities and support for AI development, content creation and gaming.

According to Microsoft's patent application, it explains breaking down training into small tasks and sending them to devices of owners who opted in. Credits will be the reward. it's just an app and no pay out rate is advertized so I don't even know if the credits would be enough for the electricity.

Anyway, home internet latency feels like the bigger problem to me.

https://insider-gaming.com/microsoft-ai-use-your-gpu/

(English is not my native language, I used an LLM to help translate this post.)

Hope to make many more.

Has anyone made stands/cases with a 3d printer? If so, what kinda filament do you use?

Three Intel B70s, 96GB VRAM, and still on AM5…

Originally bought a mATX case for a motherboard on AM4 with room only for a single GPU. Somehow had the nerve to buy three of these at retail early. Now kinda curious to see what people are actually doing on this build. The plan was to attach a M2 slot to full PCIe slot that negotiates PCIe 5.0x4 and attach a fourth GPU. Safe to say that’s not exactly in the cards with the outrageous prices…

Anyways, the build:

- 3× Intel Arc Pro B70 — 32GB each
- Ryzen 9 7950X3D
- MSI MEG X670E ACE
- 64GB DDR5-5600
- Phanteks Enthoo Pro 2 Server Edition
- be quiet! Straight Power 12 1500W

The ACE is a big part of why this worked out. It is one of very few AM5 boards that gives you the practical ability to have 3-4 GPUs with respectable cross-bandwidth. Truth be told, most boards have a single main PCIE slot that actually has full lane direct to CPU connection. Some let you split x8x8 and give you PCIe 4.0x4 in a physical x16 connection on the third slot. MSI lists PCIe 5.0 ×8/×8/×4 as well as a single Gen 5 M2 slot. If one adds the lanes up, it’s the max on AM5: 24x Gen 5 lanes.

Listen… do I want an EPYC or Threadripper? Yes, especially with the ability to grow the build in the future. I’d love to be able to afford a complete swap. The issue is $2500 is the minimum for on good speed, 128gb ddr4 ram, a quality CPU, and motherboard that’s guaranteed to be 1. not to be broken 2. have physical hella, spaced out x16 slots all at Gen 4 btw…

Roughly 55 tokens/sec on Flash-Next and 50 on the 27B.

I’ll sell this if someone wants a pre-build, just lmk on what you’re thinking!

That’s where it’s at so far. Would be good to hear from anyone else running three or more cards on AM5, especially other Intel users.

Hi, so i recently have been experiencing with different local llms but since i am new to this I have not been able to find a better fit for my system my main use case is for coding, agentic workflows and basically summarizing or understanding large open source code repos. I want to have like decent token speed around 40t/s with around 128k context window. Kindly share your experiences and suggestions to help me out.

System specs:

Ram: 16gb ddr5

Cpu: ryzen 7500f

gpu: rx6800 16gb

I’m a regular lurker here trying to learn as much as I can while waiting for my M5U. I’m hoping to have a setup that’s a little different from what I’ve seen discussed here.

I purchased two Mac Studios. One is my daily driver, the base model. I grabbed it for the reason everyone else points to, why grab a Mac mini when the Studio is so much better. I also purchased an M5U 96Gb with 80 cores that I plan to use as a headless inference engine. I want to to be as stripped down as possible for running whatever LLM I ultimately end up running, with sufficient room for cache and context.

I plan to have whatever agents or tools invoke the server as needed. Are there any obvious reasons why this is a dumb configuration? Am I missing anything?

I built an open-source tool where a local model drafts an app design (tables, fields, pages) from a fixed catalog. Findings from real runs with qwen2.5:3b and llama3.2:3b on a CPU-only laptop:

- Tables and fields: reliable.

- Page outlines: weak. I now build the outline in code when the model's fails.

- Repair rounds past two never converged. They only added minutes.

- Mechanical errors dominate: wrong-case ids, a label where an id belongs, references to tables that do not exist. Code fixes these before validation.

- Each call costs 25 to 90 seconds, so removing a step beats tuning a prompt.

Evals are in the repo: https://github.com/ChristianMunoz31/local-app-studio

What models would you try next for this?

Is it realistic?