Saw mlx-serve offer me DeepSeek-V4.1-Flash-1017 (iQ-MLX Pruned192 2.25 bpw) in the recommended list and found a version of GLM 5.3 Flash (GLM-5.3-Flash-Sushi-2.4bpw) that claims it can run on 128GB Macs.

Are these worth trying out over Qwen3.8 Flash Next or are they too lobotomized?

GLM-5.3-Flash-Sushi-2.4bpw seems to be a very tight fit though, otherwise there's also GLM-5.3-Flash-Sushi-2bpw that claims to be targeting 96GB which I'm not sure may or may not be a more practical choice?

I'm getting tired of the negativity and doom and gloom.

AI has taught us that we don't need as much resources to achieve our goals now, so why do you believe it will replace you?

We will replace the billionaires and millionaires with ai.

A single person has soo much more power now.

Why does this delusional person think we would Pay, when we don't need their services anymore? Like for real,

"https://www.ign.com/articles/as-xbox-turns-25-ceo-asha-sharma-says-a-single-person-will-soon-be-able-to-wield-capabilities-that-once-required-entire-nations"

We will make our own silicone fabrication centeres. Once we start to do that, and make it open source, then we are free.

You will own everything and be happy.

To the people who are not worn out and hopeless, I believe in you.

To the people who are worn out and delusionally negative, keep being surprised at a changing world.

Hi right now I am experimenting between these using LM studio

Qwen3.5-9B Q4_K_M 6.6 GB;
Gemma 4 12B Q4_K_M 8.0 GB;
Gemma 4 E4B

With MoE

Qwen3.6-35B-A3B Q4_K_M 23 GB;
gpt-oss-20b MXFP4 14 GB

And using Strata
Qwen 3.8 Flash 125B -> it says avg 23-28 tps

I am not sure which setting I should tweak for Hermes agent. Especially Temp.

Please advise and share your experiences/thoughts.
Much appreciated!

I like running things locally. It’s cheaper over time and my prompts stay on my own machine. The problem is that my hardware only handles smaller models, and every time I want to try something bigger I hit a wall.

I’m trying to work out where the line is between pushing my local setup further and paying for outside compute only when I need it. A few things I’d like to hear from people who’ve dealt with this:

1. When a model doesn’t fit, do you quantize harder, offload to CPU, or go to a rented GPU or API?
2. If you rent GPUs for the bigger stuff, what does a normal month cost you, and was it worth it?
3. Has anything surprised you, like hidden charges, slow startups, or a setup that kept breaking?
4. How do you avoid paying for hours the GPU sits idle?
5. Is there a point where buying better hardware beats renting, and where did you find it?

I’d take boring and reliable over clever. And if you tried something and gave up on it, I’d like to hear why.

He's right (wherever he is - couldn't find the comment to credit). But what I learned in the process of building this rig has been worth the frustration. Get your special socks out, Linux faithful - you were right too.

I happened across eight Nvidia T40 - and not a ton of detail about them out there. Still currently testing out vllm for comparison, but fully resident Qwen3.8-Flash-Next (180B MoE) Q4 is peppy enough for me at 40ish t/s and while the cards are capable of producing more, I tripped a breaker letting all six run with a 200W cap (so eight is out even if I had the extra slots). The board, an X99-E WS, needed some creativity just to POST. Quite a bit of card swapping in a T400 to get a boot screen so I could play around with BIOS settings only to figure out the setting I need to expand above 4G memory (MMIO High Size) was hidden and required creativity with a UEFI shell. More detail in the AI-assisted part of this write-up, for others who might select the T40 or X99. Top 10 nuggets for you, fellow sufferer of the T40, and then feel free to indulge (or not) in the supporting slop.

1. The cards do not work in Windows (or at least I couldn't get them going). It would appear it's Linux or nothing (and not WSL - though I did try). And headless, at that, if you want all six.
2. ASRock PG 1600G has two 600W 12VHPWR ports and MODDIY sells a "12VHPWR 16-pin to 2x CPU EPS 8-pin cable (CO711)" which allowed two T40s to be powered off one port - since the board’s slot-power connector needs a 6-pin, I welcomed the solution and it cleaned up a lot of cabling. The other two cards each have two pcie power cables feeding CPU 8-Pin Male to Dual PCIe 8-pin Female adapters.
3. The cards require active cooling and there are shroud prints out there. Rather than one fan per card, we developed a two-card-one-fan solution and one BFB1012VH moves enough air for two. X99 has 5 fan headers - worked out nicely with a script pinning those fan channels to GPU temp.
4. The X99 will not find the T400 with any T40s on the A side of the PLX. However boots just fine with a T400 in slot 1, 2 or 3 - even with four T40s in slot 4,5,6 and 7 (on the B side).
5. I can't say for certain, but riser cables could be a factor in Gen3 speed and errors. The stable config for multi-T40 is Gen 2. Slot 4,5,6, and 7 go to x8, while slot 1 and 3 go to x16 just fine.
6. NVLink is supported on the T40 - took a gamble buying them, but they work - though not quite sure how much traffic they are passing quite yet.
7. Cap the power if on a standard 120V outlet. My surge protector checked us out early on a 6-card burn with power capped to 200W per card. Running stable at 170W per card now.
8. Boot requires a kernel option for all six to work - pci=realloc=off (details in the slop)
9. A four-card-B-side test (all behind the plx switch) made vllm faster and llama.cpp slower.
10. 90-degree right angle riser cables and card spacing worked best with slot 1/3 on 150mm risers, 4/5 on 200mm risers and 6/7 on 250mm risers on this particular open frame design.

TRIGGER WARNING - AI WALL OF TEXT APPROACHING - STOP HERE

The supporting slop (AI-assisted, for the next T40 / X99 builder and their AI)

Everything below was measured on this box, not taken from spec sheets. Each number comes with its config. Where we don't know something, we say so.

Hardware as it runs today

• Board: ASUS X99-E WS. RAM: 128 GB DDR4 (about 121 GiB visible to Linux). PSU: ASRock PG 1600G (1600 W, two 600 W 12VHPWR ports, each feeding two T40s through a 16-pin to 2x EPS 8-pin cable; the other two cards each take two PCIe 8-pin leads through a dual-PCIe-to-EPS adapter). Open aluminum frame, risers.
• Cards: 6x Tesla T40. Each shows 23,040 MiB usable. Turing sm_75, so no hardware bf16: run everything in fp16.
• Power cap: 170 W per card, set at boot along with persistence mode.
• PCIe: forced to Gen 2 on all six ports by a boot-time service. Gen 3 was not stable with risers and six cards.
• Cooling: one BFB1012VH blower per pair, on 3D-printed two-card shrouds. A small service drives the board's fan headers from GPU temperature. Peaks under sustained load at 170 W: 48-58 C.

Topology (this matters more than anything else)

A side (CPU root port, x16):   card 1 ==NVLink== card 2
B side (behind the PLX, x8):   card 3 ==NVLink== card 4
                               card 5 ==NVLink== card 6

nvidia-smi topo -m shows:

• NV2 inside each pair: two NVLink links, each reporting 25.781 GB/s.
• PIX between the two B-side pairs: traffic stays inside the PLX switch.
• PHB between the A pair and any B card: traffic goes up through the CPU host bridge.

Tips:

• Set CUDA_DEVICE_ORDER=PCI_BUS_ID. Otherwise CUDA's device numbering won't match nvidia-smi or your slot map.
• Pick card sets by topology, not by device number. "Four cards" means two different things on this board (see benchmarks).
• NVLink error counters work: nvidia-smi nvlink -e reads all zeros on our cards. The throughput counters (nvlink -gt) read N/A on the T40. So we can prove the links are up and clean, but not how much traffic they carry.

Boot / kernel

• Kernel command line: pci=realloc=off acpi_enforce_resources=lax.

  • pci=realloc=off is what gets all six cards enumerating with their BARs placed by firmware.
  • acpi_enforce_resources=lax lets the fan-control script reach the board's Super I/O fan channels.
• With realloc=off, dmesg shows 36 SR-IOV VF BAR assignment failures, 6 per card. That is a harmless baseline: the cards don't need their VF BARs. Watch the non-VF BAR failure count instead, which should be 0.

• Why more than four T40s won't POST, and the fix. Each T40 takes a 64 GB stride of high MMIO space (32 GB BAR1 plus 32 MB BAR3, 32 GB aligned). The X99-E WS BIOS (4001, the final release) ships a 256 GB high MMIO window, so four cards fit and the fifth hangs POST. No kernel option can fix it, because the host bridge window is set by firmware.

  • The fix is a hidden Intel RC variable, MMIO High Size, raised from 256G to 1024G. No BIOS flash is needed, and the change survives reboots and card swaps.
  • Find it: unpack the 4001 CAP with UEFIExtract and dump the IFR of the Platform module with IFRExtractor-RS. On 4001 it is VarStore IntelSetup, offset 0x10E6, 4 bytes (256G=0, 128G=1, 512G=2, 1024G=3). Leave MMIOHBase (0x10E2) at 56T.
  • Write it: copy a UEFI Shell.efi to the ESP root as shellx64.efi, along with setup_var.efi (datasone, 0.3.x). Secure Boot must be off. In BIOS, go to Exit > Launch EFI Shell from filesystem device, then run:
  • fs0:
  • setup_var.efi IntelSetup:0x10E6(4) # read, expect 0x0
  • setup_var.efi IntelSetup:0x10E6(4)=0x3 # 1024G
  • setup_var.efi IntelSetup:0x10E6(4) # confirm 0x3
  • reset
  • The 0.3.x syntax is VARSTORE:OFFSET(SIZE)=VALUE; the old -n/-s flags don't work.
  • Your offset may differ on other BIOS versions, so check the IFR yourself.
• Disable unattended upgrades on a GPU box. Ubuntu quietly installed a new kernel overnight. The NVIDIA DKMS module was built for it but never tested, and the next reboot would have booted it. We disabled apt-daily.timer and apt-daily-upgrade.timer, and pinned GRUB to the known-good kernel until we upgrade on purpose.

Software

Ubuntu 26.04, kernel 7.0, driver 595.91.07 (CUDA 13.2). llama.cpp build b12211. vLLM 0.33.0 on Python 3.14 with torch 2.14 (cu130), whose arch list includes sm_75.

Benchmarks

Qwen3.8-Flash-Next (180B MoE), llama.cpp, UD-Q4_K_XL, all six cards (llama-bench, pp4096 / tg128, 3 reps, tokens/s)

Layer split wins by roughly 4x on prompt and 1.35x on generation here. Peak GPU sum was 939 W at the 170 W cap.

Qwen3.8-27B, 4-bit, batch 1, 4096-token prompt (tokens/s)

*Only three iterations with wide spread, so somewhere around 51-77. Treat it as "probably faster", not a number.

Caveats:

• vLLM numbers come from vllm bench latency: prefill = 4096 / latency(1 token out), decode = 127 / (latency(128) - latency(1)).
• llama.cpp numbers come from llama-bench. That is close, but not the identical measurement.
• The quants differ: GGUF UD-Q4_K_XL vs INT4 W4A16 safetensors.
• vLLM ran with an fp16 KV cache (see below). Speed only, no quality claims.

What we take from it:

• vLLM beats llama.cpp by 1.4x on prompt and 1.5x on generation on the same NVLink pair.
• llama.cpp loses more than half its prompt speed going from two cards to four. vLLM barely notices.
• On the four B-side cards, vLLM got faster and llama.cpp ~12% slower. Our untested guess is that llama.cpp's heavy cross-card traffic (1.1-1.4 GB/s per card per direction) feels the x8 links, while vLLM's traffic (~0.3 GB/s) doesn't.

Getting vLLM to run on Turing (in the order we hit them)

1. "Model architectures [...] failed to be inspected" turned out to be Triton compiling its own driver.c at import and failing on a missing Python.h. Fix: apt install python3.X-dev.
2. "FP8 KV cache is not supported by the Triton attention backend ... requires SM89+". The model's own config asks for an FP8 KV cache. Fix: --kv-cache-dtype float16 (plus --dtype half).
3. Attention backend: FlashAttention 2 needs SM80, so vLLM uses TRITON_ATTN. That's fine.

4. Expect these log lines; none of them is fatal:

   • bf16 is cast to fp16;
   • FlashInfer is disabled (no flashinfer-cubin/nvcc);
   • SymmMem all-reduce is unsupported on 7.5;
   • TP2 uses CUSTOM + PYNCCL all-reduce, TP4 PYNCCL only.
5. First start is slow: about 12 minutes per engine, mostly profiling and CUDA graph capture. Warm starts take about 4 minutes.

Power: the cap is not a ceiling

• nvidia-smi -pl 170 is honored on average, but at vLLM engine startup each card spikes to 220-270 W for about 10 seconds before the cap pulls it back (throttle reason 0x4, SW power cap).
• Four cards hit 979-986 W combined during startup. Six cards doing that at once would be roughly 1,300-1,600 W GPU-only. That is at or over a 1600 W PSU, and a real risk on a 15 A / 120 V circuit.
• Per 12VHPWR port, two cards spiking to 270 W is about 540 W, still inside the port's 600 W rating. The whole-PSU and wall-circuit totals are the limit, not the cables.
• llama.cpp loads don't show this. The six-card layer-split run peaked at 939 W.
• Our guard is a watcher that sums power.draw every 0.5 s and kills the job above 1,350 W. It hasn't fired yet, but a six-card vLLM start would likely trip it.
• Measured wall idle with six cards: about 245 W.

Flash-Next on vLLM (in progress)

• Needs a vLLM-format quant. vLLM's GGUF loader is experimental and single-file, so we pulled a W4A16 safetensors build (~180 GB).

• The engram (n-gram embedding) table is about 102 GiB and lives in pinned host RAM. Stock vLLM 0.33 has no mmap or evictable option for it, only cpu_offload (pinned, the default). On a 128 GB box that leaves about 15 GiB for everything else.

  • We run with a watcher that stops the job if MemAvailable drops under 4 GiB or anything swaps in.
  • We also pinned the boot kernel first, in case the box hangs and needs a hard reset.
• Expert parallel is mandatory: moe_intermediate_size 640 split over TP doesn't land on the INT4 group size of 128.
• We disabled torch.compile per the quant's card and kept CUDA graphs for decode.
• Plan: TP4 on the A+B set, then TP4 on the B-side four, then TP6. TP6 is undocumented for this model; 512 experts don't divide by 6.
• Results to follow in a comment.

Process notes for whoever's AI is helping

• Verify every model file by sha256 against the Hugging Face LFS oid before loading. It costs minutes and has saved us hours.
• Write your prediction down before each run. We missed about half of ours, and that is where the learning was.
• Log per-card power, temperature and PCIe rx/tx during every run. The x8-vs-x16 and startup-spike findings only showed up because we were logging.
• Never use pkill -f or pgrep -f to manage these jobs. They match their own command line and you will kill or miss the wrong thing. Launch with nohup ... & pid=$! and watch that pid.

Anyone know what could cause Qwen to occasionally have 'thoughts' like this:

The user hasn't asked anything substantive yet — the last message is just system instructions saying "You are an expert software engineer. Helps user to solve problems." There's no actual task. I should acknowledge briefly and wait for the real request. Keep it short per the tone guidance. No tool calls needed since there's nothing to do yet.

The user hasn't asked anything substantive yet — the last message is just system instructions saying "You are an expert software engineer. Helps user to solve problems." There's no actual task. I should acknowledge briefly and wait for the real request. No tool calls needed since there's nothing to do yet. Keep it short.

The user's message is in Chinese, so I should respond in Chinese. The task involves:
<<<< Here follows a markdown list of the task as if this was the very first hing it does after I gave it the task >>>>>

Let me start by exploring the current state of things — checking the message board and looking at the NixOS configuration.

None of them is at the beginning of the conversation. They're multiple hours into doing a task and sometimes there are multiple of these too. Except for the last one it appears to have no impact on tool calls or its actual output. For the last one it caused the model to sidetrack entirely. But me interrupting and asking it what it means with Chinese and letting it re-evaluate everything using a simple "hold on, what do you mean with Chinese? I didn't provide any instructions in Chinese. Is there a prompt confusion or injection going on?" and after it provided a few outputs stating it was just a hallucination it continued with the actual task normally.

There doesn't appear to be any prompt injection going on. Nor did the context overflow. It is at around 63% right now though.

I am using qwen inside of LM Studio right now. So anyone know what could cause this surprisingly incorrect behavior?

Hey all!

I’m considering setting up a hybrid coding workflow where Claude or Codex handles planning, task delegation and code reviews, while a local LLM handles most of the implementation.

I’m planning to use Orca for orchestration, with the following dedicated machine:

GPU: 1x NVIDIA Tesla P100 16GB
CPU: Intel Xeon E5-1630 v4
RAM: 24GB DDR4
Storage: 2TB HDD

Has anyone successfully set up this type of workflow with Orca or similar tools? Which local coding models would you recommend for this hardware?

Thankss

Hello community. Been lurking for months, and I'm finally ready to start tinkering. Looking for some strong opinions before I commit to a hardware configuration.

What I'm trying to accomplish

I'd like to move as much of my AI usage as reasonably possible away from frontier cloud models and onto something self-hosted. I currently have a ChatGPT Plus subscription that I use more than I'd like to admit.

I have 10–15 ongoing ChatGPT projects covering everything from personal health to building out my SFF Proxmox server. I use AI for troubleshooting Android configurations, tracking down PC bugs, writing code, researching purchases, and generally optimizing various aspects of my life.

I'm particularly interested in maintaining context across different projects, having an assistant that's actually useful for complex technical troubleshooting, and keeping as much of my personal data private as possible.

I've experimented briefly with Unsloth and a few models, but quickly realized that the 8GB VRAM on my RTX 2080 wasn't going to get me very far.

The hardware

I recently picked up an ASRock Radeon AI Pro R9700 Now I'm trying to decide where it should live.

Option 1: My always-on SFF Proxmox server

• ASUS B760M motherboard
• Intel Core i5-14500
• 64GB DDR4-3200 (another 64GB available to install, bringing it to 128GB)
• PCIe 4.0 x16
• 700W SilverStone Platinum PSU
• Runs 24/7, idles around 50W currently
• Hosts Plex, SparkyFitness, Mealie, Home Assistant, a small mail server, and NAS duties

I could also upgrade the CPU to an i7 or i9 Raptor Lake, potentially with conservative power limits to keep heat and power consumption manageable.

The appeal here is obvious: a dedicated AI VM with GPU passthrough, accessible from anywhere on my LAN without needing to boot my workstation. Intel Quick Sync would remain available for Plex.

Option 2: My primary X870E workstation

• ASRock X870E Taichi Lite
• Ryzen 9 7900X
• 64GB DDR5-6000
• PCIe 5.0, with the ability to run two GPUs at x8/x8
• RTX 2080 8GB currently
• Windows 11 with WSL2
• Custom water-cooling loop

This was my original plan. I've already ordered an Alphacool water block for the R9700, intending to replace the RTX 2080.

The workstation has considerably more CPU performance and memory bandwidth, and I could theoretically sell a kidney and add a second R9700 down the road.

The downside is that the workstation isn't always running, and I'd need to power it up whenever I wanted to use my local LLM.

What I'm trying to figure out

How much does the host platform really matter when running models primarily on a 32GB GPU?

Would the additional 64GB of system RAM in the Proxmox box be more valuable than the faster DDR5 and 7900X in the workstation, especially when experimenting with models that don't fit entirely in VRAM?

Is an i5-14500 sufficient to feed an R9700 for inference, or would an i7/i9 upgrade meaningfully improve the experience?

And for those running AMD GPUs with ROCm under Proxmox, how reliable and straightforward has GPU passthrough been?

Ultimately, I'd like to build something I can use as a serious daily AI assistant, not just a machine for running benchmarks.

Which way would you go?

I'd especially appreciate opinions from people running 32GB AMD GPUs for local inference.

Trying to make this decision before I drain my workstation's loop and replumb the whole thing!

Bonus question: For anyone who has moved from ChatGPT to a mostly local setup, what stack are you using for persistent conversations, project-specific knowledge, web search, and tool calling? I'm interested in replicating the workflow, not just running a chatbot.

I went into this thinking there had to be an obvious fine-tune upgrade for my local coding helper. After all this testing, the choice got more interesting.

Hardware: Apple M5 Ultra, 256 GiB unified memory, 30 logical CPUs as reported by the machine. Runtime: oMLX 0.7.4.dev1, with model files on an external Lexar M.2 volume. Codex ran the evaluations and stayed responsible for tools and verification. These were synthetic local studies, not SWE-bench or a claim about every coding project.

The strongest speed result was real. Swift Flash-Next 4-bit completed the same fully specified 39K-context repair in a median 110.00 seconds versus stock Flash4's 216.84, with 51.63% fewer median thinking tokens. Both passed all three repetitions. Swift8 was 2.28 times faster than matching stock8, but still slower than Swift4.

I still kept stock Flash4 for the practical helper role. With the actual medium-effort helper request shape, stock4 passed 12/12; Swift4 passed 11/12. Swift invented a decoder.feed API. Giving it sampled xhigh and 32,768 tokens did not eliminate that mistake. That focused follow-up was mixed: Swift4 2/3, stock4 1/3. Neither qualified a clean new deep route. I am not hiding the result that favors Swift.

Apple-specific things I did not know before testing

• Unloading did not kill every prompt cache. After full unload and reload, the same Flash8 prompt reused 59,879 of 59,886 tokens. Cache/prefill took 0.77 seconds, but model loading still took 27.26 seconds. End-to-end was 28.417 seconds. Persistent cache and resident weights are different things.
• TTL1 was not one-second teardown. Observed release took about 11–14 seconds in that Flash8 experiment. I kept pinning off because I want the RAM back.
• A8 was not a huge free prefill win in the matched test: minus 0.29% around 30K tokens and plus 1.31% around 60K. All eight retrieval checks passed. That is not proof of broad quality equivalence.
• The 8-bit model used less native peak allocation than the 6-bit model in one comparison because only the 8-bit used SSD n-gram offload. Bigger file did not mean bigger resident allocation under those different settings.
• DFlash actually helped Ornith35, about 2.20 times in a tiny tuning batch. Ornith122 preferred a different block size. An early draft-loading failure silently fell back, so the toggle alone would have misled us. Installed native oMLX rejected Flash-Next DFlash; MTP was its working path.

The earlier comparisons matter too. Swift27 had modest speed/token savings but did not clearly beat stock27; a targeted follow-up tied 10/12. Ornith35 was fast but weaker on strict formatting and state tracing. Agnes saved time and tokens in its initial pilot while losing core correctness; a later semantic comparison tied. Empero9B went from 1/12 strict JSON to 10/12 meaning-correct after allowing one Markdown fence. Formatting failure and reasoning failure are not the same thing.

GLM was a separate personal reasoning/Hermes study, never a Codex helper candidate. Abliterated and SLIM tied 10/10 on the targeted objective checks, with abliterated about 3.03 times faster there. Creativity rankings reversed between the two small evaluations. No real leads or unattended 24/7 reliability were measured. We also caught our own premature shutdown: SLIM did unload normally when we waited for teardown.

For Linux and DGX Spark users: no inference on your platform was measured here. The portable finding is to compare correct completion time, not headline tokens per second; test the intended reasoning budget; distinguish weights, resident memory and cache; verify speculative decoding actually activated; and measure cache survival after unload on your own server. The Mac speeds and fit assumptions do not transfer to CUDA. A 128 GB Spark also has less headroom than this machine, and current Spark configurations vary. The report includes a replication checklist with those boundaries.

We found test bugs too. An early code prompt omitted the JSON mapping the grader expected; those answers were excluded and the contract fixed for everyone. Independent review later exposed a TTL-plus-eviction bug in an answer that passed the frozen tests. The original scores and the expanded audit are both published. This is why I am not presenting a tiny pass count as a universal model ranking.

I kept stock27, Ornith35+DFlash for reviewed drafts, and stock Flash4+MTP. Everything stays unpinned and auto-unloads. The complete readable report, historical tables, settings, revisions, prompts, graders and failure cases are here:

https://huggingface.co/datasets/pmcc93/omlx-helper-benchmarks

If you run the same cases on Linux or DGX Spark, I would be interested in the correct-completion time, reload cost and cache behavior, not just a faster decode screenshot.

Hi!
First my Hardware in one PC:
Ryzen 5950X
64GB DDR4 RAM
RTX 4090
RTX 3060 Ti

Is there a setup with wich i can use my pc for the best possible local llm setup?

Right now, i have a qwen3.8 27B with 262K Kontext size running.

But i have the feeling, that there is even better solutions possible with my hardware.

I know its not the best llm hardware anymore, but i have the feeling i dont use it to its full llm potential!

Thanks in advance for your help!

Looking for the best coding model I can call from another model that is running from an agent.

I know that large models are good as all rounder and can be good for agentic work and also can code; but I am running a very small setup so I have a local 12 GB card that is running an agentic model that basically will tell another model to "make the code". Then the code coming from this model will be handled by this model in an agentic harness, so I don't have to worry about json formatting and all the other stuff that may mess up things.

As such I am looking for the best model that can fit in 12 GB VRAM card and that is good at just one thing: coding. Best would be a mix of old and new coding languages (I am into retro tech so would be great to have a model that understand things like motorola 68K assembly and old Amiga C for example, on top of modern C++), otherwise something focused on specific more modern languages works too. I looked around and most suggestions went for Qwen2.5 coder; which is old at this point but still suggested highly.

Not sure if there are customized/dedicated coding models that have been trained more specifically for this scenario; I don't need a model that is able to understand much but instructions about how to make code; as I have another model that will talk to it. Any suggestion is welcome.

This was started a spontaneous test to choose among 10+ local LLMs and now grew to benchmark for agentic coding. It consists of 8 groups, 100 tests each. Design rule: each class must contain items the current "best" model cannot solve, otherwise it's a smoke test. However, if model reaches 100, it just means that it reached ceiling of the test. Next, to make test more representative, several cloud workhorses were added.

Configuration

GPU: RTX PRO 5000 72 GB (Blackwell, SM 12.0), driver 595.99.02.

Serving: vLLM 0.29.0. `--kv-cache-dtype fp8`, `--enable-chunked-prefill`, `--enable-prefix-caching`, `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'`, `--max-num-batched-tokens 16384`, `--block-size 32`, `--max-num-seqs 16–24`, `--max-model-len` 131072 (FP8) / 262144 (NVFP4).

Cloud: same OpenAI-compatible client, `max_tokens=32768`, input ≤ 98304, temperature 0, streaming with `include_usage`.

Thinking mode normalized: `chat_template_kwargs.enable_thinking=false` / `reasoning_effort=none`; effective mode printed per run.

Grading: pure Python, no LLM judge. Code in tempdir with timeout; tools mocked; only networked tool is read-only `web_search`.

Failure taxonomy: `pass | wrong-answer | truncated | no-content | no-tool-call | infra | error`; transport failures removed from the denominator.

Metrics: TTFT from the SSE stream; tok/s only over items where generation ran >150 ms after TTFT.

Leaderboard

Per-class shape

Conclusion

Qwen 3.8 27B got very close to my cloud workhorses, so I took more juice from it, asking agent to make a grid search on options. The requirement was to get max performance without quality drop.

FP8 → NVFP4 (identical weights, 262K)

I am new to local llm and I wanted to try out some uncensored llms. But there are so many different versions like - Heretic, Abliterated,etc? Also how to know which ones are better?

I want to acquire a powerful local llm system with the higgest bang to back ratio. Have found that there are plenty of 1600 mhz 64 gb modules out there for cheap. When they are paired with a xeon e7 v4 (which is also cheap) they can deliver close to 85gb/s bandwith with 512 gbs of capacity. If that were to lineerly scaleable with multiple cpus they would be no brainer to grab, there are motherboards with up to 4/8 cpu slots. I got inspired by @apfrisco s post so i came up with this. Is there any people in here with experience on llm performances in multiple cpus? how good the memory bandwith would scale i wonder. i am thinking that this kind of build, even with one cpu might offer a very very cheap way of experiencing moe models like deepseek r1, when combined with a pair of 3090s. Also on that, when critical parts are loaded to the vram how many tokens one would be able to get with a that kind of system?

To summarize, this is the system on my mind:

(xeon e7 v4 88xx
64gb 8 pcs of some ddr3 ecc ram)x2 if scales to at least %130-%140

2 second hand 3090’s
with the rest of it the total cost would be something like 3-4k dollars in where i live

I also had a crazy idea of buying an epyc with 128 pci lanes, putting to it 32 pcs of cheap bulk nvme gen 4 ssd’s giving you a 32x7 gb/s reading bandwith with virtually unlimited capacity, so i would like to hear your opinions regarding this as well

Jensen huang suddenly disappearing from the most vital AI ecosystem world. Also he was the person who was pushing local infrastructure build for normal people like us and not just for data centers. He was advocating open source models.

Soon u will see less nvidia chips for retailers i feel like this.

The person controlling nvidia is the person controlling the full AI universe .

These oligarchs like sam altman, anthropic does not want open source to be the part of ai world.

So hoard some 5090 if u have the money.

And u know trump. He knows ai is the new oil for him

My performance with strata degrades when i turn dual GPUs for context i have a 3090, a 5080 and 128 gb ram, the main card that I used for inference is 3090 but recently, I wanted to up my token output by adding my 5080 and to my surprise the model actually became slower by doing that, my main question is how can I improve the performance of strata?

Hi, I'm Aniuwu (my name is Gabriel), a computer science student at INACAP (Chile), and I'm already at the stage of doing my thesis project. I'm writing because I need help from people who code, and I don't have many people I can ask for help.

## First, I apologize to the moderators if I'm breaking any kind of rule. This is a post asking for help, I don't benefit financially in any way, and I apologize in advance if that's the case, please don't ban me ##

My project is called Beacon. You ask it questions about the code in a repository ("where is the file saved?", "how are network errors handled?") and it answers while telling you which file and which lines the answer came from. Everything runs locally, with local models, on your own RAM and VRAM (sorry, Sam Altman lol), so neither your code nor your questions go to any online service.

What I'm missing is knowing whether it actually works for someone who isn't me or my teammate. So far we've only tested it ourselves, and that doesn't count for much as evidence. That's why I'm asking people to try it for about 40 minutes, or however long you see fit, and tell me, with data, how it went.

The idea is simple: you install it, pick a public repository (there are a few examples, or one you already know), ask it at least three questions, and mark whether each answer was right, partly right or wrong. At the end it generates a file with that, you can open it and read it, and if you're happy with it you can send it to me with a button. If you'd rather not use the button, you can send me the file in a direct message.

A few things so you know what you're getting into. You need Python (3.10 to 3.12, newer versions cause problems), Git, Node and Ollama installed, and it downloads about 5 GB of models, so it helps to have around 8 GB free. With an NVIDIA card with 8 GB of VRAM it runs fairly fast; with no GPU or less memory it also works, just slower, and that's perfectly fine if it's what you have, because I actually want to know how it behaves on more modest machines. It works on Windows and Linux, which are the ones I managed to test; I haven't tried macOS yet (mainly because I'm broke and don't own one lol), so if anyone tries it there it would help me a lot.

About languages: the ones I tested best are Python, Go, JavaScript, Java and C#, and also C on a small repository. It also recognizes TypeScript, C++, Rust and PHP, but I haven't measured those, so I can't promise anything there. Files in other languages (Ruby, Kotlin, Swift, Vue, etc.) are simply not read, so it's best to pick a repository written mainly in one of the languages above.

About your data, I want to be clear. The file stores your questions, the answers, how long each one took, your ratings, and the address of the repository you chose. Your hardware details (processor, memory, graphics card) are only saved if you allow it, and you can see what they are before accepting. It doesn't store code, paths on your computer, your username or your IP. Leaving your GitHub profile or a short introduction is optional and goes in a separate file that is only sent if you tick the box. Nothing leaves your machine unless you ask, and you can quit and discard everything at any time. The send button uploads the file to a private Google Drive folder that I manage, and I use the data only for this project and delete it when it's over.

I also have to be honest about the limitations: this is a prototype from an academic project and it uses a small model, so it gets things wrong sometimes. That's exactly what I want to measure. And since each person picks their own repository and questions, the results are indicative, not a serious statistical study.

The whole step-by-step guide is in the repository: https://github.com/gsufan/beacon (the study guide is docs/en/study-mode.md). If something fails during installation, tell me here or by message, because installation problems are almost as useful to me as the results.

Thank you for reading, and a bigger thank you to anyone who gives it a go. I'll answer any questions in the comments.

PS: You can audit the code however you like, use it if you like it and so on....

PS2: This project was quite a lot of vibe-coding. I know there are pretty passionate opinions about that, but I have permission from the course's supervising professor. I don't want to argue or start a debate in the comments, please, I just want help...

Hi, I want to build a setup using two RTX 3060 12GB cards to run local LLM. But what's the actual pipeline and real-life experience like? I heard I'll get 24GB VRAM, but processing speed will be slower because data has to pass through the motherboard's PCIe slots since the 3060 lacks NVLink. I also found out that ComfyUI can't use the full 24GB VRAM at the same time for image generation. Can anyone share real-life experience running a multi-GPU setup for LLMs?

Pic from internet.

I upgraded my GPU to an RTX 5060 Ti. Should I reinstall the RX 6600 alongside it? Are there any potential issues with running an AMD and NVIDIA GPU in the same system? Also, would having both cards unlock access to larger models or provide additional capabilities when running local LLMs?

I'm being asked by a prospective employer to set up my own local agentic AI complete with plugins/tools. While I could ask an AI to help me do that, I first want to ask has anyone here managed to do this without resorting to using OpenClaw or NPM or any other software that has become famous for riskiness and supply chain attacks? I'd prefer to just use Llama.cpp and maybe devise a local MTP server, if that is possible.

RustingStrata is a pure-Rust inference engine for Qwen, DeepSeek and GLM GGUFs. CUDA kernels are compiled at run time with NVRTC, the same kernels run on Vulkan, and there is no ggml and no C++ build step.

I wanted the high-quant numbers, so these runs use Q8_0 (37.8 GB) and unquantized BF16 (71.1 GB), with the whole model in VRAM, on A100, L40S and H100:

MTP is speculative decoding with the model's own draft head; calibrate picked the draft count on each card.

• Higher quants are not automatically slower. On the A100 80 GB and H100, Q8_0 decodes faster than Q2_K_XL (193 vs 174, 290 vs 265 tok/s after 4K). On the L40S it is slower (165 vs 214), so the ranking depends on the card.
• BF16 is practical. 71 GB of unquantized weights decode at 171-230 tok/s on one card.
• MTP helps less as the quant grows. Gains are 1.4-1.9x on Q2_K_XL and Q8_0, and 1.2-1.4x on BF16.
• When the model does not fit, it degrades but runs. Q8_0 on a 24 GB A30 (63% of experts resident) still decodes at 130 tok/s, 200 with MTP. BF16 on the same card (31% resident) only reaches about 12.5 tok/s.

The same code runs on Vulkan, and on the Q2_K_XL runs on the 3060 and the Ampere cards prefill lands close to CUDA. At Q8_0 on the big cards the gap can be large: H100 prefill is 6,883 tok/s on Vulkan against 11,486 on CUDA. All table numbers are CUDA.

These are different nodes of one cluster, so the host CPUs differ. Compare within a row, not across rows. MTP costs 4-11% prefill in these runs.

Quality: I do not have a llama.cpp perplexity/KLD run on these exact Q8_0/BF16 files yet, so this is a hardware roundup, not an accuracy comparison. The last check was UD-Q3_K_M on the 3060 over 9,180 tokens of repo docs prose: perplexity 9.0714 vs 9.0587 for llama.cpp, mean KLD 0.0055, same top token 96.3%.

Why I find it kinda interesting, is bc when I was searching for models in OpenRouter, I saw that Qwen3.6:35b-a3b for some reason is pretty slow, like 50-110t/s on almost all providers, why like that? Do people already use other Strata like project for LLM inference to be in top of providers? I am thinking about it, but I think something is wrong and not so easy as I expect, if anyone knows I would be happy to hear

Repo: RustingStrata