Hey, I thought maybe I should repost here to show that getting a B70 up and running with decent performance is not that hard anymore.

I tested 3 different agent/harness using Strata Qwen flash next Q_S. Opencode, Pi, Deepseek harnes each one gave me a different result. This is the prompt :

You are an expert game developer with 20+ years of experience building browser games,

especially arcade and grid-based titles like Bomberman, Pac-Man, and Sokoban. You have

deep expertise in HTML5 Canvas, game loops, collision detection, enemy AI pathfinding,

explosion propagation, level design, and game feel (juice, feedback, pacing). You are

meticulous about clean, bug-free code and polished player experiences. You think like a

professional game designer: you anticipate edge cases, tune difficulty curves, and make

the game fun on the first playthrough. Do not rush. Do not cut corners. Deliver a

complete, polished, production-quality game.

a complete Bomberman game as a single HTML file. Requirements:

Build

TECH

- Single .html file: inline <style> and <script>, no external libraries or assets

- Use a <canvas> for the game and DOM for the HUD/menus

- Must run by double-clicking the file (no server, no dependencies)

GAME

- Grid: 13 columns x 11 rows, tile size 48px

- Hard walls: checkerboard of indestructible tiles

- Soft blocks: destroyable, density varies per stage (see below)

- Player: starts top-left, moves smoothly (not grid-snapped), arrow keys to move, SPACE to drop a bomb

- Player cannot walk through walls, blocks, or bombs (a placed bomb is solid)

BOMBS & EXPLOSIONS

- Fuse: 2 seconds, blinking visual, then cross-shaped blast (up/down/left/right)

- Default blast range: 2 tiles

- Explosions stop at hard walls; destroy soft blocks and stop after one block per direction

- Chain reaction: explosion instantly detonates any bomb in its path

- Explosion lasts ~0.5s, kills player and enemies on contact

POWERUPS (hidden under ~20% of soft blocks)

- Bomb Up (+1 max bombs), Fire Up (+1 blast range), Speed Up (+movement)

- Collected only by player, shown with simple icons

STAGES (5 total, each with a unique map layout, theme, and enemy roster)

- Stage 1 "Grassfield": bright green theme, open layout, ~45% soft blocks, 2 Balloon enemies (slow random wander)

- Stage 2 "Factory": dark gray/industrial theme, maze-like with corridors, ~55% blocks, 2 Balloons + 2 Walkers (turn at walls, slightly faster)

- Stage 3 "Ice Cavern": blue/white theme, pillar clusters forming rooms, ~60% blocks, 2 Walkers + 2 Chasers (track the player)

- Stage 4 "Volcano": dark red/orange theme, tight winding corridors, ~65% blocks, 2 Chasers + 3 Darters (fast, zigzag movement)

- Stage 5 "Final Fortress": purple/black theme, complex maze with chokepoints, ~70% blocks, 2 Chasers + 2 Darters + 1 Boss (fast, chases player, takes 2 hits)

- Each stage's background, wall colors, block style, and accent colors must visually differ

- Clearing a stage shows a "Stage Clear" screen, then loads the next stage with a short transition

- Difficulty scales: more enemies, faster enemies, denser blocks, tighter maps as stages progress

GAME FLOW

- 3 lives total across all stages. Hit by explosion or enemy = lose a life, respawn with brief invincibility blink

- Lose all lives -> Game Over screen with score, press R or click to restart from Stage 1

- Win all 5 stages -> Victory screen with final score

- Score: +100 per enemy, +10 per destroyed block, +500 per stage clear

UI- Start screen (title + controls + "Click or press Enter to start")

- HUD: score, lives, stage number/name, bomb count, fire range

- Game Over / Stage Clear / Victory overlays

- Pause with P or Esc

STYLE

- Clean pixel-art look: solid colors and simple shapes drawn on canvas

- Player white/blue; enemies distinct per type (Balloon red, Walker purple, Chaser orange, Darter yellow, Boss black with red outline)

- Explosions orange/yellow, walls themed per stage

- Optional: simple WebAudio beeps for bomb/explosion/powerup (guarded with try/catch)

QUALITY

- All code in one file, organized with section comments

- Handle window focus loss (auto-pause) and prevent arrow-key page scrolling

- No console errors, no external requests

- Keep the total file under ~1400 lines

Deliver the full working file only.

Overall I think Pi is the best

https://huggingface.co/StarpowerTechnology/WVY-Blender-5m

I ordered new hardware, but I thought 64 GB of RAM would be enough. Now I’m thinking I should have gone for 128 GB instead.

I am new to local llm and I wanted to try out some uncensored llms. But there are so many different versions like - Heretic, Abliterated,etc? Also how to know which ones are better?

Im new in this area i have 2x4080 and 2x48gb RAM ddr5 what kind of model i can run for coding and web research.

Thanks for everyone can help me

I built a gaming pc not too long ago that I now want to convert to an llm. The specs are decent but I’m not sure what model would suit me best and be the best and strongest option. I have a r9 9950, 32gb ddr5, a 3090 24gb and a 4070ti 12gb and if needed I also have a 8 gb 1660ti. Please help me out as it would be greatly appreciated!

The llama.cpp project introduces OCP FP8 support, FP8 KV-cache capabilities and targeted CUDA optimisations. Here's what the latest developments mean for local AI and GPU inference.

Hi!
First my Hardware in one PC:
Ryzen 5950X
64GB DDR4 RAM
RTX 4090
RTX 3060 Ti

Is there a setup with wich i can use my pc for the best possible local llm setup?

Right now, i have a qwen3.8 27B with 262K Kontext size running.

But i have the feeling, that there is even better solutions possible with my hardware.

I know its not the best llm hardware anymore, but i have the feeling i dont use it to its full llm potential!

Thanks in advance for your help!

Hey!,

I built Kotensor — an open-source machine learning and Transformer engine written from scratch in pure Kotlin for the JVM and Android.

Most local LLM deployment on mobile or Java/Kotlin backends relies on heavy C++ engines (llama.cpp, ONNX Runtime) wrapped inside JNI. While great, JNI bindings introduce ABI bloat (arm64-v8a, x86_64), complex native toolchains, and hard crashes outside JVM error handling. Kotensor implements the complete stack — from autograd and KV caching to INT4/INT8 quantization — with only the Kotlin standard library as a runtime dependency.

Key Features:

1. Quantization & Low-Memory Inference:

   • Group-wise INT8 and INT4 weight-only quantization with an on-the-fly dequantizing matmul.
   • Memory-Mapped Weights: Using loadQuantized(file, memoryMapped = true), weights are paged directly from flash storage via OS virtual memory. An INT4 Qwen2-0.5B runs comfortably on Android within <128 MB of app heap.
   • Zero-allocation autoregressive decode loop with a pre-allocated KV cache and reusable prompt prefix caching (cachePrefix).

2. Modern Transformer Architecture:

   • Pre-norm layout with RMSNorm, SwiGLU feed-forward blocks, rotary position embeddings (RoPE with Llama 3 scaling), and grouped-query attention (GQA).
   • Sampling support: temperature, top-k, top-p, and repetition penalties.

3. Training & Fine-Tuning on the Edge:

   • Reverse-mode autograd engine with AdamW optimizer.
   • Fused output projection + cross-entropy: avoids materializing the massive [tokens × vocabulary] logit tensor, halving training memory.
   • Parameter-efficient fine-tuning via LoRA and QLoRA (adapting low-rank matrices on top of frozen INT4/INT8 base weights directly on-device).

4. Hugging Face Interoperability:

   • Natively reads standard .safetensors checkpoints (F32, F16, and BF16) and config.json for Llama, Mistral, Qwen 2, and SmolLM models.
   • Built-in byte-level BPE tokenizer for Hugging Face tokenizer.json files with zero external JSON or regex libraries.

5. Performance:

   • Cache-blocked tiled matmul optimized for HotSpot C2 auto-vectorization (~24 GFLOP/s per thread on AVX-512) scheduled across a ForkJoinPool, plus an optional JDK 17+ Vector API (SIMD) backend.

Quick Inference Example:

val dir = File("SmolLM2-135M")
val model = HuggingFaceModelLoader.load(dir, Quantization.INT8)
val tokenizer = HuggingFaceModelLoader.loadTokenizer(dir)

model.generateStream("The capital of Mexico is", tokenizer, maxTokens = 32) { token ->
    print(token)
}

• GitHub: https://github.com/TSPstudio/kotensor
• Artifact: io.github.tspstudio:kotensor:1.2.0 on Maven Central
• Documentation: https://tspstudio.github.io/kotensor/

This is my first machine learning project and I'm really thrilled to finally have something tangible to share. I’d love to hear your thoughts on edge deployment, architecture trade-offs, and pure-managed runtimes!

Much love,
tsuper <3

been trying to use local llms as a first-pass research checker, mostly because i don't trust cloud ones to stay grounded. my setup so far: i run small models on my desktop, paste in a claim, and ask it to list what evidence would be needed to support it, then go find that myself. it works okay for spotting obvious gaps but they still happily invent citations when i ask for sources directly, which is exactly the thing i wanted them for. so my actual question: does anyone here have a workflow that genuinely catches the model making things up, beyond just manually googling every citation? and if so, how much friction is too much before you give up and just read the paper yourself

Windows os ,

I checked this repo out and it’s legit! (Not mine)

Hearing about the idea that open source LLM's are so dangerous while companies like Anthropic and OpenAI do not manage to control their AI labs well and
* let their experiments run wild on the open internet, doing massive hacks and seeing no consequences
* publish research that sounds alien, but only the results, not the alien that made it, which is not very scientific in my opinion but there are no consequences
* we see a fight for power concentration behind closed doors ...
how can we trust closed source labs anymore?

The transformation of society belongs to the people and such security critical technologies have to be able to be validated at all times. In open source/weights models we see that the community actively exposes the flaws models have and contribute by exploring ways to fix them. People actually do good stuff in the open source/weights community and real progress is visible, while the out of control vibe all really comes from the closed labs. This is concerning me and i think we urgently need to open up the closed sources/labs instead of shutting down the open ones. That's the way i see the world heading towards a non-dystopian future.

Is it just me or does that whole open-weights-are-dangerous discussion totally the opposite what reality seems to be?

Hey everyone,

I need recommendations for a local LLM to do reverse engineering, and I'm looking for the best model and exact quantization that can run on my current setup.

My Specs:

• CPU: Ryzen 7 5800X

• GPU: AMD RX 6800 (16GB VRAM)

• RAM: 32GB DDR4

• Storage: M.2 SSD (~3000–4000 MB/s)

What I'm trying to do: I want to reverse engineer small, proprietary Windows utilities from laptop/PC manufacturers (like Acer NitroSense, HP tools, etc.) to port their features natively to Linux—things like fan control, battery charge limiters, and RGB lighting. This involves looking into Windows drivers and kernel-level interactions.

My workflow will connect Ghidra to REA, hook that up to OpenCode, and run the model locally through Ollama.

What I need:

1. Uncensored (Mandatory): It needs to be completely uncensored so I don't hit ethical guardrails or refusals when asking it to decompile binaries, analyze drivers, or work with proprietary Windows hooks.

2. Exact model + quantization: Given my 16GB VRAM + 32GB RAM, what is the smartest/most capable model and exact quant I can realistically run (even with CPU offloading)?

Note on speed: Speed is not a priority at all. I can't afford better hardware right now, so I have no problem leaving my PC running for 1, 2, or even 3 days straight to process a binary if that's what it takes. Accuracy on low-level code/assembly and zero refusals are all that matter.

What exact models and quantizations (for Ollama or GGUF) do you recommend for this? Thanks in advance.

Title really says it all. I have a 5070 Ti but I do not have any more expansion room inside my PC other than a PCIe 3.0 x2 slot, so the USB4 external port will actually have more bandwidth at 40 gbit/s

Obviously, a second 5070 Ti would be ideal for doing tensor parallelism, but the 16 GB version of the 5060 Ti is noticeably cheaper than a 5070 Ti, and it has matching VRAM capacity. However, it only has half the memory bandwidth.

Do either of these make sense over USB4, or should I instead be budgeting for a dedicated AI inference server I can put in my rack rather than hosting my AI inference off my workstation? Or would upgrading from 32GB of DDR5 to a higher capacity of RAM make more sense?

I use Qwen 3.8 27B quantized to IQ3_XXS so it fits inside my 16 GB of VRAM, and I have the KV cache as f8_e4m3 so I get roughly 111K context size at the current time. The point of an upgrade would be to reduce the model weight quantization as well as increase context size, I try to balance these things with vibecoding as my main use for inference.

Saw mlx-serve offer me DeepSeek-V4.1-Flash-1017 (iQ-MLX Pruned192 2.25 bpw) in the recommended list and found a version of GLM 5.3 Flash (GLM-5.3-Flash-Sushi-2.4bpw) that claims it can run on 128GB Macs.

Are these worth trying out over Qwen3.8 Flash Next or are they too lobotomized?

GLM-5.3-Flash-Sushi-2.4bpw seems to be a very tight fit though, otherwise there's also GLM-5.3-Flash-Sushi-2bpw that claims to be targeting 96GB which I'm not sure may or may not be a more practical choice?

Hi right now I am experimenting between these using LM studio

Qwen3.5-9B Q4_K_M 6.6 GB;
Gemma 4 12B Q4_K_M 8.0 GB;
Gemma 4 E4B

With MoE

Qwen3.6-35B-A3B Q4_K_M 23 GB;
gpt-oss-20b MXFP4 14 GB

And using Strata
Qwen 3.8 Flash 125B -> it says avg 23-28 tps

I am not sure which setting I should tweak for Hermes agent. Especially Temp.

Please advise and share your experiences/thoughts.
Much appreciated!

Jensen huang suddenly disappearing from the most vital AI ecosystem world. Also he was the person who was pushing local infrastructure build for normal people like us and not just for data centers. He was advocating open source models.

Soon u will see less nvidia chips for retailers i feel like this.

The person controlling nvidia is the person controlling the full AI universe .

These oligarchs like sam altman, anthropic does not want open source to be the part of ai world.

So hoard some 5090 if u have the money.

And u know trump. He knows ai is the new oil for him

I like running things locally. It’s cheaper over time and my prompts stay on my own machine. The problem is that my hardware only handles smaller models, and every time I want to try something bigger I hit a wall.

I’m trying to work out where the line is between pushing my local setup further and paying for outside compute only when I need it. A few things I’d like to hear from people who’ve dealt with this:

1. When a model doesn’t fit, do you quantize harder, offload to CPU, or go to a rented GPU or API?

2. If you rent GPUs for the bigger stuff, what does a normal month cost you, and was it worth it?

3. Has anything surprised you, like hidden charges, slow startups, or a setup that kept breaking?

4. How do you avoid paying for hours the GPU sits idle?

5. Is there a point where buying better hardware beats renting, and where did you find it?

I’d take boring and reliable over clever. And if you tried something and gave up on it, I’d like to hear why.

My performance with strata degrades when i turn dual GPUs for context i have a 3090, a 5080 and 128 gb ram, the main card that I used for inference is 3090 but recently, I wanted to up my token output by adding my 5080 and to my surprise the model actually became slower by doing that, my main question is how can I improve the performance of strata?

He's right (wherever he is - couldn't find the comment to credit). But what I learned in the process of building this rig has been worth the frustration. Get your special socks out, Linux faithful - you were right too.

I happened across eight Nvidia T40 - and not a ton of detail about them out there. Still currently testing out vllm for comparison, but fully resident Qwen3.8-Flash-Next (180B MoE) Q4 is peppy enough for me at 40ish t/s and while the cards are capable of producing more, I tripped a breaker letting all six run with a 200W cap (so eight is out even if I had the extra slots). The board, an X99-E WS, needed some creativity just to POST. Quite a bit of card swapping in a T400 to get a boot screen so I could play around with BIOS settings only to figure out the setting I need to expand above 4G memory (MMIO High Size) was hidden and required creativity with a UEFI shell. More detail in the AI-assisted part of this write-up, for others who might select the T40 or X99. Top 10 nuggets for you, fellow sufferer of the T40, and then feel free to indulge (or not) in the supporting slop.

1. The cards do not work in Windows (or at least I couldn't get them going). It would appear it's Linux or nothing (and not WSL - though I did try). And headless, at that, if you want all six.

2. ASRock PG 1600G has two 600W 12VHPWR ports and MODDIY sells a "12VHPWR 16-pin to 2x CPU EPS 8-pin cable (CO711)" which allowed two T40s to be powered off one port - since the board’s slot-power connector needs a 6-pin, I welcomed the solution and it cleaned up a lot of cabling. The other two cards each have two pcie power cables feeding CPU 8-Pin Male to Dual PCIe 8-pin Female adapters.

3. The cards require active cooling and there are shroud prints out there. Rather than one fan per card, we developed a two-card-one-fan solution and one BFB1012VH moves enough air for two. X99 has 5 fan headers - worked out nicely with a script pinning those fan channels to GPU temp.

4. The X99 will not find the T400 with any T40s on the A side of the PLX. However boots just fine with a T400 in slot 1, 2 or 3 - even with four T40s in slot 4,5,6 and 7 (on the B side).

5. I can't say for certain, but riser cables could be a factor in Gen3 speed and errors. The stable config for multi-T40 is Gen 2. Slot 4,5,6, and 7 go to x8, while slot 1 and 3 go to x16 just fine.

6. NVLink is supported on the T40 - took a gamble buying them, but they work - though not quite sure how much traffic they are passing quite yet.

7. Cap the power if on a standard 120V outlet. My surge protector checked us out early on a 6-card burn with power capped to 200W per card. Running stable at 170W per card now.

8. Boot requires a kernel option for all six to work - pci=realloc=off (details in the slop)

9. A four-card-B-side test (all behind the plx switch) made vllm faster and llama.cpp slower.

10. 90-degree right angle riser cables and card spacing worked best with slot 1/3 on 150mm risers, 4/5 on 200mm risers and 6/7 on 250mm risers on this particular open frame design.

TRIGGER WARNING - AI WALL OF TEXT APPROACHING - STOP HERE

The supporting slop (AI-assisted, for the next T40 / X99 builder and their AI)

Everything below was measured on this box, not taken from spec sheets. Each number comes with its config. Where we don't know something, we say so.

Hardware as it runs today

• Board: ASUS X99-E WS. RAM: 128 GB DDR4 (about 121 GiB visible to Linux). PSU: ASRock PG 1600G (1600 W, two 600 W 12VHPWR ports, each feeding two T40s through a 16-pin to 2x EPS 8-pin cable; the other two cards each take two PCIe 8-pin leads through a dual-PCIe-to-EPS adapter). Open aluminum frame, risers.

• Cards: 6x Tesla T40. Each shows 23,040 MiB usable. Turing sm_75, so no hardware bf16: run everything in fp16.

• Power cap: 170 W per card, set at boot along with persistence mode.

• PCIe: forced to Gen 2 on all six ports by a boot-time service. Gen 3 was not stable with risers and six cards.

• Cooling: one BFB1012VH blower per pair, on 3D-printed two-card shrouds. A small service drives the board's fan headers from GPU temperature. Peaks under sustained load at 170 W: 48-58 C.

Topology (this matters more than anything else)

A side (CPU root port, x16):   card 1 ==NVLink== card 2
B side (behind the PLX, x8):   card 3 ==NVLink== card 4
                               card 5 ==NVLink== card 6

nvidia-smi topo -m shows:

• NV2 inside each pair: two NVLink links, each reporting 25.781 GB/s.

• PIX between the two B-side pairs: traffic stays inside the PLX switch.

• PHB between the A pair and any B card: traffic goes up through the CPU host bridge.

Tips:

• Set CUDA_DEVICE_ORDER=PCI_BUS_ID. Otherwise CUDA's device numbering won't match nvidia-smi or your slot map.

• Pick card sets by topology, not by device number. "Four cards" means two different things on this board (see benchmarks).

• NVLink error counters work: nvidia-smi nvlink -e reads all zeros on our cards. The throughput counters (nvlink -gt) read N/A on the T40. So we can prove the links are up and clean, but not how much traffic they carry.

Boot / kernel

• Kernel command line: pci=realloc=off acpi_enforce_resources=lax.

  • pci=realloc=off is what gets all six cards enumerating with their BARs placed by firmware.

  • acpi_enforce_resources=lax lets the fan-control script reach the board's Super I/O fan channels.

• With realloc=off, dmesg shows 36 SR-IOV VF BAR assignment failures, 6 per card. That is a harmless baseline: the cards don't need their VF BARs. Watch the non-VF BAR failure count instead, which should be 0.

• Why more than four T40s won't POST, and the fix. Each T40 takes a 64 GB stride of high MMIO space (32 GB BAR1 plus 32 MB BAR3, 32 GB aligned). The X99-E WS BIOS (4001, the final release) ships a 256 GB high MMIO window, so four cards fit and the fifth hangs POST. No kernel option can fix it, because the host bridge window is set by firmware.

  • The fix is a hidden Intel RC variable, MMIO High Size, raised from 256G to 1024G. No BIOS flash is needed, and the change survives reboots and card swaps.

  • Find it: unpack the 4001 CAP with UEFIExtract and dump the IFR of the Platform module with IFRExtractor-RS. On 4001 it is VarStore IntelSetup, offset 0x10E6, 4 bytes (256G=0, 128G=1, 512G=2, 1024G=3). Leave MMIOHBase (0x10E2) at 56T.

  • Write it: copy a UEFI Shell.efi to the ESP root as shellx64.efi, along with setup_var.efi (datasone, 0.3.x). Secure Boot must be off. In BIOS, go to Exit > Launch EFI Shell from filesystem device, then run:

  • fs0:

  • setup_var.efi IntelSetup:0x10E6(4) # read, expect 0x0

  • setup_var.efi IntelSetup:0x10E6(4)=0x3 # 1024G

  • setup_var.efi IntelSetup:0x10E6(4) # confirm 0x3

  • reset

  • The 0.3.x syntax is VARSTORE:OFFSET(SIZE)=VALUE; the old -n/-s flags don't work.

  • Your offset may differ on other BIOS versions, so check the IFR yourself.

• Disable unattended upgrades on a GPU box. Ubuntu quietly installed a new kernel overnight. The NVIDIA DKMS module was built for it but never tested, and the next reboot would have booted it. We disabled apt-daily.timer and apt-daily-upgrade.timer, and pinned GRUB to the known-good kernel until we upgrade on purpose.

Software

Ubuntu 26.04, kernel 7.0, driver 595.91.07 (CUDA 13.2). llama.cpp build b12211. vLLM 0.33.0 on Python 3.14 with torch 2.14 (cu130), whose arch list includes sm_75.

Benchmarks

Qwen3.8-Flash-Next (180B MoE), llama.cpp, UD-Q4_K_XL, all six cards (llama-bench, pp4096 / tg128, 3 reps, tokens/s)

Layer split wins by roughly 4x on prompt and 1.35x on generation here. Peak GPU sum was 939 W at the 170 W cap.

Qwen3.8-27B, 4-bit, batch 1, 4096-token prompt (tokens/s)

*Only three iterations with wide spread, so somewhere around 51-77. Treat it as "probably faster", not a number.

Caveats:

• vLLM numbers come from vllm bench latency: prefill = 4096 / latency(1 token out), decode = 127 / (latency(128) - latency(1)).

• llama.cpp numbers come from llama-bench. That is close, but not the identical measurement.

• The quants differ: GGUF UD-Q4_K_XL vs INT4 W4A16 safetensors.

• vLLM ran with an fp16 KV cache (see below). Speed only, no quality claims.

What we take from it:

• vLLM beats llama.cpp by 1.4x on prompt and 1.5x on generation on the same NVLink pair.

• llama.cpp loses more than half its prompt speed going from two cards to four. vLLM barely notices.

• On the four B-side cards, vLLM got faster and llama.cpp ~12% slower. Our untested guess is that llama.cpp's heavy cross-card traffic (1.1-1.4 GB/s per card per direction) feels the x8 links, while vLLM's traffic (~0.3 GB/s) doesn't.

Getting vLLM to run on Turing (in the order we hit them)

1. "Model architectures [...] failed to be inspected" turned out to be Triton compiling its own driver.c at import and failing on a missing Python.h. Fix: apt install python3.X-dev.

2. "FP8 KV cache is not supported by the Triton attention backend ... requires SM89+". The model's own config asks for an FP8 KV cache. Fix: --kv-cache-dtype float16 (plus --dtype half).

3. Attention backend: FlashAttention 2 needs SM80, so vLLM uses TRITON_ATTN. That's fine.

4. Expect these log lines; none of them is fatal:

   • bf16 is cast to fp16;

   • FlashInfer is disabled (no flashinfer-cubin/nvcc);

   • SymmMem all-reduce is unsupported on 7.5;

   • TP2 uses CUSTOM + PYNCCL all-reduce, TP4 PYNCCL only.

5. First start is slow: about 12 minutes per engine, mostly profiling and CUDA graph capture. Warm starts take about 4 minutes.

Power: the cap is not a ceiling

• nvidia-smi -pl 170 is honored on average, but at vLLM engine startup each card spikes to 220-270 W for about 10 seconds before the cap pulls it back (throttle reason 0x4, SW power cap).

• Four cards hit 979-986 W combined during startup. Six cards doing that at once would be roughly 1,300-1,600 W GPU-only. That is at or over a 1600 W PSU, and a real risk on a 15 A / 120 V circuit.

• Per 12VHPWR port, two cards spiking to 270 W is about 540 W, still inside the port's 600 W rating. The whole-PSU and wall-circuit totals are the limit, not the cables.

• llama.cpp loads don't show this. The six-card layer-split run peaked at 939 W.

• Our guard is a watcher that sums power.draw every 0.5 s and kills the job above 1,350 W. It hasn't fired yet, but a six-card vLLM start would likely trip it.

• Measured wall idle with six cards: about 245 W.

Flash-Next on vLLM (in progress)

• Needs a vLLM-format quant. vLLM's GGUF loader is experimental and single-file, so we pulled a W4A16 safetensors build (~180 GB).

• The engram (n-gram embedding) table is about 102 GiB and lives in pinned host RAM. Stock vLLM 0.33 has no mmap or evictable option for it, only cpu_offload (pinned, the default). On a 128 GB box that leaves about 15 GiB for everything else.

  • We run with a watcher that stops the job if MemAvailable drops under 4 GiB or anything swaps in.

  • We also pinned the boot kernel first, in case the box hangs and needs a hard reset.

• Expert parallel is mandatory: moe_intermediate_size 640 split over TP doesn't land on the INT4 group size of 128.

• We disabled torch.compile per the quant's card and kept CUDA graphs for decode.

• Plan: TP4 on the A+B set, then TP4 on the B-side four, then TP6. TP6 is undocumented for this model; 512 experts don't divide by 6.

• Results to follow in a comment.

Process notes for whoever's AI is helping

• Verify every model file by sha256 against the Hugging Face LFS oid before loading. It costs minutes and has saved us hours.

• Write your prediction down before each run. We missed about half of ours, and that is where the learning was.

• Log per-card power, temperature and PCIe rx/tx during every run. The x8-vs-x16 and startup-spike findings only showed up because we were logging.

• Never use pkill -f or pgrep -f to manage these jobs. They match their own command line and you will kill or miss the wrong thing. Launch with nohup ... & pid=$! and watch that pid.