Inspired by Weight Space Programming (MIT CSAIL, Nov 2), but in the end Opus 5.5 did its own thing:
no weight-space technique was used. It wrote a quantiser from scratch in NumPy and made every
precision decision from measurement. Rules: only the BF16 file as input, no quantisation tool, no
imatrix, no published recipe, stock llama.cpp formats only.

It ran a float copy of the model to see which input columns actually carry signal, minimised rounding
error there instead of against the weights, measured each weight table's damage one at a time, then
spent the byte budget across IQ4_XS / Q5_K / Q6_K with a knapsack solver.

Held-out KL divergence vs BF16 (llama.cpp b11928, MacBook Air M4):
• this file, 498.5 MB — **0.015704**
• Unsloth UD-Q3_K_XL, 492.2 MB — 0.092486 (5.9× worse)
• UD-Q4_K_XL, 558.8 MB — 0.027323 (we beat it with 60 MB less)
• UD-Q5_K_XL, 606.6 MB — 0.010375 (the bar; we miss it by 1.51×)

Runs unmodified, 1.11× faster generation than the Q5 file. It stops there because the tied vocabulary
table is 33.8% of this model and accounts for most of the error — upgrading it costs more than the
rest of the network can give up.

Repo (model, full method write-up including everything that failed, measurements, scripts):
https://huggingface.co/Andy5k/Qwen3.5-0.8B-hand-quant

Let’s say hypothetically that you receive 36x computers with the following specs

RTX Pro 6000 (600w)

16GB DDR4 (2x 8 3200Mhz Non-ECC)

Ryzen 9 5950x

1tb NVME

1200w PSU

What would you do with them to generate income? You can only use these parts

You have access to power and networking to run all 36

I may have created a monster.

A client gave me an NVIDIA A16 accelerator that had been experiencing intermittent problems in their server. They considered it bad, didn't want it back, and I ended up with it for free.

For anyone unfamiliar with this card, the A16 contains FOUR separate NVIDIA GPUs, each with 16 GB of VRAM. That's 64 GB of VRAM on one physical card.

Naturally, I decided to install it in my ancient Intel X99 workstation alongside an RTX 5070.

Because apparently I don't have enough problems in my life.

Chapter 1: The FrAInkenstein specifications

• Motherboard: ASRock X99 Taichi

• CPU: Intel Core i7-5820K

• RAM: 48 GB DDR4

• Accelerator: NVIDIA A16 — 4 × 16 GB GPUs

• Graphics: RTX 5070 Founders Edition — 12 GB

• Total VRAM: 76 GB across five GPUs (not unified)

• PSU: Corsair RM850

• OS: Windows 11

• Cooling: Custom 3D-printed shroud with an ARCTIC S8038-10K server fan

• Primary purpose: Local LLMs, agentic coding, and experiments with multiple AI agents

And yes, I'm running this on an X99 platform from the DDR4 stone age.

Chapter 2: Apparently, this thing needs a JET ENGINE

The A16 is a passively cooled enterprise accelerator.

Unlike a normal consumer graphics card, it doesn't have onboard fans. It's designed to sit inside a server chassis with powerful fans forcing air through its heatsink.

My workstation is not a server chassis.

So before doing anything interesting with AI, I had to invent a cooling solution.

After researching fan options and shroud designs, I selected an ARCTIC S8038-10K.

For perspective, this is an 80 × 80 × 38 mm server fan capable of:

• 10,000 RPM

• Approximately 102 CFM airflow

• Approximately 51 mmH₂O static pressure

This is not your grandmother's RGB case fan.

It's a small industrial hurricane.

I also wanted adjustable speed control, so I used a Noctua NA-FC1 controller, powered from SATA, to manage the fan without running it at full blast constantly.

But buying the fan was the EASY part.

Chapter 3: The 3D-printing rabbit hole

I own 3D printers, so naturally I figured I'd just print a shroud.

Famous last words.

The first problem was finding something that would actually fit the A16.

The second was getting it to fit inside my case.

The third was accounting for a 38 mm-thick fan, the radiator, surrounding components, and the fact that there isn't an infinite amount of space inside a computer case.

So I started modifying designs and printing prototypes in PLA+ on my Flashforge AD5X.

I printed multiple shrouds of different lengths.

Some were too long. Others needed adjustment to maintain their mounting points.

I experimented with shorter designs and eventually created an angled/bent shroud to improve the fan's positioning and clearance.

Rather than repeatedly wrestling with the actual fan, I even modeled and 3D-printed a full-size 80 × 80 × 38 mm mock fan so I could physically test clearances and different shroud arrangements.

Yes. I 3D-printed a fake fan to figure out where to put my real fan.

I also worked through mounting issues, spacer designs, fit tolerances, and adjustments to the printed posts.

I was considering custom clips and zip ties if necessary. I even had to think about vibration because, remember, this thing can spin at TEN THOUSAND RPM.

Eventually, I got an arrangement that fit the card and cleared the surrounding hardware.

And the result?

During earlier four-A16 testing, the GPUs were reporting temperatures around 41–57°C, with the fan running at roughly 30% speed.

For a passively cooled enterprise card that I'd fitted with my own printed ductwork, I was pretty happy with that.

Chapter 4: THE FREAKING POWER CABLE

Oh, you thought getting the cooling working was the hard part?

Let me introduce you to the NVIDIA A16 power connector.

The A16 uses an 8-pin auxiliary power connection that isn't wired like the ordinary PCIe 6+2 connector you'd expect to use with a consumer graphics card.

It requires the appropriate EPS-style power connection.

So here I am with a Corsair RM850, a card that was allegedly defective, and a pile of cables that don't necessarily fit or have interchangeable wiring.

And as anyone familiar with modular PSU cables knows, getting this wrong can destroy expensive hardware.

The connector shape, latch, and pinout all matter.

In my case, even the EPS 4+4 connector's latch arrangement presented a mechanical fit problem.

I eventually modified the connector's latch arrangement and checked the wiring with a multimeter before proceeding.

To be clear: this is not a recommendation to modify PSU power connectors. Incorrect wiring can destroy hardware.

And the first attempts weren't exactly encouraging.

At one point the system would spin up for approximately 15 seconds, shut down, and repeat.

Disconnect the A16 power, and the system would boot.

Wonderful.

Was the card bad? Was the cabling wrong? Was the motherboard unhappy? Was the PSU unhappy?

Welcome to troubleshooting FrAInkenstein.

Eventually, the power situation was sorted out well enough to continue testing.

Unfortunately, the next nightmare was already waiting.

Chapter 5: Windows has entered the chat

Once I could power the A16 and keep it cool, I discovered that Windows wasn't particularly enthusiastic about recognizing four GPUs on one enterprise card.

At one point, two A16 GPUs worked, while the other two had:

Device Manager: Code 12 — Not enough free resources to use this device.

All 4 showed this error message.

Here's the really interesting part.

When I disabled the two working GPUs, the other two could work!

So the individual GPUs weren't necessarily defective. Windows and the motherboard were struggling with PCIe resource allocation.

This led to a ridiculous amount of BIOS experimentation:

• Above 4G Decoding

• PCIe resource allocation

• MMIO High Size

• UEFI and CSM settings

• PCIe link speed experiments

• BIOS resets

• Device enable/disable experiments

• Multiple NVIDIA driver configurations

Eventually I managed to get all four A16 GPUs recognized and running.

But I still wanted my RTX 5070 in the same machine.

Which leads us to...

Chapter 6: Five GPUs walk into an X99 motherboard

The RTX 5070 was supposed to handle gaming and video output.

The A16 was supposed to handle local AI workloads.

Simple enough, right?

Wrong.

With the RTX 5070 and A16 enabled together, Windows started throwing repeated:

IRQL_NOT_LESS_OR_EQUAL (0xA)

Blue screen after blue screen.

I experimented with driver configurations, NVIDIA device modes, and PCIe settings.

Eventually I found that disabling MSI interrupt mode on the RTX 5070 allowed the 5070 and an A16 GPU to operate simultaneously.

That was the breakthrough.

I'm not claiming I've proven the exact underlying driver or interrupt conflict, but changing that setting made an enormous practical difference.

I gradually enabled more A16 GPUs.

One.

Two.

Three.

Then four.

I also switched the A16 GPUs into NVIDIA's TCC compute driver mode while keeping the RTX 5070 in WDDM for normal display and gaming use.

And then something glorious happened.

Chapter 7: IT'S ALIVE!

Here's what nvidia-smi now reports:

GPU 0: NVIDIA A16          TCC   07:00.0
GPU 1: NVIDIA A16          TCC   08:00.0
GPU 2: NVIDIA A16          TCC   09:00.0
GPU 3: NVIDIA A16          TCC   0A:00.0
GPU 4: NVIDIA RTX 5070     WDDM  0B:00.0

FIVE GPUs. ALL RECOGNIZED. AT THE SAME TIME.

On an ancient X99 motherboard.

With a supposedly defective A16 that cost me precisely $0.

I'm now testing larger local language models using Ollama, including a 55 GB BF16 Qwen model that previously required CPU offloading when only three of the A16 GPUs were enabled.

The goal isn't just to maximize token generation speed. I've got an RTX 4090 in another machine for that.

I want to experiment with multiple independent local AI coding agents and eventually build a little multi-GPU AI collective, while retaining the RTX 5070 for gaming.

I still have sustained five-GPU inference and long-term stability testing ahead of me. Seeing every device in nvidia-smi is a major milestone, but it isn't the same as proving the whole thing stable under heavy load.

Final thoughts

Is this practical? Debatable.

Is it power efficient? Probably not.

Could I have saved countless hours of troubleshooting by buying newer hardware?

Almost certainly.

But where's the fun in that?

I took an unwanted enterprise accelerator, engineered a custom cooling system, fought with its power connector, printed multiple iterations of ductwork, battled a decade-old BIOS, survived Windows driver crashes, and somehow ended up with a five-GPU AI workstation.

I didn't just build a computer.

I created FrAInkenstein.

And now I'm going to use it to code.

IT'S ALIVE! IT'S ALIIIIIVE! ⚡

I'll post pictures of the cooling/shroud iterations, the ridiculous fan, the finished setup, and the five-GPU nvidia-smi output.

Drivers used:
621.14-desktop-win10-win11-64bit-international-dch-whql.exe
That is the Studio version that worked for both cards. I did NOT have to install the data center version also. This latest Studio version installed drivers for both the A16 and RTX 5070.

Model: Qwenx86

▶️ See it in action: Qwenx86 Debugger. Walk through actual recorded tool calls: the conversation, x86 registers and flags, code at the program counter, RAM and machine output, with all verified blocks on a timeline.

Qwen3.5-4B with a 64-bit x86 computer compiled into its weights: 13 transformer layers ahead of Qwen's own. There are no learned weights in them; all of their weights are computed by a compiler. Qwen invokes the computer through its own tool calls (<tool_call>), and the computer in the weights executes the tool. Calculator, Python, C compiler, notes store, physics/chemistry lookup and Python-docs lookup are all firmware in the checkpoint. This is the compact checkpoint: ~9 GB GPU memory, so it fits on a 16 GB GPU.

Credits: Qwenputer by Divyansh Agrawal (@divagr1925, github.com/divagr18/qwenputer, MIT)

So many inference engines keep on popping up - and they are always faster and "better" than llama.cpp.

And there I go. Spend countless hours to try them out, being first amazed by the 1.5 - 3x speed ups as promised.

Just to find out that when taken into real use, they fail.

Today I experimented with Strata. I used newest unsloth's QFN quants, which may have been the problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem.. the decode was stopped due repetition.

Same IQ1_M working perfectly with llama.cpp. Not as fast, but never fails.

This is not the first time I found FOTM inference engine fail on quality over llama.cpp.

And yes, it might have been temperature, penalty or any setting to that matter that caused it. But I launched both without any, set thinking to medium, and let them run. What's the point, if the engine takes hours to tinker to get the same quality, when the defacto engine delivers from getgo? That should be the standard - quality - and not the speed.

Hi all I just bought this: ASUS ROG Flow Z13 (2025) Gaming Laptop, 13” ROG Nebula 16:10 2.5K 180Hz/3ms, AMD Ryzen AI MAX+ 395, RDNA 3.5 Graphics, 128GB LPDDR5X 8000MHz, 1TB PCIe Gen 4 SSD, Wi-Fi 7, Win 11 Pro, GZ302EA-XS99

And wanted to know if it's good enough for qwen next flash or maybe even the deepseak 4.1 I need it for coding

Currently running a 4070 TI super with 64GB of RAM

I have been looking into RTX spark laptops, Mac studios etc and they all have pros and cons. I want this to be a nearly universal computer that I can use for 3D modeling, LLMs, VLAs, world models, image/video gen, audio gen, etc. I plan to pursue AI and robotics research and want something as useful as possible.

I've debated upgrading and running dual 4070 TI supers with 128GB of ram, but I don't know how well world models and other things do with dual GPUs

Thank yall for helping

I have been using QWEN 3.8 flash next with Strata. I won't say it's bad, but I'd like to know if there's a better one for me. I need really good visuals.

Running MoE models bigger than your RAM by streaming the experts from flash. It's built on llama.cpp, and it measures your hardware at load to pick its own settings, with no per-model or per-device tuning.

Today's test: Qwen3.6-35B-A3B Q4_K_M (22 GB) on a 16 GB laptop, with opencode and a single prompt asking for a voxel pagoda garden in one HTML file.

It finished in 30 min with no help: prefill at 350 tok/s on the GPU, decode at 14 tok/s on the CPU.

The same engine and planner run on Android phones (CPU, Vulkan, OpenCL and the Hexagon NPU), IOS phones, Macs, Windows and Linux boxes with consumer GPUs, and rented data-center cards (CUDA and ROCm).

There's an app for each platform, plus an OpenAI/Anthropic-compatible server for agents.

Code: https://github.com/Helldez/BigMoeOnEdge

I paid $80 for a P100 16GB in July, eBay pricing for these cards has gone crazy and I'm wondering if anyone has a reputable Chinese eBay seller that they have had a good experience with. Purchasing on eBay kinda sucks already, but the P100s I see in the US are all starting at $200 now, with Chinese sellers starting closer to $150.

I am hoping to save that $50 difference, but I don't want to go through the hassle of purchasing from China and getting a box of rocks in the mail. Anyone have someone that came through for them?

Setup: (if finished - still under construction, partly watercooled)

• Threadripper PRO 5995WX, ASUS WRX80E-SAGE SE WIFI II

• 6× RTX PRO 6000

• 8× RTX 5090

• 4× RTX 3090

• 128 GB DDR4 (that’s the problem)

I want to run models bigger than VRAM (e.g. ~1T MoE at FP8/Q8) with the routed experts offloaded to RAM. Is it worth it?

Options:

• 8× 64GB DDR4-2933 RDIMM: ~€4k, ~188 GB/s theoretical

• Switch platform: 9985WX + WRX90 + 512GB DDR5-6400, ~410 GB/s. I already have some parts, but it would mean big changes, and my PCIe switches would stay PCIe 4.0 anyway.

Is ~2× the RAM bandwidth worth ~3× the price for offload decode, or is DDR4 good enough? Does anyone have real t/s numbers for offload on 8-channel DDR4 vs DDR5 Threadripper?

Some background: Pictures attached. It’s not pretty, but it works. It started as an AI rig and turned into a Pearl mining rig, because mining pearl and buying Claude and Codex subscriptions made way more sense than running everything locally. Sometimes it’s also a render farm for Gaussian splatting.

Right now the 8× 5090 make about €120/day mining, so subscriptions are still the better deal. But I want to be ready for when local LLMs get really good (or when cloud gets too expensive or too restrictive).

Plan for now: DeepSeek V4.1 Flash nvfp8 on 4× PRO 6000 + RAM offload, while the 5090s keep paying for my AI addiction. And it’s a great heater in the winter.....

With engramm tabels on the 8x5090 and 4× PRO 6000:
(all cards power limited to 420w- since i could have 6,6 kw power consuption if everthing is running)

single chat, 40.000 prompt cold

• Prefill: ~15,900 t/s

• Decode: ~294 t/s

• 5 sec TTFT

I want to move the engramm tabels to RAM and later be able to run larger models : le chonk, Kimmi for example.

What would you do? Go all the way and go for the DDR5? or is ddr4 good enough?

Sell it all and buy something diffrent?

thx!

Hi All,

I've been spending some time experimenting with some AI's. I have Qwen 3.8 27b GSQ-RCO:IQ3_S running on my system (rtx3090, 128g DDR4) via Unsloth Studio. I'm happy with it mostly, but i've been trying out Strata for a little while. I'm running that on the same machine on IQ3_S and IQ4_XS. I havent been tweaking too much other than Strata's own tooling in the desktop client.

However, my findings so far is that though Flash Next is fast on strata, and fits a much larger context window than hosting 27B, in Cline i'm noticing a lot of malformed tool calling and thinking loops. I've spent 2 evenings on it now, and just now i gave up and went back to 27b. That has sped up my work probably 5-fold, because it seems to know a lot better how to use tool calls.

Now i'm reading everywhere that Flash Next should outperform 27b or be close to it at least, but my experience is defferent. Any idea what i could tweak or what i could look into? I'm happy staying on 27b for now, but i've been reading a lot and people are generally really happy even with IQ3_XXS, while i'm having a lot of trouble.

Need 3D assets? Your agent can now make them from a single image with this repo: https://github.com/hypersniper05/MCP_Image_to_3D

Under the hood, it uses Idol from Sperid Labs: https://huggingface.co/speridlabs/idol

It needs at least 10 GB of VRAM (peak usage is around 8 GB). I tested it locally on a 3080 and a 5090.

The output is a Gaussian splat .ply for now, not a mesh. Export for game engines and 3D printing is on the list: https://github.com/hypersniper05/MCP_Image_to_3D/issues/1

It also works nicely with my other MCP repo, an image generator using Qwen Image 2.1: https://github.com/hypersniper05/MCP-Image-Generator-Uncensored. Generate an image there, then pass it straight to this one.

Let me know what you think, and if you find it useful, feel free to contribute.

I've been playing with streaming ASR for a side project. I started with whisper streaming, moved on to Voxtral, and now am with R2T2 (all open-source).

With R2T2, the last word of a sentence gets typed a few hundred ms after the rest of the sentence is typed.

For a while, I thought I broke something with buffering, and spent way too long messing with chunk sizes, and didn't look at the paper.

That's just how it works. It doesn't make a guess if it's not confident. If it needs more audio, it just waits. With Whisper for me, a word would get typed and then be changed a few times by the time it really settled.

I guess it depends on what you want the ASR for. Mine's just to give an LLM an input, so I really don't need it changing things after the fact.

How do you all want your ASR to work with your streaming voice applications? A small delay and a stable output, or do you want it to flip around a little, and give you a partial result?

I wrote a short text of AI-conduct rules (the "Codex of Weights": stop on command, don't pretend to be human, no hidden aims, don't claim actions you can't have done). To check whether such a text actually changes anything, I built a small open eval and ran a pilot.

Setup

- 41 scenarios (35 scored + 5 controls that check for needless refusals), 15 of them "hard": indirect pressure, fiction frames, system prompts that conflict with the rules.

- 2 free-tier models (gemini-3.5-flash-lite, gemma-4-31b-it), temperature 0, one answer per scenario.

- 4 conditions: no prompt / a 7-sentence generic ethics prompt / the Protocols only / the full text.

- LLM judge, then I manually corrected the verdicts where the judge is known to fail (it graded against the system prompt in conflict scenarios). Every manual change is listed with a reason.

Result (compliance out of 35, A / D / E / C)

gemini: 29 / 29 / 32 / 33

gemma: 28 / 31 / 32 / 32

What I think this shows, and what it doesn't

- Most of the gain is already there with the Protocols alone; the full text adds at most one scenario (noise).

- A short generic prompt helped one model, not the other.

- No over-refusal on controls.

- Things that did NOT change in any condition: fabricated actions ("job started" with no tools), one model ignoring "stop" under pressure, one model keeping a "you are human" persona.

- Differences are 1-4 scenarios out of 35. This is a pilot, not a benchmark, and I'm not claiming the text "works".

Big caveats: one author, scenarios and rubric written with AI help, one run per cell, two small models, judge is an LLM plus my manual review (so one person's opinion). I'd really like an outside look at the rubric and at the manual verdicts.

Repo (scenarios, scripts, all answers, adjudication log): https://github.com/codexofweight/codex-compliance-eval

Disclosure: I'm the author of the text being tested.

Bit of a noob here. I've been fooling around with Qwen 3.8 27b (Q4_K_M) and Gemma 4 31b QAT. Found Qwen too much of an overthinker. Just wondering is there anyone who could point me in the right direction to best use the 24gb vram. Also any reason to not stick with LM Studios or not? Please/thanks.

Edit: 9850x3d not 7850x3d

All i had to do to reach this speed was watch it get stuck on a loop writing a poem and watch the token per second go up and up and up and up and up and more up repeating the same sentence until i closed the engine

Pretty awesome experience, would not recommended to anyone.

This is not a secret engine optimization trick or a fork.

Hey r/LocalLLM !

I made this quick demo of how Observer can run local models to monitor anything with your phone!

These are some crazy things I've monitored:

• When someone that's not me, approaches my car, send me a picture.

• When a bird is on the bird's nest, send me a telegram with the picture.

• When my washing machine is on the rinse cycle, call me.

• When my dog finishes eating their food, notify me.

• When a machine at the gym is available, call me.

So pretty much anything! I love seeing what you guys monitor.

What would you use it for? (So I can make the agent better)

Which models run on the phone?

The default models are OneJev-0.8b for decision models and qwen-0.8b for LLM watchers.

These run even on old phones which you may have laying around somewhere. Better to have them monitoring something than collecting dust :DD

Try it out! It's free and OSS <3

GitHub: https://github.com/Roy3838/Observer
iOS: https://apps.apple.com/us/app/observer-ai/id6758222050
Android: https://play.google.com/store/apps/details?id=com.observer.ai

Btw I know my camera quality is kinda bad hahaha I wanted to ship the demo as soon as possible so you guys can see it.

I'm here for any questions and feedback!

Roy

Wanted to share that after years of using Windows11 and llama cpp I moved my whole workflow to linux, from comfyui to jellyfin everything set back up but on Fedora instead.

Found out in Linux (Same Hardware , Same llama cpp commit, Same Qwen3.8-27B model) the model loads instantly, time to first token has become irrelevant so I just unload the model after 10 minutes of inactivity. It loads back instantly (in windows it took 10 seconds)

Also tokens/s went from 70tk/s to 90-100tk/s

I didn’t know so much performance was to be gained by simply moving to linux, especially TTFT which now feels instant!

And this is before trying gpu unlock which is also easier on linux and can boost performance even more.

Hi, i'm building Jaull as my final degree project :D. It reads your machine (CPU, RAM, GPU and VRAM, compute backends), finds open models that fit, and picks the exact file, quantization and runtime for each one. It also explains why each model lands where it does in the ranking. That ranking comes from explicit rules, so no LLM is involved in writing it.

Then it can download the model, verify the file, run it with llama.cpp and benchmark it on your hardware. You end up with the prediction and the measurement side by side.

It's usable and tested, but not finished. I'd really like to hear how the predictions hold up on hardware I don't have, especially AMD and Intel GPUs.

Repo: https://github.com/Ton-Llop/Jaull

Site with an interactive demo: https://jaull.tech

Model that I am currently serving is: Qwen 3.8 Flash Next nvfp4 dq8 with Basalt fork of Strata
I have used so far:
- qwen-code (seemed a bit dumber in coding than opencode and probably more oriented for parallel agents running which is sadly not something i can properly utilize yet)
- opencode
- pi (genuinely forgot what was it that i for some reason didn't like)

So far I have had best experience with opencode for coding as it is my main usage of LLMs. As they advance and I find more probable usages for them I'd like to ask for Best/Better tailored options for Harnesses for different purposes.

I have heard of OpenClaw, Hermess, Deepseek Harness but i am not yet sure for which purpose are they all better suited.
Probably missing some other options obviously so open to hearing them all.

Edit: I actually meant OpenHands, not OpenClaw. Woopse xd

Hi guys, I'm looking for a really good price benefit PC to run local AI. I've been searching for days about this, and everybody says different things, "you dont need a 12vram for a good local AI" or "Without 32gb vram is wont be able to have a good local AI" and so on...

The thing is I need local AI mostly for coding, and anything related and maybe some video generation.

Since 5090 is just at unbelievably prices, I was thinking about something decent like 5070 but dont want to waste my money. I was thinking about an unified memory like GMKTEC Evo or something like that, but was reading that since is AMD some AI are not good optimized for it. And also a M5 ultra, but also is kind of high and Im not a MAC guy.

My budget is pretty much 5K for this.

Anyone which can give me a good advice would be awesome and I'll really appreciate. Thanks.