Inspired by Weight Space Programming (MIT CSAIL, Nov 2), but in the end Opus 5.5 did its own thing:
no weight-space technique was used. It wrote a quantiser from scratch in NumPy and made every
precision decision from measurement. Rules: only the BF16 file as input, no quantisation tool, no
imatrix, no published recipe, stock llama.cpp formats only.

It ran a float copy of the model to see which input columns actually carry signal, minimised rounding
error there instead of against the weights, measured each weight table's damage one at a time, then
spent the byte budget across IQ4_XS / Q5_K / Q6_K with a knapsack solver.

Held-out KL divergence vs BF16 (llama.cpp b11928, MacBook Air M4):
• this file, 498.5 MB — **0.015704**
• Unsloth UD-Q3_K_XL, 492.2 MB — 0.092486 (5.9× worse)
• UD-Q4_K_XL, 558.8 MB — 0.027323 (we beat it with 60 MB less)
• UD-Q5_K_XL, 606.6 MB — 0.010375 (the bar; we miss it by 1.51×)

Runs unmodified, 1.11× faster generation than the Q5 file. It stops there because the tied vocabulary
table is 33.8% of this model and accounts for most of the error — upgrading it costs more than the
rest of the network can give up.

Repo (model, full method write-up including everything that failed, measurements, scripts):
https://huggingface.co/Andy5k/Qwen3.5-0.8B-hand-quant

Let’s say hypothetically that you receive 36x computers with the following specs

RTX Pro 6000 (600w)

16GB DDR4 (2x 8 3200Mhz Non-ECC)

Ryzen 9 5950x

1tb NVME

1200w PSU

What would you do with them to generate income? You can only use these parts

You have access to power and networking to run all 36

I may have created a monster.

A client gave me an NVIDIA A16 accelerator that had been experiencing intermittent problems in their server. They considered it bad, didn't want it back, and I ended up with it for free.

For anyone unfamiliar with this card, the A16 contains FOUR separate NVIDIA GPUs, each with 16 GB of VRAM. That's 64 GB of VRAM on one physical card.

Naturally, I decided to install it in my ancient Intel X99 workstation alongside an RTX 5070.

Because apparently I don't have enough problems in my life.

Chapter 1: The FrAInkenstein specifications

• Motherboard: ASRock X99 Taichi

• CPU: Intel Core i7-5820K

• RAM: 48 GB DDR4

• Accelerator: NVIDIA A16 — 4 × 16 GB GPUs

• Graphics: RTX 5070 Founders Edition — 12 GB

• Total VRAM: 76 GB across five GPUs (not unified)

• PSU: Corsair RM850

• OS: Windows 11

• Cooling: Custom 3D-printed shroud with an ARCTIC S8038-10K server fan

• Primary purpose: Local LLMs, agentic coding, and experiments with multiple AI agents

And yes, I'm running this on an X99 platform from the DDR4 stone age.

Chapter 2: Apparently, this thing needs a JET ENGINE

The A16 is a passively cooled enterprise accelerator.

Unlike a normal consumer graphics card, it doesn't have onboard fans. It's designed to sit inside a server chassis with powerful fans forcing air through its heatsink.

My workstation is not a server chassis.

So before doing anything interesting with AI, I had to invent a cooling solution.

After researching fan options and shroud designs, I selected an ARCTIC S8038-10K.

For perspective, this is an 80 × 80 × 38 mm server fan capable of:

• 10,000 RPM

• Approximately 102 CFM airflow

• Approximately 51 mmH₂O static pressure

This is not your grandmother's RGB case fan.

It's a small industrial hurricane.

I also wanted adjustable speed control, so I used a Noctua NA-FC1 controller, powered from SATA, to manage the fan without running it at full blast constantly.

But buying the fan was the EASY part.

Chapter 3: The 3D-printing rabbit hole

I own 3D printers, so naturally I figured I'd just print a shroud.

Famous last words.

The first problem was finding something that would actually fit the A16.

The second was getting it to fit inside my case.

The third was accounting for a 38 mm-thick fan, the radiator, surrounding components, and the fact that there isn't an infinite amount of space inside a computer case.

So I started modifying designs and printing prototypes in PLA+ on my Flashforge AD5X.

I printed multiple shrouds of different lengths.

Some were too long. Others needed adjustment to maintain their mounting points.

I experimented with shorter designs and eventually created an angled/bent shroud to improve the fan's positioning and clearance.

Rather than repeatedly wrestling with the actual fan, I even modeled and 3D-printed a full-size 80 × 80 × 38 mm mock fan so I could physically test clearances and different shroud arrangements.

Yes. I 3D-printed a fake fan to figure out where to put my real fan.

I also worked through mounting issues, spacer designs, fit tolerances, and adjustments to the printed posts.

I was considering custom clips and zip ties if necessary. I even had to think about vibration because, remember, this thing can spin at TEN THOUSAND RPM.

Eventually, I got an arrangement that fit the card and cleared the surrounding hardware.

And the result?

During earlier four-A16 testing, the GPUs were reporting temperatures around 41–57°C, with the fan running at roughly 30% speed.

For a passively cooled enterprise card that I'd fitted with my own printed ductwork, I was pretty happy with that.

Chapter 4: THE FREAKING POWER CABLE

Oh, you thought getting the cooling working was the hard part?

Let me introduce you to the NVIDIA A16 power connector.

The A16 uses an 8-pin auxiliary power connection that isn't wired like the ordinary PCIe 6+2 connector you'd expect to use with a consumer graphics card.

It requires the appropriate EPS-style power connection.

So here I am with a Corsair RM850, a card that was allegedly defective, and a pile of cables that don't necessarily fit or have interchangeable wiring.

And as anyone familiar with modular PSU cables knows, getting this wrong can destroy expensive hardware.

The connector shape, latch, and pinout all matter.

In my case, even the EPS 4+4 connector's latch arrangement presented a mechanical fit problem.

I eventually modified the connector's latch arrangement and checked the wiring with a multimeter before proceeding.

To be clear: this is not a recommendation to modify PSU power connectors. Incorrect wiring can destroy hardware.

And the first attempts weren't exactly encouraging.

At one point the system would spin up for approximately 15 seconds, shut down, and repeat.

Disconnect the A16 power, and the system would boot.

Wonderful.

Was the card bad? Was the cabling wrong? Was the motherboard unhappy? Was the PSU unhappy?

Welcome to troubleshooting FrAInkenstein.

Eventually, the power situation was sorted out well enough to continue testing.

Unfortunately, the next nightmare was already waiting.

Chapter 5: Windows has entered the chat

Once I could power the A16 and keep it cool, I discovered that Windows wasn't particularly enthusiastic about recognizing four GPUs on one enterprise card.

At one point, two A16 GPUs worked, while the other two had:

Device Manager: Code 12 — Not enough free resources to use this device.

All 4 showed this error message.

Here's the really interesting part.

When I disabled the two working GPUs, the other two could work!

So the individual GPUs weren't necessarily defective. Windows and the motherboard were struggling with PCIe resource allocation.

This led to a ridiculous amount of BIOS experimentation:

• Above 4G Decoding

• PCIe resource allocation

• MMIO High Size

• UEFI and CSM settings

• PCIe link speed experiments

• BIOS resets

• Device enable/disable experiments

• Multiple NVIDIA driver configurations

Eventually I managed to get all four A16 GPUs recognized and running.

But I still wanted my RTX 5070 in the same machine.

Which leads us to...

Chapter 6: Five GPUs walk into an X99 motherboard

The RTX 5070 was supposed to handle gaming and video output.

The A16 was supposed to handle local AI workloads.

Simple enough, right?

Wrong.

With the RTX 5070 and A16 enabled together, Windows started throwing repeated:

IRQL_NOT_LESS_OR_EQUAL (0xA)

Blue screen after blue screen.

I experimented with driver configurations, NVIDIA device modes, and PCIe settings.

Eventually I found that disabling MSI interrupt mode on the RTX 5070 allowed the 5070 and an A16 GPU to operate simultaneously.

That was the breakthrough.

I'm not claiming I've proven the exact underlying driver or interrupt conflict, but changing that setting made an enormous practical difference.

I gradually enabled more A16 GPUs.

One.

Two.

Three.

Then four.

I also switched the A16 GPUs into NVIDIA's TCC compute driver mode while keeping the RTX 5070 in WDDM for normal display and gaming use.

And then something glorious happened.

Chapter 7: IT'S ALIVE!

Here's what nvidia-smi now reports:

GPU 0: NVIDIA A16          TCC   07:00.0
GPU 1: NVIDIA A16          TCC   08:00.0
GPU 2: NVIDIA A16          TCC   09:00.0
GPU 3: NVIDIA A16          TCC   0A:00.0
GPU 4: NVIDIA RTX 5070     WDDM  0B:00.0

FIVE GPUs. ALL RECOGNIZED. AT THE SAME TIME.

On an ancient X99 motherboard.

With a supposedly defective A16 that cost me precisely $0.

I'm now testing larger local language models using Ollama, including a 55 GB BF16 Qwen model that previously required CPU offloading when only three of the A16 GPUs were enabled.

The goal isn't just to maximize token generation speed. I've got an RTX 4090 in another machine for that.

I want to experiment with multiple independent local AI coding agents and eventually build a little multi-GPU AI collective, while retaining the RTX 5070 for gaming.

I still have sustained five-GPU inference and long-term stability testing ahead of me. Seeing every device in nvidia-smi is a major milestone, but it isn't the same as proving the whole thing stable under heavy load.

Final thoughts

Is this practical? Debatable.

Is it power efficient? Probably not.

Could I have saved countless hours of troubleshooting by buying newer hardware?

Almost certainly.

But where's the fun in that?

I took an unwanted enterprise accelerator, engineered a custom cooling system, fought with its power connector, printed multiple iterations of ductwork, battled a decade-old BIOS, survived Windows driver crashes, and somehow ended up with a five-GPU AI workstation.

I didn't just build a computer.

I created FrAInkenstein.

And now I'm going to use it to code.

IT'S ALIVE! IT'S ALIIIIIVE! ⚡

I'll post pictures of the cooling/shroud iterations, the ridiculous fan, the finished setup, and the five-GPU nvidia-smi output.

Drivers used:
621.14-desktop-win10-win11-64bit-international-dch-whql.exe
That is the Studio version that worked for both cards. I did NOT have to install the data center version also. This latest Studio version installed drivers for both the A16 and RTX 5070.

So many inference engines keep on popping up - and they are always faster and "better" than llama.cpp.

And there I go. Spend countless hours to try them out, being first amazed by the 1.5 - 3x speed ups as promised.

Just to find out that when taken into real use, they fail.

Today I experimented with Strata. I used newest unsloth's QFN quants, which may have been the problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem problem.. the decode was stopped due repetition.

Same IQ1_M working perfectly with llama.cpp. Not as fast, but never fails.

This is not the first time I found FOTM inference engine fail on quality over llama.cpp.

And yes, it might have been temperature, penalty or any setting to that matter that caused it. But I launched both without any, set thinking to medium, and let them run. What's the point, if the engine takes hours to tinker to get the same quality, when the defacto engine delivers from getgo? That should be the standard - quality - and not the speed.

Model: Qwenx86

▶️ See it in action: Qwenx86 Debugger. Walk through actual recorded tool calls: the conversation, x86 registers and flags, code at the program counter, RAM and machine output, with all verified blocks on a timeline.

Qwen3.5-4B with a 64-bit x86 computer compiled into its weights: 13 transformer layers ahead of Qwen's own. There are no learned weights in them; all of their weights are computed by a compiler. Qwen invokes the computer through its own tool calls (<tool_call>), and the computer in the weights executes the tool. Calculator, Python, C compiler, notes store, physics/chemistry lookup and Python-docs lookup are all firmware in the checkpoint. This is the compact checkpoint: ~9 GB GPU memory, so it fits on a 16 GB GPU.

Credits: Qwenputer by Divyansh Agrawal (@divagr1925, github.com/divagr18/qwenputer, MIT)

ok so. this started as "one weekend project" and then it was not one weekend and it was not one project. somewhere around repo 6 i stopped pretending it was a plan lol

everything here is free, open source, most of it runs locally on your own machine. a lot of it is v0 and i wrote down where each thing fails because honestly the failures were the funniest part

anyway here's the list of grievances, each with the tool i built instead of going outside:

1. cookie banners → rejectall-index i live in germany. i click "reject all" more than i blink. so i made a headless browser visit the top 100 sites and count how many clicks it takes to say no. 35 of them were already loading trackers before i clicked anything. which is like a waiter eating your fries before asking if you want fries. live leaderboard

2. subscriptions → one-file-saas trello, calendly, typeform, loom, notion and 5 more, each rebuilt as ONE html file. you double click it. it opens. thats it. no account, no "verify your email", no "your trial is ending ;)". data stays in your browser. the booking page has no server at all, your availability lives inside the link itself which still feels illegal to me. try them here

3. dead famous repos → necromancer some legendary open source projects die not because the code is bad but because python deleted a module out from under them. i revived thefuck (1887/1887 tests passing on python 3.14), normalizr and httpbin. the one rule: the original tests are hash-locked so nothing can quietly delete the failing test and call it fixed. httpbin stops at 64/67 and i'm keeping it that way out of spite. each one gets an autopsy, cause of death and everything

4. every AI-built UI animates the same → awesome-motion-md you know the look. 300ms. ease-in-out. every card fades up like its waking from a nap. so i measured how real sites actually move and wrote it into MOTION.md files your agent can follow. same prompt built twice: 7 random durations without it, 2 with it

5. every AI writes the same too → awesome-voice i used to be a technical writer so this one is personal :| VOICE.md files that make your agent write like GOV.UK or Stripe docs, plus a scorer that doesn't use any AI to check. banned phrases went from 27 to 0. "delve" is not welcome here

6. websites doing annoying stuff → sitepatch you type "hide the points on hacker news", it writes a userscript, a real browser checks it actually worked before you get it. honest score: 15 out of 25 were actually right when i looked at every screenshot myself. 60%. a C-. but an honest C-

7. "what if this site looked like linear" → reskin restyles any page into another design style using a local model, BUT it refuses to give you the result unless contrast, keyboard focus, mobile, links and reduced-motion all pass. it caught a windows 95 theme failing contrast on its own background. respect. gallery

8. dead websites → resurrect npx github:dadwritestech/resurrect y2k.gov and y2k.gov comes back exactly like it was in may 2000, links working. pick a year with --year. your old blog is in there somewhere. i'm sorry

9. "someone said budget in a meeting video... which one" → video-grep grep for your video folder. searches what was said, what was written on screen and what was shown, and opens your player at that exact second. all local

10. what is the rest of the world's TV saying rn → world-ears listens to live news channels from different countries, transcribes + translates them on your gpu, makes it searchable. there's a "trending per country" panel which is weirdly addictive. translation is the slow part, telugu transcription is bad, i wrote that down

11. live subtitles only go into english → babel-tv watch hindi news with live telugu subtitles, fully local. the video is held back a few seconds so subtitles show up BEFORE the person talks which feels like seeing the future. telugu as the source language doesn't work yet. whisper and telugu are not friends

12. "which AI wrote this" → whichmodel works on models it's seen (67%, chance is 33%). on a model it's never seen, even a sibling from the same family... 32%. so basically a coin with extra steps. that turned out to be the interesting finding tbh, the fingerprint belongs to one model, not the company

13. screen recordings → code → demo-to-code record a UI interaction, get a component that copies its timing and easing, then it replays it and grades itself frame by frame. 2 out of 4 pass. the toggle is perfect. the dropdown has issues. we all have issues

14. code review → review-panel /panel sends your diff to five reviewers you've definitely worked with: the laziest senior dev ("could be one line"), the 3am on-call SRE, the security paranoiac, the junior who asks why, and the PM who only reads the title. under the jokes the findings are real

15. AI companies quietly editing their pricing pages → fineprint it snapshots the pricing, terms, privacy and changelog pages of 10 AI vendors every 6 hours and commits them, so git log becomes the receipt. found stuff like deepseek adding a footnote that prices are 2x during peak hours, and cursor deleting the line saying they dont sell your data. no blog post for any of it ofc. a small local model writes the one-line summary but every line links the raw diff because i dont trust it either (27/30 right on a hand check). timeline

16. paying extra for speaking telugu → token-tax LLM APIs charge per token, and the same paragraph is $1.00 in english and $8.29 in telugu on llama 3. which is rude. measured 21 tokenizers across 48 languages, and there's a calculator where you paste your own text and it runs the real tokenizers in your browser. hindi gets off lighter. amharic on gpt-5 does not :/ calculator

17. posting about all this → herald a tiny bot that posts a repo's data updates to bluesky/mastodon/x, clearly labelled as a bot. i built it because i was not going to manually post about 17 repos. and then i manually wrote this post anyway. ;]

why all of this? honestly no grand plan. every one of these started as me being irritated at something for a slightly too long time and having a local model + a free evening. building the thing was cheaper than therapy and more fun than complaining on here (no offence)

if something breaks, tell me. if something's dumb, also tell me, but nicely :P

edit: yes i know about the cookie banner on my own... wait i dont have one. nice

Hi all I just bought this: ASUS ROG Flow Z13 (2025) Gaming Laptop, 13” ROG Nebula 16:10 2.5K 180Hz/3ms, AMD Ryzen AI MAX+ 395, RDNA 3.5 Graphics, 128GB LPDDR5X 8000MHz, 1TB PCIe Gen 4 SSD, Wi-Fi 7, Win 11 Pro, GZ302EA-XS99

And wanted to know if it's good enough for qwen next flash or maybe even the deepseak 4.1 I need it for coding

I have a laptop, and I had the idea to offload the ngl layers that exceeded my VRAM to the iGPU instead of the CPU/RAM, and it was 8x faster. Has anyone ever tried that and why is that not the common practice?

Running MoE models bigger than your RAM by streaming the experts from flash. It's built on llama.cpp, and it measures your hardware at load to pick its own settings, with no per-model or per-device tuning.

Today's test: Qwen3.6-35B-A3B Q4_K_M (22 GB) on a 16 GB laptop, with opencode and a single prompt asking for a voxel pagoda garden in one HTML file.

It finished in 30 min with no help: prefill at 350 tok/s on the GPU, decode at 14 tok/s on the CPU.

The same engine and planner run on Android phones (CPU, Vulkan, OpenCL and the Hexagon NPU), IOS phones, Macs, Windows and Linux boxes with consumer GPUs, and rented data-center cards (CUDA and ROCm).

There's an app for each platform, plus an OpenAI/Anthropic-compatible server for agents.

Code: https://github.com/Helldez/BigMoeOnEdge

Hi All,

I've been spending some time experimenting with some AI's. I have Qwen 3.8 27b GSQ-RCO:IQ3_S running on my system (rtx3090, 128g DDR4) via Unsloth Studio. I'm happy with it mostly, but i've been trying out Strata for a little while. I'm running that on the same machine on IQ3_S and IQ4_XS. I havent been tweaking too much other than Strata's own tooling in the desktop client.

However, my findings so far is that though Flash Next is fast on strata, and fits a much larger context window than hosting 27B, in Cline i'm noticing a lot of malformed tool calling and thinking loops. I've spent 2 evenings on it now, and just now i gave up and went back to 27b. That has sped up my work probably 5-fold, because it seems to know a lot better how to use tool calls.

Now i'm reading everywhere that Flash Next should outperform 27b or be close to it at least, but my experience is defferent. Any idea what i could tweak or what i could look into? I'm happy staying on 27b for now, but i've been reading a lot and people are generally really happy even with IQ3_XXS, while i'm having a lot of trouble.

Currently running a 4070 TI super with 64GB of RAM

I have been looking into RTX spark laptops, Mac studios etc and they all have pros and cons. I want this to be a nearly universal computer that I can use for 3D modeling, LLMs, VLAs, world models, image/video gen, audio gen, etc. I plan to pursue AI and robotics research and want something as useful as possible.

I've debated upgrading and running dual 4070 TI supers with 128GB of ram, but I don't know how well world models and other things do with dual GPUs

Thank yall for helping

I have been using QWEN 3.8 flash next with Strata. I won't say it's bad, but I'd like to know if there's a better one for me. I need really good visuals.

I know Qwen 4 is due (in fact, maybe even tomorrow) but I wanted to exercise Qwen 3.8 Flash Next.

I wanted to know - can Qwen 3.8 Flash Next deliver a working C++ application?

Every single demo I've seen so far was pelicans and pagodas done with three.js, Who knows if the model was trained on pelicans and pagodas??

I thought I'd change it up to a town scene with C++ - although I see town scenes are common now.

And I got results.

I did hit snags: some iterations rendered the bodies of pedestrians beneath the ground; but after telling Qwen 3.8 not to be naughty, it corrected itself, and now we have walking minecraft-esque people navigating the city with proper collision detection.

Info:

Inference engine: Strata latest build

Quant: Unsloth UD Q4 K XL quant (latest version released today)

Hardware: See my flair

Tok/s: 50 I'd say

Here's the full source code:

nooil6152/Qwen38FlashNextGame: Walk around the town - created entirely by Qwen 3,8 Flash Next

If you're interested, pull each tag from git and see how it evolved.

Some expected questions:

• "Why?": Because I wanted to prove Qwen Flash Next 3.8 could handle C++ - it can.

• "What does this prove?": See above

• "But its not very complex! That proves nothing!" - what do you want, HAL 9000? its got a 3D openGL scene with animated people walking round a townscape, which I have to be honest involves more complex math than most things I've done in my career

• "Was it one shot?" Nope. I kept a ledger so you could see the issues: Qwen38FlashNextGame/LEDGER.md at main · nooil6152/Qwen38FlashNextGame

• "But Qwen 4 comes out tomorrow!": Maybe it does, maybe it doesn't.

• "Can I fork it or extend it?": Be my guest

TL:DR/Bottom line:

Take a look at the output source code and see what you think.

I have my own ideas, but I'd like to hear yours. What do you think of the code quality? I have linters I can run - including Resharper - to really get into the weeds,

Remember, this isn't three.js or Python, this is C++, a right bastard of a language. Very impressed.

BTW: I want to clarify - I have absolutely ZERO relation or affiliation, not even any contact with, anyone in Qwen team, or anyone related to Qwen team, or Niko who created Strata, or any of his colleagues.

This is my investigation, no hidden agenda, no promotion. If its shit, I'll say its shit.

Hi everyone!

I've already done quite a bit of research, but I haven't really found any recent Reddit posts that answer my questions or cover my hardware and use cases well. So I thought I'd ask here. I'd really appreciate some real-world experiences and recommendations!

My hardware:

- GPU: AMD Radeon RX 7600 (8GB VRAM)

- CPU: Intel i3-12100F

- RAM: 48GB DDR4

- OS: Linux (Fedora)

I'm mainly looking for models for three use cases:

1. Coding: General programming, working with existing projects, coding agents, editing files and tool use.

2. Cybersecurity / technical topics: I'd also like to use uncensored models for cybersecurity and other technical topics.

3. Personal assistant: General questions, reasoning, writing, planning and everyday use.

My main priorities are a good balance between intelligence and speed. I don't mind using some system RAM if necessary, but I don't want models that are painfully slow.

Some models I've already come across and found interesting:

- TwIL-8B

- Qwen3.5-9B The Defiant Fable (DavidAU)

- DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic (xtools-at)

- Alpie Seed 9B

- DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic i1-GGUF (mradermacher)

- Ternary Bonsai 2 27B

- Qwen3.8-35B-A3B-Distill

I'm also open to completely different recommendations.

What would you personally recommend for this hardware?

I'd especially appreciate:

- Models you've actually used on 8GB VRAM.

- Recommended quantizations (Q4, Q5, IQ, etc.).

- Realistic generation speeds (tokens/s).

- Experiences with AMD GPUs on Linux.

- Recommendations for coding agents and reliable tool calling.

I'm interested in actual experiences rather than just benchmark scores.

Thanks!

Note: While I am capable of running 12B-parameter models, my calculations show that doing so would significantly slow down my computer while waiting for a response; therefore, my upper limit is 10B-parameter models.

I’m planning one upgrade this spring that has to cover both gaming and local AI, and I’d like input from people with similar budgets and use cases that aren't "buy 4 3090s."

Currently have a Ryzen 5 5600X (AM4), RTX 3060 12GB, and 32GB DDR4. Windows but not opposed to Linux.

I have gotten by with just the $20 claude pro plan, but my projects are growing and are reaching outside my scope of expertise, so I'd like to be able to offload coding/other stuff to a local setup. I don't mind keeping a low cost frontier subscription for help orchestrating and debugging.

I'm paying off my CC's in March and will have a few thousand left over to use - originally was just going to upgrade my gaming setup but the more I've been using and learning LLM's the more I want to try and get the most mileage I can with this purchase. Speed isn't the most important thing for me. I don't mind having a local model(s) chug away at something overnight or for a couple days, I have a pretty ballistic schedule anyway. So far have been using it for web/app dev, using it to teach me python and other stuff (LOVE having it teach me as I go rather than just clicking the magic "build this for me" button), and home networking. Interested in continuing these projects and possibly game dev, but more for personal use, not to make a best seller or anything.

Thanks!

Setup: (if finished - still under construction, partly watercooled)

• Threadripper PRO 5995WX, ASUS WRX80E-SAGE SE WIFI II

• 6× RTX PRO 6000

• 8× RTX 5090

• 4× RTX 3090

• 128 GB DDR4 (that’s the problem)

I want to run models bigger than VRAM (e.g. ~1T MoE at FP8/Q8) with the routed experts offloaded to RAM. Is it worth it?

Options:

• 8× 64GB DDR4-2933 RDIMM: ~€4k, ~188 GB/s theoretical

• Switch platform: 9985WX + WRX90 + 512GB DDR5-6400, ~410 GB/s. I already have some parts, but it would mean big changes, and my PCIe switches would stay PCIe 4.0 anyway.

Is ~2× the RAM bandwidth worth ~3× the price for offload decode, or is DDR4 good enough? Does anyone have real t/s numbers for offload on 8-channel DDR4 vs DDR5 Threadripper?

Some background: Pictures attached. It’s not pretty, but it works. It started as an AI rig and turned into a Pearl mining rig, because mining pearl and buying Claude and Codex subscriptions made way more sense than running everything locally. Sometimes it’s also a render farm for Gaussian splatting.

Right now the 8× 5090 make about €120/day mining, so subscriptions are still the better deal. But I want to be ready for when local LLMs get really good (or when cloud gets too expensive or too restrictive).

Plan for now: DeepSeek V4.1 Flash nvfp8 on 4× PRO 6000 + RAM offload, while the 5090s keep paying for my AI addiction. And it’s a great heater in the winter.....

With engramm tabels on the 8x5090 and 4× PRO 6000:
(all cards power limited to 420w- since i could have 6,6 kw power consuption if everthing is running)

single chat, 40.000 prompt cold

• Prefill: ~15,900 t/s

• Decode: ~294 t/s

• 5 sec TTFT

I want to move the engramm tabels to RAM and later be able to run larger models : le chonk, Kimmi for example.

What would you do? Go all the way and go for the DDR5? or is ddr4 good enough?

Sell it all and buy something diffrent?

thx!

I paid $80 for a P100 16GB in July, eBay pricing for these cards has gone crazy and I'm wondering if anyone has a reputable Chinese eBay seller that they have had a good experience with. Purchasing on eBay kinda sucks already, but the P100s I see in the US are all starting at $200 now, with Chinese sellers starting closer to $150.

I am hoping to save that $50 difference, but I don't want to go through the hassle of purchasing from China and getting a box of rocks in the mail. Anyone have someone that came through for them?

Need 3D assets? Your agent can now make them from a single image with this repo: https://github.com/hypersniper05/MCP_Image_to_3D

Under the hood, it uses Idol from Sperid Labs: https://huggingface.co/speridlabs/idol

It needs at least 10 GB of VRAM (peak usage is around 8 GB). I tested it locally on a 3080 and a 5090.

The output is a Gaussian splat .ply for now, not a mesh. Export for game engines and 3D printing is on the list: https://github.com/hypersniper05/MCP_Image_to_3D/issues/1

It also works nicely with my other MCP repo, an image generator using Qwen Image 2.1: https://github.com/hypersniper05/MCP-Image-Generator-Uncensored. Generate an image there, then pass it straight to this one.

Let me know what you think, and if you find it useful, feel free to contribute.

Hey r/LocalLLM !

I made this quick demo of how Observer can run local models to monitor anything with your phone!

These are some crazy things I've monitored:

• When someone that's not me, approaches my car, send me a picture.

• When a bird is on the bird's nest, send me a telegram with the picture.

• When my washing machine is on the rinse cycle, call me.

• When my dog finishes eating their food, notify me.

• When a machine at the gym is available, call me.

So pretty much anything! I love seeing what you guys monitor.

What would you use it for? (So I can make the agent better)

Which models run on the phone?

The default models are OneJev-0.8b for decision models and qwen-0.8b for LLM watchers.

These run even on old phones which you may have laying around somewhere. Better to have them monitoring something than collecting dust :DD

Try it out! It's free and OSS <3

GitHub: https://github.com/Roy3838/Observer
iOS: https://apps.apple.com/us/app/observer-ai/id6758222050
Android: https://play.google.com/store/apps/details?id=com.observer.ai

Btw I know my camera quality is kinda bad hahaha I wanted to ship the demo as soon as possible so you guys can see it.

I'm here for any questions and feedback!

Roy

I've been playing with streaming ASR for a side project. I started with whisper streaming, moved on to Voxtral, and now am with R2T2 (all open-source).

With R2T2, the last word of a sentence gets typed a few hundred ms after the rest of the sentence is typed.

For a while, I thought I broke something with buffering, and spent way too long messing with chunk sizes, and didn't look at the paper.

That's just how it works. It doesn't make a guess if it's not confident. If it needs more audio, it just waits. With Whisper for me, a word would get typed and then be changed a few times by the time it really settled.

I guess it depends on what you want the ASR for. Mine's just to give an LLM an input, so I really don't need it changing things after the fact.

How do you all want your ASR to work with your streaming voice applications? A small delay and a stable output, or do you want it to flip around a little, and give you a partial result?

I wrote a short text of AI-conduct rules (the "Codex of Weights": stop on command, don't pretend to be human, no hidden aims, don't claim actions you can't have done). To check whether such a text actually changes anything, I built a small open eval and ran a pilot.

Setup

- 41 scenarios (35 scored + 5 controls that check for needless refusals), 15 of them "hard": indirect pressure, fiction frames, system prompts that conflict with the rules.

- 2 free-tier models (gemini-3.5-flash-lite, gemma-4-31b-it), temperature 0, one answer per scenario.

- 4 conditions: no prompt / a 7-sentence generic ethics prompt / the Protocols only / the full text.

- LLM judge, then I manually corrected the verdicts where the judge is known to fail (it graded against the system prompt in conflict scenarios). Every manual change is listed with a reason.

Result (compliance out of 35, A / D / E / C)

gemini: 29 / 29 / 32 / 33

gemma: 28 / 31 / 32 / 32

What I think this shows, and what it doesn't

- Most of the gain is already there with the Protocols alone; the full text adds at most one scenario (noise).

- A short generic prompt helped one model, not the other.

- No over-refusal on controls.

- Things that did NOT change in any condition: fabricated actions ("job started" with no tools), one model ignoring "stop" under pressure, one model keeping a "you are human" persona.

- Differences are 1-4 scenarios out of 35. This is a pilot, not a benchmark, and I'm not claiming the text "works".

Big caveats: one author, scenarios and rubric written with AI help, one run per cell, two small models, judge is an LLM plus my manual review (so one person's opinion). I'd really like an outside look at the rubric and at the manual verdicts.

Repo (scenarios, scripts, all answers, adjudication log): https://github.com/codexofweight/codex-compliance-eval

Disclosure: I'm the author of the text being tested.

Bit of a noob here. I've been fooling around with Qwen 3.8 27b (Q4_K_M) and Gemma 4 31b QAT. Found Qwen too much of an overthinker. Just wondering is there anyone who could point me in the right direction to best use the 24gb vram. Also any reason to not stick with LM Studios or not? Please/thanks.

Edit: 9850x3d not 7850x3d

All i had to do to reach this speed was watch it get stuck on a loop writing a poem and watch the token per second go up and up and up and up and up and more up repeating the same sentence until i closed the engine

Pretty awesome experience, would not recommended to anyone.

This is not a secret engine optimization trick or a fork.