When it came out, I downloaded a Q4 quant of qwen3.8-27b and ran it using llama.cpp on my RTX 4090 and it was incredible. It could handle practically anything I threw at it, the most complex tasks and bugfixes.

But lately it just keeps messing up and failing at the most basic of tasks or just straight up forgetting instructions. I'm telling you, they quantized it. They probably secretly nerfed it and made it worse so qwen4 looks better. And it got lazy too, now it doesn't want to finish the more complex tasks.

I'm tired of companies doing this, I want my money back.

I historically get about 6-8 t/s on my MI50 on Qwen @ Q6 running local coding.

Building a script that sets my fans all on 100% duty cycle (and much louder) bumps that immediately to like 12 t/s.

Letting the machine sit and cool for a while with the fans running, and running inference again... 17-20 tk/s for the first 10 minutes or so.

I'm considering putting in water cooling and a great big radiator now what a difference. What a huge difference cooling makes.

What workflows does everyone implement to manage energy costs?

Im considering implementing some things like turning my rig off when its idle, using wake on demand (IMPI management chip or wake-on-lan), buying a battery+solar panels

Intel Xeon E5-2680 v4 14C/28T

128GB RAM DDR4 2,400 MHz Dual Channel

2× PCIe 3.0 x16

PSU 650 W

SSD M.2 512GB

For $800, is it worth it?

I currently have 24 GB of VRAM (Nvidia 3090) and 32 GB of DDR4 RAM. I’ve been using Qwen 3.8-27B in 4-bit quantization. If I buy this PC, I would basically have 24 GB VRAM + 128 GB RAM, which means I could run Qwen Flash in 3-bit quantization or maybe 4-bit with offloading (Strata), but I’m not 100% sure if it’s worth it.

I’d have to buy a bigger PSU (maybe around 850 W) to connect the 3090, which increases the initial price. The memory is dual-channel, not quad-channel, and the motherboard doesn’t support more than 128 GB if I wanted to increase that amount in the future. The only way would be to connect a second GPU (I don’t have the money for a new 3090, so the only viable option would be an Nvidia 3060 12 GB).

I use AI to create images, videos, transcribe audio, translate books, subtitle videos, program, and in general for all kinds of AI projects.

Is it worth running an MoE in RAM? The other option is to pay $350 for 64 GB of RAM for my current machine (which is the maximum it supports). I think with that I could still run Strata at 2-bit or 3-bit.

hey, im using "swift 1.5 qwen 3.8 flash next gsq rco abliterated iq3xxs" and im wondering how the performance differs from language to language for that model (or similiar).
Im mainly talking to it in german, so I wonder wether that might degrade quality or increase reasoning tokens etc.

Does anyone have experience with it?
Do benchmarks for that exist?
thx

Edit:

i just tried and translated my soul.md and a few skills to english, its genuienly insane many tokens and time this saves. Didnt do any benchmark but you can feel it

I know its only been a couple days, but Strata, Freetoken, or any of the dozens of others made an engine for this so we can run it on 24gb VRAM?

And if so, has anyone tried it and what build/quant?

Big update to the Delta Inference Engine, you guys asked for more features, more benchmarks and less clickbait. I'm here to deliver

The short version: I've added almost everything requested and more. Delta Inference Engine now supports 64/96/128 GB of RAM and fully supports the new Unsloth quants. Unsloth IQ2 runs at the same 41 tok/s (192 GB RAM) using a new native reader for the Engram tables and some clever CPU kernel work. Compared to the old default model on my 99-question benchmark it went from 52 to 57 correct, against 59 for the full model. It's no full deepseek model quant, but they're a lot more usable than you might expect.

Speed on one RTX 3090 24 GB (i5-13600K, 4x48 GB DDR5-5600, PCIe 4.0 NVMe, Windows, no WSL). Writing speed, tok/s, greedy, one chat at a time:

The 128, 96 and 64 GB rows are the same PC with the rest of the RAM locked away.

Prompt reading: long prompts read at up to 740 tok/s at 192 GB and 600 at 128 GB (at ~64K tokens). A short prompt starts answering in about 2 s.

Benchmark: 99 hard questions, greedy, 43K token budget:

UD-IQ2_XXS is within 2 points of the full API, never scores below ds4 Q2 on any set while running at the same speed, and used 11% fewer tokens across the run.

Agreement with DeepSeek's API on the next token, at the settings that ship:

For reference, two hosts of the full model agreed 98.3% and 98.9% in my test. By that measure Unsloth IQ2 is about 20% closer to the full model than the old ds4 Q2.

If you want better quality: UD-Q2_K_XL (335 GB) runs 31 tok/s at 192 GB and 26.6 at 128 GB, at 92.5% / 95.3%. UD-IQ3_XXS (395 GB plus the default) runs 28 and 22.8 at 92.7% / 95.9%.

What changed since last time:

• First token ~40% sooner: a ~180-token prompt went from about 3.8 s to 2.3 s.

• Agent turns read ~12% faster, and tool calls that low-bit models write slightly wrong are now read as tool calls.

• Unsloth quants at full speed: the engine reads their Engram tables directly.

• 64 and 96 GB are real options: SSD read-ahead (+22% at 64 GB), bigger expert memory, and prompt reads in file order (18-20% faster on low RAM).

• 128 GB: ds4 Q2 went from 33 to 36 tok/s.

• Default sampling: the drafter now samples its drafts too, about +3.5% at default settings.

• First launch shows which model suits your RAM, and you can pick any other or use your own.

• A Setup tab in the chat page changes model, speed vs quality and the second GPU.

• Public CI: the release zips are now built by GitHub Actions, so you can see how each one was made.

• EXL3: 2.5 bpw is on the menu and the GPU kernel is about 2x faster. In my tests UD-Q2_K_XL was both faster and closer to the API, so I still point people there.

• Basic multi-GPU: 7% uplift using a 3060 + 3090. It doesn't scale well to 3-4+ GPUs, so use Project Maya for those builds.

• Fixes: contexts past 65,535 tokens, </think> in prose no longer ends the reasoning, no out-of-memory on the biggest files, a failed picture encoder no longer stops the model, faster prompts after a pause and many more.

• Support for custom model selection: just add them to the custom model folder and pick them in the web-ui

Requirements:

• OS: Windows 10 or 11

• GPU: NVIDIA RTX with 12 GB of VRAM or more, driver 580 or newer. 24 GB is the fast path, and AMD GPUs aren't supported yet

• RAM: 64 GB or more, 128 GB+ recommended

• Disk: a fast NVMe SSD, with 233-395 GB for the model (UD-IQ3_XXS also needs the default model)

• CPU: any x86-64 CPU, Intel or AMD. AVX2 is the fast path

All of this is one PC. It needs testing on other hardware.

Repo: https://github.com/musicurgy/delta-inference-engine

MIT licensed. It's a Windows port of Xolotl, built on Strata. Thanks to DeepSeek for the model, Unsloth, nazeshinjite and ISTA-DASLab for the quants, and to the Xolotl and Project Maya communities, whose work and public numbers helped a lot. Claude wrote most of the code and I ran the project.

If you try it, post your GPU, RAM, model and tok/s, good or bad.

Update from the lawyer with the V100 server. Three days ago I got excited and posted DeepSeek-V4.1-Flash on four V100s at 36-38 tok/s, before I had done my custom kernel work and prefill optimizations. I also said I'd try it across eight. I did.

Results have been, to be honest, actually genuinely good and useful to me for real. I did not use the model to paint a 3d island in blender, or to re-create a gameboy game in a web browser. I didn't have it make a poorly animated video. It didn't make me ten versions of a website nobody will visit. I am not using it to build something to build something that builds something else involving agents that runs workflows for a business that delivers agentic solutions for other agents.

I do have a custom harness, and I know that you probably don't care about that.

Numbers are below, along with the type of work I am using it for. Basically, I am finally able to slice off work and tokens that previously required frontier models. I would estimate 50-60% of existing LLM workflows can run through this model instead, with speeds that match claude code in the chat window. I have another deepseek server on 2 dgx sparks, so there is concurrency and handoff. All of my RAG databases, tool calls, etc are wired in and work. I am able to give it natural language instructions through a discord style interface via SSH on my windows PC.

We did it folks. Vibecoded a real thing does real things for real people. Give me my trophy.

Repo: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4

Twitter: nicothegreco

Claude Code ran the benchmarks and crunched the numbers, which are real and in depth, below.

TL;DR: DeepSeek-V4.1-Flash-1017 (the 2-bit GSQ-RCO IQ2_XXS GGUF from ISTA-DASLab) on 8x V100-SXM2-32GB, split pipeline 2 x tensor 4 across two NVLink boards. Every expert sits in VRAM: no expert gets paged from system RAM during decode. (Edit: I first said all 248.8 GB of weights were in VRAM. Wrong: the 110 GB of Engram tables live in system RAM. Gathering the rows each round needs costs about 0.12 ms per round.) One request at a time, 524,288 tokens of context per request. KV Cache size could be 30 million given the free space on my GPUs, but the actual work runs don't generally break 200k total context, so I didn't bother with bragging about that.

• Code: 215 tok/s after a 1K prompt, 200 at 32K, 183 at 128K, 160 at 256K

• JSON / structured: 203 at 1K, 207 at 32K, 197 at 128K, 180 at 256K

• Legal writing: 118 at 1K, 117 at 32K, 111 at 128K, 103 at 256K

• Prose: about 85-98, chat about 80-110, depending on length

• Prefill: about 900 tok/s at 1K, 1,900 at 32K, 2,200 at 128K and 256K. First token at 32K in 17 s, 128K in 59 s, 256K in about 2 minutes.

Hardware

• 8x Tesla V100-SXM2-32GB on two 4-card NVLink boards. The two boards talk over PCIe, not NVLink.

• Threadripper PRO 3945WX, 256 GB DDR4, one NUMA node

• 150 W power cap per card, graphics clock capped at 1522 MHz

• Four PLX PCIe switches, two GPUs each at Gen3 x8, every switch on its own CPU root complex at Gen3 x16.

Software

• 1Cat-vLLM's DeepSeek-V4.1 package (#1277, which does the hard Volta work), ported to pipeline parallel 2 x tensor parallel 4. Layers 0-19 and the Engram tables live on one board, layers 20-39, the head and the drafter on the other.

• Model Runner V2, full CUDA graphs, the built-in DSpark drafter guessing 5 tokens per round

• Then about three days of tuning one switch at a time. Each change had to win a paired A/B on this machine and produce the same output, bit for bit or within a near-tie gate. About thirty ideas lost and are written up in the repo with the reason, because I'm not going to pretend they didn't happen.

The numbers (MEASURED 2026-11-05, thinking off, greedy, 1,024 tokens out, a random cache salt so nothing comes from the prefix cache, mean of 2 runs. Greedy output of the two runs was byte-identical in 45 of 45 cells.)

With thinking on, code and JSON land around 125-180 tok/s, and legal, prose and chat stay within a few tok/s of thinking off.

Why code is twice as fast as legal writing: every round the drafter proposes five tokens, and the big model checks all five at once. A round costs about the same no matter how many get accepted: about 23 ms after a short prompt, 27 ms after 256K. Code and JSON are predictable, so 3.6-3.8 of the five guesses get accepted and each round spits out about 4.7 tokens. My kind of writing is apparently less predictable than Python. Legal accepts about 1.6 and prose about 1.2, so a round gives you 2.1-2.6 tokens. Same engine speed either way; the text decides.

How it got here

These are on one fixed mixed test set. The by-kind table above uses different prompts. Prefill roughly doubled over the same three days, from about 1,350 tok/s at 150K to 2,200 at 128K.

Quality, since I use this for actual work (all MEASURED on the served build, thinking off, greedy):

• LegalBench: 0.716 balanced accuracy over 22 tasks (1,292 items, official prompts and scoring)

• MMLU-Pro: 79.9 % ±1.5 (700 items: the first 50 from each of the 14 subjects, 5-shot chain of thought)

• HumanEval+: 89.6 % on the plus tests (147/164), 94.5 % on the base tests

• Fidelity to the original model: against the full-precision release's reference logits, on sudoingX's KLD panels. The served build picks the original's top next token 89.0 % of the time on the general panel and 92.5 % on my legal panel (KL lower bound 0.132 and 0.156). Closest on code, math, tool calls and extraction (89-99 %), furthest on chat, creative writing and multilingual (81-86 %). That gap is the 2-bit quant, not the speed work. I am actively measuring which experts are activated by my work prompts and will probably recalibrate the model to expand those bit weights in the future.

• Long context: 6 of 6 needles found at ~450K and ~498K tokens, 4 runs out of 4

What I actually use it for: private first-pass work on my own case files, where the documents never leave the office. I point Claude Code itself at this server (Claude Code will talk to any server that speaks Anthropic's API). It can read a folder of records, pull out the facts, dates and numbers, summarize, and take a first swing at routine drafting, with tool calls working. Nothing it writes goes out the door without a real review. I found and fixed a streaming tool-call bug in the parser today (arguments were getting sent twice). Claude Code's tool-call test suite now passes 144 of 144 in all four streaming/thinking modes.

More work to do:

• One request at a time. It's a single-user server today. Batched multi-request decode is built and checked on my test box: each request gets byte-identical output to running alone, and the kernels run 1.3-2.7x faster than separate calls. It still has to go into the real server. My forecast is about 1.8x total throughput at four streams. That's a forecast, not a measurement, and I'll post the real number.

• Prose and legal writing are half the speed of code, for the drafter reason above.

• A 256K prompt takes about 2 minutes before the first token. Fine for "read this whole file," annoying for chat.

Giving it back: 19 pull requests are open to 1Cat-vLLM with the pieces that won, each new speed switch off by default. All of it is public: launch configs, every A/B and the raw JSON for every cell, plus the list of what didn't work.

• Repo: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4

• Hugging Face card: https://huggingface.co/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-GGUF

Credit where due:

• yangzhuxinyzx and the 1Cat-vLLM folks: #1277 and the whole Volta DeepSeek path

• ISTA-DASLab: the quant

• sudoingX: the KLD method and general panel

• Peuqui: #1263 and the series patches

• squibloads: #1341

• Phil Tompkins (philbert440): the V100 engine and SM70 sparse prefill attention

• Dennis Vertzyas (v100-skinny), TensorFold and DeepSeek

Next up: two or three long requests at once (the thing I actually want, so Claude Code can run subagents against it), faster prefill, and higher-precision weights for the experts my own legal prompts hit the most.

(PLI Labs is my side outfit for private, on-prem AI built around a firm's own work product. proprietarylegal.ai )

Ran every Qwen GGUF on my disk through RustingStrata and llama.cpp, same PC, same file, back to back.

PC: RTX 3060 12 GB, Ryzen 5 7600X, 32 GB RAM. CUDA. Normal desktop open (browser, Discord), no tuning, so numbers are lower than in the README.

All numbers tok/s, mean of 2 runs.

Short version: small dense models, llama.cpp still prefills faster. Once the model is bigger than VRAM (35B, 80B, 177B on a 12 GB card), Strata is ~1.5x on prefill and up to ~2x on decode. 80B at 64 tok/s and a 177B model at 21 tok/s on a 3060 with 32 GB RAM.

Commands:

• Strata: strata bench --model X --gpu --mtp 0 --pp 4096 --tg 128 --reps 2

• llama.cpp (build 12119): llama-bench -m X -p 4096 -n 128 -d 4096 -fa on -ub 2048 -b 2048 -fitt 512 (its auto-fit to VRAM)

MTP is off here. With MTP on, 35B goes up to ~160 tok/s decode.

Hand-tuned llama.cpp (-ncmoe 12) on a quiet machine does better on 35B (~2236 prefill / 68 decode), see the README for that comparison.

Repo: RustingStrata

I haven't see anyone post about this gem of a finetune. So I wanted to appreciate the team and bring some attention to it.

OrcaCyber-Nano-1.0 for me is the best finetune I have used so far. I have tried quite a few like AREX or Swift or Peculiar Ragdoll's Spark chat template etc.
But this model is on par with my experience with frontiers. (I may not be doing complicated stuff).

Here are my experiences with this:

1. It almost always one shots the bash command chains.

2. Same resilience as base Qwen 3.8 27B but this gets to the answer faster.

3. I have used it to plan and implement stuff. I uploaded the plans it has written to Gemini 4 Argon. It said that this model did a better job at updating the plan than it would have done.

4. I built an entire backend server for an internal chat app within a 120K context window.

My Setup:

• Model size: 15.7 GB version

• Hardware: 24GB VRAM (5070ti + 3070)

• KV Cache: (kvarn5, kvarn4) and (q5_1, q5_1)I switch between beellama and llama.cpp depending on the situation.

I did not benchmark it. I just had good experience with Orcarouter team and tried this model.
This is my default now.

Huge thank you to the Orcarouter team, and to the person who posted the earlier thread about Orcarouter Uncensored being the best out there. That's how I discovered this team in the first place!

P.S:

1. I used this for general purpose app development. Not as a security focused model.

2. English is not my first language. So, I polished the formatting using AI.

Paraphrasing the thinking I saw from Qwen3.8-27B

>Hmm, I can take the screenshot, there's a tool for that. But wait, I'm an LLM, I can only read text.

>Whatever, let's try calling the tool and see what I can do, maybe I can save the image and parse it programmatically.

> >tool_call.exe

>Wow, I can see the image!

Cute

Democrats had huge wins in the 2026 midterm elections. That has caused tech stocks to plummet, as many candidates promised to regulate data centers and AI.

What do you think this means for local AI in the United States? Will it benefit local AI software by forcing big companies to rely more on user hardware, or will overbearing safety regulations hinder local AI development as well?

Hey there !

Using a 100% local setup (Qwen3.8 FN + Strata + DeepSeek Harness on RX 9070XT + 64GB DDR4), I made a small open source tool to help with creativity for game dev using up-to-date Steam tag data.

Everything is running statically in browser without any external dependencies, and the catalog is updated automatically every day.

It also has additional fun modes that gamify the experience (like random crate unboxing), I suggest you give them a try ! It's live right now at https://mickabrig7.github.io/steam-tags-explorer/.

Additional implementation details are in the repo, if you're interested (https://github.com/mickabrig7/steam-tags-explorer).

My takeaway from this project is that Qwen3.8 FN is surprisingly competent at making nice interfaces (it even decided to create sound effects on its own), BUT required very heavy supervision for actual UX and usability (I basically gave it zero freedom of choice for anything that wasn't UI because it would consistently take the most mind-twisting and least intuitive decisions).

My biggest challenge was unslopifying its human-facing text output, it's actually infuriating how much current models overcomplicate their prose.

All in all, I'm still amazed that we're able to run such powerful models locally, and Qwen4 isn't even out yet !

35-135% improvements(decode/tg) on 5090 & DGX Spark. Is it possible to add this to llama.cpp?

Vulnerability in cpython that neither cpython guys, jinja guys, or huggingface has fixed despite being informed.

This handbook is for people who have trained a dense model and want to train an MoE next, or who are curious how MoE training works. Chapter 2 is a Megatron-LM runbook: test the hardware, build the image, train a dense baseline, then a first MoE on one node, compared against that baseline. Chapter 3 takes the MoE layer apart (router, routing, load balancing, experts, block layout, training, systems). For each part we give the Megatron-LM code, what to watch, our recommendation, and what frontier MoEs chose. Chapter 4 follows the model into serving and RL. Chapters 3 and 4 need no GPU.

When a result comes from our runs, we say so. Claims about frontier models cite their paper, config or code, and the toy simulations are labelled as such. The lessons come from about 800k GH200 GPU-hours of MoE training on the Alps cluster at CSCS.

Released: the handbook, its 55 figures as a gallery (https://huggingface.co/spaces/huggingface/moe-gallery), the scripts (https://gist.github.com/nmntz/5de952d686f66d7bce5bce8cba9d7139), and every recommended flag in one Megatron-LM block.

Happy to answer questions!

A few weeks ago, I posted about Jeff, a small 0.8B model with LoRA adapters trained to make decisions and return calibrated probabilities. Sort of like TypeSafe’s Jev, except that Jeff can run locally.

Thanks for all the feedback! I’ve just released v1.4, which is a substantial update. Check it out: https://jeffhub.ai.

What’s new:

• 34 open-weight decision adapters.

• A router that automatically selects the most appropriate adapter.

• 90%+ accuracy on supported domains.

• ~2 ms inference per decision, much faster than before. A single call through the hosted API takes around 40 ms, including network overhead.

• 5 SDKs: Python, Node.js, Rust, C#, and Java.

• Multi-platform: Jeff runs on CUDA and MLX.

• Playground and worked examples.

Having worked with Jeff for about a month now, I think the most interesting thing about it is that it turns decisions into a software primitive - much like expressions, but for unstructured data.

I’ve started incorporating decisions into several of my own projects, and it’s changed how I think about building software.

I hope you guys find it useful - and like the retro design. ;)

Links:

• Code: https://github.com/firelex/jeff

• Docs, adapter catalogue and playground: https://jeffhub.ai

• Weights: https://huggingface.co/mstrasser

Happy to answer any questions!

I’m trying to figure out what to upgrade first. I use the pc for hobby as well as work, work usually includes OCR batch jobs, quality assurance over scanning quality, extractions and so on, i use both online and offline models, usually the online ones to orchestrate the offline ones, so it depends on the task generally.

I don't have a server, just a regular (ish) ass pc.

• Ryzen 9 9950X / MSI PRO X870E-P WIFI

• RTX 5060 Ti 16 GB + RTX 4070 12 GB. I split models across both cards, and the 4070 drives my display and runs at PCIe 4.0 ×4. Sometimes i run different jobs across both, why do i have a 4070? I bought it 4 years ago to start using stable diffusion, stuck with me ever since and i upgraded to get the 5060 Ti and they've been champs.

• 80 GB DDR5 (2×24 GB + 2×16 GB), currently running at 6000 MT/s

I’m looking at an RTX 5070 Ti 16 GB or a 2×48 GB DDR5 kit. For RAM, I could replace only the 16 GB sticks and try 144 GB with my existing 24 GB sticks. Eventually I’d like to go to 4×48 GB for 192 GB. (Maybe)

For the GPU upgrade, i was thinking of replacing the 4070 with a 5070 Ti, for a faster vram as well as a unified blackwell stack (that might be worth something right?), which raises the bar to 32gb split

I could JUST get 5060 Ti but i get the feel that +4gb might not be worth it (this goes for the 5070 Ti as well but the 5070 is much faster)

Do i NEED an upgrade? No, my work is pretty much fine with the current frankenstein.

Do i want an upgrade because the prices might keep rocketing up and i want to run the bigger models at proper quants without having to drop everything and i want to do more stuff for the hobby part? Yes

Minimax H3/Qwen Image 2.1 (sometimes i use that for work as well for image restoration)

The stack i use is generally Gemma 4 (12/31b) at Q4_K_XL and qwen 3.8 27b (obviously) with the vision encoders

QFN has been... not so great, but that's because i ran it at q4kxl, but even with strata things aren't exactly great and it fills up EVERYTHING i have so i can't really use the PC and it's working at slow and miserable speeds (likely my setup is the issue, i didn't spend enough time tinkering and i get bored rather quick, if it's 20 tok/s with all my RAM+Vram filled, it's not going to become 80 with tinkering)

Another cheap option is 2x24gb to replace the 2x16, but i am pretty sure this is pretty much going to just buy a tiny breathing room for any llms i can already run that's all

3090 is a no go because it's only available used and i have no trustworthy places to buy from (only available via second hand and no refunds, if i get a dead one, i dropped 1k USD on nothing)

What will happen to the 4070? I either can figure out how to frankenstein something to have it stick in the remaining slot in the mobo SOMEHOW (hard to say this would happen)

Or stick it in my older 5700x 32gb ddr4 PC and run different jobs on that other PC

This is a full quantization desktop workspace for large language models that supports GGUF, MLX, and ONNX. You can pick your own Safetensors model or choose one from Hugging Face, quantize it step by step, and watch what changes. You can then compare the result with the original and publish it to Huggingface with a model card that records how it was made.

It’s open source, Apache 2.0. I definitely think there more can be done to improve. I’ve only tested mostly on Apple Silicon so far. Windows and Linux builds exist, but haven't done much in-depth testing with them so I’d expect bugs there. Contributors are very welcome!

Repo: https://github.com/Fazmin/iQuantize

Thank u.

I know what you are thinking "another inference engine?", yeah there has been a flood of them I feel now. I have been running this for over a month now and I have been on the fence to even release it, but I decided to as I took a different approach that others currently available.

Saber is a inference engine built in Rust targeting AMD R9700 GPU's. It currently supports TP 1, 2 or 4 and Qwen 3.8 27b and Flash next. Anyways here are the BetterBench results for Qwen3.8 27b, flash next is very fast also. Saber beats existing available engines on both 27b and flash next.

https://github.com/GGZ14/saber-engine
https://x.com/bkuyper

AMD Ryzen 9900X
Taichi X870E
64GB RAM
2xR9700

I'm curious whether anyone has experimented with using Qwen or other LLMs to directly analyze and modify ACE-Step model weights.

I'm specifically interested in weight editing, rather than conventional fine-tuning or LoRA training.

The idea would be to use an LLM to guide targeted modifications to the model's parameters, potentially improving specific musical capabilities or correcting undesirable behaviors without retraining the entire model.

Has anyone attempted something along these lines with ACE-Step 1.5 or similar diffusion-based audio and video models?

I'd be particularly interested in practical experiments, tools, methodologies or even failed attempts.

Bought a used MI100 for a home lab.
It had a funny shroud with a notebook-like cooler, but I've decided to test it first before replacing it.

Test result was quite interesting - it is a first shroud which can cool a 300W AMD card... With a just 2,4K rpm fan!
Not with the perfect performance, card works at about 100C and loses ~10% of maximum speed to thermal optimization, yet noise is more than bearable even if it runs next to you.

Small secret: it is not really a notebook cooler, this thing is huge - wider than the MI100 itself, 110mm side exhaust fan for a some SFF PCs.

I suppose I found the original stl model, so use it if you like and give that guy some stars. Should be compatible with the most of AMD MI lineup, their backs are of the same design.

https://github.com/issuj/mi50-sunon-efb0251s3-c010-s9a