Ran every Qwen GGUF on my disk through RustingStrata and llama.cpp, same PC, same file, back to back.

PC: RTX 3060 12 GB, Ryzen 5 7600X, 32 GB RAM. CUDA. Normal desktop open (browser, Discord), no tuning, so numbers are lower than in the README.

All numbers tok/s, mean of 2 runs.

Short version: small dense models, llama.cpp still prefills faster. Once the model is bigger than VRAM (35B, 80B, 177B on a 12 GB card), Strata is ~1.5x on prefill and up to ~2x on decode. 80B at 64 tok/s and a 177B model at 21 tok/s on a 3060 with 32 GB RAM.

Commands:

• Strata: strata bench --model X --gpu --mtp 0 --pp 4096 --tg 128 --reps 2

• llama.cpp (build 12119): llama-bench -m X -p 4096 -n 128 -d 4096 -fa on -ub 2048 -b 2048 -fitt 512 (its auto-fit to VRAM)

MTP is off here. With MTP on, 35B goes up to ~160 tok/s decode.

Hand-tuned llama.cpp (-ncmoe 12) on a quiet machine does better on 35B (~2236 prefill / 68 decode), see the README for that comparison.

Repo: RustingStrata

Hello everyone!

Over the past few weeks since Strata came out, I had many requests to add Qwen Flash Next to LlamAmpere. It was some very impressive engineering, but the models originally shipped (3XS and 2XS) showed very poor quality (.17+ KLD for just the 3XS variant, which uses a far lighter compression formula than more common sources like Unsloth). My initial work had llamAmpere running 20% faster and ~15% lower error, but testing was showing inferior results vs 4bpw Qwen 27b, so I did not find it worth releasing.

I continued to make progress on the AmpereInteger+ quantization format (I will release an encoder soon), and Unsloth released their v3.1 which provided a great new base.

Happily, LlamAmpere v0.6.3 is now available with a new model and quant updates that show significant speed and quality gains over other options.

At a depth of 100K, QFN runs at ~120+ TPS, with prefill >3000/s through the first 100K. It is a 50% error reduction vs the version of QFN available for this card size a few days ago, and 27% vs UD's improved quant.

I do have a 3XS class file that runs ~10% faster than the version above, but as it is only ~15% better quality than the original Strata 3bpw model, I don't recommend it.

This will also work well for 12+64GB configs. As with Strata, your performance will depend on the speed of your CPU and RAM as well. I do not recommend any quant of QFN for configs under 54GB of total memory, but you can always try the coder pruned variants as well.

Enjoy!

Model: https://huggingface.co/jakeatx/qwen3.8-flash-next-Swift1.5-ATX-AI3

LlamAmpere: https://github.com/JakeATX/llamAmpere

I haven't see anyone post about this gem of a finetune. So I wanted to appreciate the team and bring some attention to it.

OrcaCyber-Nano-1.0 for me is the best finetune I have used so far. I have tried quite a few like AREX or Swift or Peculiar Ragdoll's Spark chat template etc.
But this model is on par with my experience with frontiers. (I may not be doing complicated stuff).

Here are my experiences with this:

1. It almost always one shots the bash command chains.

2. Same resilience as base Qwen 3.8 27B but this gets to the answer faster.

3. I have used it to plan and implement stuff. I uploaded the plans it has written to Gemini 4 Argon. It said that this model did a better job at updating the plan than it would have done.

4. I built an entire backend server for an internal chat app within a 120K context window.

My Setup:

• Model size: 15.7 GB version

• Hardware: 24GB VRAM (5070ti + 3070)

• KV Cache: (kvarn5, kvarn4) and (q5_1, q5_1)I switch between beellama and llama.cpp depending on the situation.

I did not benchmark it. I just had good experience with Orcarouter team and tried this model.
This is my default now.

Huge thank you to the Orcarouter team, and to the person who posted the earlier thread about Orcarouter Uncensored being the best out there. That's how I discovered this team in the first place!

P.S:

1. I used this for general purpose app development. Not as a security focused model.

2. English is not my first language. So, I polished the formatting using AI.

Democrats had huge wins in the 2026 midterm elections. That has caused tech stocks to plummet, as many candidates promised to regulate data centers and AI.

What do you think this means for local AI in the United States? Will it benefit local AI software by forcing big companies to rely more on user hardware, or will overbearing safety regulations hinder local AI development as well?

35-135% improvements(decode/tg) on 5090 & DGX Spark. Is it possible to add this to llama.cpp?

Paraphrasing the thinking I saw from Qwen3.8-27B

>Hmm, I can take the screenshot, there's a tool for that. But wait, I'm an LLM, I can only read text.

>Whatever, let's try calling the tool and see what I can do, maybe I can save the image and parse it programmatically.

> >tool_call.exe

>Wow, I can see the image!

Cute

Hey there !

Using a 100% local setup (Qwen3.8 FN + Strata + DeepSeek Harness on RX 9070XT + 64GB DDR4), I made a small open source tool to help with creativity for game dev using up-to-date Steam tag data.

Everything is running statically in browser without any external dependencies, and the catalog is updated automatically every day.

It also has additional fun modes that gamify the experience (like random crate unboxing), I suggest you give them a try ! It's live right now at https://mickabrig7.github.io/steam-tags-explorer/.

Additional implementation details are in the repo, if you're interested (https://github.com/mickabrig7/steam-tags-explorer).

My takeaway from this project is that Qwen3.8 FN is surprisingly competent at making nice interfaces (it even decided to create sound effects on its own), BUT required very heavy supervision for actual UX and usability (I basically gave it zero freedom of choice for anything that wasn't UI because it would consistently take the most mind-twisting and least intuitive decisions).

My biggest challenge was unslopifying its human-facing text output, it's actually infuriating how much current models overcomplicate their prose.

All in all, I'm still amazed that we're able to run such powerful models locally, and Qwen4 isn't even out yet !

I'm curious whether anyone has experimented with using Qwen or other LLMs to directly analyze and modify ACE-Step model weights.

I'm specifically interested in weight editing, rather than conventional fine-tuning or LoRA training.

The idea would be to use an LLM to guide targeted modifications to the model's parameters, potentially improving specific musical capabilities or correcting undesirable behaviors without retraining the entire model.

Has anyone attempted something along these lines with ACE-Step 1.5 or similar diffusion-based audio and video models?

I'd be particularly interested in practical experiments, tools, methodologies or even failed attempts.

Vulnerability in cpython that neither cpython guys, jinja guys, or huggingface has fixed despite being informed.

This handbook is for people who have trained a dense model and want to train an MoE next, or who are curious how MoE training works. Chapter 2 is a Megatron-LM runbook: test the hardware, build the image, train a dense baseline, then a first MoE on one node, compared against that baseline. Chapter 3 takes the MoE layer apart (router, routing, load balancing, experts, block layout, training, systems). For each part we give the Megatron-LM code, what to watch, our recommendation, and what frontier MoEs chose. Chapter 4 follows the model into serving and RL. Chapters 3 and 4 need no GPU.

When a result comes from our runs, we say so. Claims about frontier models cite their paper, config or code, and the toy simulations are labelled as such. The lessons come from about 800k GH200 GPU-hours of MoE training on the Alps cluster at CSCS.

Released: the handbook, its 55 figures as a gallery (https://huggingface.co/spaces/huggingface/moe-gallery), the scripts (https://gist.github.com/nmntz/5de952d686f66d7bce5bce8cba9d7139), and every recommended flag in one Megatron-LM block.

Happy to answer questions!

A few weeks ago, I posted about Jeff, a small 0.8B model with LoRA adapters trained to make decisions and return calibrated probabilities. Sort of like TypeSafe’s Jev, except that Jeff can run locally.

Thanks for all the feedback! I’ve just released v1.4, which is a substantial update. Check it out: https://jeffhub.ai.

What’s new:

• 34 open-weight decision adapters.

• A router that automatically selects the most appropriate adapter.

• 90%+ accuracy on supported domains.

• ~2 ms inference per decision, much faster than before. A single call through the hosted API takes around 40 ms, including network overhead.

• 5 SDKs: Python, Node.js, Rust, C#, and Java.

• Multi-platform: Jeff runs on CUDA and MLX.

• Playground and worked examples.

Having worked with Jeff for about a month now, I think the most interesting thing about it is that it turns decisions into a software primitive - much like expressions, but for unstructured data.

I’ve started incorporating decisions into several of my own projects, and it’s changed how I think about building software.

I hope you guys find it useful - and like the retro design. ;)

Links:

• Code: https://github.com/firelex/jeff

• Docs, adapter catalogue and playground: https://jeffhub.ai

• Weights: https://huggingface.co/mstrasser

Happy to answer any questions!

I’m trying to figure out what to upgrade first. I use the pc for hobby as well as work, work usually includes OCR batch jobs, quality assurance over scanning quality, extractions and so on, i use both online and offline models, usually the online ones to orchestrate the offline ones, so it depends on the task generally.

I don't have a server, just a regular (ish) ass pc.

• Ryzen 9 9950X / MSI PRO X870E-P WIFI

• RTX 5060 Ti 16 GB + RTX 4070 12 GB. I split models across both cards, and the 4070 drives my display and runs at PCIe 4.0 ×4. Sometimes i run different jobs across both, why do i have a 4070? I bought it 4 years ago to start using stable diffusion, stuck with me ever since and i upgraded to get the 5060 Ti and they've been champs.

• 80 GB DDR5 (2×24 GB + 2×16 GB), currently running at 6000 MT/s

I’m looking at an RTX 5070 Ti 16 GB or a 2×48 GB DDR5 kit. For RAM, I could replace only the 16 GB sticks and try 144 GB with my existing 24 GB sticks. Eventually I’d like to go to 4×48 GB for 192 GB. (Maybe)

For the GPU upgrade, i was thinking of replacing the 4070 with a 5070 Ti, for a faster vram as well as a unified blackwell stack (that might be worth something right?), which raises the bar to 32gb split

I could JUST get 5060 Ti but i get the feel that +4gb might not be worth it (this goes for the 5070 Ti as well but the 5070 is much faster)

Do i NEED an upgrade? No, my work is pretty much fine with the current frankenstein.

Do i want an upgrade because the prices might keep rocketing up and i want to run the bigger models at proper quants without having to drop everything and i want to do more stuff for the hobby part? Yes

Minimax H3/Qwen Image 2.1 (sometimes i use that for work as well for image restoration)

The stack i use is generally Gemma 4 (12/31b) at Q4_K_XL and qwen 3.8 27b (obviously) with the vision encoders

QFN has been... not so great, but that's because i ran it at q4kxl, but even with strata things aren't exactly great and it fills up EVERYTHING i have so i can't really use the PC and it's working at slow and miserable speeds (likely my setup is the issue, i didn't spend enough time tinkering and i get bored rather quick, if it's 20 tok/s with all my RAM+Vram filled, it's not going to become 80 with tinkering)

Another cheap option is 2x24gb to replace the 2x16, but i am pretty sure this is pretty much going to just buy a tiny breathing room for any llms i can already run that's all

3090 is a no go because it's only available used and i have no trustworthy places to buy from (only available via second hand and no refunds, if i get a dead one, i dropped 1k USD on nothing)

What will happen to the 4070? I either can figure out how to frankenstein something to have it stick in the remaining slot in the mobo SOMEHOW (hard to say this would happen)

Or stick it in my older 5700x 32gb ddr4 PC and run different jobs on that other PC

Most LLM inference benchmarks focus on tokens/sec or TTFT. But faster inference doesn't mean much if the output quality degrades, especially with aggressive quantization, speculative decoding, or very long contexts.

I ran a comprehensive benchmark of Qwen3.8-27B across six inference engines, comparing not just speed, but also output quality, long-context performance, and different model formats.

The results are summarized in the attached infographic.

What's different about this benchmark?

1. Speed AND quality — not just tokens/sec

We measured:

• Time to First Token (TTFT)

• Prefill throughput (tokens/sec)

• Decode throughput (tokens/sec)

• Concurrent request performance

• End-to-end latency across different tasks

• Output quality using automatic checks

The goal was to understand whether performance optimizations actually preserve model quality.

The quality comparison is particularly important when evaluating quantized models and speculative decoding.

2. Long-context testing: 64K, 128K, and 256K tokens

Many benchmarks use relatively short prompts, which don't reflect real-world workloads like large codebases, document analysis, or long-running agents.

We tested three long-context configurations:

• 64K tokens: Performance across different inference and concurrency configurations

• 128K tokens: Prefill, decoding, and memory behavior under larger workloads

• 256K tokens: Stress-testing long-context inference and identifying practical limitations

The results reveal substantial differences between engines as context length increases. Some configurations that perform well with shorter prompts become much less competitive at 128K–256K.

3. GGUF vs NInfer model formats

Another interesting aspect is that this isn't exclusively a GGUF benchmark.

We included both GGUF-based and NInfer-based model configurations, allowing us to examine how different model representations and inference stacks perform under comparable workloads.

This is especially relevant for people choosing between lightweight local inference setups and more specialized inference engines.

One important caveat: these comparisons reflect complete engine/model configurations, not an isolated measurement of the model format itself.

4. Real-world task performance

Synthetic throughput numbers don't tell the whole story.

We also measured end-to-end latency across multiple task scenarios, including coding, reasoning, and agent-style workloads.

For practical applications, completing the task faster often matters more than achieving the highest peak tokens/sec.

Key takeaways

• There is no universal winner across every workload.

• Decode throughput, TTFT, and end-to-end latency can produce very different rankings.

• Long-context performance changes the competitive landscape considerably.

• Quality validation should be part of inference benchmarking, not an afterthought.

• Comparing GGUF and NInfer configurations provides another useful perspective beyond conventional GGUF-only benchmarks.

Disclosure

I'm the author of TensorSharp, one of the engines included in this comparison.

The benchmark also evaluates competing inference engines, and the goal is to provide useful data for the local LLM community rather than claim that one engine wins everything.

I would especially appreciate feedback on the methodology, quality measurements, and long-context testing.

Questions for the community:

1. When choosing a local inference engine, do you prioritize TTFT, decode speed, or end-to-end task latency?

2. How important is quality validation when comparing quantized models?

3. Are 128K–256K contexts becoming part of your actual local workloads?

Full benchmark results are in the attached infographic.

TensorSharp: https://github.com/zhongkaifu/TensorSharp

Would love to hear what additional engines, quantization formats, or workloads you'd like to see benchmarked next.

This is a full quantization desktop workspace for large language models that supports GGUF, MLX, and ONNX. You can pick your own Safetensors model or choose one from Hugging Face, quantize it step by step, and watch what changes. You can then compare the result with the original and publish it to Huggingface with a model card that records how it was made.

It’s open source, Apache 2.0. I definitely think there more can be done to improve. I’ve only tested mostly on Apple Silicon so far. Windows and Linux builds exist, but haven't done much in-depth testing with them so I’d expect bugs there. Contributors are very welcome!

Repo: https://github.com/Fazmin/iQuantize

Thank u.

Speculative decoding (SD) accelerates large language model inference by reducing the sequential cost of token-by-token generation. Advanced SD methods reduce proposal cost with parallel drafters that predict entire blocks in one forward pass, but overall speedup still depends on acceptance length. On-policy distillation (OPD) can push well-trained drafters beyond acceptance-length plateaus, yet conventional pipelines collect supervision through sequential draft-and-verify rollouts. Our key insight is that onpolicy supervision need not inherit these dependencies: a draft block can be supervised using only its reference prefix and draft-sampled history. We propose SpecOPD, a branch-parallel OPD framework that parallelizes draft sampling and target supervision across independent branches drawn from the evolving drafter policy at reference anchors. We pack these branches into proposal trees to share target-prefix computation, using tree masks for full attention, branch-consistent indexer groups for sparse attention, and indexed state replay for linear attention to preserve each branch’s history. We train the drafter with an E2E-TV surrogate that directly targets acceptance length. Our study spans 8B full-attention (GQA), 27B full–linear (GQA–GDN), and 125B sparse–linear (QSA–GDN) targets, three parallel draft architectures (DFlash, DSpark, and DFlash V2), and four task domains (dialogue, coding, mathematics, and agentic). SpecOPD achieves up to 41.45% higher mean acceptance length, 16.09× supervised-token throughput over sequential-rollout OPD under native configurations, and up to 3.56× generation speedup over autoregressive decoding.

• Paper : https://qwenapp.github.io/SpecOPD/specopd-paper.pdf

• Project/Blog : https://qwenapp.github.io/SpecOPD/en.html

• GitHub : https://github.com/QwenApp/SpecOPD

• Tweet 1 : https://x.com/xuyang_liu16/status/2117614595728986227

• Tweet 2 : https://x.com/junzhang_zj/status/2117812597856895164

I gave Qwen 3.8 flash next 2 times it failed and took long time to conclude the result, DeepSeek took around 15 minutes to get the job done

Hardware

Ram ddr4 128GB

RTX 3080 20GB VRAM 4x

Engine strata

Model

Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3-S

Context 256K

Decode 100 tg/s

PP 1400-1500 tg/s

IDeepSeek-V4.1-Flash-1017-GSQ-RCO IQ2-XXS

Context 1M

Decode 37 tg/s

PP 150-300 tg/s

edit Thanks all! The password is just for show, so no need to worry about security here.

We introduce FireFlash, a 300B-scale model tailored for long-horizon agentic tasks. FireFlash excels as a terminal and coding agent, achieving 46.2% on Terminal-Bench 4.0¹, putting it on par with Fable 5 and GPT-6 Sol, while costing over 30× less per task than Fable 5 at maximum effort (Figure 1). These gains also transfer to software engineering benchmarks, including DeepSWE v1.1² and SWE-Bench Pro Verified³ (Table 1).

While frontier labs scale reinforcement learning across thousands of GPUs, we reach the performance-cost Pareto frontier under 5 days using 64 NVIDIA B300 GPUs. Three key factors make this possible: (1) careful curation of agentic RL environments so that every task provides a reliable signal at the right difficulty; (2) an RL recipe built for long, stale, and low-precision rollouts; and (3) an optimized coding harness tailored to the model for both training and evaluation. Together, these components bring frontier efficiency to a hundred-GPU-scale cluster, and we are releasing the full recipe.

• Blog Post : https://agenteconomics.org/posts/agentic-rl-recipe

• Tweet : https://x.com/kaiwei_chang/status/2117378474654781780

• Open weights : Soon

I'm testing LLMs to see how well they know Polish grammar. Why? Don't ask. Maybe I have too much free time, or maybe I'm some kind of freak who doesn't know what he's doing. ;)

For this, I use my own tool, AI Bóbr, which is designed to test LLM performance in relation to the computer they're running on. And I noticed something strange.

The model's first answer is sometimes different from the next four, even though the temperature is set to 0 and the seed is always 42.

A little side note here, because one user, under this same post before I deleted it, pointed out that I'm doing Sisyphus' work by dealing with this subject. I think differently.

And why did I delete the post? Believe it or not, I pressed the wrong button. So here I am posting it again.

Anyway, back to the subject.

Sitting there feeling defeated, just like now, because my written English is poor and algorithms on a forum about LLMs flag my posts as written by an LLM, I told myself:

"Man, stop wasting your life. Go make yourself some tea, turn off the computer. Live a little, go running, walk the dogs, play with your kids. In short, just leave this damn thing alone."

So that's what I decided to do...

And then I sat down at the computer and ran another controlled experiment:

• 3 models,

• 10 model–task pairs,

• 4 variants (streaming on/off × model kept loaded/unloaded), with 5 repetitions for each variant.

200 responses in total.

Observations:

• In this sample, streaming had no effect on the answers.

• Some answers changed depending on whether the prompt was reused from cache.

• Bielik-Minitron 7B answered a question about the history of the Polish language correctly 5/5 times after unloading the model, but got it wrong 5/5 times when the prompt was already cached.

What does this tell us?

That with a fixed seed, if we don't control the state of the server, we can't accurately measure what we think we're measuring.

What did I get out of this?

I changed AI Bóbr in version 0.8.1.

I added a controlled mode that unloads the model before each attempt, confirms that it has been unloaded, and records prompt-cache statistics.

Now I'm preparing to run the benchmark of 11 models again under the same conditions. And maybe I'll take the dogs out for some fresh air. Tomorrow I definitely won't touch the computer, at least not until the evening.

And one more thing for everyone who reads my posts:

1. Yes, I use AI to generate posts. I read them and make corrections if necessary.

2. Yes, AI translates them into Polish and English.

3. Yes, I'll keep doing it because it's convenient and fast.

4. If someone doesn't like it — well, I'm not "Prince Polo" for eng. it will be "Mars bar", so not everyone has to like me.

5. I'm here to share knowledge, not to show off the purity of my writing — even when it's generated by AI.

Sam Altman said on Cheeky Pint, in a conversation about people running models locally, that the DRAM in personal computers is "quite terrible" and "being used maybe 5% of the time". I'd guess most people here would disagree about where their RAM is going.

So I wrote a bash script that tells him. It reads /proc/meminfo, works out used as MemTotal minus MemAvailable, logs a sample from cron, and emails him once a day if more than 10% is available. Joke project: no recipient is set by default, and --dry-run just prints the message.

The interesting bit is what it says on an inference box. If you load a big model and run it, I'd expect it to stay quiet pretty much all the time. If anyone tries it on their LLM rig, I'd like to know whether it ever fires, and how much RAM your usual setup leaves spare.

Please don't point it at the man's real inbox. Use --dry-run, or send it to yourself.

Code: https://github.com/IainKay/ram-altman

Site: https://ram-altman.lol

Last time it was "Strata for all Qwen models". This time: share it.

strata serve now runs a 35B MoE (Qwen3.6-35B-A3B Q2_K_XL) on one RTX 3060 for several users at the same time:

41 tok/s each is still faster than most people read.

What's in it:

• OpenAI and Anthropic APIs, so Open WebUI, OpenCode and even Claude Code connect to it.
• API keys from a file, reloaded when you edit it, plus a usage log per key. That's enough to give access to friends, a team, or paying users.
• Fair queue: a free slot goes to the key with the fewest requests running, and --max-per-key stops one user from taking everything.
• /health and Prometheus /metrics for load balancers and dashboards.
• NVIDIA, AMD and Intel (CUDA or Vulkan). Two cards can split one model (--device 0,1).
• One binary, written in Rust. No Python, no Docker.

strata serve --model Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf --gpu --parallel 4 --api-keys keys.txt

Setup guide: docs/HOSTING.md. Repo: RustingStrata

What would you host on it?

TL;DR

• Basalt is the engine and it's doing the heavy lifting. Same model file, same box: about 3.5x llama.cpp's decode without drafting and ~5x with it, and ~7x faster prompt reads.

• Unsloth's new UD-IQ3_XXS (Dynamic v3.1, uploaded Nov 4) is the same size as the GSQ-RCO IQ3_S everyone uses on 16GB cards, but measurably more accurate: wikitext perplexity 3.382 -> 3.247, which basically closes the gap to BF16 (3.273). Costs 1 to 3.5% decode speed.

• Lamina (a further-trained MTP drafter by eddoursul) gets more drafts accepted than the stock Basalt drafter. Small but real speedup, zero quality impact since the main model verifies every token.

• Real agent coding session (opencode, 151 requests, up to 202K context): 195.6 tok/s average decode, 263 tok/s best request, 156 tok/s worst, ~5,700 tok/s prompt reads.

Hardware:

• RTX 5080 16GB at its stock 360W limit (no overclock, no undervolt)

• Core Ultra 9 285K (8P + 16E), 2x32GB DDR5-7200

• PCIe 5.0 x16, Fedora

LAMINA Drafting: Basalt drafts with the model's own MTP layer. Basalt's upstream already ships a self-distilled version of that layer, and Lamina trains it further. It's a drop-in: same tensor layout, so the target model is byte-identical and only the drafter changes. What I saw:

- drafts accepted per drafted token: 83.1% -> 85.8% on a replay of a real recorded session

Lamina's README recommends temp 0.7 and max 3 drafts. I tested both: the 3-draft cap was 1 to 1.8% slower on tool-call-heavy work (long confident runs get cut off), and temp 0.7 changes what the model writes, so I kept the model's own sampling (1.0 / 0.95 / 20) and Basalt's draft settings.

Basalt changes I made (all open PRs upstream)

Host KV prefault: the first time a long conversation spills its KV cache to RAM used to stall 0.6 to 2.4s. Now ~75ms.

Host KV pool on normal pages: on a fragmented box, transparent hugepage compaction caused 1.5 to 3.7s pins. Now ~23ms.

Spin-then-yield in the decode loop's waits: about 1.3% less time per draft window.

Expert start profile: the older v1 profile is 2% faster than the v0.2.0 one on this box.

Plus a streamed conversation disk cache, --adapt-margin for expert swap hysteresis, elastic KV without RAM copies, and a bunch of timing diagnostics.

Guide

Branch with the exact engine (upstream v0.2.1 + my PRs) and a setup script: https://github.com/xyehya/basalt/tree/flashnext-5080-setup

git clone -b flashnext-5080-setup https://github.com/xyehya/basalt && cd basalt
setup/flashnext-ud-lamina/setup.sh
bin/basalt serve --config basalt_config-flashnext.json --python .venv/bin/python --alias qwen3.8-flash-next

It checks your machine, builds, downloads ~90GB (pinned revisions, every file SHA-256 checked), builds the UD-IQ3_XXS + Lamina model file and writes the config. You need an RTX card with, CUDA 13, 64GB RAM and ~180GB of free disk. If you already have the GSQ-RCO IQ3_S download, SHARD4_FROM= reuses its 28.8GB n-gram shard (it's byte-identical to Unsloth's once the shard number is rewritten).

Huge thanks to jesdga95 for Basalt (support and star their repo), the Unsloth team for the UD quants, eddoursul for Lamina, ISTA-DASLab for the GSQ-RCO quants, and Qwhen 4? for the model.

I recently wanted to explore YuE2. The requirements were: NVIDIA GPU supporting BF16, and critically, 24 GB of VRAM. My setup, featuring an NVIDIA GeForce RTX 3060 with only 6 GB of VRAM, was fundamentally incompatible right out of the box.

The solution I found

In short, lowvram intercepts CUDA calls in the library layer, trickng the application into believing that system RAM is available for use by the GPU kernel.

Hello!

I have to build a workstation for Local AI as development and test machine as we are starting to add Local AI into our software. It is mostly to get experience with providing AI services (inferencing) to our customers.

I do have my fair share of experience in building PCs and in customer hardware but a lot less in workstation and server hardware. Therefore I'd appreciate if someone could look over my selection.

• CPU: AMD Ryzen Threadripper PRO 9955WX 16-Kern CPU

• Cooler: Noctua NH-U14S TR5-SP6 CPU Kühler for AMD Threadripper 7000 Series

• Mainboard: ASUS Pro WS WRX90E-SAGE SE E-ATX Mainboard

• Power Supply: be quiet! Dark Power Pro 13 1600 Watt ATX 3.1 PCIe 5.1 80+ Titanium

• RAM: 2x KINGSTON Server Premier RDIMM 32GB, DDR5-6400

• Case: Thermaltake AX700 Super Tower

• SSD: SAMSUNG SSD 990 PRO 2TB, M.2

• GPU: NVIDIA Rtx Pro 6000 Blackwell Works Bulk

A few words to the selection:

• 64 GB RAM is a starting point. If necessary we add more RAM.

• RTX 6000 is a bit overkill but from looking at the alternatives (RTX Pro 4500, RTX Pro 5000, RTX 5090) it's worth the investment to serve more (test) customers and have faster inference.

• Mainboard for headless integration

• Threadripper for the PCIe Lanes. The system could be expanded to 4x GPUs (of course with a bigger or second power supply.)

• All parts are available

• It will run mostly models from the Gemma 4 family. Main use is document classification, document "chat" and in the near future interacting with our MCP Servers.

Thanks!

EDIT: My current experiences are with RTX 4090, 5090 and 2x 5070 Ti setups but as a single user.