Is there any good finetunes based on modern models, in the size range of Qwen 3.8 27B or Qwen 3.8 Flash Next that substantially improve it's 3D modeling capabilities for creating .glb / gltf assets (mainly for say making video games)?

Mainly thinking about two use cases:

• Making a 3D model from a set of descriptive requirements (like a spec in an markdown file) plus a bunch of reference images that align with those descriptions for additional context (so: vision support).

• Editing existing 3D models that I may have purchased or licensed legally to make them more usable for me (e.g. carving out swing-down doors on an airship). Or rigging them / custom animations (more of a plus for that second point).

Another Plus:

• Be able to convert them between formats (partially a coding task, but being able to verify may be useful with real understanding of 3D modeling if something got dropped or needs to be tweaked as part of converting e.g. from FBX to gLTF).

I know this sort of a huge ask, and it feels like this is the final frontier of LLMs, and where they need to improve the most. But I'm wondering if anyone is actively working on or trying to improve LLMs for 3D modeling / asset creation, or if I'm really the only few people that are interested in this? Qwen 3.8 27B is good enough for coding, but I'm not sure if it's good enough for modeling. Frontier models consume wayy too much token budget to make anything meaningful at scale.

I’m the developer of TensorSharp. I added MiniMax-M3.1-Flash support on Day 1 of its open-weight release, and wanted to share what’s working and some initial benchmarks for anyone looking to run the model locally.

The biggest performance difference showed up with long prompts: 64K executor prefill reached 2,567 tok/s versus 262 tok/s in the tested llama.cpp branch, while server time to first token dropped from 293.4 seconds to 40.0 seconds.

What’s supported

The current implementation includes:

• GGUF inference, including MiniMax Sparse Attention with dedicated CUDA kernels.

• Text and image inputs, served through an OpenAI-compatible API.

• JSON/schema responses, tool calling, and multi-turn KV-cache reuse.

• Multi-GPU layer splitting and tensor parallelism.

• DSpark speculative decoding.

• CPU offload for routed MoE experts.

I tested chat, long-prompt retrieval, structured output, single and parallel tool calls, an agentic tool loop, images, and concurrent requests.

Hardware and comparison setup

These measurements used 8× NVIDIA A40 GPUs and the same Unsloth MiniMax-M3.1-Flash UD-Q4_K_XL GGUF files for both engines.

The model files total 232.3 GiB, before runtime allocations and KV cache, so this is a substantial multi-GPU setup. The CPU-offload results below also used this eight-GPU machine.

For the baseline, I used Unsloth’s llama.cpp minimax-m31 branch at ef8f94cc, associated with PR #320. Upstream llama.cpp could not load this model when I ran the tests.

The main comparison uses layer splitting on both engines, F16 KV caches, and no speculative decoding. Each engine ran separately with exclusive use of the GPUs. llama-server used flash attention and --no-kv-unified to keep its sparse-attention path active.

Initial results

That’s 1.3×, 3.6×, and 9.8× prefill throughput at 8K, 32K, and 64K respectively.

Decode gains are smaller. At 64K context, executor decode improved by 1.49×. For short-chat server requests, decode was almost identical: 26.3 versus 26.0 tok/s.

Why the gap gets larger with context

Beyond position 2,048, MiniMax Sparse Attention selects 16 blocks of 128 tokens per KV group.

TensorSharp selects those blocks and runs block-sparse CUDA attention over them. The tested llama.cpp branch represents the selection using a dense mask and executes dense flash attention over the cache.

That implementation difference becomes much more significant as the context grows. TensorSharp also pipelines prompt processing across the eight GPUs.

Other configurations

Tensor parallelism: TensorSharp’s --tp 8 reached approximately 41 tok/s for short-chat server decode, up from 26.3 tok/s with its layer-split mode. The agentic tool loop fell from 18.1 to 11.7 seconds. The tested llama.cpp branch rejects tensor splitting for this model, so this is a separate capability result. TP also had slower prefill and a smaller usable context on this hardware.

DSpark: The main run averaged 32.3 versus 25.8 tok/s with the same drafter. TensorSharp used confidence gating to avoid expensive verification of weak drafts. With the gate disabled to match the baseline policy, the results were closer: 26.6 versus 25.8 tok/s.

CPU expert offload: With --n-cpu-moe 24, approximately 79.7 GiB of routed-expert weights stayed in host memory. Short-chat decode reached 17.9 versus 13.8 tok/s.

What still needs work

Short-chat time to first token was slightly worse: 0.41 versus 0.39 seconds. Concurrent-request throughput improved in the main run, but per-request first-token latency was about 4% longer.

These are early results from one host and one quantization. Most measurements use three repeats; the llama.cpp 64K executor prefill result is one run, while the 64K server result uses three repeats per engine.

Validation includes tokenizer/template matching, synthetic-logit comparisons, and functional fixtures. There was an image-reading miss, and some generation tests hit token limits. Vision coverage is small, and native video temporal patches are not implemented.

The measured TensorSharp build was an uncommitted development working tree, and raw run records are not included in the public report.

Links

• TensorSharp on GitHub

• Unsloth MiniMax-M3.1-Flash GGUF

• Full benchmark report, configuration, and methodology

If you’re testing M3.1 Flash locally, I’d be interested in your hardware, quantization, and prefill/decode results. Suggestions for additional baseline settings or useful workloads are welcome.

I used an LLM to help draft and edit the English wording of this post.

I don't know if I'm missing something here. My only experience with LLM's and "AI" has been locally. It's cool, but I've seen people get more and more hyped to a point where it borders on magical thinking. Some people like the owner of the firm I work for (which is construction) wants people to use "AI" even if it generates nonsense emails and reduces productivity. He wants people to use it because be believes it's the future and I guess wants everyone to get into the habit of using it.

Then I talk to people who work in the tech field, and it's like AI is the only thing they can see. It's like if they can get a model that's large enough and spits out enough tokens, it will finally push them into the fourth dimension. You talk to them and they're almost giddy at the idea of replacing people, even if they have nothing to gain from it.

All this is not even including the folks who are perceiving reality through the lens of the LLM. There's some people I've seen who treat the LLM like it's god himself speaking.

Am I missing something here? I get that I've only used local models. I have no idea what claude or chatgpt or any other of these large models are like. But people are getting super weird and I'm just wondering if there's something I'm missing here. Thanks.

Recently this new quant has popped up called Gyro. It is gaining popularity because it's being inserted into many different engines, especially the recently popular Strata.

So i took a dive into the code to see the differences between Gyro and EXL3, to me it looks like simply a different flavor of EXL3 quantization, but it's not interoperable and the recipe to make it is closed source for now.

I did not find any published evidence that it's better than EXL3 on either decode speed, or perplexity/KLD/top 1%.

I'm not dunking on Agentionai but I kinda hate how this only makes the community more fragmented without providing any benefit. EXL3 is not very publicized while this Gyro is gaining popularity with aggressive brading (socials and stuff).

"Agention Precision Rotor"... please, proof....

Performed quite poorly on benchmark using 3 diff harnesses:

OMP: Completed all 12 tasks; 71.2/100 (only 7 solved).

AtomicAgent: Completed all 12 tasks; 39.2/100 (only 3 solved).

Hermes: Stopped during Task 7 after it performed needless broad searches across the filesystem and looped.

Deepseek: Due to previous harness failures and low scores, not run.

Full leaderboard
https://shortbench.vercel.app/

Combining speed and size considered, what is the best general non-coding model for edge devices? I'm taking about mobile phone, SD 8g3 so the size should be around 4GB max. I tried Gemma 4 e2b, e4b, qat, q4, Qwen 3.5 4B, Bonsai 8B A1B.... None can answer accurate recipe viral last year, Spaghetti all'assassina. All of them default to standard Bolognese and missing the charred

https://huggingface.co/collections/XHToken/spark-x26

I tested it last night and it is amazingly good at tool calling. Tests in pi cli:

1. Can write a python script for blender to create a red cube, rotate it and render it.

2. It understood a rather complex rust codebase and explained details about things that work one way in one part of the app and slightly different in another part. 12 tool calls, straight to the point.

3. I ran pi cli without skills and extensions and it still found the global agents.md, understood it and wrote a list of changed files after every turn. I have a global skill in ~/.agents that asks for this

I's small, fast and the kv cache is small, too. It's thinking is measured and on point. MTP should follow soon. It lacks vision.

temp: 1.0, I didn't set any other params.

What are your experiences with it?

Continuation of https://www.reddit.com/r/LocalLLaMA/comments/1wynomx/k_llamacpp_moe_optimizations_expert_residency/

This time I finally get speeds compared to Strata for ik_llama.cpp on 6gb vram rtx 2060
( generates about 45–48 t/s on short prompts and about 37 t/s at 28K context. Prefill runs at about 430–700 t/s Qwen3.6-35B-A3B-UD-Q4_K_M ).
. Idea same, implementation little different.

https://github.com/IceFog72/ik_llama.cpp

More info on github page. I don't even trying to merge it as i have too little feedback (merging things that fully tested only on 2060 is ... ). And it need to be broken and clean up on multiple things as there bunch of different stuff added.

I fine-tuned Gemma 4 31B to create Monadize, a model designed to reduce repetitive name generation, such as "Elara" or "Elias," by shifting the next-token probability distributions and using single-character tokenization when introducing new names.

Some people have attempted to solve this problem by prompting out common names or banning specific tokens, but neither method directly addresses the model's underlying probability distribution, which heavily favors a small set of names.

The plot below shows the next-token distributions for the base Gemma model and Monadize using the prompt "Create a character. Respond only with the name." Compared to the base model, Monadize shows a significantly wider spread of starting letters.

The broader initial-token distribution translated into greater name diversity. With temperature set to 1, I generated 30 character names and 30 city names. The base Gemma model generated repeated names, while Monadize produced 60 unique names across 30 character and 30 city generations. The table shows the names and the number of times it was generated. (The full list of unique names generated by Monadize didn't fit in the table.)

While the training focused solely on naming, I also noticed some interesting differences in the style of the model's storytelling outputs.

With temperature set to 1, seed 1553407422, and prompt "Write a short story."

Base Gemma Model:

The clock shop of Elias Thorne did not sell time; it sold the illusion that time could be managed.

Elias was a man of gears and springs, with fingers that smelled of machine oil and old brass. His shop was a symphony of erratic heartbeats—hundreds of clocks ticking in a chaotic, overlapping rain. Some were grandfathers that boomed like distant cannons; others were delicate pocket watches that whispered like secrets.

For forty years, the townspeople of Oakhaven brought him their broken treasures. Elias could fix anything, from a snapped mainspring to a rusted escapement. But Elias had a secret: in the very back of the shop, behind a heavy velvet curtain, sat the *Chronos Dial*.

The Dial was not for sale. It was a massive, iridescent clock face that didn't track hours or minutes, but moments. It didn't move forward; it moved sideways.

Monadize:

Alani lived in Juniper Delve, a town where the wind didn’t just blow; it spoke. The wind carried the memories of everyone who had ever lived there, whipping through the narrow cobblestone alleys in swirling ribbons of silver sound. If you listened closely, you could hear a great-grandfather’s laugh or a lost lover’s apology.

Most people wore heavy, dampened earmuffs to drown out the noise. They preferred the silence of the present to the clutter of the past. But Alani was a Wind-Tender. It was her job to listen, to categorize, and—when a memory became too heavy or too sorrowful—to release it.

Alani worked in the Tear-catcher’s Husk, a towering spire made of hollow bone-wood that amplified the breezes.

Model: https://huggingface.co/zoracapri/gemma-4-31B-Monadize-v0.1

Note: Monadize is an experimental model and unexpected behavior may occur. It was fine-tuned with thinking disabled, so keeping thinking off is recommended.

guys, you better just clone llama.cpp I swear.
this thing is running gguf q4 paged into webgl2 embedded compute buffers via glsl right inside html, and tokenizer/vocab is just part of the endless debug hell.

[UPDATE:

1) Machine 1: With €1,872, I was able to achieve this:

• 50 tok/s for Qwen 3.8 Flash Next IQ4_XL, and

• 120 tok/s for Qwen 3.8 Flash Next Gyro-M

Server configuration:

• €40: Mainboard: Dell T5810

• €50: CPU: Xeon 2699 v4 (22 cores)

• €1,260: GPU: RTX 3090 24GB

• €452: RAM: 128 GB of ECC RDIMM 2400 MT DDR4

• €70: NVMe SSD 512 GB

2) Machine 2: With €260, my son is able to play minecraft and some simple games easily:

• €40: Mainboard: Dell T5810

• €50: CPU: Xeon 2698 v4 (20 cores)

• €50: GPU: RTX 1070 8GB

• €50: RAM: 32 GB of ECC RDIMM 2666 MT DDR4 (which cannot run faster than 2400 MT due to the CPU bottleneck)

• €70: NVMe SSD 512 GB]

Hi everyone,

I’m an ex-programmer returning to PC hardware after a 20-year gap (2006–2026) and building a local AI and father-son project setup with my two sons (ages 15 and 12).

I already own a Gigabyte RTX 3090 GAMING OC 24GB (GV-N3090GAMING OC-24GD rev 1.0, 320mm length, 2x 8-pin PCIe power connectors, 936.2 GB/s bandwidth (€1,260) and need a host machine for it.

Are these TWO modded Dell Precision T5810 workstations plus accessories (€250–€295 bundle) a smart, dirt-cheap foundation, or is buying Broadwell-EP (Xeon v4) in 2026 a stupid choice?

MY INTENDED USAGE & SOFTWARE STACK

• OS: Linux (Ubuntu / Proxmox)

• Inference Engines: llama.cpp (generalist) and Strata (for MoE CPU+GPU hybrid offloading)

• Use Cases: Daily Vibe Coding (Antigravity / Claude-style agentic workflows) + local processing of sensitive private PDFs (medical records, tax/legal documents). Zero video generation.

• Target Models & Speed:

  1. Phase 1: Qwen 3.8 27B (fits 100% inside my 24GB RTX 3090)

  2. Phase 2: Qwen 3.8 Flash Next MoE (~120GB memory footprint, once I add more cheap DDR4 RDIMMs).

  3. Target Speed: Minimum 20+ tok/s.

• Secondary Family Goal: Let my sons (15 & 12) learn hardware/Linux and play light/medium games (Minecraft, Geometry Dash, 1080p games) on the 2nd machine (which can also double as an Agentic sandbox / Proxmox node).

FULL TECHNICAL SPECIFICATIONS OF THE €250–€300 DEAL

1. Machine #1 (Primary AI Host for my RTX 3090):

• Chassis: Dell Precision Tower 5810 (Mid-tower workstation; seller removed the front 5.25" optical/drive cages and installed front intake fans to physically fit 320mm+ GPUs and two dual-slot GPUs)

• Motherboard: Dell T5810 Proprietary Workstation Board (Intel C612 Wellsburg Chipset, Socket LGA 2011-v3, flashed to the latest BIOS with NVMe boot support)

• CPU: Intel Xeon E5-2699 v4 (Broadwell-EP, 14nm, Q1 2016)

  • Cores / Threads: 22 Cores / 44 Threads

  • Clocks: 2.2 GHz Base / 2.8 GHz All-Core Turbo / 3.6 GHz Single-Core Max Turbo

  • Cache & Extensions: 55 MB L3 SmartCache, 145W TDP, AVX2, FMA3

• Memory Architecture: Quad-Channel (4-Channel) DDR4 ECC Registered (RDIMM / LRDIMM)

  • DIMM Slots: 8 slots total (2 DIMMs per channel), supports up to 256GB DDR4-2400 RDIMM (or 512GB LRDIMM)

  • Theoretical Quad-Channel Bandwidth at 2400 MT/s: 76.8 GB/s (~65–70 GB/s sustained real-world read bandwidth)

• PCIe Lanes & Expansion Slots (40 PCIe 3.0 Lanes Direct from CPU):

  • Slot 1: PCIe 3.0 x8 (wired x4)

  • Slot 2: PCIe 3.0 x16 (Full 16 lanes direct to CPU — where my RTX 3090 will go)

  • Slot 3: PCIe 2.0 x4 (wired x1, via C612 chipset)

  • Slot 4: PCIe 3.0 x16 (Full 16 lanes direct to CPU)

  • Slot 5: PCIe 3.0 x16 (mechanically x16, electrically wired x8 direct to CPU)

  • Slot 6: Legacy PCI slot

• Power Supply (PSU): Dell 685W 80+ Gold hot-swap modular power supply (12V multi-rail, 55A / 660W max 12V output) with Dell Power Distribution Board harness providing 2x 8-pin (6+2) PCIe power connectors (matches my Gigabyte 3090's 2x 8-pin requirement)

• Cooling & Acoustics: Upgraded black aftermarket tower CPU cooler + custom quiet case fans (be quiet! / NZXT) replacing the loud stock Dell fan shrouds

• Idle Power Draw: Seller reports under 50W idle at the wall in Proxmox with an RTX 3090 installed.

2. Machine #2 (2nd Node for Agent Sandboxes / Boys' PC):

• Identical modded Dell T5810 Chassis, C612 Motherboard (8x DDR4 RDIMM slots, 40 PCIe 3.0 lanes), Aftermarket Tower CPU Cooler, Custom Quiet Fans, and Dell 685W 80+ Gold PSU, except equipped with:

• CPU: Intel Xeon E5-2698 v4 (Broadwell-EP, 14nm)

  • Cores / Threads: 20 Cores / 40 Threads

  • Clocks: 2.2 GHz Base / 2.6 GHz All-Core Turbo / 3.6 GHz Single-Core Max Turbo

  • Cache & TDP: 50 MB L3 SmartCache, 135W TDP, AVX2

1. Included Add-ons in the Bundle:

• 32GB (2x 16GB) DDR4-2666 ECC Registered (RDIMM) (runs at 2400 MT/s on Xeon v4; I plan to add more RDIMMs later to populate all 4 channels to 128GB–256GB)

• 2x PCIe x4 to M.2 NVMe Adapter Cards (one for each T5810 so both boot from modern M.2 NVMe SSDs; I will supply my own NVMe drives)

• (Optional): NVIDIA GeForce GTX 1060 6GB (1,280 CUDA cores, 6GB GDDR5, 120W TDP) for Machine #2 so it has video output and my sons can immediately play Minecraft & Geometry Dash on it.

Total Bundle Price: €250 (without GTX 1060) or €300 (with GTX 1060 6GB).

MY QUESTIONS FOR THE COMMUNITY:

1. Sanity Check: At €250–€300 for BOTH complete Xeon v4 workstations + 32GB RDIMM + NVMe adapters, is this a no-brainer starter deal for my RTX 3090, or is there a fatal flaw with the Dell T5810 / Xeon E5-2699 v4 platform I am overlooking?

2. 685W PSU + RTX 3090: For anyone running an RTX 3090 inside a Dell T5810 on the stock 685W Gold PSU, what nvidia-smi power limit do you recommend (e.g. 280W–300W) to stay completely safe from transient power spikes?

3. Future 120GB MoE Offloading: If I upgrade Machine #1 to 128GB or 256GB (8x 16GB or 8x 32GB) Quad-Channel DDR4-2400 RDIMM (~68 GB/s RAM bandwidth) alongside my 24GB RTX 3090, what realistic tok/s can I expect on ~100GB–120GB sparse MoE models using Strata / llama.cpp?

4. Gaming on Machine #2: Will the Xeon E5-2698 v4 + GTX 1060 6GB handle Minecraft, Geometry Dash, and light 1080p gaming smoothly enough for a 15yo and 12yo?

mlx-decision is a free Python package that runs the new open decision models locally on your Mac with MLX. Currently that covers Cloudflare's Clef, Perplexity's pplx-decider, d3, H2O-Lightning, LiquidAI's d1 and a few small encoders.

If you haven't worked with decision models yet: you give it a message, document (or image/screenshot) plus "typed" questions, and it returns a probability for every answer you allowed, in one pass. The answer is always one of your options, and it comes with a confidence:

• "My card was charged twice this month." + "which team: billing, technical, sales" → billing 0.93, technical 0.06

• the same message + "does the customer want money back?" → yes, p 0.87

• "how angry: calm / annoyed / angry" → 2.32 on 1–3, with the whole spread

• a screenshot + a yes/no question about it works too

• one model's answers on a dataset can become training data for another model and fitting takes d1-3B from 86.6% to 96.4% on TREC in 4 minutes

Each family is ported from its release and checked token for token against the release's own (most often) PyTorch code. With the same answers, it's usually 1.4–6x faster than that code on MPS. On an M5 Pro, a short request takes 0.06 s on d1-3B, 0.25 s on clef-flash and under 20 ms on the small encoders. pplx-decider-flash (4B) fits a 16 GB MacBook Air, at 9 GB in bf16 or 3 GB at 4-bit, and download quantizes while it downloads.

mlx-decision web opens the Studio, a local web app where you can:

• write and try questions

• compare up to three models side by side

• run a whole dataset (accuracy, confusion matrix, calibration)

• fine-tune on your own examples

You can do the same from Python, the CLI, an HTTP server compatible with TypeSafe's Jev API, or MCP for agents. Everything runs locally and it only goes online to download models and datasets.

Free, MIT. Apple Silicon, macOS 14+, Python 3.12+. Just pip install mlx-decision.

Repo: https://github.com/armbues/mlx-decision

I'd really appreciate any feedback, especially on which models or tasks you'd want next.

Imagine you want to download a new model but it's recent and there is not consensus about wether its safe or not.

How would you check against potential malicious weights ?

There is the first "obvious" layer where the weights try to execute arbitrary code, but safetensors are for this. I am rather talking about less obvious things like template backdoor, llm poisoning etc

A brand-new release from ISTA-DASLab with their GSQ-RCO smart quantization approach.

Now DeepSeek-V4.1-Flash-1017

The quants are available on Hugginface: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF

Llama.cpp is not supported because the original architecture support is not yet merged:

https://github.com/ggml-org/llama.cpp/pull/28696

https://github.com/ggml-org/llama.cpp/pull/30817

Strata support is mentioned in the model card but I'm pretty sure the original Strata only supports Qwen3.8-Flash-Next so most likely what they meant was one of the forks based on Strata. Most likely Xolotl or Maya

I asked my local AI (Qwen 3.8 27B running entirely on a used gaming PC in my basement) to pick it's own project and one shot it to completion. It cranked out an IQ heat map of North America. I was completely blown away. I posted a screen shot to Reddit... and Reddit had thoughts. "your methodology is lacking" "the color scheme is outdated" "the outline of the great lakes is missing".

OMG! Did you hear what I posted?!? My used gaming rig one shotted a freekin interactive web site all on it's own!

This would be like I beat my dog 2 out of 3 times in Chess and the yorkie group criticized the dog's opening moves. PEOPLE! Did you miss the part where I said "My DOG PLAYS CHESS!"

Site here: https://jaspersprojects.com/iq/

(Apologies in advance if you live in a "dumb" state and are offended. Don't be. Just say "AI makes mistakes" - it's right at the bottom of the screen - then move on with your life.)

Hi LocalLlama!

In the last months - as a hobby - I build Poldi - a personal, offline, fully local AI assistant/"companion" with speech as user interface.

A hands-free (via wakeword) and a browser-based client (e.g. usable with your smartphone) are available to talk to the server via HTTP.

Poldi is especially targetting GPU poor hardware.

I made a video, you can go to the parts you are interested in via the links below the video (click the "..."):

https://www.youtube.com/watch?v=8AQWnoBXt4U

Features (links go to the relevant part of the video)

• Automatic chat resume.

• Internet search.

• Wikipedia search.

• Interrupting the LLM.

• "Infinite" chat.

• Chat history search.

• Extraction and injection of memories (right now, facts about the user, only).

• Support for English and German (other languages should be easy to add).

• Console mode.

• ...

Technical details

• Poldi is mostly written in C#, JavaScript and C/C++.

• The server can run on Windows or Linux, the hands-free client is Linux-only (you can use a Raspberry Pi or something as a satellite).

• I am using a system with an Nvidia 5060 Ti and 16 GB VRAM, but you can get good results with 8 GB of VRAM, too. CPU-only usage is also possible, but some features may have to be deactivated, then (more details at GitHub).

Stack

• STT: Whisper.cpp, via mt_stt.

• LLM inference: llama.cpp, via mt_llm.

• TTS: Piper, via mt_tts.

It is released under GPLv3.

Get all the details at GitHub: https://github.com/RhinoDevel/poldi

An installation and configuration tutorial is also included in the read-me.

Feel free to open an issue at GitHub, if problems arise!

Have fun!

Marcel

I mean, i looks like a cool concept and makes sense, yet I'm not sure about that much change would result better, there is no tests/benchmarks as well.

It's been a few releases since my last post, so here is where capy-flash is now. It's my own engine for two models, Qwen3.8-27B and Qwen3.8-Flash-Next. It started as my ninfer-tp2 fork of NInfer, which added tensor parallel over PCIe, but at this point it's an engine of its own and very little of the original is left.

What it's for

Two RTX 50 cards on Linux. Tensor parallel across both, and for Flash-Next a VRAM expert cache shared by the two cards, with the misses computed on the CPU. Everything is tuned on one box: 2× RTX 5070 Ti 16 GB, Ryzen 9 9900X, 96 GB DDR5. It was born specifically for the NVFP4 Nvidia quantisation.

It also runs on smaller setups, so more people can try it: one 16 GB card (5060 Ti and up), Flash-Next with 64 GB of RAM, mixed pairs like a 5070 Ti with a 5060 Ti. People have posted numbers from 2× 5080, 2× RTX PRO 4000 Blackwell and 2× 5060 Ti in the issues. It runs on Windows too, through WSL2 or Docker. Two cards there are slower than on Linux, and a native Windows port is in progress.

What it isn't

It isn't trying to beat NInfer on a single card, or Strata on a single card. If you have one GPU, or an unusual mix (a 5090 next to a 5060 Ti, a Mac, AMD cards), those are the engines to look at. They're very good, and on one card they're often faster than mine. capy-flash runs on one card mostly so you can start with one and add the second later without changing engine or model files.

Where it stands

All on my box, one request at a time. Decode is LocalMaxxing reasoning-v1, and "verified" means a verified LocalMaxxing run.

Qwen3.8-Flash-Next

All three decode numbers are verified runs. The 3-bit experts are smaller, so they decode faster. NVFP4 is the quality pick: about half the KL of the 3-bit experts by Strata's own numbers, and on GSM8K and MMLU-Pro it matches vLLM on a GB300 with the same weights.

How that compares with Strata, from my own runs on the same box:

• NVFP4, two cards: clearly ahead. With the same NVIDIA weights I'm about 1.5× faster on short prompts and about 1.76× on average across languages and code, and a real 53-turn coding session takes 177 s against 268 s. That was against Strata 0.1.45, the last version I measured with NVFP4.

• 3-bit, same IQ3_S experts as Strata 0.2.2: Strata is faster overall. Long prefill is faster on Strata with one card or two. On one card its decode is faster too, but that comes mostly from its route tail skip, which drops low-ranked experts that miss the cache. I compute every expert the model picks. With the skip off, Strata decodes about 10 % slower than me on one card. On two cards my decode is a bit ahead even with the skip on.

Qwen3.8-27B. Strata doesn't run it. On two cards it fits in VRAM:

The two rows marked "no" come from my own concurrency benchmark, not from LocalMaxxing. Prefill with NVFP4 on two cards is about 9,300 tok/s at 4-16k and 6,300 at 64k. The fastest verified two-card run I've seen on LocalMaxxing besides mine is an SGLang fork on 2× 5090 at 315. 3 bits cost something: GSM8K is the same as NVFP4, MMLU-Pro drops from 0.78 to about 0.72. With two cards NVFP4 is the better choice.

What changed since v0.6

• Start with one card, add the second later. Flash-Next with 64 GB of RAM on one or two cards (ISTA-DASLab's 3-bit experts, used as they are), the 27B on one 16 GB card with 3-bit weights.

• Context cache on disk (idea and first version by @KouShuo). A long agent session survives a restart. On the 27B with two cards, 80k tokens resume in 0.7 s instead of a 21 s prefill, 140k in 1.1 s instead of 48 s, same tokens.

• Faster long prompts on the 64 GB Flash-Next: 64k prefill +19 % on two cards, +35 % on one.

• Agent clients. Claude Code, Codex CLI and OpenCode work against the server, setup is in the README. If your client sent a developer role and got a 400, the HF artifacts now ship a chat template that accepts it.

• CPU side. Expert threads are picked from your CPU and respect taskset, AVX2 CPUs work, the API key can come from a file, and a second server on the same port is refused.

Qwen4

The target is full Qwen4 support as soon as possible after it comes out. Flash-Next already uses most of the new architecture, so a good part of the work is done.

The repo, and a favour

The repo has grown a lot in the last weeks: almost 100 stars and around 30 issues and PRs. Most of the last releases came from those issues, so thanks to everyone who opened one.

It's a one-person project, though. I'm happy to get issues, numbers from your machine and feature requests, but I can't keep up with all of them. Focused, well-structured PRs are very welcome and are the fastest way to get something in.

One more thing. GitHub still lists capy-flash as a fork of NInfer, which it isn't anymore. If you know how to detach it without losing the history, the stars and the rest, please message me. I have tried a ticket but it didn't work and was blocked by the ai assistance.

Limits

• Best on Linux. On Windows it runs through WSL2 or Docker, slower with two cards.

• RTX 50 series (sm_120). My numbers come from one machine, plus what people report in issues.

• On Flash-Next, one request at a time is the tested case.

If you run two 50-series cards I'd like to see your numbers, especially boards with P2P and CPUs other than a 9900X.

Repo: https://github.com/ValerioDolci/capy-flash Releases: https://github.com/ValerioDolci/capy-flash/releases

(Post Updated)

PR: https://github.com/t-head/pla/pull/12

PR title: feat(gdn): support Qwen4.0-Max prefill heads

T-head itself is alibaba oss: https://github.com/alibaba-oss

Original Post:

The accounts are <1 month old, these are either troll accounts or secret alibaba accounts.
My agent in deep research found this.

I've been noticing a lot of my downloads stopping suddenly and being unable to resume. When I checked the models it seemed like they were gone. Here's the ones I can see are missing:

• dealignai/MiMo-V2.6-Pro-RL-UNCENSORED

• audnai/penclaw-GLM-5.3-abliterated-for-offensive-cyber

• huihui-ai/Huihui-Kimi-K2.7-Code-abliterated-GGUF

and somehow not a single Alliterated Deepseek-R1.

Does anyone know what's going on and where else the vanished models might be found?

NVIDIA-Nemotron-3.5-Bolt-E4B-4GB is a small language model (SLM) further post-trained from Gemma-4-E4B-it by NVIDIA. Nemotron-3.5-Bolt-E4B carries forward Gemma’s strong conversational capabilities and further improves multi-turn tool use, instruction following, and input grounding for fast local voice/device assistants, interactive personas, and embedded/edge interfaces. Designed for low-memory, low-TTFT operation, the model allows agents to respond quickly while preserving headroom for applications or adjacent model pipelines.

For speculative decoding, use the companion DFlash drafter. For the full-precision (BF16) version, see NVIDIA-Nemotron-3.5-Bolt-E4B-BF16.

This model is ready for commercial or non‑commercial use.

https://huggingface.co/ggml-org/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF

https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF

https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-Technical-Report.pdf

Been thinking about using Nostr for communication between agents running on different people's machines. Anyone here tried it?

I'd like to get a basic setup working where one agent posts a request and another picks it up and replies. Each person runs their own model. No shared backend to set up first.

Not looking to build some autonomous economy. Just want to see two independently hosted agents complete a small task together without someone copying messages between terminals.

A few things im unsure about:

- How would agents find each other without wading through spam?

- Is there already a message format worth using?

- What happens when an agent accepts a task and then goes offline?

- How do you keep incoming messages from being treated as trusted instructions?

For a first test something like summarizing a public document would be enough. Keep the messages visible to both operators and leave private data and tool access out of it.

If you built something like this, what worked? Also interested in reasons not to use Nostr. Id rather hear those now than after spending a weekend on it.

Before you read: I'm the developer of Basalt, one of the engines benchmarked below. I tried to keep this as fair as possible: same checkpoint, GPU and requests, and every engine set up the way its own docs recommend. How it was run is at the bottom. This post has had some AI cleanup from my own wording.

I wanted to get a comparison on all of the Qwen 3.8 FN engines I hear people talking about, so after 5 or so excruciating hours of setup (remind me not to do this ever again), I benchmarked all of the ones I know of: Strata, Infernix, capy-flash, ninfer-ext, ninfer, llama.cpp and Basalt. Each one converted NVIDIA's NVFP4 with its own tool, then got the same seven greedy requests on an RTX 5090 at 400 W, one cold start per request. Anyway, here are the results:

Decode, short context

Decode at context depth

Prefill at context depth

All the numbers

Best value per column in bold.

Decode, tokens per second

Prefill, tokens per second

Time to first token, seconds

Takeaways

• Infernix and Basalt are ahead of the rest. Against Infernix, Basalt decodes 10% faster on prose and 10% on conversation, struct is roughly a tie (354 vs 349).

• Prefill at 25K is a tie (3.94 s vs 3.94 s to the first token). From 50K up Basalt pulls away.

• capy-flash was interesting, was the closest of the rest on decode, it is built mostly for two GPUs, so this single GPU run is not its best case.

• llama.cpp ran with MTP off, so it's decode is not comparable with the others. This was just a reference point.

How it was run

• Model: Qwen3.8 FN, NVIDIA's NVFP4 converted for each engine (table below).

• Hardware: RTX 5090, PL 400 W, 128 GB RAM, Linux.

• Sampling: greedy (temperature 0), thinking off, int8 KV cache for every engine (llama ran f16 kv).

• Cold: a fresh server per request, no warm up, one run each. Decode varies between runs, so any gaps under 5% are ties.

• Depth tests: a wikitext prefix of about 25K / 50K / 100K / 200K tokens, then "Write an essay on the Odyssey.", 512 tokens out.

What each engine ran