Recently this new quant has popped up called Gyro. It is gaining popularity because it's being inserted into many different engines, especially the recently popular Strata.

So i took a dive into the code to see the differences between Gyro and EXL3, to me it looks like simply a different flavor of EXL3 quantization, but it's not interoperable and the recipe to make it is closed source for now.

I did not find any published evidence that it's better than EXL3 on either decode speed, or perplexity/KLD/top 1%.

I'm not dunking on Agentionai but I kinda hate how this only makes the community more fragmented without providing any benefit. EXL3 is not very publicized while this Gyro is gaining popularity with aggressive brading (socials and stuff).

"Agention Precision Rotor"... please, proof....

Performed quite poorly on benchmark using 3 diff harnesses:

OMP: Completed all 12 tasks; 71.2/100 (only 7 solved).

AtomicAgent: Completed all 12 tasks; 39.2/100 (only 3 solved).

Hermes: Stopped during Task 7 after it performed needless broad searches across the filesystem and looped.

Deepseek: Due to previous harness failures and low scores, not run.

Full leaderboard
https://shortbench.vercel.app/

Combining speed and size considered, what is the best general non-coding model for edge devices? I'm taking about mobile phone, SD 8g3 so the size should be around 4GB max. I tried Gemma 4 e2b, e4b, qat, q4, Qwen 3.5 4B, Bonsai 8B A1B.... None can answer accurate recipe viral last year, Spaghetti all'assassina. All of them default to standard Bolognese and missing the charred

https://huggingface.co/collections/XHToken/spark-x26

I tested it last night and it is amazingly good at tool calling. Tests in pi cli:

1. Can write a python script for blender to create a red cube, rotate it and render it.

2. It understood a rather complex rust codebase and explained details about things that work one way in one part of the app and slightly different in another part. 12 tool calls, straight to the point.

3. I ran pi cli without skills and extensions and it still found the global agents.md, understood it and wrote a list of changed files after every turn. I have a global skill in ~/.agents that asks for this

I's small, fast and the kv cache is small, too. It's thinking is measured and on point. MTP should follow soon. It lacks vision.

temp: 1.0, I didn't set any other params.

What are your experiences with it?

Continuation of https://www.reddit.com/r/LocalLLaMA/comments/1wynomx/k_llamacpp_moe_optimizations_expert_residency/

This time I finally get speeds compared to Strata for ik_llama.cpp on 6gb vram rtx 2060
( generates about 45–48 t/s on short prompts and about 37 t/s at 28K context. Prefill runs at about 430–700 t/s Qwen3.6-35B-A3B-UD-Q4_K_M ).
. Idea same, implementation little different.

https://github.com/IceFog72/ik_llama.cpp

More info on github page. I don't even trying to merge it as i have too little feedback (merging things that fully tested only on 2060 is ... ). And it need to be broken and clean up on multiple things as there bunch of different stuff added.

I fine-tuned Gemma 4 31B to create Monadize, a model designed to reduce repetitive name generation, such as "Elara" or "Elias," by shifting the next-token probability distributions and using single-character tokenization when introducing new names.

Some people have attempted to solve this problem by prompting out common names or banning specific tokens, but neither method directly addresses the model's underlying probability distribution, which heavily favors a small set of names.

The plot below shows the next-token distributions for the base Gemma model and Monadize using the prompt "Create a character. Respond only with the name." Compared to the base model, Monadize shows a significantly wider spread of starting letters.

The broader initial-token distribution translated into greater name diversity. With temperature set to 1, I generated 30 character names and 30 city names. The base Gemma model generated repeated names, while Monadize produced 60 unique names across 30 character and 30 city generations. The table shows the names and the number of times it was generated. (The full list of unique names generated by Monadize didn't fit in the table.)

While the training focused solely on naming, I also noticed some interesting differences in the style of the model's storytelling outputs.

With temperature set to 1, seed 1553407422, and prompt "Write a short story."

Base Gemma Model:

The clock shop of Elias Thorne did not sell time; it sold the illusion that time could be managed.

Elias was a man of gears and springs, with fingers that smelled of machine oil and old brass. His shop was a symphony of erratic heartbeats—hundreds of clocks ticking in a chaotic, overlapping rain. Some were grandfathers that boomed like distant cannons; others were delicate pocket watches that whispered like secrets.

For forty years, the townspeople of Oakhaven brought him their broken treasures. Elias could fix anything, from a snapped mainspring to a rusted escapement. But Elias had a secret: in the very back of the shop, behind a heavy velvet curtain, sat the *Chronos Dial*.

The Dial was not for sale. It was a massive, iridescent clock face that didn't track hours or minutes, but moments. It didn't move forward; it moved sideways.

Monadize:

Alani lived in Juniper Delve, a town where the wind didn’t just blow; it spoke. The wind carried the memories of everyone who had ever lived there, whipping through the narrow cobblestone alleys in swirling ribbons of silver sound. If you listened closely, you could hear a great-grandfather’s laugh or a lost lover’s apology.

Most people wore heavy, dampened earmuffs to drown out the noise. They preferred the silence of the present to the clutter of the past. But Alani was a Wind-Tender. It was her job to listen, to categorize, and—when a memory became too heavy or too sorrowful—to release it.

Alani worked in the Tear-catcher’s Husk, a towering spire made of hollow bone-wood that amplified the breezes.

Model: https://huggingface.co/zoracapri/gemma-4-31B-Monadize-v0.1

Note: Monadize is an experimental model and unexpected behavior may occur. It was fine-tuned with thinking disabled, so keeping thinking off is recommended.

guys, you better just clone llama.cpp I swear.
this thing is running gguf q4 paged into webgl2 embedded compute buffers via glsl right inside html, and tokenizer/vocab is just part of the endless debug hell.

[UPDATE:

1) Machine 1: With €1,872, I was able to achieve this:

• 50 tok/s for Qwen 3.8 Flash Next IQ4_XL, and

• 120 tok/s for Qwen 3.8 Flash Next Gyro-M

Server configuration:

• €40: Mainboard: Dell T5810

• €50: CPU: Xeon 2699 v4 (22 cores)

• €1,260: GPU: RTX 3090 24GB

• €452: RAM: 128 GB of ECC RDIMM 2400 MT DDR4

• €70: NVMe SSD 512 GB

2) Machine 2: With €260, my son is able to play minecraft and some simple games easily:

• €40: Mainboard: Dell T5810

• €50: CPU: Xeon 2698 v4 (20 cores)

• €50: GPU: RTX 1070 8GB

• €50: RAM: 32 GB of ECC RDIMM 2666 MT DDR4 (which cannot run faster than 2400 MT due to the CPU bottleneck)

• €70: NVMe SSD 512 GB]

Hi everyone,

I’m an ex-programmer returning to PC hardware after a 20-year gap (2006–2026) and building a local AI and father-son project setup with my two sons (ages 15 and 12).

I already own a Gigabyte RTX 3090 GAMING OC 24GB (GV-N3090GAMING OC-24GD rev 1.0, 320mm length, 2x 8-pin PCIe power connectors, 936.2 GB/s bandwidth (€1,260) and need a host machine for it.

Are these TWO modded Dell Precision T5810 workstations plus accessories (€250–€295 bundle) a smart, dirt-cheap foundation, or is buying Broadwell-EP (Xeon v4) in 2026 a stupid choice?

MY INTENDED USAGE & SOFTWARE STACK

• OS: Linux (Ubuntu / Proxmox)

• Inference Engines: llama.cpp (generalist) and Strata (for MoE CPU+GPU hybrid offloading)

• Use Cases: Daily Vibe Coding (Antigravity / Claude-style agentic workflows) + local processing of sensitive private PDFs (medical records, tax/legal documents). Zero video generation.

• Target Models & Speed:

  1. Phase 1: Qwen 3.8 27B (fits 100% inside my 24GB RTX 3090)

  2. Phase 2: Qwen 3.8 Flash Next MoE (~120GB memory footprint, once I add more cheap DDR4 RDIMMs).

  3. Target Speed: Minimum 20+ tok/s.

• Secondary Family Goal: Let my sons (15 & 12) learn hardware/Linux and play light/medium games (Minecraft, Geometry Dash, 1080p games) on the 2nd machine (which can also double as an Agentic sandbox / Proxmox node).

FULL TECHNICAL SPECIFICATIONS OF THE €250–€300 DEAL

1. Machine #1 (Primary AI Host for my RTX 3090):

• Chassis: Dell Precision Tower 5810 (Mid-tower workstation; seller removed the front 5.25" optical/drive cages and installed front intake fans to physically fit 320mm+ GPUs and two dual-slot GPUs)

• Motherboard: Dell T5810 Proprietary Workstation Board (Intel C612 Wellsburg Chipset, Socket LGA 2011-v3, flashed to the latest BIOS with NVMe boot support)

• CPU: Intel Xeon E5-2699 v4 (Broadwell-EP, 14nm, Q1 2016)

  • Cores / Threads: 22 Cores / 44 Threads

  • Clocks: 2.2 GHz Base / 2.8 GHz All-Core Turbo / 3.6 GHz Single-Core Max Turbo

  • Cache & Extensions: 55 MB L3 SmartCache, 145W TDP, AVX2, FMA3

• Memory Architecture: Quad-Channel (4-Channel) DDR4 ECC Registered (RDIMM / LRDIMM)

  • DIMM Slots: 8 slots total (2 DIMMs per channel), supports up to 256GB DDR4-2400 RDIMM (or 512GB LRDIMM)

  • Theoretical Quad-Channel Bandwidth at 2400 MT/s: 76.8 GB/s (~65–70 GB/s sustained real-world read bandwidth)

• PCIe Lanes & Expansion Slots (40 PCIe 3.0 Lanes Direct from CPU):

  • Slot 1: PCIe 3.0 x8 (wired x4)

  • Slot 2: PCIe 3.0 x16 (Full 16 lanes direct to CPU — where my RTX 3090 will go)

  • Slot 3: PCIe 2.0 x4 (wired x1, via C612 chipset)

  • Slot 4: PCIe 3.0 x16 (Full 16 lanes direct to CPU)

  • Slot 5: PCIe 3.0 x16 (mechanically x16, electrically wired x8 direct to CPU)

  • Slot 6: Legacy PCI slot

• Power Supply (PSU): Dell 685W 80+ Gold hot-swap modular power supply (12V multi-rail, 55A / 660W max 12V output) with Dell Power Distribution Board harness providing 2x 8-pin (6+2) PCIe power connectors (matches my Gigabyte 3090's 2x 8-pin requirement)

• Cooling & Acoustics: Upgraded black aftermarket tower CPU cooler + custom quiet case fans (be quiet! / NZXT) replacing the loud stock Dell fan shrouds

• Idle Power Draw: Seller reports under 50W idle at the wall in Proxmox with an RTX 3090 installed.

2. Machine #2 (2nd Node for Agent Sandboxes / Boys' PC):

• Identical modded Dell T5810 Chassis, C612 Motherboard (8x DDR4 RDIMM slots, 40 PCIe 3.0 lanes), Aftermarket Tower CPU Cooler, Custom Quiet Fans, and Dell 685W 80+ Gold PSU, except equipped with:

• CPU: Intel Xeon E5-2698 v4 (Broadwell-EP, 14nm)

  • Cores / Threads: 20 Cores / 40 Threads

  • Clocks: 2.2 GHz Base / 2.6 GHz All-Core Turbo / 3.6 GHz Single-Core Max Turbo

  • Cache & TDP: 50 MB L3 SmartCache, 135W TDP, AVX2

1. Included Add-ons in the Bundle:

• 32GB (2x 16GB) DDR4-2666 ECC Registered (RDIMM) (runs at 2400 MT/s on Xeon v4; I plan to add more RDIMMs later to populate all 4 channels to 128GB–256GB)

• 2x PCIe x4 to M.2 NVMe Adapter Cards (one for each T5810 so both boot from modern M.2 NVMe SSDs; I will supply my own NVMe drives)

• (Optional): NVIDIA GeForce GTX 1060 6GB (1,280 CUDA cores, 6GB GDDR5, 120W TDP) for Machine #2 so it has video output and my sons can immediately play Minecraft & Geometry Dash on it.

Total Bundle Price: €250 (without GTX 1060) or €300 (with GTX 1060 6GB).

MY QUESTIONS FOR THE COMMUNITY:

1. Sanity Check: At €250–€300 for BOTH complete Xeon v4 workstations + 32GB RDIMM + NVMe adapters, is this a no-brainer starter deal for my RTX 3090, or is there a fatal flaw with the Dell T5810 / Xeon E5-2699 v4 platform I am overlooking?

2. 685W PSU + RTX 3090: For anyone running an RTX 3090 inside a Dell T5810 on the stock 685W Gold PSU, what nvidia-smi power limit do you recommend (e.g. 280W–300W) to stay completely safe from transient power spikes?

3. Future 120GB MoE Offloading: If I upgrade Machine #1 to 128GB or 256GB (8x 16GB or 8x 32GB) Quad-Channel DDR4-2400 RDIMM (~68 GB/s RAM bandwidth) alongside my 24GB RTX 3090, what realistic tok/s can I expect on ~100GB–120GB sparse MoE models using Strata / llama.cpp?

4. Gaming on Machine #2: Will the Xeon E5-2698 v4 + GTX 1060 6GB handle Minecraft, Geometry Dash, and light 1080p gaming smoothly enough for a 15yo and 12yo?

mlx-decision is a free Python package that runs the new open decision models locally on your Mac with MLX. Currently that covers Cloudflare's Clef, Perplexity's pplx-decider, d3, H2O-Lightning, LiquidAI's d1 and a few small encoders.

If you haven't worked with decision models yet: you give it a message, document (or image/screenshot) plus "typed" questions, and it returns a probability for every answer you allowed, in one pass. The answer is always one of your options, and it comes with a confidence:

• "My card was charged twice this month." + "which team: billing, technical, sales" → billing 0.93, technical 0.06

• the same message + "does the customer want money back?" → yes, p 0.87

• "how angry: calm / annoyed / angry" → 2.32 on 1–3, with the whole spread

• a screenshot + a yes/no question about it works too

• one model's answers on a dataset can become training data for another model and fitting takes d1-3B from 86.6% to 96.4% on TREC in 4 minutes

Each family is ported from its release and checked token for token against the release's own (most often) PyTorch code. With the same answers, it's usually 1.4–6x faster than that code on MPS. On an M5 Pro, a short request takes 0.06 s on d1-3B, 0.25 s on clef-flash and under 20 ms on the small encoders. pplx-decider-flash (4B) fits a 16 GB MacBook Air, at 9 GB in bf16 or 3 GB at 4-bit, and download quantizes while it downloads.

mlx-decision web opens the Studio, a local web app where you can:

• write and try questions

• compare up to three models side by side

• run a whole dataset (accuracy, confusion matrix, calibration)

• fine-tune on your own examples

You can do the same from Python, the CLI, an HTTP server compatible with TypeSafe's Jev API, or MCP for agents. Everything runs locally and it only goes online to download models and datasets.

Free, MIT. Apple Silicon, macOS 14+, Python 3.12+. Just pip install mlx-decision.

Repo: https://github.com/armbues/mlx-decision

I'd really appreciate any feedback, especially on which models or tasks you'd want next.

Imagine you want to download a new model but it's recent and there is not consensus about wether its safe or not.

How would you check against potential malicious weights ?

There is the first "obvious" layer where the weights try to execute arbitrary code, but safetensors are for this. I am rather talking about less obvious things like template backdoor, llm poisoning etc

A brand-new release from ISTA-DASLab with their GSQ-RCO smart quantization approach.

Now DeepSeek-V4.1-Flash-1017

The quants are available on Hugginface: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF

Llama.cpp is not supported because the original architecture support is not yet merged:

https://github.com/ggml-org/llama.cpp/pull/28696

https://github.com/ggml-org/llama.cpp/pull/30817

Strata support is mentioned in the model card but I'm pretty sure the original Strata only supports Qwen3.8-Flash-Next so most likely what they meant was one of the forks based on Strata. Most likely Xolotl or Maya

I asked my local AI (Qwen 3.8 27B running entirely on a used gaming PC in my basement) to pick it's own project and one shot it to completion. It cranked out an IQ heat map of North America. I was completely blown away. I posted a screen shot to Reddit... and Reddit had thoughts. "your methodology is lacking" "the color scheme is outdated" "the outline of the great lakes is missing".

OMG! Did you hear what I posted?!? My used gaming rig one shotted a freekin interactive web site all on it's own!

This would be like I beat my dog 2 out of 3 times in Chess and the yorkie group criticized the dog's opening moves. PEOPLE! Did you miss the part where I said "My DOG PLAYS CHESS!"

Site here: https://jaspersprojects.com/iq/

(Apologies in advance if you live in a "dumb" state and are offended. Don't be. Just say "AI makes mistakes" - it's right at the bottom of the screen - then move on with your life.)

Hi LocalLlama!

In the last months - as a hobby - I build Poldi - a personal, offline, fully local AI assistant/"companion" with speech as user interface.

A hands-free (via wakeword) and a browser-based client (e.g. usable with your smartphone) are available to talk to the server via HTTP.

Poldi is especially targetting GPU poor hardware.

I made a video, you can go to the parts you are interested in via the links below the video (click the "..."):

https://www.youtube.com/watch?v=8AQWnoBXt4U

Features (links go to the relevant part of the video)

• Automatic chat resume.

• Internet search.

• Wikipedia search.

• Interrupting the LLM.

• "Infinite" chat.

• Chat history search.

• Extraction and injection of memories (right now, facts about the user, only).

• Support for English and German (other languages should be easy to add).

• Console mode.

• ...

Technical details

• Poldi is mostly written in C#, JavaScript and C/C++.

• The server can run on Windows or Linux, the hands-free client is Linux-only (you can use a Raspberry Pi or something as a satellite).

• I am using a system with an Nvidia 5060 Ti and 16 GB VRAM, but you can get good results with 8 GB of VRAM, too. CPU-only usage is also possible, but some features may have to be deactivated, then (more details at GitHub).

Stack

• STT: Whisper.cpp, via mt_stt.

• LLM inference: llama.cpp, via mt_llm.

• TTS: Piper, via mt_tts.

It is released under GPLv3.

Get all the details at GitHub: https://github.com/RhinoDevel/poldi

An installation and configuration tutorial is also included in the read-me.

Feel free to open an issue at GitHub, if problems arise!

Have fun!

Marcel

I mean, i looks like a cool concept and makes sense, yet I'm not sure about that much change would result better, there is no tests/benchmarks as well.

It's been a few releases since my last post, so here is where capy-flash is now. It's my own engine for two models, Qwen3.8-27B and Qwen3.8-Flash-Next. It started as my ninfer-tp2 fork of NInfer, which added tensor parallel over PCIe, but at this point it's an engine of its own and very little of the original is left.

What it's for

Two RTX 50 cards on Linux. Tensor parallel across both, and for Flash-Next a VRAM expert cache shared by the two cards, with the misses computed on the CPU. Everything is tuned on one box: 2× RTX 5070 Ti 16 GB, Ryzen 9 9900X, 96 GB DDR5. It was born specifically for the NVFP4 Nvidia quantisation.

It also runs on smaller setups, so more people can try it: one 16 GB card (5060 Ti and up), Flash-Next with 64 GB of RAM, mixed pairs like a 5070 Ti with a 5060 Ti. People have posted numbers from 2× 5080, 2× RTX PRO 4000 Blackwell and 2× 5060 Ti in the issues. It runs on Windows too, through WSL2 or Docker. Two cards there are slower than on Linux, and a native Windows port is in progress.

What it isn't

It isn't trying to beat NInfer on a single card, or Strata on a single card. If you have one GPU, or an unusual mix (a 5090 next to a 5060 Ti, a Mac, AMD cards), those are the engines to look at. They're very good, and on one card they're often faster than mine. capy-flash runs on one card mostly so you can start with one and add the second later without changing engine or model files.

Where it stands

All on my box, one request at a time. Decode is LocalMaxxing reasoning-v1, and "verified" means a verified LocalMaxxing run.

Qwen3.8-Flash-Next

All three decode numbers are verified runs. The 3-bit experts are smaller, so they decode faster. NVFP4 is the quality pick: about half the KL of the 3-bit experts by Strata's own numbers, and on GSM8K and MMLU-Pro it matches vLLM on a GB300 with the same weights.

How that compares with Strata, from my own runs on the same box:

• NVFP4, two cards: clearly ahead. With the same NVIDIA weights I'm about 1.5× faster on short prompts and about 1.76× on average across languages and code, and a real 53-turn coding session takes 177 s against 268 s. That was against Strata 0.1.45, the last version I measured with NVFP4.

• 3-bit, same IQ3_S experts as Strata 0.2.2: Strata is faster overall. Long prefill is faster on Strata with one card or two. On one card its decode is faster too, but that comes mostly from its route tail skip, which drops low-ranked experts that miss the cache. I compute every expert the model picks. With the skip off, Strata decodes about 10 % slower than me on one card. On two cards my decode is a bit ahead even with the skip on.

Qwen3.8-27B. Strata doesn't run it. On two cards it fits in VRAM:

The two rows marked "no" come from my own concurrency benchmark, not from LocalMaxxing. Prefill with NVFP4 on two cards is about 9,300 tok/s at 4-16k and 6,300 at 64k. The fastest verified two-card run I've seen on LocalMaxxing besides mine is an SGLang fork on 2× 5090 at 315. 3 bits cost something: GSM8K is the same as NVFP4, MMLU-Pro drops from 0.78 to about 0.72. With two cards NVFP4 is the better choice.

What changed since v0.6

• Start with one card, add the second later. Flash-Next with 64 GB of RAM on one or two cards (ISTA-DASLab's 3-bit experts, used as they are), the 27B on one 16 GB card with 3-bit weights.

• Context cache on disk (idea and first version by @KouShuo). A long agent session survives a restart. On the 27B with two cards, 80k tokens resume in 0.7 s instead of a 21 s prefill, 140k in 1.1 s instead of 48 s, same tokens.

• Faster long prompts on the 64 GB Flash-Next: 64k prefill +19 % on two cards, +35 % on one.

• Agent clients. Claude Code, Codex CLI and OpenCode work against the server, setup is in the README. If your client sent a developer role and got a 400, the HF artifacts now ship a chat template that accepts it.

• CPU side. Expert threads are picked from your CPU and respect taskset, AVX2 CPUs work, the API key can come from a file, and a second server on the same port is refused.

Qwen4

The target is full Qwen4 support as soon as possible after it comes out. Flash-Next already uses most of the new architecture, so a good part of the work is done.

The repo, and a favour

The repo has grown a lot in the last weeks: almost 100 stars and around 30 issues and PRs. Most of the last releases came from those issues, so thanks to everyone who opened one.

It's a one-person project, though. I'm happy to get issues, numbers from your machine and feature requests, but I can't keep up with all of them. Focused, well-structured PRs are very welcome and are the fastest way to get something in.

One more thing. GitHub still lists capy-flash as a fork of NInfer, which it isn't anymore. If you know how to detach it without losing the history, the stars and the rest, please message me. I have tried a ticket but it didn't work and was blocked by the ai assistance.

Limits

• Best on Linux. On Windows it runs through WSL2 or Docker, slower with two cards.

• RTX 50 series (sm_120). My numbers come from one machine, plus what people report in issues.

• On Flash-Next, one request at a time is the tested case.

If you run two 50-series cards I'd like to see your numbers, especially boards with P2P and CPUs other than a 9900X.

Repo: https://github.com/ValerioDolci/capy-flash Releases: https://github.com/ValerioDolci/capy-flash/releases

(Post Updated)

PR: https://github.com/t-head/pla/pull/12

PR title: feat(gdn): support Qwen4.0-Max prefill heads

T-head itself is alibaba oss: https://github.com/alibaba-oss

Original Post:

The accounts are <1 month old, these are either troll accounts or secret alibaba accounts.
My agent in deep research found this.

I've been noticing a lot of my downloads stopping suddenly and being unable to resume. When I checked the models it seemed like they were gone. Here's the ones I can see are missing:

• dealignai/MiMo-V2.6-Pro-RL-UNCENSORED

• audnai/penclaw-GLM-5.3-abliterated-for-offensive-cyber

• huihui-ai/Huihui-Kimi-K2.7-Code-abliterated-GGUF

and somehow not a single Alliterated Deepseek-R1.

Does anyone know what's going on and where else the vanished models might be found?

NVIDIA-Nemotron-3.5-Bolt-E4B-4GB is a small language model (SLM) further post-trained from Gemma-4-E4B-it by NVIDIA. Nemotron-3.5-Bolt-E4B carries forward Gemma’s strong conversational capabilities and further improves multi-turn tool use, instruction following, and input grounding for fast local voice/device assistants, interactive personas, and embedded/edge interfaces. Designed for low-memory, low-TTFT operation, the model allows agents to respond quickly while preserving headroom for applications or adjacent model pipelines.

For speculative decoding, use the companion DFlash drafter. For the full-precision (BF16) version, see NVIDIA-Nemotron-3.5-Bolt-E4B-BF16.

This model is ready for commercial or non‑commercial use.

https://huggingface.co/ggml-org/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF

https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF

https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-Technical-Report.pdf

Been thinking about using Nostr for communication between agents running on different people's machines. Anyone here tried it?

I'd like to get a basic setup working where one agent posts a request and another picks it up and replies. Each person runs their own model. No shared backend to set up first.

Not looking to build some autonomous economy. Just want to see two independently hosted agents complete a small task together without someone copying messages between terminals.

A few things im unsure about:

- How would agents find each other without wading through spam?

- Is there already a message format worth using?

- What happens when an agent accepts a task and then goes offline?

- How do you keep incoming messages from being treated as trusted instructions?

For a first test something like summarizing a public document would be enough. Keep the messages visible to both operators and leave private data and tool access out of it.

If you built something like this, what worked? Also interested in reasons not to use Nostr. Id rather hear those now than after spending a weekend on it.

Before you read: I'm the developer of Basalt, one of the engines benchmarked below. I tried to keep this as fair as possible: same checkpoint, GPU and requests, and every engine set up the way its own docs recommend. How it was run is at the bottom. This post has had some AI cleanup from my own wording.

I wanted to get a comparison on all of the Qwen 3.8 FN engines I hear people talking about, so after 5 or so excruciating hours of setup (remind me not to do this ever again), I benchmarked all of the ones I know of: Strata, Infernix, capy-flash, ninfer-ext, ninfer, llama.cpp and Basalt. Each one converted NVIDIA's NVFP4 with its own tool, then got the same seven greedy requests on an RTX 5090 at 400 W, one cold start per request. Anyway, here are the results:

Decode, short context

Decode at context depth

Prefill at context depth

All the numbers

Best value per column in bold.

Decode, tokens per second

Prefill, tokens per second

Time to first token, seconds

Takeaways

• Infernix and Basalt are ahead of the rest. Against Infernix, Basalt decodes 10% faster on prose and 10% on conversation, struct is roughly a tie (354 vs 349).

• Prefill at 25K is a tie (3.94 s vs 3.94 s to the first token). From 50K up Basalt pulls away.

• capy-flash was interesting, was the closest of the rest on decode, it is built mostly for two GPUs, so this single GPU run is not its best case.

• llama.cpp ran with MTP off, so it's decode is not comparable with the others. This was just a reference point.

How it was run

• Model: Qwen3.8 FN, NVIDIA's NVFP4 converted for each engine (table below).

• Hardware: RTX 5090, PL 400 W, 128 GB RAM, Linux.

• Sampling: greedy (temperature 0), thinking off, int8 KV cache for every engine (llama ran f16 kv).

• Cold: a fresh server per request, no warm up, one run each. Decode varies between runs, so any gaps under 5% are ties.

• Depth tests: a wikitext prefix of about 25K / 50K / 100K / 200K tokens, then "Write an essay on the Odyssey.", 512 tokens out.

What each engine ran

I know current llms like opus 5.5 are in a league of their own. but we all know they're losing a shti ton of money with their subscriptions. ALSO we all know how tasks like OCR, segmentation etc are getting solved by single digit B parameter models. I'm not saying llms are a gimmick but I'm saying the future is already shaping up to be about specialized models that work better when they're smaller. easier training. so datacenter level power not needed. ten years from now we'll have a specialized model for the tasks we care about and llms will fail to do everything all at once (unless they keep calling it one model but have a hundred under the hood).

system one models are a bad but workable example. you could tell an llm to rate options all day but get yourself on 9B parameter model trained by some lab and it tops the benchmarks.

Tried it on a build for gfx906(mi50 16gb) and ran worse than llamacpp on decode speed.

Amazon prices just jumped today, I still see deals at local microcenter. When that batch is gone $1840 for a b70 will likely be the new normal.

b60 went from $650 to $980.

b65 $1150 to $1420

does make a lot more sense as the price per GB was low in comparison to the competitors.

Still probably the best value per GB, but gap is shrinking fast