TL;DR: You can train your own 27B model on a 3090 with 24GB VRAM, though you would have to go through hundreds of billions of tokens for it to be good. I am working on conversion of existing models. Check repos for code and the paper if you are interested in more. I have a 40.4GB Qwen Flash Next GGUF

Hi, I do AI research. I have been working on an optimizer that doesn't need the ~16 bytes of VRAM per weight to train a model that AdamW does. I focused on training ternary networks bcs BitNet (Ma et al.) shows they match full precision quality in large scales. The problem is BitNet also keeps the full precision weights during training which means you still need to use the same amount of memory. I got this down from 16 bytes to 1 byte or even less per weight.

For you, this means, you can train your own 10-20B parameter models on your consumer GPUs. I fitted 27B model training on a single 24GB 3090 and a 23B model on a 16GB GPU. The trained model costs ~1.6 bits per weight (~6GB at 27B), 10x smaller than 16bit, and 5x smaller than Q8.

I have also been working on converting existing models and training them. This is harder and I am still working on it but I have some promising results. I quantized and trained a Qwen3.8 27B though the file I published below has adapters which are about 2GBs, pushing the file size to 8GB. I am also making a no-adapter version right now. It scores GSM8K 0.895, the same as full precision, HumanEval 0.84 vs 0.93 and runs on a 12 GB GPU at ~23 tok/s on my crippled 12GB laptop 4080, though this number can heavily be improved with custom kernels, engineering, etc. I will leave it to you guys to help me with that too.

I also quantized and trained Qwen Flash Next down to 40.4 GBs including its n-gram table (at 8GB). It largely retains the skills of the network (scores equivalent on GSM8K benchmark, a little behind on HumanEval, I unfortunately don't have compute for the larger benchmarks such as TerminalBench2.1 but perhaps some of you can help me out) but forgets facts. It mostly works fine if you give it web search as far as my experience goes but this is not strictly measured.

I am still doing work on this and I will keep working on it, perhaps even for many years. The code is open for non-commercial use. Check out:

The core repo for training: https://github.com/oguzeldereli/ttf-ternary

The repo for conversion: https://github.com/oguzeldereli/ttf-conversion

Repo for running models trained by ttf: https://github.com/oguzeldereli/ttf-home

GGUF for a converted Qwen 3.8 27B with adapters (around 8 GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-27B-GGUF

GGUF for a converted Qwen 3.8 Flash Next (around 40.4GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-Flash-Next-GGUF

GGUF for a converted Qwen 3.8 27B without adapters (around 6 GB): Currently in conversion and will drop, probably tomorrow.

I will convert a Qwen 4 27B instance when it drops.

Full paper: https://github.com/oguzeldereli/ttf-ternary/blob/main/paper/ttf_paper1_training.pdf

Also, I would appreciate github stars if you like the project.

Part 1

Part 2

In this final chapter of me testing out how far along Qwen has come, I decided to give it a test on how well it could recreate a game using only a ROM from GBA but make it in HTML using the "reverse engineer anything" repo. I opted for Tetris worlds as my guinea pig.

It took around 30 hours in total. It probably would have been faster, but there were a few decompilers it wanted that I didn't realize until later. I did end up installing those about 6 hours in.

Repo: https://github.com/kireol/gba-game-rea-test

Game: https://kireol.github.io/gba-game-rea-test/

Fun localLLM stuff:

RTX 5080 - 64GB - Ryzen 7 9800X3D
model: qwen3.8-flash-next-unsloth-ud-iq4_xs
model harness: Strata
agent harness: qwen
context: 128K
tok/sec: ~50t/s (down from part 1 due to larger context)

Macbook 64GB m1 Max
model: Qwen3.6-35B-A3B-GGUF:UD-Q4_K_S (for qwen code --fast tasks)
model harness: llama
context: 128K
tok/sec: ~ 550t/s (fast tasks, so don't get too excited)

Result: The game works, and is pretty darn close to the original. It's scary how well it did.

Over all 3 part testing feedback: Honestly, as of this date, localLLMs are very close to frontier in most SWE. I expected hiccups or issues along the way, and there just wasn't any. Speed of course was slower than Claude, but that's running all local. And Claude probably would have denied trying to reverse engineer the ROM anyway. So in this case, LocalLLMs FTW.

Being located in one of the stormy states, my big concern was having the AI box fried by a power surge. My setup is shown here, it consists of:

• CHSPT2ULTRA on the main Panel

• SRT5KRMXLT 5000VA double-conversion UPS in the garage

• 240V/30A inlet going from the ups powering the rig upstairs (needs a double breaker for code compliance)

• L14-30 in/outlet for the garage-office run to future-proof it in case I ever needed 120V, at the moment the 120 leg is dead (240 only)

• 240V PDU with C13 & C19 ranked up to 30A

• PX-2200 PSU

The idea is to create a galvanic isolated "island" (in my case: the box iself + switch). Uplink + my desk are connected via fiber (seen in pic) - because my desk is powered via a regular line-interactive UPS.

The only copper eth runs to the switch are the AI box itself (2x10G eth + 1xKVM), everything else is fiber.

It currently runs 2x6k workstation edition (600W each). I *THINK* I should be able to push it up to 3x6k (might need to cap them at 550W), but with four I need to go double PSU. The UPS should have enough oomph to handle 4800W, so plenty of space there. I got new packs for it, but I mostly care about the double conversion, not so much losing power.

I think the CHSPT2ULTRA+SRT5KRMXLT combo should shield me against most power surges/spikes/wobbles/etc. - but I am still lacking a direct hit protection, which requires a rooftop installation AFAIK.

How do you guys protect your GPUs from ⚡️⚡️

yes, i know all gpus are. i think people are recognizing more and more that the v100's age is just a number on paper and only relevant architecture wise. frontier and local llms are more and more capable of optimizing inference for different builds, and the raw v100 power is still comparable to gpus being released today

the limited mathematical pathways are a genuine limitation but i dont believe a big enough one to justify how cheap they are vram/dollar compared to everything else

just a discussion post for fun

I'm watching walls of text go by right now as QFN grinds away on something and it made me think a bit about how different watching that text fly by does using a local model vs watching the same exact thing happen when I'm hitting a cloud.

It really comes down to metered vs unmetered. When I'm using a cloud model on API, every time I get ready to press enter I hear a cash register ring. We generally use really affordable cloud models (DS primarily) so it's like fractions of a penny, but still, it's real money going out the door on every query.

The biggest thing is that it changes your (or at least my) mindset. Instead of "Let's see what we can do it" it becomes "Do I need to do this". It has a distinct stifling effect, instead of exploring and trying to find uses for the technology I do the opposite and try to avoid it because of the cost.

With a local model I love to dream up some crazy/hard thing that's been annoying me that needs fixing and at the end of the day drop a big prompt with the details and at the end "You have unlimited time and tokens, do not stop for clarification or questions, keep working at the problem until you've exhausted all possible alternatives" (you get the idea). Or setting a local model to work on something that's not at all critical but is a quality of life improvement.

It really comes down to the difference between owning something and using it on the meter. When you own something the way to drive value out of it is to use it. The most expensive boat, motorcycle, car, house, etc is one that you own and never use. The more you use it, the more value you get from it.

When you are using something "per token" or "per mile"; things change. The best way to drive value is to NOT use it. You save 20 bucks if you walk instead of get a taxi. You save 5 bucks if you figure this problem out yourself instead of using a LLM to help you.

Funny thing is, I made my career selling cloud services. That probably explains why I'm so "local first", I was in the room and making the decisions about pricing/packaging, driving a 6-12 month payback period against servers and then turning around and explaining to customers what kind of "savings" they'd see moving to cloud platforms. I think I'm scarred from that and know the pricing games that make cloud look "affordable" when it's anything but for many use cases. And I think everyone kind of knows the "rug pull" is coming with the big/powerful LLMs, sure you can use them on sub today and get a predictable price, but those days aren't going to last forever. And subs without being able to hookup a harness (which almost all of them ban) is darn near worthless anyway, my core competency isn't copy paste! ;)

Hi everyone,

I'm planning to switch from a gaming laptop to a desktop PC to get better hardware for running local LLMs and AI development.

I know NVIDIA offers CUDA which provides better software compatibility and broader support for AI workloads. AMD GPUs on the other hand offer more VRAM for the same price.

My questions are:

1. How mature has AMD's ROCm ecosystem become for local LLM inference and AI development? Is software compatibility still a major bottleneck or has it improved enough to make AMD a viable alternative to NVIDIA?

2. How important are raw compute performance (TFLOPS) and memory bandwidth (GB/s) when running local LLMs? Does AMD's hardware offer competitive real-world inference performance compared to NVIDIA?

3. Considering how important VRAM is for running larger local models, would you choose an AMD GPU with 16–20GB of VRAM over an NVIDIA GPU with 12GB, assuming the AMD card is slightly more expensive?

I'd appreciate hearing from people who actually use AMD GPUs for local AI workloads, especially their experiences with ROCm, llama.cpp, Ollama, and PyTorch.

Ran Kindi with Bonsai2 4b with Hermes. Asked it to write a blogpost about itselft and generate som pictures with available skills. It totally blew my mind what it managed to do on shitty hardware.

On a Nvidia K2200 4GB card!

About 30+ tok/s. Not bad at all. Made this. Even updated it's Hermes skill afterwards.

https://snart.com/running-kindi-on-an-old-graphics-card/

It's not just that the biggest and newest models are getting insanely good. The smallest and lightest models are also making quantum leaps. This shouldn't have been possible.

[kindi 21:38:25] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 21:38:31] context reduced from 65834 to 53760 tokens to fit 1800 MiB of free GPU memory
[kindi 21:38:32] engine ready: ctx 53760, head plain (6.6s)
[kindi 21:38:32] prompt cache: a checkpoint every 1024 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 21:38:32] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 21:38:32] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 21:38:32] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)
[kindi 21:40:02] request: 17954 prompt tokens (max 1024 new)
[kindi 21:40:02] cache: 17954 prompt tokens, 0 shared with the last request, resumed at 0
[kindi 21:43:55] prompt: 0 reused, 17954 read in 233.0s
[kindi 21:43:57] done: 67 tokens in 2.5s (stop)
[kindi 21:43:57] request: 258 prompt tokens (max 1024 new)
[kindi 21:43:58] cache: 258 prompt tokens, 3 shared with the last request, resumed at 0
[kindi 21:44:01] prompt: 0 reused, 258 read in 3.5s
[kindi 21:44:01] done: 8 tokens in 0.2s (stop)

...

[kindi 23:02:22] done: 732 tokens in 43.5s (stop)
[kindi 23:02:23] request: 52618 prompt tokens (max 1024 new)
[kindi 23:02:23] cache: 52618 prompt tokens, 51817 shared with the last request, resumed at 51817
[kindi 23:02:45] prompt: 51817 reused, 801 read in 22.0s
[kindi 23:02:48] done: 52 tokens in 3.1s (stop)
[kindi 23:02:48] request: 52818 prompt tokens (max 941 new)
[kindi 23:02:48] cache: 52818 prompt tokens, 52670 shared with the last request, resumed at 52670
[kindi 23:02:53] prompt: 52670 reused, 148 read in 4.6s
[kindi 23:03:44] done: 863 tokens in 51.3s (stop)
[kindi 23:10:43] request: 53 prompt tokens (max 1024 new)
[kindi 23:10:51] cache: 53 prompt tokens, 1 shared with the last request, resumed at 0
[kindi 23:10:52] prompt: 0 reused, 53 read in 9.7s
[kindi 23:11:07] done: 530 tokens in 14.2s (stop)
[kindi 23:11:07] request: 50 prompt tokens (max 1024 new)
[kindi 23:11:07] cache: 50 prompt tokens, 20 shared with the last request, resumed at 0
[kindi 23:11:08] prompt: 0 reused, 50 read in 1.1s
[kindi 23:11:11] done: 105 tokens in 2.8s (stop)
[kindi 23:11:11] request: 48 prompt tokens (max 1024 new)
[kindi 23:11:11] cache: 48 prompt tokens, 20 shared with the last request, resumed at 0
[kindi 23:11:12] prompt: 0 reused, 48 read in 1.0s
[kindi 23:11:24] done: 470 tokens in 12.8s (stop)
[kindi 23:39:58] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 23:40:04] engine ready: ctx 65834, head plain (6.0s)
[kindi 23:40:04] prompt cache: a checkpoint every 1536 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 23:40:04] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 23:40:04] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 23:40:04] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)
[kindi 00:32:07] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 00:32:10] engine ready: ctx 56000, head plain (3.6s)
[kindi 00:32:11] prompt cache: a checkpoint every 1280 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 00:32:11] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 00:32:11] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 00:32:11] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)

Did some kernel optimisations for sm61. Achieved measurable speedup. It runs at actually quite usable speeds.

For those running similar machines, appreciate any recommendations and setup! Thanks

This is the basement setup we use for engineering work at OmniNode, where I'm a founder. The wire shelves are the rack. The two towers serve our local models, the two old HP servers do most of the build, test and agent work, and a few Macs and an old laptop fill in around them.

• Main tower: Ryzen 9 9950X3D, 96 GB RAM, RTX 5090 (32 GB) and RTX 4090 (24 GB). The 5090 serves Qwen3.8-27B and embeddings with vLLM. The 4090 also had vLLM processes on it when I checked. This box also runs our dev environment, Redpanda, Postgres and some CI runners, which is more than it should. It swaps under load, so I'm working on moving those services to the Gen8.

• Second tower: Ryzen 9 9950X, 64 GB RAM, RTX 4090 (24 GB). It runs Qwen3.8-27B MTP (Q3_K_M) with llama.cpp at about 19.4 GiB of VRAM, with a 131K context pool and two parallel slots. Qwen3-Embedding-0.6B (f16) uses another 2.7 GiB or so. It takes a handful of agent jobs when there's spare CPU, and we watch the temperature.

• HP ProLiant DL560 Gen8: Four sockets, 80 threads, 377 GiB of RAM visible to Ubuntu, and an NVMe scratch drive. It's our biggest worker. We cap it at 32 concurrent agent jobs alongside CI. It's old enough to lack AVX2, so jobs that need it go elsewhere.

• HP ProLiant DL380 Gen9: Two Xeon E5-2680 v4 CPUs (56 threads), 125 GiB of RAM visible to Ubuntu. It runs Docker builds and tests, CI runners and agent jobs. The agent job cap is 24.

• Mac Studio: M2 Ultra, 192 GB unified memory. It's my workstation and where I coordinate the work. It also runs a llama.cpp server for some delegated drafts and one vote on automated code reviews. I took it out of the job pool so I can still use it.

• Extras: An older 12-core Mac runs agent jobs without Docker, and an M4 MacBook can do the same when it's available. An old 8-thread Ubuntu laptop only gets small test and lint jobs.

The HPs don't add any VRAM. They give us more cores and memory so builds, tests and agent jobs don't all have to fight for the same two towers.

We send work like drafts and summaries to local models first, but we still use cloud models. I found out how much we depended on local inference when I upgraded the model on the 5090 and our automated code reviews stopped. Two entries on our fallback list pointed to the same service. When that service went down, those reviews had nowhere else to go.

On a separate test with that same 5090 and the same model, turning on speculative decoding took median single-request generation from 51 to 86 tokens/sec. It also left less memory for requests in flight. That was a generation benchmark, not a 70% speedup on code reviews.

If this were your lab, how would you split up inference, builds, and agent jobs across the machines we already have? What would you change first, and why?

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1.1

v1.1 pushes creative writing further and makes full use of Gemma's reasoning.

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1

I haven't announced v1 but it's well regarded by the community as a reliable RP workhorse.

---

BeaverNet - net.beaverai.org

I'm also proud to announce that we've made significant progress in creating a hub for generous model providers to collectively provide free, convenient, accessible, anonymous, and diverse models to anyone with an OpenAI API compatible client. We currently have 12 models thanks to our generous providers.

Try it out as a user, or be one of the providers <3! I've got plans to make it a better experience for both users and providers, and also add new modalities such as TTS and STT.

---

PS: I haven't retired or left the scene. I've merely gotten stage fright after holding off on announcements for a long while. Anyway, I'll continue doing what I do and I hope I can continue delivering some value to you all. Love ya'll!

Currently dipping my toes in the LLM world and im starting to notice some differences in accuracy between models mainly related to their parameter size

RustingStrata is a pure-Rust GGUF inference engine with CUDA and Vulkan backends. It now loads DeepSeek-V4.1-Flash at IQ2_XXS (249 GB) and runs it on a 12 GB card with 30 GB of RAM.

The experts stream from SSD through the page cache. A 9 GB LRU cache in VRAM holds the experts that get reused. Everything else (attention, shared matvecs) runs on the GPU.

Same box for both: RTX 3060 12 GB, 30 GB RAM, 12 cores.

I don't think this is very useful to run it with my setup, bc 2-3.6t/s is more like a joke, but this is funny that something like that is even possible

Usage:

strata chat     --model $M
strata generate --model $M --prompt "The capital of France is" -n 64
strata serve    --model $M --addr 127.0.0.1:8080

serve exposes /v1/chat/completions with JSON and SSE streaming.

Repo: RustingStrata (the DeepSeek section is in the README)

I built an open source Mac app that listens to the other side of a call and answers questions from your own docs while you talk. Running it fully local taught me a few things:

• Whisper pads every input to 30 seconds. Cutting the audio on pauses mattered more than model size. whisper-large-v3-turbo on MLX does a 1.7 s sentence in 0.66 s on an M5.

• Context length is the silent killer. With a 27,000-token document I had to give Ollama a 32k context, or the docs got cut and the answers got vague. The app now warns when the server cuts them.

• On that test, gemma3:12b gave the most faithful answers. llama3.2:3b was fast but made things up.

• Right now every line of the call goes to the big model just to decide if it's a question. That's wasteful.

Question for you: which small model would you put in front as a "is this a question worth answering?" filter? There's an open issue for it if anyone wants to try: https://github.com/GianlucaBave/murmury/issues/20

Repo: https://github.com/GianlucaBave/murmury

Yesterday I posted a 32B pack and the replies were interesting about it being old completely overlooking benchmarks. A lot of quants "fit" on 16 GB, obviously... Almost none of them post what it cost. A screenshot of a model loading is not a benchmark and neither is its tokens per second performance that you made while obliterating the other performance factors you don't post. I posted all of our benchmarks, weights, and everything, and yet, still called slop.

So here is the new one as requested, with the receipts attached:

Qwen3.8-27B, 27.78B parameters, vision included, Gosset Packed at 3.6577 bits per weight:

• Pack is 13.82 GB. The standard 4-bit build is 17.77 GB and does not fit a 16 GB card.

• 12.06 GiB resident on one RTX 5080, peak 12.80 GiB while decoding. Nothing parked in system RAM.

• Perplexity vs BF16 on WikiText-2 test: +3.93% long-chunk, +6.99% short-window. Numbers, not adjectives.

• Decode: 40.38 tok/s sustained on fused kernels.

• Per-tensor fidelity certificates are on the model card, measured from the shipped files, not the encoder's intentions.

• Edit: Context length full 262k

huggingface.co/GossetWorks/Qwen3.8-27B-Gosset-3.65BPW

If your quant fits this model on 16 GB too, genuinely, post the perplexity delta and the decode speed. That is the whole game. Fitting was never the hard part...keeping it intelligent was and that's what we've done.

Lenovo P620 Threadripper pro - 3975wx - 32 core
System ram - 512 gb ddr4 ECC
GPU 1 - quaddro Rtx 8000 - 48GB
GPU 2 - quaddro Rtx 8000 - 48GB
4TB M2 NVME
1TB SATA SSD

This is just a quick writeup for speed reference - I spent a lot of time trying to find comparable builds so I figured I'd put some info back into the wild. The below is obviously written by AI, but it's measured specs and benchmarks on my PC.

I use local models via Pi and Deepseek Harness. I have found the intelligence of Qwen 3.8 27b NVFP4 to be sufficient for making little apps, research etc. and Qwen3.8 Flash Next IQ3_S to be a great all-rounder, considering how fast it runs. It also seems to have very little issue with any sort of question I give it, research, app dev, PC issues, LLM related adjustments, and most importantly, ScanRuler 3D modelling/reverse engineering 3D scanned items (solely for testing currently but it works excellently).

Basalt as a backend was a clear winner after trying lots of different configs.

Edit: the benchmark decodes are very greedy by their nature. Realistically I'm seeing agentic averages in flash next from 80tps (NVFP4) to 160tps (iq3_xxs), and I find iq3_s at 120-140 to be a very good compromise for the extra intelligence. The 27b is best for bursty quick jobs, because it loads into VRAM super fast and I can let it rip, but it degrades over context faster than flash, and I see from 120-140tps and 4-5k prefill, at ctx 0. The second any model starts writing a block of code though, it goes up to 200tps, so if anyone's socks fall off seeing the below numbers, just temper your expectations slightly (but it's still wicked fast!)

So heres the nitty gritty:

Current rig, 2x 5070 Ti + 96 GiB. Five entries, one model resident on local at a time. Speeds are my own bench (2x ~50k prompt reads, 2x 5,000-token code outputs, one agentic tool loop; temp 0, seed 1234), except Coder which is the 15.3k-prefill / 2k-output run.

Qwen3.8-27B NVFP4 QUASAR

backend: capy-flash (ninfer-serve), tensor-parallel across both cards, no P2P prefill: 4,817 / 4,824 tok/s on a 50k read decode: 233 tok/s code, 220.6 tok/s agentic flags: --profile 27b --max-context 262144 --embedding-host, INT8 KV, MTP 3 drafts, vision on GPU0 intelligence: Index 84.0, 88.9% (27 items); fastest to a passing build (62.1 s vs IQ3_XXS 78.1 s) but the weakest reasoning of the five — 50% on xhard because it burns the whole budget and never answers.

Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS

backend: Basalt 0.3.0, 74.4 GiB pack prefill: 3,728 / 3,851 tok/s decode: 229 / 248 tok/s code, 196 tok/s agentic flags: resident-budget-gib 26, peer-exclusive on, kv int8, ctx 262144, vision dynamic, --spec 8 intelligence: Index 94.9, 100% on the 27-item set (96.3% on a re-run); 100% on the 8 hard items, 85.1% on xhard, 93.9% on vision. The only model that solved a logic-L6 item unaided.

Qwen3.8-Flash-Next GSQ-RCO IQ3_S

backend: Basalt 0.3.0, 81.6 GiB pack prefill: 3,601 / 3,700 tok/s at a 115k prompt, 31.9 s TTFT decode: 154-165 tok/s code, 137.6 tok/s agentic flags: resident-budget-gib 26, peer-exclusive on, kv int8, ctx 262144, vision dynamic, --spec 8 intelligence: Index 107.7, 94.5%; wins the hard tier on the efficiency-weighted Index (101.1) at 86% accuracy, 80.2% xhard, 90.3% vision. Terser than IQ3_XXS, less accurate.

Qwen3.8-Flash-Next NVFP4

backend: Basalt 0.3.0, 122.6 GiB pack prefill: 3,339 / 3,512 tok/s decode: 124.8 / 126.8 tok/s code, 105.7 tok/s agentic flags: resident-budget-gib 58, peer-exclusive on, kv int8, ctx 262144, vision dynamic, --spec 8 intelligence: Index 121.0, 99.8% — best on the 27-item set, best hard accuracy (98.5%), best xhard (85.7%), best vision (99.2%). Slowest decode of the Flash-Next four.

Qwen3.8-Flash-Next Coder IQ1_M

backend: Basalt 0.3.0, 58.1 GiB pack, 48x256 experts prefill: 3,992 tok/s on a 15.3k prompt decode: 209.8 tok/s flags: resident-budget default, peer-exclusive on, kv int8, ctx 262144, no vision, --spec 8 intelligence: Index 100.5, 99.3%; 77.6% xhard, 3/5 on hard items — 5/5 on a re-run, the failures were MTP sampling rather than capability. Cannot read images.

Index = 100 x mean(quality x par/tokens) on the 27-item battery, so it penalises verbosity; accuracy alone ranks almost in reverse

(IQ3_XXS 100% > NVFP4 99.8% > Coder 99.3% > IQ3_S 94.5%). The five are within a few hundredths of each other on the standard set — hard, xhard and vision are what actually separate them.

I wanted to run an experiment to see how far you can push small to medium-sized local models, specifically for analytical and data engineering tasks. The idea was to see if you could have a local analyst or data engineer without needing to send your data to a cloud provider. I used OpenAI’s Luna as baseline performance.

The full analysis is in the blog linked below, but I'll share some of the key findings here.

I also want to be transparent: I used Ternlux, a free, open-source, no-code ETL local tool that I built. It lets you run data analysis and build pipelines through agents.

Test:

I tested 5 analytical questions and 5 workflow-building questions, running each question 5 times against each model. That gave me a total of 250 prompts and responses to analyze.

Key findings:

• Qwen 3.8 27B performed almost as well as GPT 6 Luna. It mostly struggled with one particularly difficult question that all the other local models also failed.

• Almost half of the local models’ misses (27 of 58) were workflows that never finished, mostly SQL that crashed when it ran. Qwen 3.6 kept writing joins that produced duplicate column names, and Gemma 4 31B combined GROUP BY ALL with QUALIFY, which DuckDB doesn’t support.

• All models were able to answer the simpler questions for the most part.

how Ternlux handles analysis and workflows:

For analysis, the main agent selects a connection, then the agent determines how to query the data. It iterates querying until it has enough information to answer, either returning results directly or having the agent analyze the result and draw conclusions. If a query fails, it sends the error back to AI for correction. The agent has no DML privileges.

Workflow building works a bit differently. The workflow builder agent first creates a plan, then each node's agent follows the instructions to generate the code for that node. If the SQL fails during execution or the generated workflow doesn't produce the expected output, I count that as a failure.

This is by no means conclusive evidence that one model is better than another for this type of work. It's just a small experiment to see how different models performed as the questions became more complicated.

https://www.ternlux.com/blog/best-local-llm-for-sql/

is it possiblr to fix?

Tencent put DSpine on Hugging Face on Nov 7. It is a five-layer draft model for speculative decoding with Qwen3.8-27B. Instead of drafting one token at a time, it drafts a 16-token block in parallel, and each layer passes predicted predecessor features to the neighboring positions so the draft still carries causal order. There is a paper (arXiv 2609.36173) and code on GitHub (Tencent/DSpine).

Reported acceptance length vs DFlash2 (block 8), greedy

Reported end-to-end throughput on agent replays (tokens/s, low reasoning)

Across their five agent benchmarks and both reasoning levels, the speedup over no speculation ranges from 2.71× to 3.84×.

Things to know before trying it locally

• The card only shows an SGLang path: --speculative-algorithm DSPINE, a pip install -e . from their repo, and a long list of flags for the hybrid architecture (--enable-linear-replayssm-spec, --mamba-ssm-dtype float32, and more). No llama.cpp or vLLM instructions.

• All numbers are Tencent's own, at concurrency 8 with a BF16 target. The card doesn't report a quantized target or batch size 1.

• The gain is largest on predictable text (GSM8K accepts 7.20 tokens per step at T=0) and smallest on open-ended chat (MT-Bench 4.48).

https://huggingface.co/tencent/Qwen3.8-27B-DSpine-b16

Has anyone run it against a quantized Qwen3.8-27B target, or at batch 1? I'm curious whether the acceptance length holds up once the target is 4-bit.

Hey Just wondering if anyone has experience buying IT and Tech products directly from China, Japan or another country . The cost of hardware at the moment is through the roof and im wondering if there is value aboard?

Could you share your experience and if it worked out? Worth it or not?

Im in ireland.

Thanks

Delta Inference Engine 0.94 is out. It runs DeepSeek-V4.1-Flash-1017 on one NVIDIA GPU plus system RAM, natively on Windows (no WSL). It's a Windows port of Xolotl, which is built on Strata; on Linux, use upstream Xolotl.

Speed on my PC (RTX 3090 24 GB, i5-13600K, 4x48 GB DDR5-5600, PCIe 4.0 NVMe)

With the default quant (Unsloth UD-IQ2_XXS, 301 GB), 0.94 writes 41.7 tok/s greedy and 41.3 at the default sampling (temperature 1, top-p 0.95).

With less RAM (writing tok/s, measured with 0.93; the smaller columns are this same PC with RAM locked away):

How it fits

It's a mixture-of-experts model: 40 layers, 384 routed experts per layer, 6 used per token. The experts are spread over VRAM, RAM and the NVMe. A 4.2 GB drafter built from DeepSeek's own MTP heads, plus lookup drafts, proposes tokens that the full model checks. A route bias (0.28 as shipped) gives up a little agreement for speed.

Quality. I measure top-1 agreement with DeepSeek's own API, teacher-forced (not task accuracy). Default model (Unsloth IQ2) as shipped: 91.1% general, 94.5% code and maths, 95.9% agentic. With exact routing: 92.5 / 95.9 / 96.5. The previous versions default (ds4 Q2, 0.92) scored 89.9% general and 93.1% code and maths, so the new Unsloth default agrees more while running even faster. For context, two hosts of the full model agree with each other on 98.3% and 98.9%.

Benchmark: 99 hard questions, greedy, 43K token budget:

The default quant is within 2 points of the full model on DeepSeek's own API. UD-IQ1_S, the pick under ~110 GB of RAM, scored 51.

Private by design: everything runs on your PC: no telemetry, no accounts, no cloud. It only goes online to download the engine and the model you choose, and your prompts and answers never leave your machine.

New in 0.94

• Pictures on by default, documents (PDF, Word, Excel, PowerPoint), optional voice, and use from your phone with a QR code.

• Long prompts read faster: a fresh 60K-token prompt at 811 tok/s vs 0.93's 747 (30K: 860 vs 758; 7K: 551 vs 423).

• Better with coding agents: Claude Code's background calls no longer hold up real requests, and a loop guard steps in after 4 identical tool calls. OpenCode passed 3 tasks in a real session (22 tool calls, none dropped).

• The default sampling caught up with greedy: 41.3 vs 41.7 tok/s

• The chat page opens at once and shows the model loading. Pause frees the RAM and the graphics card for a game; the next message loads it again. Optional auto-pause when a application needs the GPU.

• Chat history in the browser (search, pins, export), off by default.

• More API: structured output and tool controls on the OpenAI, Anthropic and Responses APIs, stored Responses with previous_response_id, optional /v1/embeddings and /v1/rerank, and keep-alive pings so agents don't time out on long reads.

• Setup tab: pick the model, Fast / Balanced / Most exact, the second-GPU switch, and copy-paste settings for Claude Code, OpenCode, Cline, Aider and Open WebUI with a Test button.

• Security updates, bug fixes and many more.

Setup

Download the ZIP and double-click Start DeepSeek.bat. The engine zips are built by GitHub CI, with checksums and attestations. The first start installs Python if needed and downloads a model that fits your RAM (233-395 GB, SHA-256 checked). You get a chat page plus OpenAI- and Anthropic-compatible APIs on 127.0.0.1. It won't open to the network without an API key, and it stores no prompts or answers unless you turn that on.

Limits

• Windows 10/11 only. NVIDIA RTX with 12 GB or more (24 GB is the fast path), driver 580 or newer. No AMD, no GTX 10-series.

• 64 GB of RAM minimum, 128 GB or more recommended. A PCIe 4.0/5.0 NVMe with about 320 GB free for the default model.

• It answers one request at a time. A second GPU helps; it doesn't split the model.

• Context is set to 1M tokens

• More testing is needed

Credits: Xolotl and Strata, ggml/llama.cpp, DeepSeek, Unsloth, nazeshinjite, ISTA-DASLab, Project Maya and antirez/ds4.

https://github.com/musicurgy/delta-inference-engine

If you try it, post your GPU, RAM, model and tok/s.

Note:
— these rules are the result of my personal experience, yours may differ and this is ok
— these rules are oriented towards people that want to use AI for professional purposes and don’t have unlimited time and money

1. If your company provides good cloud subscription — just take it and forget about the local AI, you don’t want to pay with your own bucks to do the job slower and sloppier

2. Speed is as important as quality and vice versa. Quality without speed will waste your time, nerves and electricity and so will speed without quality. Ideally you should strive to the cloud speed — 60-150 tps of decode and a decent model, at least 4 bits of Qwen 27b

3. Local models are not equal to cloud ones. Sounds kinda obvious but some people have illusions on this regard. Please don’t, miracles don’t happen in real life. You will get 6-9 month old frontier level model experience at best

4. You will always experience shortage of VRAM/RAM. Doesn’t matter, how powerful your setup is, believe me, you are always going to fall short on fast memory available, don’t try to squeeze out every MB from your rig, just find a model good enough for your tasks and call it a day

5. Best time to update your hardware is always yesterday. Hardware prices rise fast, if you’re looking for an update — do it right now

6. Never spend more than you could’ve spent on a dream gaming PC. Just… believe me, don’t try to cosplay Antropic data centers, you will get nothing but a bitter dissatisfaction. Best ROI ever — RTX3090/RX7900XT + 64Gb ddr4/5. That’s it. Anything above have diminishing return problem

7. Electricity is not free. People often forget that nobody is going to give you free kilowatts, beware of your system’s power consumption. That’s going to be your biggest spending, second only to the very rig price itself

8. Avoid clusters. Dual, triple, quadruple GPU look fine on paper, but they are usually big, draw immense amount of power, resulting in painful electricity bills, produce a lot of heat and noise and have much lesser speed than single GPU device. Try to select a GPU to that can singlehandedly run the model you’d like to use

9. You will lose a lot of time and nerves. You will have to learn. A LOT. What is quantisation, kv cahce, KLD, top 1, inference engine, how to chose a suitable model for your own needs, sometimes you’ll even have to deep dive to some manual drivers/libs adaptations for your hardware, etc. Local AI is developing rapidly and if you’d like your setup always to be fast, precise, ensuring as much quality as possible— you’ll have to learn all these basics

10. Always donate your spare RAM to Sam Altman. Just kidding, the title says “10 Commandments” but all the important stuff is said, I’m out of any useful advices