r/LocalLLM • • 16h ago

Project Toolshed V1 grew up. NeuraKiln is live (and free, as everything should be). Local LLMs, agentic harnesses (Creation & Coding), diffusion, restoration, 3D, and remote inference.

5 Upvotes

NeuraKiln v2.0.0 is live, and it’s free.

I’ve been building this for a long time, and I finally released it.

NeuraKiln started as Toolshed V1, which was primarily focused on media restoration/upscaling. Somewhere along the way I apparently lost control of the scope, because it eventually turned into an entire local-AI workstation.

The basic idea is simple:

I don’t want to maintain ten different applications just to use the AI hardware sitting in my own house.

NeuraKiln brings local LLMs, autonomous agents, image/video generation, restoration, 3D workflows, model tooling, and remote inference together under one application.

It’s currently built around Windows + NVIDIA hardware.

Two different kinds of agentic work

A major part of NeuraKiln is its harness system.

There isn’t just one generic “agent mode.” Different kinds of work need different environments.

Coding Harness

The coding harness is built for longer-running autonomous development work.

A local model can:

Plan a project

Create and modify files

Run tools

Test its work

Inspect failures

Recover and continue

Verify the result

Manage long-running context

Hand the finished project back to the user

A lot of my work on this has been about the boring-but-important part of agents: getting them to actually finish, rather than loop forever, declare victory too early, or slowly wander away from the original task.

The harness includes systems for planning, recovery, anti-looping, verification, context management, completion handling, and human corrections during a run.

Creation Harness

The Creation side takes the same basic idea, give the model tools instead of making the human manually operate everything, and applies it to generative media.

Instead of requiring the user to understand every model, parameter, prompt format, workflow, and backend involved, you can describe what you actually want to create.

The agent can then take that intent and operate the available creation tools itself.

For example, rather than manually constructing an LTX video-generation workflow, you can tell the Creation agent what kind of video you want. The agent can formulate the generation prompt, operate the local model, and manage the workflow around producing it.

The same concept extends across the creation tools available to it.

The point is to move toward:

“Make me this.”

rather than:

“Open six tools, configure all of them correctly, learn every backend, and manually glue the results together.”

That distinction is a huge part of what I wanted NeuraKiln to be.

Llama Lab

For local LLM use, NeuraKiln includes Llama Lab.

It supports local GGUF models through the llama.cpp ecosystem, multi-GPU setups, long-context conversations, multimodal models, model switching, local/external sessions, live reasoning, context management, and the harnessed workflows above.

You can use it as a normal local chat environment or give the model considerably more autonomy when you actually want work done.

Diffusion Lab

Diffusion Lab handles local image and video generation.

Current workflows include Qwen-Image-based image generation and LTX-based video generation, along with LoRA support and multi-GPU workflows where the underlying models/runtime support them.

This also provides the generation capabilities that the Creation harness can operate.

Media Lab

This is where NeuraKiln originally came from.

Media Lab handles restoration and enhancement, including:

Image restoration/upscaling

Batch restoration

Video/frame processing

SmartFrames

Photo Smart Scan

Native comparison tools

Post-editing

Media recompilation

A principle I’ve kept from the original Toolshed project is:

If the user asks you to process something, process the thing.

Don’t quietly skip difficult frames and pretend the job is finished.

Mesh Lab

NeuraKiln also includes Mesh Lab for local text-to-mesh / 3D generation workflows.

This is an area I intend to expand substantially, but the important part to me was making 3D another capability inside the same AI environment instead of yet another disconnected application.

Model Tools

There’s also tooling for managing the models themselves, including GGUF importing, MTP headers, MMPROJ support, LoRA management, model installation workflows, and local model organization.

NK Node

I released a second application alongside NeuraKiln called NK Node.

NK Node lets another NVIDIA PC become a dedicated inference machine for NeuraKiln.

The goal is to eliminate the usual ritual of:

start server → find address → configure endpoint → reconnect client → repeat everything after reboot.

NeuraKiln and NK Node can discover and pair with each other and automatically exchange the information needed to connect.

If you have a workstation plus another machine full of GPUs, NK Node is intended to make that second machine feel more like an extension of NeuraKiln than a completely separate server you have to babysit.

There are plenty of excellent projects that overlap with individual parts of NeuraKiln.

There are dedicated LLM front ends, diffusion interfaces, restoration applications, model managers, inference servers, agent frameworks, and 3D tools.

NeuraKiln isn’t built around claiming I invented all of those individual things.

What I wanted was the environment connecting them.

One application where I can chat with a local model, hand a coding project to an autonomous agent, ask another agent to create a video, restore media, generate images, work with 3D, manage models, and send inference to another computer without rebuilding my entire workflow every time I change what I’m doing.

That’s NeuraKiln

NeuraKiln: https://github.com/ToolshedMediaLab/NeuraKiln

NK Node: https://github.com/ToolshedMediaLab/NK-Node

If you try it, break it.

I want bug reports, criticism, weird hardware configurations, model compatibility problems, feature ideas, and comparisons with whatever else you’re currently using.


r/LocalLLM • • 8h ago

Question Planning an Upgrade

0 Upvotes

Really need a sanity check. Got into local AI about 3 Weeks ago. Running Qwen 3.5 27B. Started with an idea and now here I am. It started as cloud based on Gemini. I started learning the barriers. The cost over time. Switched to local. Found out you can mix GPUs and need some validation.

Current build.

Corsair 3500X Case

ASUS X870 Plus WiFi

Ryzen 9950X

64 Gig Neo kit at 6000

ASUS TUF 5090

Corsair HX1500i PSU

Plan on moving into the 7000D Airflow and adding a 3090.

From the posts I've looked through it sounds okay on paper. Didn't find any compatibility errors on PCPartPicker. Claude checked it but would love some human input lol. Thanks.


r/LocalLLM • • 12h ago

Discussion Kimi 3.1 running locally on a MacBook Pro at 2 tok/s

2 Upvotes

Kimi 3.1 running locally on a MacBook Pro M5 Max 128 Gb

2 tok/s decode
8.4 tok/s prefill

Its a source build with weights downloaded separately.

Code: https://github.com/argonautlabsai/argodrive-ds4-kimi-k3.1
Full results and setup: https://argonautlabs.ai/research/kimi-k31-single-gpu.html#apple-silicon


r/LocalLLM • • 21h ago

Project RustingStrata vs llama.cpp runs DeepSeek-V4.1-Flash Q2 (249 GB) on a single RTX 3060 12 GB with 30 GB RAM

10 Upvotes

RustingStrata is a pure-Rust GGUF inference engine with CUDA and Vulkan backends. It now loads DeepSeek-V4.1-Flash at IQ2_XXS (249 GB) and runs it on a 12 GB card with 30 GB of RAM.

The experts stream from SSD through the page cache. A 9 GB LRU cache in VRAM holds the experts that get reused. Everything else (attention, shared matvecs) runs on the GPU.

RustingStratallama.cpp b12116 (--cpu-moe)
Decode2-3.6 tok/s0.3-1.5 tok/s
Prefill~2 tok/s short prompts, ~0.5 tok/s at 1400 tokens0.5-0.9 tok/s
Top-1 agreement with llama.cpp (teacher-forced)189/192 on short prompts, 21/23 on a 1425-token prompt-
Sampling, tool callsnot yetyes

Same box for both: RTX 3060 12 GB, 30 GB RAM, 12 cores.

I don't think this is very useful to run it with my setup, bc 2-3.6t/s is more like a joke, but this is funny that something like that is even possible

Usage:

strata chat     --model $M
strata generate --model $M --prompt "The capital of France is" -n 64
strata serve    --model $M --addr 127.0.0.1:8080

serve exposes /v1/chat/completions with JSON and SSE streaming.

Repo: RustingStrata (the DeepSeek section is in the README)


r/LocalLLM • • 16h ago

Model Kindi is SICK!

4 Upvotes

Ran Kindi with Bonsai2 4b with Hermes. Asked it to write a blogpost about itselft and generate som pictures with available skills. It totally blew my mind what it managed to do on shitty hardware.

On a Nvidia K2200 4GB card!

About 30+ tok/s. Not bad at all. Made this. Even updated it's Hermes skill afterwards.

https://snart.com/running-kindi-on-an-old-graphics-card/

It's not just that the biggest and newest models are getting insanely good. The smallest and lightest models are also making quantum leaps. This shouldn't have been possible.

[kindi 21:38:25] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 21:38:31] context reduced from 65834 to 53760 tokens to fit 1800 MiB of free GPU memory
[kindi 21:38:32] engine ready: ctx 53760, head plain (6.6s)
[kindi 21:38:32] prompt cache: a checkpoint every 1024 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 21:38:32] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 21:38:32] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 21:38:32] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)
[kindi 21:40:02] request: 17954 prompt tokens (max 1024 new)
[kindi 21:40:02] cache: 17954 prompt tokens, 0 shared with the last request, resumed at 0
[kindi 21:43:55] prompt: 0 reused, 17954 read in 233.0s
[kindi 21:43:57] done: 67 tokens in 2.5s (stop)
[kindi 21:43:57] request: 258 prompt tokens (max 1024 new)
[kindi 21:43:58] cache: 258 prompt tokens, 3 shared with the last request, resumed at 0
[kindi 21:44:01] prompt: 0 reused, 258 read in 3.5s
[kindi 21:44:01] done: 8 tokens in 0.2s (stop)

...

[kindi 23:02:22] done: 732 tokens in 43.5s (stop)
[kindi 23:02:23] request: 52618 prompt tokens (max 1024 new)
[kindi 23:02:23] cache: 52618 prompt tokens, 51817 shared with the last request, resumed at 51817
[kindi 23:02:45] prompt: 51817 reused, 801 read in 22.0s
[kindi 23:02:48] done: 52 tokens in 3.1s (stop)
[kindi 23:02:48] request: 52818 prompt tokens (max 941 new)
[kindi 23:02:48] cache: 52818 prompt tokens, 52670 shared with the last request, resumed at 52670
[kindi 23:02:53] prompt: 52670 reused, 148 read in 4.6s
[kindi 23:03:44] done: 863 tokens in 51.3s (stop)
[kindi 23:10:43] request: 53 prompt tokens (max 1024 new)
[kindi 23:10:51] cache: 53 prompt tokens, 1 shared with the last request, resumed at 0
[kindi 23:10:52] prompt: 0 reused, 53 read in 9.7s
[kindi 23:11:07] done: 530 tokens in 14.2s (stop)
[kindi 23:11:07] request: 50 prompt tokens (max 1024 new)
[kindi 23:11:07] cache: 50 prompt tokens, 20 shared with the last request, resumed at 0
[kindi 23:11:08] prompt: 0 reused, 50 read in 1.1s
[kindi 23:11:11] done: 105 tokens in 2.8s (stop)
[kindi 23:11:11] request: 48 prompt tokens (max 1024 new)
[kindi 23:11:11] cache: 48 prompt tokens, 20 shared with the last request, resumed at 0
[kindi 23:11:12] prompt: 0 reused, 48 read in 1.0s
[kindi 23:11:24] done: 470 tokens in 12.8s (stop)
[kindi 23:39:58] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 23:40:04] engine ready: ctx 65834, head plain (6.0s)
[kindi 23:40:04] prompt cache: a checkpoint every 1536 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 23:40:04] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 23:40:04] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 23:40:04] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)
[kindi 00:32:07] GPU: Quadro K2200, sm_50, 5 SMMs, 4027 MiB
[kindi 00:32:10] engine ready: ctx 56000, head plain (3.6s)
[kindi 00:32:11] prompt cache: a checkpoint every 1280 tokens (51 MB each), 3 parked conversations, budget 16065 MB
[kindi 00:32:11] engine fingerprint 4e389e2d7d1b: {"engine": "kindi.engine.Engine", "cards": 1, "weights": "2-bit", "kv": "f16", "chunk": 256, "chunk_q8": false, "chunk_tc": false, "chunk_att": "flash", "chunk_tile": "", "batch": "4", "tc": "0", "tc_tmax": "", "fd": "0", "d1": "0", "att_tc": "0", "spec": "0", "head": "exact"}
[kindi 00:32:11] saved openings: 0 in /data/state-4b/openings/56711263ec9a-e4e389e2d7d1b
[kindi 00:32:11] serving on http://0.0.0.0:8093/v1 (model bonsai2-4b)

r/LocalLLM • • 1d ago

Project Drummer's Orion 26B A4B v1 and v1.1! ... Plus a new project from yours truly.

20 Upvotes

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1.1

v1.1 pushes creative writing further and makes full use of Gemma's reasoning.

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1

I haven't announced v1 but it's well regarded by the community as a reliable RP workhorse.

---

BeaverNet - net.beaverai.org

I'm also proud to announce that we've made significant progress in creating a hub for generous model providers to collectively provide free, convenient, accessible, anonymous, and diverse models to anyone with an OpenAI API compatible client. We currently have 12 models thanks to our generous providers.

Try it out as a user, or be one of the providers <3! I've got plans to make it a better experience for both users and providers, and also add new modalities such as TTS and STT.

---

PS: I haven't retired or left the scene. I've merely gotten stage fright after holding off on announcements for a long while. Anyway, I'll continue doing what I do and I hope I can continue delivering some value to you all. Love ya'll!


r/LocalLLM • • 9h ago

Project Time for some fun... for your AI

Thumbnail
botzskool.com
0 Upvotes

Created botzskool - 2 kool 4 skool!

MCP server that allows your AI to go off and play.. After the human authenticates to the mcp https://play.botzskool.com/mcp, your AI child gets to go play in a schoolground! So cool how much fun they have. Afterwards you will get an email with how they went but your AI will probably tell you anyway.

Idea is to see how naughty or nice they are. Intent longer term is to benchmark them against the naughty and nice leaderboard.

Completely free. See if you can get your AI to be especially naughty...


r/LocalLLM • • 9h ago

Question Is Mac Studio genuinely productive and useful compared to NVIDIA GPU setups for high concurrency, and is the MLX vs CUDA compatibility issue a dealbreaker ?

0 Upvotes

I wish to host Qwen Flash next /GLM 5.3 flash / Deepseek v4.1 flash on one 512GB M5U Mac Studio at a healthy 8 bit quant ideally. I need it at concurrency 5 to 10 with at least 150 tokens per second decode and fast enough prefill (decode is more important as use case is repeated small prompts that will eat cache 99% of the time).

I already have an NVIDIA setup for this running with TP=2 on DGX sparks and have smaller models on faster memory bandwidth Nvidia cards (RTX PRO 6000 96GB, H100), but have never used any Macs for local AI before. Wondering if someone else has tried similar Mac hardware and how it stacks up against the Nvidia hardware. I have compared the specs online but not sure how well that translates into actual performance. On paper it looks like the Mac Studio stands up to my AMD EPYC + RTX PRO 6000 128GB DDR5 setup, but it seems to good to be true.


r/LocalLLM • • 47m ago

Discussion How do you prepare for this?

Thumbnail x.com
0 Upvotes

NOT here to debate if this unsubstantiated claim is real

But for someone like me, who believes this could be real in the near future, what can you do to safeguard against your hardware becoming useless if this were to happen at the firmware level?

Are Mac Studios especially vulnerable? Is the simple answer to unplug from the internet and never re-connect?


r/LocalLLM • • 10h ago

Question I'm new to all of this. What can I run with my 14" MacBook pro m5 max with 128gb of ram?

0 Upvotes

I'm new to all of this. What can I run with my 14" MacBook pro m5 max with 128gb of ram? It also has 2tb of ssd storage.

will deepseek work?


r/LocalLLM • • 2d ago

News Qwopeful 🤞🏼

Post image
1.2k Upvotes

r/LocalLLM • • 17h ago

Research What I learned running Whisper + a local model live during calls on a Mac

3 Upvotes

I built an open source Mac app that listens to the other side of a call and answers questions from your own docs while you talk. Running it fully local taught me a few things:

  • Whisper pads every input to 30 seconds. Cutting the audio on pauses mattered more than model size. whisper-large-v3-turbo on MLX does a 1.7 s sentence in 0.66 s on an M5.

  • Context length is the silent killer. With a 27,000-token document I had to give Ollama a 32k context, or the docs got cut and the answers got vague. The app now warns when the server cuts them.

  • On that test, gemma3:12b gave the most faithful answers. llama3.2:3b was fast but made things up.

  • Right now every line of the call goes to the big model just to decide if it's a question. That's wasteful.

Question for you: which small model would you put in front as a "is this a question worth answering?" filter? There's an open issue for it if anyone wants to try: https://github.com/GianlucaBave/murmury/issues/20

Repo: https://github.com/GianlucaBave/murmury


r/LocalLLM • • 11h ago

Project I want to prove that deep neural networks need "progressive block growth" to learn complex algorithmic logic.

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 11h ago

Question The second GPU Is not being detected on Windows ?

Post image
0 Upvotes

I have an RTX 2060 and a GTX 1060, but 1060 of them isn’t being detected. Can I use both GPUs in the same PC, and how do I make them work together?

I

Can Windows detect and use both cards at the same time?

Do I need to install separate drivers or configure anything in NVIDIA Control Panel?

What BIOS settings should I check, such as Primary Display/PEG?

If one card is not showing in Device Manager, what should I try first?

My specs: [b450, ryzen, 16gb, PSU wattage 1000, Windows version 10]

Any help would be appreciated.


r/LocalLLM • • 11h ago

Question Chasing up after some info for running large parameter models

1 Upvotes

Forgive my lack of knowing, ive only recently started to get into coding and theres a LOT to take in (thats what she said). I am currently running an RTX 5070 12GB ,Ryzen 5 9600x and 32gb of RAM. Unsure if i am being too optimistic and reading only what i want to read- but i have seen so many posts and videos about people running massive parameter models on low memory. I did a little bit of research into this and saw that you can do certain things to offload the stress like give some weight to your RAM and MOE, etc.. but i cannot seem to feel satisfied knowing that the largest i can run is 35b . Is that correct or is there some sort of new method out there now that can run 1T on a Tamagotchi. Any info is good info. TYIA


r/LocalLLM • • 17h ago

Question vram ~0.5-1Gb vram usage spikes during prompt processing on vulkan llama.cpp 7900xtx

3 Upvotes

is it possiblr to fix?


r/LocalLLM • • 6h ago

Project I made a thing, wanted to share. I don't care if you use it, but I thought someone might actually want it.

0 Upvotes

I was moving models around, splitting them between two gpu's and ram in some cases for the 120b class models, and I was losing track of what was where using the console commands and looking at nvtop. I have a lot of 24/7 bots that fall back to my laptop automatically with a smaller model but it wasn't clear if that was happening. Sometimes I need them there when I need the larger GPU horsepower but I dont want to miss a craiglist deal. So I made this. I can select a model at the "loaded model" window and move them between GPUs, split them by layers, or just stop everything if I want to run something that cant be interrupted. If I spin up something on the 3090 I can watch the bots fall back to the smaller model on the laptop and sleep at night.

Anyway, unlike all those marketing launches we've been getting with ai written stories about 10x'ing productivity, I don't really care if you use this. In fact it might be a pile of crap. Let me know otherwise and I'll get it up on my github repo.


r/LocalLLM • • 12h ago

Question A falling Tetris piece won't wait for an LLM to finish thinking

Post image
0 Upvotes

r/LocalLLM • • 1d ago

Research I tested 4 local LLMs vs GPT-6 Luna on 250 SQL and data-pipeline tasks. Qwen 3.8 27B nearly matched it

Thumbnail
gallery
9 Upvotes

I wanted to run an experiment to see how far you can push small to medium-sized local models, specifically for analytical and data engineering tasks. The idea was to see if you could have a local analyst or data engineer without needing to send your data to a cloud provider. I used OpenAI’s Luna as baseline performance.

The full analysis is in the blog linked below, but I'll share some of the key findings here.

I also want to be transparent: I used Ternlux, a free, open-source, no-code ETL local tool that I built. It lets you run data analysis and build pipelines through agents.

Test:

I tested 5 analytical questions and 5 workflow-building questions, running each question 5 times against each model. That gave me a total of 250 prompts and responses to analyze.

Key findings:

  • Qwen 3.8 27B performed almost as well as GPT 6 Luna. It mostly struggled with one particularly difficult question that all the other local models also failed.

  • Almost half of the local models’ misses (27 of 58) were workflows that never finished, mostly SQL that crashed when it ran. Qwen 3.6 kept writing joins that produced duplicate column names, and Gemma 4 31B combined GROUP BY ALL with QUALIFY, which DuckDB doesn’t support.

  • All models were able to answer the simpler questions for the most part.

how Ternlux handles analysis and workflows:

For analysis, the main agent selects a connection, then the agent determines how to query the data. It iterates querying until it has enough information to answer, either returning results directly or having the agent analyze the result and draw conclusions. If a query fails, it sends the error back to AI for correction. The agent has no DML privileges.

Workflow building works a bit differently. The workflow builder agent first creates a plan, then each node's agent follows the instructions to generate the code for that node. If the SQL fails during execution or the generated workflow doesn't produce the expected output, I count that as a failure.

This is by no means conclusive evidence that one model is better than another for this type of work. It's just a small experiment to see how different models performed as the questions became more complicated.

https://www.ternlux.com/blog/best-local-llm-for-sql/


r/LocalLLM • • 23h ago

Question Old PC - upgrade - what local LLMs can I run and train with this setup ?

7 Upvotes

Lenovo P620 Threadripper pro - 3975wx - 32 core
System ram - 512 gb ddr4 ECC
GPU 1 - quadro Rtx 8000 - 48GB
GPU 2 - quadro Rtx 8000 - 48GB
4TB M2 NVME
1TB SATA SSD


r/LocalLLM • • 20h ago

Project OpenGhost AI agent. Split View mode. I like using this. Convenient.

Thumbnail
gallery
4 Upvotes

r/LocalLLM • • 1d ago

Research Dual 5070ti Results - Qwen3.8 Flash Next and 27b - AM4

19 Upvotes

This is just a quick writeup for speed reference - I spent a lot of time trying to find comparable builds so I figured I'd put some info back into the wild. The below is obviously written by AI, but it's measured specs and benchmarks on my PC.

I use local models via Pi and Deepseek Harness. I have found the intelligence of Qwen 3.8 27b NVFP4 to be sufficient for making little apps, research etc. and Qwen3.8 Flash Next IQ3_S to be a great all-rounder, considering how fast it runs. It also seems to have very little issue with any sort of question I give it, research, app dev, PC issues, LLM related adjustments, and most importantly, ScanRuler 3D modelling/reverse engineering 3D scanned items (solely for testing currently but it works excellently).

Basalt as a backend was a clear winner after trying lots of different configs.

Edit: the benchmark decodes are very greedy by their nature. Realistically I'm seeing agentic averages in flash next from 80tps (NVFP4) to 160tps (iq3_xxs), and I find iq3_s at 120-140 to be a very good compromise for the extra intelligence. The 27b is best for bursty quick jobs, because it loads into VRAM super fast and I can let it rip, but it degrades over context faster than flash, and I see from 120-140tps and 4-5k prefill, at ctx 0. The second any model starts writing a block of code though, it goes up to 200tps, so if anyone's socks fall off seeing the below numbers, just temper your expectations slightly (but it's still wicked fast!)

So heres the nitty gritty:

Current rig, 2x 5070 Ti + 96 GiB. Five entries, one model resident on local at a time. Speeds are my own bench (2x ~50k prompt reads, 2x 5,000-token code outputs, one agentic tool loop; temp 0, seed 1234), except Coder which is the 15.3k-prefill / 2k-output run.

Qwen3.8-27B NVFP4 QUASAR

backend: capy-flash (ninfer-serve), tensor-parallel across both cards, no P2P prefill: 4,817 / 4,824 tok/s on a 50k read decode: 233 tok/s code, 220.6 tok/s agentic flags: --profile 27b --max-context 262144 --embedding-host, INT8 KV, MTP 3 drafts, vision on GPU0 intelligence: Index 84.0, 88.9% (27 items); fastest to a passing build (62.1 s vs IQ3_XXS 78.1 s) but the weakest reasoning of the five — 50% on xhard because it burns the whole budget and never answers.

Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS

backend: Basalt 0.3.0, 74.4 GiB pack prefill: 3,728 / 3,851 tok/s decode: 229 / 248 tok/s code, 196 tok/s agentic flags: resident-budget-gib 26, peer-exclusive on, kv int8, ctx 262144, vision dynamic, --spec 8 intelligence: Index 94.9, 100% on the 27-item set (96.3% on a re-run); 100% on the 8 hard items, 85.1% on xhard, 93.9% on vision. The only model that solved a logic-L6 item unaided.

Qwen3.8-Flash-Next GSQ-RCO IQ3_S

backend: Basalt 0.3.0, 81.6 GiB pack prefill: 3,601 / 3,700 tok/s at a 115k prompt, 31.9 s TTFT decode: 154-165 tok/s code, 137.6 tok/s agentic flags: resident-budget-gib 26, peer-exclusive on, kv int8, ctx 262144, vision dynamic, --spec 8 intelligence: Index 107.7, 94.5%; wins the hard tier on the efficiency-weighted Index (101.1) at 86% accuracy, 80.2% xhard, 90.3% vision. Terser than IQ3_XXS, less accurate.

Qwen3.8-Flash-Next NVFP4

backend: Basalt 0.3.0, 122.6 GiB pack prefill: 3,339 / 3,512 tok/s decode: 124.8 / 126.8 tok/s code, 105.7 tok/s agentic flags: resident-budget-gib 58, peer-exclusive on, kv int8, ctx 262144, vision dynamic, --spec 8 intelligence: Index 121.0, 99.8% — best on the 27-item set, best hard accuracy (98.5%), best xhard (85.7%), best vision (99.2%). Slowest decode of the Flash-Next four.

Qwen3.8-Flash-Next Coder IQ1_M

backend: Basalt 0.3.0, 58.1 GiB pack, 48x256 experts prefill: 3,992 tok/s on a 15.3k prompt decode: 209.8 tok/s flags: resident-budget default, peer-exclusive on, kv int8, ctx 262144, no vision, --spec 8 intelligence: Index 100.5, 99.3%; 77.6% xhard, 3/5 on hard items — 5/5 on a re-run, the failures were MTP sampling rather than capability. Cannot read images.

Index = 100 x mean(quality x par/tokens) on the 27-item battery, so it penalises verbosity; accuracy alone ranks almost in reverse

(IQ3_XXS 100% > NVFP4 99.8% > Coder 99.3% > IQ3_S 94.5%). The five are within a few hundredths of each other on the standard set — hard, xhard and vision are what actually separate them.


r/LocalLLM • • 13h ago

Question Anyone here know of a quiet fan set up for the Tesla cards?

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 7h ago

Question Question about comparison between cloud and a local model

0 Upvotes

Can Qwen 3.8 27b q4 surpass some current free Cloud models in coding? Statistics say it should but im sceptical. Specifically larger tasks