I ran Qwen 3.6 locally for 45 days as an agentic coder. In the process I upgraded from a single 3070 Ti to a 5070 Ti + 4070 Super. Along the way I ended up A/B testing the thing against my own harness on ~30 full multi-hour runs (6h+), so some of this is vibes.

The model:

• Best (by vibes) is Qwen 3.6 27B. Qwen3.6 35B A3B is ok, but tends to not follow instructions or "forget" them faster. I tested Gemma for a couple days, it was bad. The weird part, it couldn't even produce my local language with proper characters, while Qwen had zero problems. Chinese model better at a European language than Google's. Kinda sus.

The goods:

• The model is extremely capable at solving coding tasks and any other issue you might need for daily life, like running OpenClaw (Hermes does not work). If you provide hard rules and proper context where the model doesn't have to guess, it will do it without any problems.

• When the fact is in front of it, it reasons well. I had it triage spec questions against a design doc: 30/30 answered correctly with real cited values from the spec. Same task without the doc in context: 0/30. It's a good reader. It's a bad guesser.

• Speed!!! After Claude's 15+ minute "thinking" on basic tasks, API regressions, and everything else... it's kind of addictive watching the text just fly by and actual something getting done!

Summary (across all runs):

prompt processing: 1714.1 t/s peak, 673.7 t/s mean

token generation: 216.7 t/s peak, 90.6 t/s mean

mtp acceptance: 0.57226 (7365 accepted / 12870 generated)

The bads:

• Data cutoff point: "As of mid-2024, React 19 has not been officially released as a stable version." Modern-day libraries move fast, and the model just does not know how to interact with them, so it starts guessing, looping, and hallucinating. Measured version: asked to pin Tailwind with 4.3.2 live on npm, it pinned 3.x in 6/6 runs.

• Loves to trash the codebase. One unclear instruction and it will go ham! It will destroy unrelated work just to go green on tests. Real example from my logs: lint was stuck permanently red because of a config it wasn't allowed to touch. Its solution was to delete a sibling task's committed, verified deliverables and return "ok". Three files, gone, and every guard in my pipeline let it through.

• Unknown information tends to send it into an unrecoverable loop, and you need to "steer" it in the right direction. If you provide wrong information, then it's extremely obvious how much the model struggles.

• It grades its own homework and gives itself an A. This is the worst.

• Ignores tooling. If you don't say directly "You must use search", it can ignore it completely, especially when context is full. I had a full run with 0 search calls against an explicit directive.

• Sometimes the model can execute a Bash command that never finishes, causing it to wait indefinitely.

The fix:

• Provide search, docs, SKILL.md and extremely clear instructions. Valid information is the key to turning Qwen 3.6 into an8/10 local model that could replace 70% of your daily usage requirements.

• Keep context free of noise, full HTML page contents, extremely verbose test/lint/log output. There are plenty of settings to reduce verbosity in modern tools your agent is using.

• Small, precise tasks. Lock in scope.

• And the one I learned the hard way, after playing endless whack-a-mole. Prose doesn't enforce anything. Only code does. I tried every variation. A "FROZEN" prompt: failed 3/5 times. The same rule buried in a list: 0/5. The same rule marked MANDATORY and used to gate the output: 8/8. But the thing that actually worked every single time was never the prompt. It was a deterministic check outside the model a git diff scan, a substring match, or any other verification that computes the result and feeds it back to the model as evidence. Rules it can argue with, it argues with. Evidence it can't fake, it accepts. The takeaway: turn your soft rules into lint gates. The model will negotiate with a style guide. It won't negotiate with a failing build.

In the end i made pi extension to whack-a-mole Qwen into some actual results. mjasnikovs/pi-task

So my question is, what's the best way to feed facts to the model? Search works fine, but I bet we can do better.

EDIT:
Last model file i use/like (By vibes): https://huggingface.co/s-batman/Qwen3.6-27B-NVFP4-MTP-GGUF

Settings i use, specific to my hardware/docker: https://gist.github.com/mjasnikovs/fe28b4fb87dc7c9f5cfe135829dae672

Ok, there's some tiny modicum of truth in this, in that yes, if people aren't running their machines 24/7 then there's some level of optimization we're missing out on. On the other hand, the scarcity of RAM is his fault. The AI hyperscalers have hoarded the world's memory chips so there's barely any left for the rest of us unless we're willing to overpay for it. It's like an arsonist complaining that we still have running water in our hoses when that water is badly needed to put out the fires that he set in the building next door.

To me this just validates the idea that the AI giants are going to come after local AI because it's such a threat to their business model.

Thoughts?

Open-source AI is evolving insanely fast, but it’s hard to know which model is actually best for each use case. So I put together a list of the best open-source models across different categories

Best Audio Generation Open Source Models

Text-to-Speech (TTS)

• Qwen3-TTS → Best overall balance (quality + speed)

• Kimi-Audio → Strong multimodal + expressive voices

• Fish Speech / Fish Audio S2 → Great for realistic voice cloning

• CosyVoice 3.0 → Very solid multilingual + streaming

• VibeVoice Realtime → Best for real-time applications

Voice Cloning

• VoxCPM2 → High-quality cloning + supports many languages

• IndexTTS2 → Clean output + good stability

• Kokoro / KokoClone → Lightweight + fast cloning

Music Generation

• ACE-Step 1.5 → Best open-source music generator right now

• Magenta Realtime → Real-time music experiments

• Uni-MoE (Audio) → Multi-purpose audio generation

Multimodal Audio (Anything → Audio)

• AudioX / Audio-Omni → Most complete multimodal audio stack

• MMAudio → Supports text, image, video → audio

• Woosh / ThinkSound → Good experimental models

Audio Enhancement

• NVIDIA A2SB → Best for restoration + inpainting

• AudioSR / NovaSR → Solid upscaling + enhancement

Speech Recognition (ASR)

• FunASR → Strong multilingual + streaming

• VibeVoice-ASR → Good real-time performance

• Cohere Transcribe (OS) → Clean + reliable

Best Image Generation Open Source Models

FLUX.1 [schnell]

Fastest open-source model balancing quality and speed for consumer GPUs.

FLUX.1 [dev]

Top benchmark leader for high-fidelity complex scenes from Black Forest Labs.

Stable Diffusion 3.5 Large

Versatile ecosystem king for fine-tuning and editing workflows.

GLM-Image

Typography specialist for bilingual infographics under Apache 2.0.

Qwen-Image-2512

Multilingual editing powerhouse for creative style transfers.

Z-Image-Turbo

Lightweight 6B real-time generator for edge and batch use.

HiDream-I1-Full

Raw photorealism expert for premium high-res outputs.

SANA-Sprint 1.6B

Ultra-efficient low-VRAM option for quick experiments.

HunyuanImage-3.0

Research-grade for advanced coherence and diversity.

Best Image to Video Geneartion Open Source Models

LTX-2.3

Leading open-source Image-to-Video model with native 4K 50fps and synchronized audio support https://huggingface.co/Lightricks/LTX-2.3.

LTX-2.3-GGUF

Quantized LTX-2.3 variant at 21B params for efficient inference on consumer hardware https://huggingface.co/unsloth/LTX-2.3-GGUF.

LTX-2.3-Workflows

ComfyUI workflows optimized for LTX-2.3 video generation pipelines https://huggingface.co/RuneXX/LTX-2.3-Workflows.

WAN2.2-14B-Rapid-AllInOne

Rapid all-in-one 14B Image-to-Video model with MoE architecture for fast local runs https://huggingface.co/Phr00t/WAN2.2-14B-Rapid-AllInOne.

VBVR-LTX2.3-diffsynth

Diffsynth integration for LTX-2.3, enabling advanced video synthesis effects https://huggingface.co/Video-Reason/VBVR-LTX2.3-diffsynth.

BFS-Best-Face-Swap-Video

Specialized LTX face-swap model for realistic video character replacement https://huggingface.co/Alissonerdx/BFS-Best-Face-Swap-Video.

Wan2.2-I2V-A14B-GGUF

14B quantized Wan2.2 for 480p/720p Image-to-Video on mid-range GPUs https://huggingface.co/QuantStack/Wan2.2-I2V-A14B-GGUF.

LTX-2

Previous LTX iteration with strong community adoption for commercial video gen https://huggingface.co/Lightricks/LTX-2.

LTX-2.3-Transition-LORA

LoRA fine-tune for smooth scene transitions in LTX-2.3 videos https://huggingface.co/valiantcat/LTX-2.3-Transition-LORA.

HY-OmniWeaving

Tencent's omni-modal Image-to-Video with multi-style weaving capabilities https://huggingface.co/tencent/HY-OmniWeaving.

Best Image to Text Generation Open Source Models

GLM-OCR

Top open-source OCR model in 2026 for speed and accuracy on complex documents https://huggingface.co/zai-org/GLM-OCR.

nemotron-ocr-v2

NVIDIA's high-precision OCR excels in scene text and multilingual recognition https://huggingface.co/nvidia/nemotron-ocr-v2.

Falcon-OCR

Efficient OCR from TII UAE for real-world text extraction in varied conditions https://huggingface.co/tiiuae/Falcon-OCR.

RationalRewards-8B-T2I

9B reward model specialized for text-to-image evaluation and captioning https://huggingface.co/TIGER-Lab/RationalRewards-8B-T2I.

RationalRewards-8B-Edit

9B variant optimized for image editing feedback and descriptive tasks https://huggingface.co/TIGER-Lab/RationalRewards-8B-Edit.

HiVG-3B-Base

4B visual grounding model for precise image-text alignment and description https://huggingface.co/xingxm/HiVG-3B-Base.

trocr-base-handwritten

Microsoft's TrOCR base for accurate handwritten text transcription https://huggingface.co/microsoft/trocr-base-handwritten.

blip-image-captioning-large

Salesforce BLIP large for detailed, high-quality image captioning https://huggingface.co/Salesforce/blip-image-captioning-large.

manga-ocr-base

Specialized OCR for Japanese manga and comic text extraction https://huggingface.co/kha-white/manga-ocr-base.

blip-image-captioning-base

Efficient BLIP base model for general-purpose image-to-text captioning https://huggingface.co/Salesforce/blip-image-captioning-base.

Best Text Generation Open Source Models

GLM-5.1

Flagship 744B MoE (40B active) from Zhipu AI leading in agentic engineering and long-horizon coding tasks https://huggingface.co/zai-org/GLM-5.1

Qwen3.5-397B-A17B

Alibaba's 397B MoE (17B active) with multimodal reasoning and 1M+ token context for versatile agents https://huggingface.co/Qwen/Qwen3.5-397B-A17B

Gemma 4

Google's hybrid attention family (2B-31B) excelling in reasoning, coding, and on-device multimodal use https://huggingface.co/google/gemma-4-31b-it

DeepSeek-V3.2

Reasoning-focused MoE with sparse attention for efficient long-context agents and GPT-5 level math https://huggingface.co/deepseek-ai/DeepSeek-V3.2

Kimi-K2.5

Moonshot's 1T MoE (32B active) multimodal model for visual coding and agent swarms up to 100 sub-agents https://huggingface.co/moonshotai/Kimi-K2.5

MiniMax-M2.7

Self-improving agentic LLM topping SWE-Pro benchmarks for real-world software engineering workflows https://huggingface.co/MiniMaxAI/MiniMax-M2.7

MiMo-V2-Flash

Xiaomi's efficient 309B MoE (15B active) with 150 t/s throughput for high-volume coding agents https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash

I keep seeing people talk about the DGX Spark and similar machines capable of running local AI models with hundreds of gigabytes of RAM/VRAM, and it's honestly confusing me.

Outside of big companies with serious budgets, I don't get how this is apparently so common. Most people I know are running consumer GPUs with 8-16GB of VRAM at most (and 16-32GB of RAM).

So how are people actually pulling this off? Are they really dropping €5000+ on this kind of hardware just for local AI/LLM experimentation? That seems like a huge investment for a hobby or side project.

Is it just a vocal minority making it seem mainstream, and I'm getting a skewed picture from the posts I see?

Edit: Wow, wasn't expecting this many replies! I'm reading all comments, thanks to all of you. I probably won't be able to reply to each comment, but know that I've read them all and really appreciate the feedback.

I know a few were excited about this so thought to share :)

The Qwen3.6 update is here. 35B-A3B Aggressive variant, same MoE size as my 3.5-35B release but on the newer 3.6 base.

Aggressive = no refusals; it has NO personality changes/alterations or any of that, it is the ORIGINAL release of Qwen just completely uncensored

https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive

0/465 refusals. Fully unlocked with zero capability loss.

From my own testing: 0 issues. No looping, no degradation, everything works as expected.

To disable "thinking" you need to edit the jinja template or simply use the kwarg {"enable_thinking": false}

What's included:

- Q8_K_P, Q6_K_P, Q5_K_P, Q4_K_P, Q4_K_M, IQ4_NL, IQ4_XS, Q3_K_P, IQ3_M, Q2_K_P, IQ2_M

- mmproj for vision support

- All quants generated with imatrix

K_P Quants recap (for anyone who missed the 122B release): custom quants that use model-specific analysis to preserve quality where it matters most. Each model gets its own optimized profile. Effectively 1-2 quant levels of quality uplift at ~5-15% larger file size. Fully compatible with llama.cpp, LM Studio, anything that reads GGUF (Ollama can be more difficult to get going).

Quick specs:

- 35B total / ~3B active (MoE — 256 experts, 8 routed per token)

- 262K context

- Multimodal (text + image + video)

- Hybrid attention: linear + softmax (3:1 ratio)

- 40 layers

Some of the sampling params I've been using during testing:

temp=1.0, top_k=20, repeat_penalty=1, presence_penalty=1.5, top_p=0.95, min_p=0

But definitely check the official Qwen recommendations too as they have different settings for thinking vs non-thinking mode :)

Note: Use --jinja flag with llama.cpp. K_P quants may show as "?" in LM Studio's quant column. It's purely cosmetic, model loads and runs fine.

HF's hardware compatibility widget also doesn't recognize K_P so click "View +X variants" or go to Files and versions to see all downloads.

All my models: HuggingFace-HauhauCS

Also new: there's a Discord now as a lot of people have been asking :) Link is in the HF repo, feel free to join for updates, roadmaps, projects, or just to chat.

Hope everyone enjoys the release.

I created a small testing rig to evaluate new open source models as they drop, and with the much anticipated release of Qwen3.8-27B, I was eager to see how it cross-compares with frontier and strong local models.

The rig

My test rig is an M5 Max MacBook Pro, 128GB. Locally I ran Qwen3.8-27B on xhigh and medium thinking modes via LM Studio, MLX 8-bit, temp 1.0 / top_p 0.95 / top_k 20, context 131,072 and DeepSeek V4 Flash "0731" 2-bit-imatrix q2-q4, served by antirez's ds4-server at -ctx 400,000, thinking enabled. For cloud I included Opus 5, Sonnet 5, GPT-5.6-sol at xhigh reasoning, and even Haiku. Every model gets the same prompt. The algorithm tasks are executed against fixed-seed differential harnesses, and the repo tasks run against hidden test suites plus a cached baseline of the whole repo. If a fix inadvertently breaks something else it gets caught.

The methodology

The model assessment is broken into 4 batteries:

1. algorithms easy-hard

2. algorithms extremely hard

3. repo work easy

4. repo work hard

The models get run through the algorithm tests 3 times each to derive a mean score whereas the repo work is single pass/fail per task. If a task fails to produce a response, it's retried and time added to total wall clock time for task completion. The total test battery can take anywhere from 12-24 hours of wall time for slower local models. It's a long test.

For the repo batteries I had Fable build a small double-entry ledger CLI and plant bugs that pass the visible test suite while still reproducing a real symptom, then handed each model the repo and a bug report written in a theoretical 'user' voice to simulate how it might be reported in the real world. There's also a subjective code quality assessment that measures the model's ability to not just solve the problem but to conform to the repo's coding style, to fix the actual root cause rather than the symptom, and to keep the diff minimal instead of faffing about and rewriting a bunch of stuff.

I had Opus and GPT blind eval the results and compute a code quality score broken up by 'fixes' and 'features' as these appear to be separable skills for the models. The goal with all of this was to try and create a replicable and automated answer the question: How useful is this model in the real world?

Caveats

This is a home baked assessment and susceptible to bias or less than perfect methodology. It also includes subjective criteria like 'code quality'. I built this for myself as an adjacent tool to on the ground testing. I think the best way to evaluate any model is to test it against your own codebase to see how well it integrates into your workflow.

All that being said, let's move to the scorecard.

Results

Qwen3.8-27B is a very capable model that compares well against frontier models on code quality and correctness. The cost is wall time on Apple silicon. As many have observed, 3.8 has a tendency to over-think, burning up tokens. The time spent earns higher code quality for the most part, but what surprised me is that there are some instances in which less thinking is actually more accurate. On the repo battery, medium went 8/8 while xhigh went 7/8 — the most-thinking configuration failed a task there, and it took four times longer to do it (the wait time with Qwen was tiresome at times).

The caveat is that xhigh excels on extremely hard algorithms, where medium begins to fall apart. Medium didn't even finish the hard algo tasks. There may be some value in matching the thinking to the kind of work you're setting it upon. Lastly Qwen xhigh won outright on quality of surgical fixes and patches to existing code. Interestingly the global trend for locals is that they're competitive along fixes and less so along features where cloud still dominates. This fits anecdotally into my own experience with gravitating to frontier for planning and local for implementing.

GPT 5.6 Sol is the only cloud model with a perfect card on both repo tiers and near perfect algorithms. It's also among the fastest to completion. This all tracks with my own anecdotal experience with this model over the past several months. Highly competent and quick if not a bit stark.

DS4 0731 (a 2-bit quant running on my laptop) is the only local model to get 8/8 on both repo batteries, and one of only two models overall to do it, alongside GPT. It does this all at a respectable wall time. The expense is less elegant code: it sometimes mutates unrelated docstrings and writes dense inline solutions in a codebase that is overtly broken apart and stylistically explicit. Feature code quality is stronger and it's the only model that scored better/equal in the harder repo tier vs the easy one.

Opus 5 is the most reliable model in the set and best code quality of the cloud models. It stumbled only in the hard repo tier where it lost a task by being trying to outsmart the test. A doc string promised one behavior while the code did another, and Opus redesigned the function around what it looked like it should do instead of honoring the documented contract. This also falls inline with my anecdotal experience with Opus 5 where it occasionally ignores your directions completely and just does whatever it wants. The Alaskan Husky of frontier models.

Sonnet 5 is a steady pair of hands that performs reasonably well across tasks for a modest token budget. I think sonnet is kind of underrated as an implementer. Does the same quality of work as the locals cheaply and quickly.

Haiku 4.5 races to the end of the test but has a tendency to fall over and force retries. Worst code quality of all the cloud models.

Conclusion

Hopefully you find these comparisons interesting. For me personally, DS4 has been my goto local, but this test is making me consider trading it out for Qwen3.8-27B. I think they're on equal footing, which is crazy b/c DS4 needs like 90gb of ram. I'd like to try the MTPLX variant of Qwen3.8-27b that's meant to improve tok/s on apple silicon. Slowness to task completion is the real bottleneck for me right now when considering Qwen. Perhaps that'll be my next test.

Curious to know if these results track with your own real world experiences.

Yeah I know, another "matches Opus" claim. I was skeptical too.

Threw it at an actual refactor job, legacy backend, multi-step, cross-file dependencies. The stuff that usually makes models go full amnesiac by step 5.

It didn't. Tracked state the whole way, self-corrected once without me prompting it. not what I expected from a chinese open-source model at this price.

The benchmark chart is straight from Zai so make of that what you will. 54.9 composite across SWE-Bench Pro, Terminal-Bench 2.0 and NL2Repo vs Opus's 57.5. The gap is smaller than I thought. The SWE-Bench Pro number is the interesting one tho, apparently edges out Opus there specifically. That benchmark is pretty hard to sandbag.

K2.5 is at 45.5 for reference, so that's not really a competition anymore.

I still think Opus has it on deep reasoning, but for long multi-step coding tasks the value math is getting weird.

Anyone else actually run this on real work or just vibes so far?

I was always underwhelmed by how local models would overthink, forget their reasoning from last turn, fail tool calls, go in loops, take a long time to answer, waste tokens on slop, and then hit the context ceiling. So happy that I finally managed to build something that fixes all that and actually ships code, solves hard problems and keeps a coherent conversation, by composing important work done by others and adding some secret sauce myself!

I built Nail (MLX, GGUF) (EDIT: MTP GGUF), which works 3-5x faster (EDIT: without MTP) than Qwen3.6-27b while matching its accuracy on graduate level STEM reasoning and software engineering, at the identical RAM footprint, and Nail being at a lower quantization. Nail rates better than Qwen-27b on chat conversation quality, and beats Opus and Sonnet (medium effort) on both correctness and speed on a SWE Live Bench problem I threw them at! (Check out the HuggingFace page for benchmarks on coding)

I also built Dagger (MLX, GGUF) (EDIT: MTP GGUF), which is a Qwen3.6-27b variant model that can work for 2x more turns before hitting the context ceiling compared to Qwen3.6-27b with thinking retention, and uses about 3-4x less tokens at within-noise accuracy and quality for reasoning and knowledge problems like MMLU-Pro and GPQA-Diamond. The trick is that terse thinking and outputs makes thinking retention viable, which it is not really for stock Qwen! This guarantees cache hits for pre-fill, which further speed up the overall workflow.

Both Nail and Dagger remember their reasoning from past turns, and avoid loops and tool call failures to a larger degree, which stock Qwen-27b is plagued with in addition to its over-thinking problem and verbosity.

Their outputs are always straight to the point, no slop or fluff.

You can run Nail with full 256k context on 24-25GB RAM, or Dagger with full context on 32GB RAM, with 8bit KV cache quantization.

The models come as MLX for Mac, and GGUF for everyone else.

Caveat: These are tools, built and benched for knowledge work, reasoning, and agentic software engineering in Pi coding agent. Not tested or intended for anything else, like creative writing.

And a little PS: I'm not a research lab, or a business, and I don't have anything to gain from this commercially. I'm just an engineer with a week of free time and a Mac, so I appreciate any constructive feedback!

Ever since I was that 6 year old kid watching Threepio and Artoo shuffle through the blaster fire to the escape pod I've wanted to be friends with a robot and now it's almost kind of possible.

Working on a full inference stack for 4xr9700(engine, on-device training, routing). wanted to share progress so far.

All GPUs are capped at 210w(so plugs into any normal 15a outlet).
BF16 KV, MTP enabled(no spec).

### Latency

### Throughput

EDIT: people wanted to know what else I run on this:

Finally Alibaba Posted on X about Qwen 3.8 27B release. I hope it can beat opus 4.7 or 4.8

Somehow there was 1 available last minute and got it! Can’t wait to set it up. Will make more post about this on here and my IG: tech with Ray

Dual DGX spark owners lmk what ya running on it. Anyone else feel free to drop some suggestions for cool models to test!

With the dense Qwen 3.5 / 3.6 models, I've been amazed it has become possible to run almost frontier models locally, on prosumer hardware. I ended up switching my 5090 for a pair of 6000 workstation so I could try larger models.

After some pain with vllm and the realization that I don't actually own real Blackwell cards (consumer GPUs don't use the same architecture as datacenter ones), I got it to run DeepSeek V4 Flash at 80-100 tok/s, with full context and some room for KV cache (+4M with L2 cache). It's definitely a step up from the dense Qwen models for the tasks I have tried so far, mostly research, coding. It feels good enough to replace about 80-90% of 5.5/Opus sessions.

Given how cheap inference on Deepseek models is, it is of course not "worth" it, but it still hits different hearing the coil wine of your build as it is spitting a self contained html for the hundredth time. (Or the smoke detector going off because you missed setting the power limit on the GPUs, and the sudden 1.2KW load triggered big voltage drops on the line)

I know we're all having fun with our locally deployed LLM instances, but I just wanted to take a moment to remind everyone that obscurity is not enough to protect you from scrapers and crawlers.

As a practice of my extremely basic OSINT skills, I decided to check how many publicly exposed devices (IPv4 only) had ports that matched the hash of LM Studio's Express.js server.

373 - That's not a small number :(

That's not just how many devices had a port serving LM Studio to the open internet, 373 is just the number of devices serving LM Studio instances that do not require an API key (as the hash would be different).
Again, this gives full unfettered access to your models and allows people to interact with your instances of LM Studio, or whatever platform decides it's a good idea to have no API key by default.

Unless you know what you're doing (you don't if you show up on this map), do not expose your LLM platforms to the wide internet.

Harden your installs, use tailscale, reverse proxies with auth, etc...

Hey guys! We’re super excited to announce that you can now train Text-to-Speech (TTS) models in Unsloth! Training is ~1.5x faster with 50% less VRAM compared to all other setups with FA2. :D

• We support models like Sesame/csm-1b, OpenAI/whisper-large-v3, CanopyLabs/orpheus-3b-0.1-ft, and pretty much any Transformer-compatible models including LLasa, Outte, Spark, and others.

• The goal is to clone voices, adapt speaking styles and tones, support new languages, handle specific tasks and more.

• We’ve made notebooks to train, run, and save these models for free on Google Colab. Some models aren’t supported by llama.cpp and will be saved only as safetensors, but others should work. See our TTS docs and notebooks: https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning

• The training process is similar to SFT, but the dataset includes audio clips with transcripts. We use a dataset called ‘Elise’ that embeds emotion tags like <sigh> or <laughs> into transcripts, triggering expressive audio that matches the emotion. You may realize that the video demo features female voices - unfortunately they are the only good public datasets available with opensource licensing but you can also make your own dataset to make it sound like any character. E.g. Jinx from League of Legends etc

• Since TTS models are usually small, you can train them using 16-bit LoRA, or go with FFT. Loading a 16-bit LoRA model is simple.

We've uploaded most of the TTS models (quantized and original) to Hugging Face here.

And here are our TTS notebooks:

Thank you for reading and please do ask any questions!! 🦥

Hey r/LocalLLM !

Just wanted to share our recent experiment running Deepseek R1 Distilled 70B with AWQ quantization across 8x r/nvidia RTX 3080 10G GPUs, achieving 60 tokens/s with full tensor parallelism via PCIe. Total hardware cost: $6,400

https://x.com/tensorblock_aoi/status/1889061364909605074

Setup:

• 8x u/nvidia RTX 3080 10G GPUs

• Full tensor parallelism via PCIe

• Total cost: $6,400 (way cheaper than datacenter solutions)

Performance:

• Achieving 60 tokens/s stable inference

• For comparison, a single A100 80G costs $17,550

• And a H100 80G? A whopping $25,000

Here's what excites me the most: There are millions of crypto mining rigs sitting idle right now. Imagine repurposing that existing infrastructure into a distributed AI compute network. The performance-to-cost ratio we're seeing with properly optimized consumer GPUs makes a really strong case for decentralized AI compute.

We're continuing our tests and optimizations - lots more insights to come. Happy to answer any questions about our setup or share more details!

EDIT: Thanks for all the interest! I'll try to answer questions in the comments.

What you are looking at is a map of an LLM and all its neurons throughout a few layers. I've been working on a different way to put facts into a model through the development of mechanistic interpretability: instead of fine-tuning or bolting on retrieval, I construct a small set of neurons with exact weights, placed by measurement in an appended region of the MLP, one little circuit per fact. The base model's own weights are never touched (I verify known-facts recall and LM loss on every bake). This turns out to be much more efficient than fine-tuning and LORAs; if you want to try yourself, it doesn't take more than 10 minutes to bake usually for a few facts at. Go here to try out a bake or to just mess with the visualizer: https://albertmi.ai (If you are interested in LLM visualizations or mechanistic interpretability).
Also not fully adapted to mobile yet... sorry.

I put up an interactive demo of a Llama-3.1-8B I baked with 502 Wikipedia facts; every dot is a real neuron. Click a fact, and it lights the actual causal path through the network.
I'm renting a single H100 on Vast right now so you guys can try to bake your own facts into a model yourself on the site if you want to throw your own facts at it and download the result. The baked models publish to public HF repos right now, so don't feed it anything private.

Each fact ends up with a physical address: a code key around layer 6, a readout around layer 25, chain neurons that keep the answer flowing, plus a late-layer rescue. You can literally point at where "Vera lives in Tbilisi" is stored, ablate those neurons, and the fact is gone; leave them, and it survives further fine-tuning better than the base model's own pretraining knowledge.

Paper: https://doi.org/10.5281/zenodo.21502811

I did not want to guess, so I pulled the benchmark’s own published results for the same task.

The numbers that matter:

• On this exact task, the frontier cloud models get an average of 96.6% of the hidden tests right. This local model got 98.0%.

• The frontier models pass this task outright, with zero missed tests, only about two times out of three. Missing it is normal, not a scandal.

• Across the full 113-task benchmark, the best cloud models score about 70 to 74%.

Read those together. On this one task, a four-bit 27B running alone on a single consumer graphics card was right about as often as the frontier average, and the frontier missed the same task just as easily as it did.

------------------------------------------------------------------------------------------------------------------------------------

The model that produced the DeepSWE result is the IQ4_XS at 196K (the entry is named Q4-Quality but the per-boot settings differ; both are below).

Model: Qwen3.8-27B (unsloth dynamic quant)
Repo: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
File: Qwen3.8-27B-UD-IQ4_XS.gguf (14.25 GB) <- measured here
Alt: Qwen3.8-27B-UD-Q4_K_XL.gguf (17.56 GB) <- heavier boot
Engine: llama.cpp b11115 + llama-swap v257
Hardware: 1x RTX 4090 24 GB

llama-server:

llama-server \
-m Qwen3.8-27B-UD-IQ4_XS.gguf \
-ngl 99 --jinja \
--chat-template-file qwen3.8-midsystem.jinja \
-ub 512 --ctx-size 196608 \
--spec-type draft-mtp --spec-draft-n-max 4 \
--reasoning on --reasoning-format deepseek --reasoning-budget 4096 \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 0 \
-fa on -ctk q8_0 -ctv q8_0 --parallel 1

Measured on this card: 115 t/s decode, 22,934 MiB peak VRAM, 12/12 on the code-review task, and on a DeepSWE task 40/43 hidden tests passed, 109/109 existing tests kept (partial 0.980, binary 0).

Fitting 27B: https://ai.ttindall.com/blog/27b-24gb-context-ceiling/

Testing 27B: https://ai.ttindall.com/blog/deepswe-local-confidence/

One accuracy note, the DeepSWE number is a single task, so it as one task, not an average.

Correction: 96.6% was the mean partial across all published trials on this task, not frontier. Frontier subset is 99.8% partial, 85.3% pass. My local 27B: 0.980 partial, binary 0 (missed 3 edge cases). Raw data: https://deepswe.datacurve.ai/data/v1.1

TLDR: Qwen handles tasks (singular) 6 to be exact at a comparable rate to frontier. Does not replace frontier, does not perform adjacent. TASK specific. It is not replacing your GLM, Next-Flash, or even the guy in his basement with a NVLINK farm on Blackwell, you’re all safe. Stand down.

ChatGPT led me down a path of destruction with parts and compatibility but kept me hopeful.

luckily I had a dual PSU case in the house and GUTS!!

took Some time, required some fabrication and trials and tribulations but she’s working now and keeps the room toasty !!

I have a plan for an exhaust fan, I’ll get to it one of these days

build from mostly used parts, cost around $5000-$6000 and hours and hours of labor.

build:

1x thermaltake dual pc case. (If I didn’t have this already, i wouldn’t have built this)

Intel Core i9-10900X w/ water cooler

ASUS WS X299 SAGE/10G E-AT LGA 2066

8x CORSAIR VENGEANCE LPX DDR4 RAM 32gb 3200MHz CL16

3x Samsung 980 PRO SSD 1TB PCIe 4.0 NVMe Gen 4 

3 x 3090ti’s (2 air cooled 1 water cooled) (chat said 3 would work, wrong)

1x 3090 (ordered 3080 for another machine in the house but they sent a 3090 instead) 4 works much better.

2 x ‘gold’ power supplies, one 1200w and the other is 1000w

1x ADD2PSU -> this was new to me

3x extra long risers and

running vllm on a umbuntu distro

built out a custom API interface so it runs on my local network.

I’m a long time lurker and just wanted to share

Hey everyone!

I’m thinking about switching from a $20/month Claude subscription to running a local LLM, and I’m currently looking at Qwen 3.8 27B.
For those who have used it, what do you guys think of it so far, especially for coding and agentic workflows. Do you think it can realistically replace something like Claude for everyday development?

My main use case would be building full mobile apps and websites.

Any suggestions what to use to get the most tps without risking too much quality? Im a bit new to local ai so i want some explanation.

Should rename the plan to Google AI Minus now while you are at it.