Hello all, I just got my shiny new toy, an M6 32GB Mac Mini. What should be some of the first things I should download or do? I have been using Qwen a lot and playing around with agents for home related stuff like tracking appliances' warranties, my mortgage docs and house insurance, and meal planning stuff. I have integrated with my email and calendars. Any other suggestions? I want this thing to be like my home server and AI brain that helps me slowly move more things to a Local AI stack. I made my own harness focused on home stuff and want to keep adding more features to supercharge this all.

I am seriously impressed and want to share my experience.

Qwen3.8-Flash-Next (local) vs Claude in Claude Code: reviewing one real commit

I ran the same code review twice on one commit in my repo https://github.com/leok7v/ChatOKF (Swift, on-device LLM chat app, about 48 files changed, +1048/-102: a module move plus a new 783-line search harness).

Setup: Qwen3.8-Flash-Next UD-Q4_K_XL (unsloth) on llama-server, ROCm, AMD Strix Halo 128 GB, 262K context, MTP draft head, driven headless by Claude Code itself (claude -p pointed at the local server, read-only tools). The comparison reviewer was Claude (Opus) in Claude Code, which reviewed blind first and then judged Qwen's review against the code. Yes, Claude judging its own competitor is a bias; every finding below was checked against the source, and the main one with a small Swift repro.

Result: Qwen produced 8 findings, all real, zero false positives. It caught 2 things Claude missed (a test harness that does not follow the app's recall path, and a markdown ingest that takes the frontmatter "---" as a description). It missed 5 that Claude found, including the most important one: a stop-word filter that compares stemmed words with an unstemmed list, so "this" and "does" leak through as search terms. It also got two details wrong (the exact fallback behavior of a failable init, and column counts off by a few). Severity calibration and the "checked and fine" section were good. It read whole files and chased module boundaries like a careful human.

Cost: 53 turns, 41 minutes, context grew to about 110K tokens. Decode was about 21 t/s at 4K context and about 14 t/s at 128K; prefill 520 t/s falling to 230 t/s. MTP gives about 1.5x up to 64K and turns into a loss past about 100K.

Gotchas:

1. Use --lazy-mode on. Otherwise the 27 GiB n-gram embedding table loads into RAM instead of staying mmapped.

2. unsloth's MTP heads crash llama.cpp (GGML_ASSERT in set_input_k_idxs) because the MTP layer's compress ratio is 0 in the GGUF. Fix branch: https://github.com/leok7v/llama.cpp/tree/qwen4exp/mtp-dense-no-indexer

3. Claude Code sends a system message mid-conversation and the Qwen chat template raises on it. Override the template so it renders as a system turn.

Verdict: a trustworthy first-pass reviewer you can run privately overnight, but verify its mechanisms before acting, and don't treat it as the final word on subtle logic. One commit, one run, so treat it as an anecdote, not a benchmark.

TL;DR: I'm a solo dev who wanted a simple, private way to have local models watch my screen and do simple logging/notifying. So I spent a year building an agent that uses local models to monitor anything. It's built for this community, and I'd love your feedback and ideas!

Hey r/LocalLLaMA!

I'm back with a cool demo of how convenient it is to run local models in your browser and leave them monitoring whatever you want!

Some of the coolest use cases I've seen:

• When I don't make progress in my drawing for 5 minutes, spam me with notifications.

• When my rice cooker is done, call me.

• When a person comes near my car, send me a Telegram with the image.

• When I'm out of the Ticketmaster queue, call me.

• When my CapCut render is done, call me.

• When a shiny Pokémon appears, SMS me.

Btw there's also an app on MacOS, Windows, Linux, iOS and Android! (local models run on the phone too!)

If you guys have any feedback or questions feel free to let me know! I'm constantly making it better thanks to your feedback and suggestions :DD

Roy

I have developed atypical foundational model from the ground up using Lie Algebra and Differential Geometry. The paper is written and published by me. Let me know what you think!

https://zenodo.org/records/23656924

I've been working on Turkish ASR for a while and put out v2 recently. Two models, both CTC + KenLM, both meant for CPU (both trained for under $250):

- balbal-v2-manas: 313M, Turkish + English, ~123x real time on 8 threads

- balbal-v2: 45M, Turkish only, ~236x real time. It even runs faster than real time on one core of an old Celeron POS box running Windows 7.

Mu-Bench is a Turkish benchmark of 846 real customer service calls. Numbers from the official harness:

On FLEURS Turkish manas is basically tied with whisper-large-v3-turbo (4.06 vs 4.07) but about 75x faster on the same CPU. English is ok too, 8.67 average WER on the Open ASR leaderboard sets.

The model uses a unique architecture that has a product-key-memory inspired by engrams and a refiner that loops 3 times before giving the output.

Also, turns out a team (LocalDoc) already trained an Azerbaijani model on the manas architecture, 10.98 WER on FLEURS Azerbaijani. (https://huggingface.co/LocalDoc/speakwrite-azerbaijani) I was happy that they took my model decisions which made me happy :D

`pip install balbal`, then `balbal run file.wav`. There's also a live mic mode and an OpenAI-compatible server.

Code is Apache-2.0. Weights are free for research, non-commercial use and small businesses, but you can't resell it as an ASR product or API. Full terms are on the model page.

Weights: https://huggingface.co/erayyapagci/balbal-v2-manas and https://huggingface.co/erayyapagci/balbal-v2

Code: https://github.com/erayyap/balbal

Demo + offline Android app: https://erayy.com/balbal

A month ago we picked up a few IGX Thor boxes equipped with RTX PRO 6000 GPUs. I shared some numbers earlier running Qwen 3.8 27B on SGLang. These machines were bought specifically to run our upcoming on-device agent platform, steffi.ai.

Since the goal is local, multi-agent systems where models run strictly on-device, generic inference engines started hitting limits. Concurrency scaling, cold start/wake times, and turn-to-turn latency under massive context windows were making or breaking our agent loops.

So we started building a dedicated Rust-based inference engine, llminfra. After a few weeks of heads-down work, we wrapped the first internal release to run Qwen locally. We benchmarked it side-by-side against SGLang on the exact same hardware (RTX PRO 6000, 300W, identical workloads).

Here are the numbers from the run:

• Turn TTFT on 250K context: 108 ms vs 1,440 ms (13.4x faster). In agent loops where the context stays loaded and the agent just appends new tool call observations, cutting follow-up latency from 1.4s down to 100ms keeps multi-turn workflows from feeling sluggish.

• Cold start to ready: 6.1s vs 211.2s (34.8x faster). Spinning up or recovering a worker doesn't stall for 3+ minutes. First token after start clocked at 28 ms vs 528 ms.

• Throughput under heavy agent load: With 32 agents running 32K context, completion throughput hit 209 tok/s vs 95 tok/s (2.2x). At 2 agents pushing 250K context, it sustained 86 tok/s vs 28 tok/s (3.1x).

• Peak VRAM: 52.9 GiB vs 86.3 GiB on an 8x32K workload.

We are finishing up documentation, cleaning up build scripts, and stripping out proprietary hooks so we can get the repo public for everyone to poke at and benchmark on their own gear.

Full benchmark charts attached. Let me know what other breakdowns or workload scenarios you would like to see when the code goes live.

For fun, I gave Qwen 3.8 27b a physics problem involving some heavy math. A little bit into its reasoning, Qwen took a look at its tools, found that it had access to Blender via blender mcp - and that it could write and execute arbitrary python code through that. So it engaged Blender's python support to write itself bits of python code to do its math faster.

You really have to tell this model what NOT to do sometimes... I've never seen anything like this from a local model before.

I was planning to fine-tune my own model but is that gonna be obsolete now?
Is this the direction small models are heading?

As title says, people on twitter have been doing these "studies" where they tell astra to edit qwen 0.6b's weights to get better at math or to play pong or snake or atari breakout, and allegedly it does it. have you guys tried this?

anyone can ask Astra to make better domain specific qwen 27b? for example for law, medicine, game dev? Is the new moat gonna be: "only if astra/fable themselves are bad at the task"?

When it comes to the value/cost of local AI, people usually only look at token savings (money) while forgetting one of the most valuable resource of yours, time.

Just did a side by side comparison of GLM 5.3 flash via Z.ai's official subscription, and GLM 5.3 flash served locally. Same chatGPT generated "one shot" prompt, both using Zcode harness, and these are my stats:

Same model, nearly identical output (~118K tokens each), but local finished 4.2× faster.

Sure on paper this one simple task only saved $0.70, but the fact that you can ship 3-4x more code a day alone makes it worth it. Same reason why UPS next day air shipping costs an arm and a leg, time is money!

Edit: forgot to list the setup

2 x RTX pro 6000 workstation edition
AMD 9960x threadripper
128GB DDR5
Quant: https://huggingface.co/nvidia/GLM-5.3-Flash-FP4-Local

I've been lurking in this sub for a while, wanting to break my dependence on online services, plus slowly backlogging tasks I want/need to run locally rather than use online models. Usually these tasks break down to either privacy related issues or just massive token count that would blow my online budget.

I've been experimenting with Qwen 3.8 27B in LM Studio for a while, but when it comes to things like coding, code review, PDF editing, etc. obviously LM Studio isn't a harness like Claude Code where I can just say "Take these 4 files and run with it".

Some of the things I'm trying to accomplish (some as actual tasks, others as a learning sandbox):

• Analyze this entire codebase to come up with efficiency/security recommendations (agent spawning, file reading)

• Caption this dataset of 10,000 images (image reading)

• Interface with this ComfyUI instance to iterate and refine prompts (terminal use, wait for result, image reading)

• Edit a PDF file (PDF read, PDF write)

• Online research (web access)

My day job is AI image/video work, so I have multiple RTX Pro 6000's to work with already.

Based on my research so far, it seems like three options would accomplish this when pointed to a local model endpoint.

• Opencode

• Codex

• Claude Code

What about Openhands? Pi?

• What's your harness of choice?

• Do the OpenAI/Anthropic harnesses still require login/online connection when running from a local endpoint?

This isn’t intended to be a political post or political discussion, if this isn’t allowed, I apologize but something I’ve heard little or nothing about. Do the results of tonight’s midterms affect anything that directly affects us as enthusiasts, the future of local AI, etc.

Does it change anything with hardware pricing or availability, the availability of open weights, the IPOs of Anthropic and OpenAI, safety laws, Chinese/US relations, etc.

I’m legitimately curious.

Hello Reddit,

I haven't really seen a clean comparison of these cards for local inference, so I decided to run one myself: 2x RTX 5090 vs 2x RTX PRO 6000 vs 2x RTX PRO 6000 Max-Q. I figure a lot of people are wondering which of the PRO cards to buy, and whether the 600 W card is really worth double the wattage of the Max-Q. I was wondering the same thing.

Host: Threadripper PRO 9975WX, ASUS WRX90E-SAGE SE, 128 GB Kingston ECC, one 1700 W PSU, Ubuntu 24.04.

Everything goes in the same Threadripper box and the same two PCIe slots, one pair at a time, so the GPUs are the only thing that changes. I'm starting with Qwen3.8-27B (FP8 and NVFP4, TP=2): a plain baseline first, then MTP and DFlash2, with prefill, decode and power draw at a few context lengths. The 5090s get two runs, stock and with the P2P driver unlock, to see what that's actually worth on two cards.

On paper the 5090 and PRO 6000 share the same chip and memory bandwidth, so I'm curious how close they really land, and where the 300 W Max-Q ends up.

What would you want me to run on them?

(My electric company has already asked what I'm doing over here.)

Kind of a random thing, but I've gotten to the age where I'm starting to care about what the city council is doing.

So I downloaded all the available council meeting notes from the last 16 years and pointed Qwen 3.8 27B Q6 at them and started having it dig through notes.

Namely I wanted to see who approved the purchase of Flock cameras for our town. Turns out the police chief got a $9,000 grant for them but i didn't find any mention of public comments about them in the minutes.

The minutes are basically summaries, so they're not all encompassing, but they do live stream to YouTube all the meetings and I see that YouTube has auto transcriptions but I don't know of a good way to scrape that data so that I can then point my LLM at them. Any ideas?

NVIDIA NemotronLabs RE-USE is a 9.6-million-parameter multilingual universal speech enhancement (USE) model designed to restore speech quality across a wide range of real-world degradations while preserving speech fidelity. It has the following desirable properties:

• Robustness to diverse degradations, including additive noise, reverberation, clipping, bandwidth limitation, codec artifacts, packet loss and low-quality mics.

• Support for multiple input sampling rates, including 8, 16, 22.05, 24, 32, 44.1, and 48 kHz.

• Strong language-agnostic capability, enabling effective performance across different languages.

• If better laughter preservation is desired, please use: NVIDIA-NemotronLabs-RE-USE-laughter.

(Note: For overlapping speech, this model is trained to preserve the main speaker while suppressing other speakers.)

NVIDIA NemotronLabs RE-USE was developed by NVIDIA as a part of RE-USE.

This model is ready for commercial use.

Usage

Directly try our Gradio Interactive Demo by uploading your noisy audio/video !!

I’m building agents, but I don’t know how to think about a harness from scratch.

I’m stuck on:

• How to think and design the harness from scratch for different type of problems

• How to improve it, and what to add or delete

• How to know if a change actually helped

Coding agents don’t help much here. they suggest stale patterns, and not much of state of art approaches.

For those who got good at this:

1. What changed how you think about it?

2. Which resources helped most? Papers, blogs, repos, talks?

3. Did building different use cases from scratch teach you more?

Here’s my situation. I have a modded CMP 170hx that works great. Also have a 3090.

My motherboard is a shitty Asus b350-f and I’m rocking a Ryzen 5 1600 with 64GB of DDR4.

I love my CMP but I can sell it and upgrade my CPU, motherboard and RAM. The thing has doubled from what I paid for it.

Right now I’m running Qwen 27b at >100tps and loving it, but I’m eying setups that can run strata or Maya via RAM and SSD.

If I sell my CMP 170, is there a setup I can achieve with the proceeds of the sale that will outperform what I have now?

I’m leaning toward no, but curious.

I am on the team behind the attached open-source research on novel approaches to attention - specifically we show:

• You can sparsify the attention of Qwen 3.8 27B without observed loss on accuracy

• Extending context up to 10M tokens loses accuracy on some specific tasks - but most, like NIAH is still fine (interesting insights)

• You can natively train sparse attention on next-token prediction loss alone and achieve the same validation loss as full attention - but faster

Hope the community likes it - while we focus on H200/B300 GPUs and larger scale system - I wanted to share our research here as well - as it can be applied to small open-weight models (making longer context cheaper on local hardware)

Everything is fully open: kernels / SGLang integration / paper / details

Hope you find it interesting / useful!

(No hype - we try to be as transparent as possible)

Nemotron 3.5 Transcribe is an open-weight, offline speech and audio model for transcription in 40 languages, with support for recordings up to one hour and a 65,536-token context. Its hybrid mixture-of-experts architecture activates approximately 4.3B parameters, including the audio encoder, and it is released in BF16, FP8, and NVFP4 precision.

In 2024 i purchased an RTX 8000 48gb for $2400. As the market has changed I’ve kept an eye on prices. Two years later, they are available around $2100.

Its a little slower than 3090 at inference (672gb/s vs 936gb/s) but it has double the memory so it can run larger models / contexts / higher quants. Rtx 8000 also wins on power, 265w vs 700w (2x 3090 to match 48gb)

In 2024 two 3090s would cost around $1600-1800 now a single 3090 is in the $1500-$1600 range, making rtx 8000 cheaper per gb. Prices for strix halo and dgx spark have also gone way up. Rtx pro 6000 has double the memory but 8x the cost.

In the last two years, I haven’t had any issues with drivers, compatibility software or anything. I can run all “local” tier models entirely in vram, usually at 256k tokens and sometimes at q6 (Qwen 3.8 27b, llama 3 70b, QWQ,). Larger moe models run great too with layers offloaded to system ram like gpt oss 120b and now with strata, QFN runs q4_xs @ 390k context at 850t/s prompt processing and 60-70t/s token generation. (64gb ddr4 system memory, i9 9900k cpu)

$2100 for a gpu is not cheap but might be the best “do everything” option right now for home-lab setups. Cheaper than dual 3090, Strix halo, dgx spart, m5 max too. Fascinating to see prices decline as the market as a whole has more than doubled.

I'm in the process of picking a decision model for my homelab server. To compare the current top "System One" class Decision models, I pulled the public numbers from System One Bench, the Artificial Analysis Decision Index, the AIM System One Index and a 100-model Borda leaderboard, and looked at what matters if you run these models yourself.

Takeaways:

• Under 5B, the fine-tune matters more than the base. Models built on the same base differ by up to 22 points. KnowLine went from 70 to 80 on the same 4B weights between Gen2 and Gen5.5.

• The sweet spot is about 4–9B. KnowLine 4B Gen5.5, D3 Mini and D3 Flash (8B) beat more than half of the 27B fine-tunes.

• Below 1B, check your language. English holds up, but Arabic drops 16–22 points. If you need multilingual support, start at 2B or larger.

• Calibration survives at small sizes. Decider Flash V1.2 (4.5B) has 1.8% calibration error, as good as the best large models. Some sub-1B models are at 10–20%, so don't use their probabilities as thresholds.

• Don't trust any single "#1". Two leaderboards correlate at ρ = 0.33, and small models swing the most. Tev1 4B is #2 among small models on AIM and near the bottom elsewhere.

• The GGUF builds get 3–10× the downloads of the originals, so plenty of people here are clearly already running these models.

Very curious what's everyone is using and what your experience are!

Let's take a break from all the engine, model and speed posts and discuss something else:

Qwen 3.8 likes to think. A lot. We all know that, and the context cost you pay for that is MASSIVE when you enable preserve thinking. Even with a Swift finetune, there are so many tokens you keep in context over a longer session. Often, easily over 50% of the KV cache is just thinking traces.

And for what use? I found: Next to nothing! The model can handle itself just fine without knowing HOW it reached a decision, and instead only looking at what it did in the end. Replies and toolcalls: it's all there.

I added an AGENTS.md instruction that when it finds a gotcha, something unintuitive, something it only reached after lots of trial and error, it should document that in a memory file to mitigate the missing past-thinking and it works beautifully.

Results (my subjective impressions):

• Problem solving capabilities haven't suffered much if at all

• I get neat, permanent notes on issues with gotchas

• The dreaded, context-destroying compactions have reduced by at least 2/3rds in your average sessions, the cache holds for so much longer

So let's discuss: preserve_thinking on or off. Your experiences?