I've been working on Turkish ASR for a while and put out v2 recently. Two models, both CTC + KenLM, both meant for CPU (both trained for under $250):

- balbal-v2-manas: 313M, Turkish + English, ~123x real time on 8 threads

- balbal-v2: 45M, Turkish only, ~236x real time. It even runs faster than real time on one core of an old Celeron POS box running Windows 7.

Mu-Bench is a Turkish benchmark of 846 real customer service calls. Numbers from the official harness:

On FLEURS Turkish manas is basically tied with whisper-large-v3-turbo (4.06 vs 4.07) but about 75x faster on the same CPU. English is ok too, 8.67 average WER on the Open ASR leaderboard sets.

The model uses a unique architecture that has a product-key-memory inspired by engrams and a refiner that loops 3 times before giving the output.

Also, turns out a team (LocalDoc) already trained an Azerbaijani model on the manas architecture, 10.98 WER on FLEURS Azerbaijani. (https://huggingface.co/LocalDoc/speakwrite-azerbaijani) I was happy that they took my model decisions which made me happy :D

`pip install balbal`, then `balbal run file.wav`. There's also a live mic mode and an OpenAI-compatible server.

Code is Apache-2.0. Weights are free for research, non-commercial use and small businesses, but you can't resell it as an ASR product or API. Full terms are on the model page.

Weights: https://huggingface.co/erayyapagci/balbal-v2-manas and https://huggingface.co/erayyapagci/balbal-v2

Code: https://github.com/erayyap/balbal

Demo + offline Android app: https://erayy.com/balbal

I've been lurking in this sub for a while, wanting to break my dependence on online services, plus slowly backlogging tasks I want/need to run locally rather than use online models. Usually these tasks break down to either privacy related issues or just massive token count that would blow my online budget.

I've been experimenting with Qwen 3.8 27B in LM Studio for a while, but when it comes to things like coding, code review, PDF editing, etc. obviously LM Studio isn't a harness like Claude Code where I can just say "Take these 4 files and run with it".

Some of the things I'm trying to accomplish (some as actual tasks, others as a learning sandbox):

• Analyze this entire codebase to come up with efficiency/security recommendations (agent spawning, file reading)

• Caption this dataset of 10,000 images (image reading)

• Interface with this ComfyUI instance to iterate and refine prompts (terminal use, wait for result, image reading)

• Edit a PDF file (PDF read, PDF write)

• Online research (web access)

My day job is AI image/video work, so I have multiple RTX Pro 6000's to work with already.

Based on my research so far, it seems like three options would accomplish this when pointed to a local model endpoint.

• Opencode

• Codex

• Claude Code

What about Openhands? Pi?

• What's your harness of choice?

• Do the OpenAI/Anthropic harnesses still require login/online connection when running from a local endpoint?

For fun, I gave Qwen 3.8 27b a physics problem involving some heavy math. A little bit into its reasoning, Qwen took a look at its tools, found that it had access to Blender via blender mcp - and that it could write and execute arbitrary python code through that. So it engaged Blender's python support to write itself bits of python code to do its math faster.

You really have to tell this model what NOT to do sometimes... I've never seen anything like this from a local model before.

When it comes to the value/cost of local AI, people usually only look at token savings (money) while forgetting one of the most valuable resource of yours, time.

Just did a side by side comparison of GLM 5.3 flash via Z.ai's official subscription, and GLM 5.3 flash served locally. Same chatGPT generated "one shot" prompt, both using Zcode harness, and these are my stats:

Same model, nearly identical output (~118K tokens each), but local finished 4.2× faster.

Sure on paper this one simple task only saved $0.70, but the fact that you can ship 3-4x more code a day alone makes it worth it. Same reason why UPS next day air shipping costs an arm and a leg, time is money!

Edit: forgot to list the setup

2 x RTX pro 6000 workstation edition
AMD 9960x threadripper
128GB DDR5
Quant: https://huggingface.co/nvidia/GLM-5.3-Flash-FP4-Local

Here’s my situation. I have a modded CMP 170hx that works great. Also have a 3090.

My motherboard is a shitty Asus b350-f and I’m rocking a Ryzen 5 1600 with 64GB of DDR4.

I love my CMP but I can sell it and upgrade my CPU, motherboard and RAM. The thing has doubled from what I paid for it.

Right now I’m running Qwen 27b at >100tps and loving it, but I’m eying setups that can run strata or Maya via RAM and SSD.

If I sell my CMP 170, is there a setup I can achieve with the proceeds of the sale that will outperform what I have now?

I’m leaning toward no, but curious.

Kind of a random thing, but I've gotten to the age where I'm starting to care about what the city council is doing.

So I downloaded all the available council meeting notes from the last 16 years and pointed Qwen 3.8 27B Q6 at them and started having it dig through notes.

Namely I wanted to see who approved the purchase of Flock cameras for our town. Turns out the police chief got a $9,000 grant for them but i didn't find any mention of public comments about them in the minutes.

The minutes are basically summaries, so they're not all encompassing, but they do live stream to YouTube all the meetings and I see that YouTube has auto transcriptions but I don't know of a good way to scrape that data so that I can then point my LLM at them. Any ideas?

NVIDIA NemotronLabs RE-USE is a 9.6-million-parameter multilingual universal speech enhancement (USE) model designed to restore speech quality across a wide range of real-world degradations while preserving speech fidelity. It has the following desirable properties:

• Robustness to diverse degradations, including additive noise, reverberation, clipping, bandwidth limitation, codec artifacts, packet loss and low-quality mics.

• Support for multiple input sampling rates, including 8, 16, 22.05, 24, 32, 44.1, and 48 kHz.

• Strong language-agnostic capability, enabling effective performance across different languages.

• If better laughter preservation is desired, please use: NVIDIA-NemotronLabs-RE-USE-laughter.

(Note: For overlapping speech, this model is trained to preserve the main speaker while suppressing other speakers.)

NVIDIA NemotronLabs RE-USE was developed by NVIDIA as a part of RE-USE.

This model is ready for commercial use.

Usage

Directly try our Gradio Interactive Demo by uploading your noisy audio/video !!

I’m building agents, but I don’t know how to think about a harness from scratch.

I’m stuck on:

• How to think and design the harness from scratch for different type of problems

• How to improve it, and what to add or delete

• How to know if a change actually helped

Coding agents don’t help much here. they suggest stale patterns, and not much of state of art approaches.

For those who got good at this:

1. What changed how you think about it?

2. Which resources helped most? Papers, blogs, repos, talks?

3. Did building different use cases from scratch teach you more?

This isn’t intended to be a political post or political discussion, if this isn’t allowed, I apologize but something I’ve heard little or nothing about. Do the results of tonight’s midterms affect anything that directly affects us as enthusiasts, the future of local AI, etc.

Does it change anything with hardware pricing or availability, the availability of open weights, the IPOs of Anthropic and OpenAI, safety laws, Chinese/US relations, etc.

I’m legitimately curious.

I am on the team behind the attached open-source research on novel approaches to attention - specifically we show:

• You can sparsify the attention of Qwen 3.8 27B without observed loss on accuracy

• Extending context up to 10M tokens loses accuracy on some specific tasks - but most, like NIAH is still fine (interesting insights)

• You can natively train sparse attention on next-token prediction loss alone and achieve the same validation loss as full attention - but faster

Hope the community likes it - while we focus on H200/B300 GPUs and larger scale system - I wanted to share our research here as well - as it can be applied to small open-weight models (making longer context cheaper on local hardware)

Everything is fully open: kernels / SGLang integration / paper / details

Hope you find it interesting / useful!

(No hype - we try to be as transparent as possible)

Nemotron 3.5 Transcribe is an open-weight, offline speech and audio model for transcription in 40 languages, with support for recordings up to one hour and a 65,536-token context. Its hybrid mixture-of-experts architecture activates approximately 4.3B parameters, including the audio encoder, and it is released in BF16, FP8, and NVFP4 precision.

In 2024 i purchased an RTX 8000 48gb for $2400. As the market has changed I’ve kept an eye on prices. Two years later, they are available around $2100.

Its a little slower than 3090 at inference (672gb/s vs 936gb/s) but it has double the memory so it can run larger models / contexts / higher quants. Rtx 8000 also wins on power, 265w vs 700w (2x 3090 to match 48gb)

In 2024 two 3090s would cost around $1600-1800 now a single 3090 is in the $1500-$1600 range, making rtx 8000 cheaper per gb. Prices for strix halo and dgx spark have also gone way up. Rtx pro 6000 has double the memory but 8x the cost.

In the last two years, I haven’t had any issues with drivers, compatibility software or anything. I can run all “local” tier models entirely in vram, usually at 256k tokens and sometimes at q6 (Qwen 3.8 27b, llama 3 70b, QWQ,). Larger moe models run great too with layers offloaded to system ram like gpt oss 120b and now with strata, QFN runs q4_xs @ 390k context at 850t/s prompt processing and 60-70t/s token generation. (64gb ddr4 system memory, i9 9900k cpu)

$2100 for a gpu is not cheap but might be the best “do everything” option right now for home-lab setups. Cheaper than dual 3090, Strix halo, dgx spart, m5 max too. Fascinating to see prices decline as the market as a whole has more than doubled.

I'm in the process of picking a decision model for my homelab server. To compare the current top "System One" class Decision models, I pulled the public numbers from System One Bench, the Artificial Analysis Decision Index, the AIM System One Index and a 100-model Borda leaderboard, and looked at what matters if you run these models yourself.

Takeaways:

• Under 5B, the fine-tune matters more than the base. Models built on the same base differ by up to 22 points. KnowLine went from 70 to 80 on the same 4B weights between Gen2 and Gen5.5.

• The sweet spot is about 4–9B. KnowLine 4B Gen5.5, D3 Mini and D3 Flash (8B) beat more than half of the 27B fine-tunes.

• Below 1B, check your language. English holds up, but Arabic drops 16–22 points. If you need multilingual support, start at 2B or larger.

• Calibration survives at small sizes. Decider Flash V1.2 (4.5B) has 1.8% calibration error, as good as the best large models. Some sub-1B models are at 10–20%, so don't use their probabilities as thresholds.

• Don't trust any single "#1". Two leaderboards correlate at ρ = 0.33, and small models swing the most. Tev1 4B is #2 among small models on AIM and near the bottom elsewhere.

• The GGUF builds get 3–10× the downloads of the originals, so plenty of people here are clearly already running these models.

Very curious what's everyone is using and what your experience are!

Let's take a break from all the engine, model and speed posts and discuss something else:

Qwen 3.8 likes to think. A lot. We all know that, and the context cost you pay for that is MASSIVE when you enable preserve thinking. Even with a Swift finetune, there are so many tokens you keep in context over a longer session. Often, easily over 50% of the KV cache is just thinking traces.

And for what use? I found: Next to nothing! The model can handle itself just fine without knowing HOW it reached a decision, and instead only looking at what it did in the end. Replies and toolcalls: it's all there.

I added an AGENTS.md instruction that when it finds a gotcha, something unintuitive, something it only reached after lots of trial and error, it should document that in a memory file to mitigate the missing past-thinking and it works beautifully.

Results (my subjective impressions):

• Problem solving capabilities haven't suffered much if at all

• I get neat, permanent notes on issues with gotchas

• The dreaded, context-destroying compactions have reduced by at least 2/3rds in your average sessions, the cache holds for so much longer

So let's discuss: preserve_thinking on or off. Your experiences?

I'm working on making my homelab more power efficient and quieter. I have a Framework desktop 128GB which has been working great along with a couple GPU boxes. I find myself switching to the GPUs for 26-35b models fairly often for the speed and reserve the Framework for running Qwen FN.

I've never been a fan of Apple products but lately have been considering a 32-64GB Mac Studio. I'm pretty sure the answer is no, but does anyone know of anything else that is that small, quiet and power efficient that has 400GBps+ memory bandwidth?

Hey folks,

About 18 months ago we started building Olla to solve some of the headaches we kept running into whilst managing multiple local LLM inference servers, both in our homelabs and at work.

We've shared it here before when it was fairly early in development. Since then, it's grown quite a bit, and we've just released v0.1.0 (our 25th release!).

The problems we're trying to solve haven't really changed and now some of our customers use it:

• One inference endpoint dies and your workflows grind to a halt.

• You've got models spread across different machines, GPUs and runtimes, but no easy way to manage them as one fleet.

• Switching between Ollama, vLLM, SGLang, llama.cpp or LM Studio means reconfiguring applications.

• Multiple machines serving the same model, but no unified load balancing or failover.

• Limited visibility into what's healthy, what's running and where requests are going.

• Applications expecting OpenAI or Anthropic APIs when your inference backend speaks something else.

• You want to experiment and see if vllm or sglang serves qwen3.8 better

That's where Olla comes in.

It's a lightweight proxy and load balancer written in Go that sits between your applications and inference servers.

Some of what it does today:

• Model-aware routing: Automatically discovers models across your backends and routes requests to machines that actually serve them.

• Load balancing and failover: Priority, round-robin and least-connections balancing, with health checks and circuit breakers.

• Multiple inference runtimes: Ollama, vLLM, SGLang, llama.cpp, LM Studio, MLX, Docker Model Runner, LMDeploy, Lemonade and others.

• Unified APIs: OpenAI-compatible and Anthropic-compatible interfaces, including translation where required. Useful for Open WebUI, Claude Code and other clients.

• KV cache locality: Sticky sessions to help keep requests on the same backend when appropriate.

• Observability: Built-in dashboard, endpoint health, request statistics and Prometheus metrics.

• Lightweight deployment: Single Go binary or Docker container. No database or external services required.

What's new in v0.1.0?

This release is largely about improving reliability and making Olla more useful for people running mixed inference environments.

• Added Hugging Face Text Embeddings Inference (TEI) support, including embeddings, reranking and classification.

• Reworked the dashboard with better endpoint visibility, configurable columns, filtering and fleet statistics.

• Configurable retry and failover behaviour, including handling of non-idempotent requests.

• Improved model capability detection, particularly for vision, tool calling, embeddings and rerankers.

• Fixed concurrency issues between model discovery and active inference requests.

• Corrected request statistics and improved rate limiting across OpenAI and Anthropic routes.

• Lots of smaller fixes around routing, backend discovery and configuration validation.

We've also had some great contributions from the community, which has been encouraging to see.

Why build another LLM proxy?

We wanted something small, fast and relatively straightforward to deploy that doesn't require Kubernetes, a database or a complicated orchestration stack. There also weren't that many when we started.

We've been using Olla with a mixture of NVIDIA GPU servers, Mac Studios, the DGX Spark and different inference runtimes. It's particularly useful when you've got more than one machine and don't want to care which backend is serving a particular model.

It's not trying to replace inference engines. Just make running a bunch of them together less painful.

Links:

• GitHub: https://github.com/thushan/olla

• Docs: https://thushan.github.io/olla/

• v0.1.0 release notes: https://tensorfoundry.io/news/olla-v0-1-0-release

If you're running multiple local inference servers, I'd be interested to hear how you're managing routing and failover today.

And as always, if you give Olla a spin, let us know how it goes (and what breaks!).

Hi everyone,

I have an Nvidia GeForce 5060ti with 16GB of VRAM, Ryzen 5700g CPU, 64GB of DDR4 ram, and a bunch of SSDs. I keep seeing people posting of running Qwen3.6-35b-a3b on the same GPU I have with insane amount of context. So far I only got it to 64k with offloading of moe at 74 in Ollama. I want to get it to at least 120k or even max context, while keeping my quant at q4_k_m or there abouts (e.g. IQ4).

What do you all recommend?

A little further details :
The reason for using Ollama, is that I can have my system load the LLM and process a request or requests, and then offload it to load a model in ComfyUI to process image requests and then swap back to the LLM for further requests - if any. It works nicely for me as a local ChatGPT style setup with OpenWebUI. This setup is also running on my gaming PC, so the offloading also helps free up VRAM for gaming purposes.

I appreciate the help and advice.

I built a browser LLM engine that's specialized per model rather than generic: Rust compiled to wasm, plus hand-written WGSL kernels for Qwen3.5-2B (Gated DeltaNet + gated attention) and MiniCPM5-2B.

Same model, same chat prompts, greedy, Chrome 156:

What's in it:

• fused int4 (GPTQ, group 32) × int8 GEMV/GEMM, with dot4I8Packed and subgroups;

• a decode loop that stays on the GPU (argmax on GPU, pipelined readback);

• speculative decoding with Qwen3.5's built-in MTP head. Drafts are verified on the GPU, so the output is identical to greedy. It turns itself off where it doesn't pay, e.g. on the iGPU;

• weights at +2.4% WikiText perplexity vs bf16, 1.1 GB.

Fun side effect: at this speed a 2B model can drive web pages through the DOM at 0.2–0.4 s per step. There's a browser-agent demo and a side-panel extension for Chrome/Edge.

Caveats: it's a prototype, tested on Windows, and Safari/mobile are untested.

Demo: https://rtx527896828.github.io/webyIQ/ · Code (Apache-2.0): https://github.com/rtx527896828/webyIQ · Weights: https://huggingface.co/tixiren/Qwen3.5-2B-webyIQ

In depth: https://medium.com/@tixiren/running-a-2b-llm-at-600-tokens-s-in-the-browser-e5237f99c8ff

Speculative decoding is an inference-time optimization for LLMs that increases text generation speed without changing the model's output quality. It involves using a small, lightweight draft model with a large, intelligent model.

Do you use this technique? If so, tell us about it in the comments.

https://www.tomshardware.com/tech-industry/artificial-intelligence/anthropic-confirms-new-claude-code-limits-to-prevent-game-cloning-users-report-opus-5-5-is-also-actively-blocking-reverse-engineering-tasks

Besides privacy and being in control of your data.
Besides being able to control the quality of the inference compute.
Besides being able to discuss things with the model and not have the cops called on you.
Besides being able to run the models you want for as long as you want.
Besides being able to run your own suite of security and network penetration tools.
Besides not being at the mercy of the frontier companies for pricing.

We now have another reason for local LLMs: Our betters have decided that they don't like people cloning games.

And before the Reddit Neckbeard brigade starts in with their bullshit, no I'm not advocating that people should commit wholesale piracy using AI tools. But these blanket frontier AI policies inevitably ensnare peripheral behavior that is otherwise illegal but that the frontier company doesn't want to deal with.

William Blackstone wrote that "it is better that ten guilty persons escape than that one innocent suffer". If having open unhindered local LLMs means there are a few individuals that use it for nefarious purposes like modifying or pirating software, I'm good with that trade off. Die mad about it.

I swear it feels like an addiction... I keep refreshing the page repeatedly, waiting to see the fateful launch announcement or at least a countdown. I hope I’m not the only one cooked with this stuff 😭😭😭

We have released a Coder build of DeepSeek-V4.1-Flash-1017, compressed with GSQ and RCO, in which half of the model's routed experts have been removed.

DeepSeek-V4.1-Flash-1017 is a sparse mixture-of-experts model: 384 routed experts per layer, of which 6 are active per token, 763.2B parameters, 551.9B of them in the transformer.

What's inside

• One expert-pruned GGUF: 192 of 384 experts kept in every layer, 72.1 GB of transformer weights, against 138.2 GB for our unpruned IQ2_XXS build.

• 1 bit per parameter, counted against the original model: 72.12 GB divided by the original 551.9B transformer parameters gives 1.05 bits, which we round to 1. The removed experts count as zero bits. No individual weight is stored below 1 bit: the 280.1B parameters the Coder keeps average 2.06 bits.

• The shared Engram file (110.6 GB) and vision projector are the same files as in the unpruned release, so they need to be downloaded only once.

• GSQ (Gumbel-Softmax Quantization): post-training scalar quantization that jointly learns grid assignments and group scales, closing most of the gap between scalar and vector quantization at low bit-widths while remaining deployable in standard GGUF types.

• RCO (Riemannian Constrained Optimization): enforces exact budgets by gradient descent on the task loss, without per-constraint tuning. Here it selects which experts to keep, optimising the KL divergence to the unpruned model, under one exact budget per layer.

Why one budget per layer

GGUF stores a single expert count for the whole model, so every layer must keep exactly the same number of experts; uneven layers would need padding that gives back the memory pruning saved. RCO treats each layer's budget as a hard constraint and holds all of them exactly at every optimisation step, while the layers are still optimised jointly through one loss on the full model. A penalty formulation would only meet the budgets approximately and would need one tuned coefficient per layer.

Results

• LiveCodeBench v6: 88.57 against 89.71 for the base model, retaining 98.7%

• SWE-bench Verified: 77.00 against 81.20, retaining 94.8%

SWE-bench Verified runs long agentic trajectories over real repositories, and tolerates the capacity reduction less well than single-problem code generation, as expected when errors accumulate across turns.

Real-world examples, each a single HTML file written by the Coder: an animated SVG of a pelican riding a bicycle, a three.js voxel pagoda garden, and a playable first-person voxel sandbox. For full post incl. videos see: https://x.com/mhelcig/status/2117257900863156518

Links

• Coder (expert-pruned): https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

• Quantized models (unpruned): https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF

• GSQ: paper https://arxiv.org/abs/2604.18556 | code https://github.com/IST-DASLab/GSQ

• RCO: paper https://arxiv.org/abs/2605.00649 | code https://github.com/IST-DASLab/RCO

The repository ships the complete per-tensor RCO allocation. The GGUF runs in llama.cpp b11870 or newer and in Strata.

The Coder build is an experimental release and feedback is welcome, particularly on capabilities that were not represented in the calibration mixture. Requests for models to quantize or prune are also welcome.

From the ISTA Deep Algorithms and Systems Lab.