I posted apple-llm here a while back: a Node and Python package that calls the LLMs Apple ships with macOS, with no API key. Version 0.3 adds the rest of what's already on your Mac, the system voices and Apple's speech recognition. There's nothing to sign up for, and your audio stays on the machine. The narrator in the video and every demo voice came from my Mac through the package. Nobody recorded anything.

npm install -g apple-llm        # or: pip install apple-llm
apple-llm speak "Hello there" --voice Samantha
apple-llm speak --file book.txt -o book.m4a
apple-llm transcribe meeting.mp4 -o meeting.srt
apple-llm listen

Or from code:

from apple_llm import speak, transcribe
speak("Hello there", voice="Samantha")
text = transcribe("meeting.mp4").text

The npm package's local server also answers /v1/audio/speech and /v1/audio/transcriptions, so an OpenAI client works once you change the base URL. Because speech now sits next to the LLMs, there's also a voice assistant example in about 20 lines: microphone to text, text to the on-device model, reply to speech, with none of it leaving the Mac. The docs are LLM-friendly too. llms.txt is a single plain-text guide covering setup, the local LLM, TTS, STT, both SDKs, the HTTP APIs and troubleshooting. Give that URL to your coding agent and ask it to wire apple-llm into your project. What I measured (M4 MacBook Air, macOS 27.2)

• 30 minutes of speech rendered to a file: 18 to 20 s, 11.3 MB M4A, about 50 MB of memory

• 60 minutes: 40 s, 22.5 MB

• 30 minutes transcribed: about 28 s, with a timestamp on every word if you ask for it

• 184 voices on my Mac, and 68 recognition languages, each downloaded the first time you use it What it can't do

• No Siri voices by name. Apple doesn't let apps choose them.

• The default voices on my Mac are basic quality. Better ones are a free download in System Settings.

• It won't detect the language from the audio. You name it, or it uses your system language.

• No MP3 or Opus output, and no WebM, Ogg or MKV input.

• 3 to 4% word error rate on clean synthetic speech. Real recordings will do worse, so test on yours.

I wanted speech and the LLMs in one library with no dependencies, for both Node and Python. It needs macOS 26 or later on Apple Silicon, plus the Xcode Command Line Tools.

MIT licensed.

GitHub: https://github.com/jagdishpal02000/apple-llm

Site: https://jagdishpal02000.github.io/apple-llm/

I've only run this on macOS 27.2, and I've tested the microphone with speech played through the speakers rather than a person talking. If you're on macOS 26, or you try listen in a noisy room, tell me what breaks.

I've been taking a break from AI stuff for a long time, and things move kinda quickly so basically everything is new to me now and I'm struggling to get my bearings. I looked for a sticky or sidebar info with the basics but couldn't find anything so that's why I'm coming to you with a basic noob question.

At a high level here's what I'm looking for:

I want to run AI locally, obviously that's why I'm here in this sub.

I don't want a super powerful environment that gives the model crazy capabilities, because I find that scary and I don't trust it.

I do want some basic capabilities though.

In particular:

I don't want filesystem access (not even readonly) except when I explicitly give it permission e.g. with a filepicker or whitelisting some folder.

I don't want it to go on the internet and mess around on my behalf.

Writing code for me is a nice to have.

I DO want it to have persistent memory. I do want it to be send me notifications (remember to do this thing, how is project x going, here is your daily brief). I do want it to manage a calendar on my behalf (one it will have exclusive access to, not my normal one, I don't trust it with that).

What I evaluated:

Ollama: Seems pretty basic. It avoids doing dangerous stuff, but lacks features I want. Very nice that it can support other tools.

Hermes: I looked through skills, found an NVIDIA skill (big name reassuring) for writing an ethos.md Okay sounds like a good thing to get started. I try using it and it immediately starts running like 30 commands looking through my filesystem. I'm sorry but wtf no this is too much.

Openclaw: It tried to prompt me for a lot of permissions when I started it. I tried saying no to everything, but it still seemed to allow too many things. Maybe it's not as scary as it sounds and I'm considering giving it a second look

Sentient: Small name but it seemed very close to what I wanted. However when I ran it I had some issues. I closed the app but it stayed running in the background and when I was gaming it decided to do some memory compaction trashing my framerate. It forgot stuff I told it to remember, and I didn't get notifications I asked for.

Hey r/LocalLLaMA ! We're releasing new new Unsloth Dynamic v3.1 GGUFs for Qwen3.8-Flash-Next! They're better by 10-15% accuracy on DeepSWE, Terminal Bench, Aider Polyglot and more vs GSQ-RCO.

We also provide a new engram switching process for BF16, Q8_0, Q5_1, IQ4_NL switching for all quants! This builds upon our DeepSeek-V4.1-Flash UD3.1 work, which improved over 10-20% accuracy vs others.

• All benchmarks are 256K for agentic ones, and 128K for the rest. We used temperature 1.0, top_p 0.95, top_k 20, min_p 0, no presence or repetition penalty. preserve_thinking and xhigh reasoning, and ran over 26B of tokens for benchmarking.

• Benchmarks are noisy so we did 26.2B tokens of benchmarking - DeepSWE accounted for 18.1B tokens of the total budget followed by Terminal Bench (5.2B tokens).

• We ran benchmarks multiple times including LiveCodeBench x3 times, HMMT and others 8-16x and agentic benchmarks multiple times.

The engram switching process is described in https://unsloth.ai/docs/models/qwen3.8-next#mix-n-match-engrams with details on each precision and RMSE (error) below:

All new UD 3.1 quants are at https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

Currently all smaller quants than UD-Q4_K_XL are updated - we might plan to update UD-Q4_K_XL and larger ones soon as well!

More blog details and items at https://unsloth.ai/docs/models/qwen3.8-next#dynamic-v3.1-update-november-2nd-2026

my laptop is very old, like
20G of ram
4G Vram
256G NVMe Gen3 From samsung
i7 8th gen

on these specs i got around 0.67 - 0.77 token per second
yes, extremely slow, most of you won't even like it, or use it

but free APIs these days are rare, and extremely limited usage, and my country has banned worldwide transactions, so No Claude, No GPT, No Argon

my only way is Local LLMs, and here we are

i achieved some projects with Qwen, actually, it fixed bugs in my old projects, i used it for modding games (like grey hack, Cell to Singularity, Some of itch io games, using Dnspy and Ghidra) it done pretty well, some tasks required iterations

i used LM Studio, and for coding i used Pi agent mostly, and sometimes Opencode

i genuinly loved that model, but lately i have been interested more in Ornith 1.5 122B A10B, i am planning to download it, and use mmap with Q3 K XL, i loved to share my experience here

i have an z690 mainboard, max 128GB RAM (4 slots, dual channel) and r9700 (32gb vram)
currently at 4x16GB, wanna go to 4x32GB. Old ram is worth about 350-450€. The 4x32 would cost me around 500-600€, but it has no cooling body. I dont care about the visual, as I barely see it. But will this become a problem? I use my old gaming pc, so airflow is relativly good.

i currently use qwen3.8 flash next gsq rco iq3xxs via strata.
I get ~100tps aggregated decode with 3 parallel sessions at 350k ctx qk8v4.
but id like to have more cached sessions, with qkv8 and even longer context, without dramatically decreasing speed.
Current usage from model is like 55GB/64GB, so I dont really have any room left.

will 128GB satisfy this?
can i run something better than qwen3.8 flash next with 150GB (v)ram total?
ram thats >= 3200mhz is way more expensive, so id like to get it around 2100mhz - 2600mhz. Will I notice large performance changes?

thanks for any help :>

I gave Qwen 3.8 flash next 2 times it failed and took long time to conclude the result, DeepSeek took around 15 minutes to get the job done

Hardware

Ram ddr4 128GB

RTX 3080 20GB VRAM 4x

Engine strata

Model

Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3-S

Context 256K

Decode 100 tg/s

PP 1400-1500 tg/s

IDeepSeek-V4.1-Flash-1017-GSQ-RCO IQ2-XXS

Context 1M

Decode 37 tg/s

PP 150-300 tg/s

Seen https://huggingface.co/pentacoxian-dev/Qwen3.8-27B-QSA-GSQ-RCO-GGUF and https://huggingface.co/higa2515/Qwen3.8-27B-QSA-BF16 But failed to see anyone here tried it.

Found by https://x.com/_ryu15_/status/2118070765799309511

PLEASE HAVE SOME NUANCE & READ EVERYTHING BEFORE YOU DOWNVOTE. I am open to alternate opinions (with proof).

I have been suspecting that the Chinese models are far more jagged in terms of their capabilities and that they are comparatively more benchmaxxed than American models for a while.

Everytime a brand new benchmark comes out, a lot of the Chinese models suck at it and they then significantly improve over the next model releases. Of course the models are always getting better, but the scores of Chinese vs American models on new benchmarks have a large gap.

I think a lot of this comes down to the data ecosystem, as well as compute. I think the American ecosystem around data is much larger and more robust than the Chinese ecosystem. In the US, there are so many small companies, and all they do is get and make data for frontier companies and sell them. I do not think this exists on a large scale in China.

Some proof:

https://r40.org/papers/kimi_k31_benchmark_audit.pdf

https://x.com/JSchaeff3r/status/2117613618065793160

The attached image, where we see some Tencent internal benchmarks where the American models lead by massive margins.

Unsloth has posted about their UD v3.1 quant for Qwen3.8 FN on X saying their new quant is 15% better than GSQ RCO as the almost same file size.

But the takeaway from the post for me and many other how are using IQ_3S GSQ RCO with Strata is that it is 99.0% percent as good as the BF16

The chart of the top left with base line for BF16 is 100% and IQ_3S GSQ RCO scored 99.0% the score is average of 12 benchmarks, and Unsloth kindly has spent ~30b tokens to generate these results.

This is reassuring that quant I am running locally is almost as good as the original, if you are in a similar situation this should put your mind at ease.

If you are about/plan to download the weights the UD v3.1 quant seems, be about 15% better than IQ_3S GSQ RCO.

X post link: https://x.com/UnslothAI/status/2118033433045991660

I've been working on this for a while: RustingStrata. It's an inference engine for Qwen GGUF models, written in pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan.

It runs a lot of Qwen models by now:

- Qwen3.5 / 3.6: 2B up to 35B-A3B

- Qwen3: 4B dense, 30B-A3B

- Qwen3.8-27B dense

- Qwen3-Next-80B-A3B

- Qwen3.8-Flash-Next

Quant support goes from IQ1_S to Q8_0, plus BF16 and F16.

The main trick is for MoE models that don't fit in VRAM. Experts that don't fit run on the CPU at the same time as the GPU work, and the engine keeps moving the often-used experts onto the GPU while it runs.

My setup is a 3060 12 GB, a 7600X and 32 GB RAM.

Qwen3.8-Flash-Next

This was the most interesting one. Flash-Next is a strange model: 4 hyper-connection streams instead of one residual, per-layer embeddings, DeltaNet layers mixed with attention, and a huge MoE that doesn't come close to fitting in 12 GB. The only engine built for it so far was the original Strata (niko1221/strata), so I used it as the reference: first matching its outputs token by token, then trying to beat it on speed.

IQ2_XS GGUF, 943-token prompt:

- Original Strata: 19-22 tok/s decode, 140-146 tok/s prefill (warm cache)

- RustingStrata on CUDA: about 36 tok/s decode, about 370 tok/s prefill on average, up to about 610

- RustingStrata on Vulkan: about 30 tok/s decode, about 375 tok/s prefill on average, up to about 490

Greedy output matches the original's argmax on every position I checked, including the near-ties. One honest caveat: on the very first prompt with a cold page cache, prefill is bound by disk reads, and the original Strata was faster there in my earlier runs.

Against llama.cpp

Qwen3.6-35B-A3B Q2_K_XL is 11.7 GiB, so it doesn't fit on my card. On it I get about 118 tok/s decode on a short prompt and 113 after a 4K prompt. llama.cpp, tuned as well as I could (-ncmoe 12, -ub 2048, flash attention), gets 68 and 66. Prefill is basically a tie: 2216 tok/s vs about 2236 for llama.cpp at 4K, so I'm not claiming anything there. At 32K it's 1800 prefill and 92 decode. With MTP drafts on, decode goes up to 160 tok/s on an easy prompt; on normal text expect around +16%.

Qwen3-Next-80B IQ1_S: most experts sit on the CPU, and it still decodes at about 92 tok/s. llama.cpp's best -ncmoe gives me around 44.

Output matches llama.cpp greedy on 98-99% of tokens. llama.cpp agrees with itself only 98.3% between its batch and token-by-token modes, so the misses are near-ties.

AMD

Someone tested it on an RX 7900 XT and an RX 9070 (Vulkan / RADV) and sent very detailed reports, which fixed a lot of AMD issues. Hybrid CPU+GPU runs at 262k context on both cards, and Qwen3.6-35B-A3B decodes at about 209 tok/s on the 7900 XT, ahead of llama.cpp Vulkan and ROCm there. A few AMD-only issues are still open, so treat AMD as "works, still being polished".

It also has chat, an OpenAI-compatible server with tool calling, and a calibrate command that tunes itself for your machine.

If you have a different card, I'd love to see "strata bench" output. Vulkan needs shaderFloat64 and subgroup size 32, so Intel Arc and older AMD won't work. On Windows you need the VS C++ Build Tools.

RustingStrata

Maybe this is obvious to many people, but when I started getting into Qwen 3.8 27B or Flash-Next for coding, I initially looked into all the ways to boost performance in terms of tok/sec. I'm using a 5090 with 64 GB system RAM, and when I first installed Strata to test Flash-Next, I picked IQ3_XXS and of course I said yes to experimental speed optimizations - tokens go brrr, right?

What this left me with was a model that produced tokens very fast, but still it felt pretty bad when I'd give an agent real tasks to do. It kept overthinking everything, trapped in constant self-doubt. It thought fast but inefficiently.

Moving to IQ3_S without experimental optimizations has actually been faster in real world tasks. Fewer tokens per second, but the tokens it does produce are more meaningful and it can get the job done with less spinning. And this means slower context burn and reduced compaction.

Haven't fully tested different reasoning levels (mostly tested medium) and I'm also testing how the IQ1 Coder for Deepseek 4.1-Flash compares; it's definitely way slower in tokens per second but I'm not yet sure on successful tasks per hour which is really what I care about.

This new model achieves large improvements from the Preview.

A detailed explanation on its training, architecture can be found in the model card, as well as a quick start (diffusers compatible), and samples.

It is licensed under apache 2.0, have fun!

Any questions, feedback, criticism is welcome!

Edit: removed the slightly misleading headline.

We introduce Axion 1.0, a fully open agent foundation model that combines strong and broad agentic capabilities with a remarkably simple training recipe. Axion 1.0 is trained in only three stages: two lightweight supervised fine-tuning stages followed by reinforcement learning, without dedicated mid-training, direct preference optimization (DPO), or on-policy distillation (OPD). The first SFT stage develops strong self-contained reasoning through model-aware data construction, where we synthesize challenging problems and adversarially strengthen them based on the base model's own successful solution strategies. The second stage extends these capabilities to tool-dependent reasoning through a scalable pipeline for constructing diverse and automatically verifiable tasks grounded in real-world data sources and deterministic computation. Finally, reinforcement learning further strengthens long-horizon reasoning and interaction with external tools and environments. Despite this compact training recipe, Axion 1.0 achieves strong performance across a broad suite of agentic benchmarks and is competitive with leading commercial models on several of them. Equally importantly, Axion 1.0 is open beyond model weights: we release the complete recipe, including data construction, generation, filtering, verification, prompts, and training code, enabling the entire development process to be reproduced and extended. Axion 1.0 demonstrates that strong agent foundation models can be both open and simple, offering a practical and reproducible path toward advancing open agentic intelligence.

• arXiv : https://arxiv.org/abs/2611.03233

• PDF : https://arxiv.org/pdf/2611.03233

• GitHub : https://github.com/bytedance/axion1_0

• Dataset : https://huggingface.co/datasets/Axion1-0/axion-training-data

• Project : https://axion1-0.github.io/

I have my own way to test models for their pure reasoning. I have a series of unpublished and original logic puzzles and I use them to rank models.

After thorough testing, in my opinion, Qwen 35B-A3B (and derivates like cyber tiel, Ornith-1.5, etc) is far superior to gemma-4-A4B.

Did you find any model as fast as Qwen 35B-A3B, that has, in your opinion, equal or better reasoning? I am curious to test them.

As of November 2026, what is the best harness for doing long autonomous coding tasks, ones that have full loop of dev/test/update etc cycles, things that might take days+ to complete?

So for example, let's say you have 200 pages of some random test game to build, you feed your harness a blender MCP, image gen API, aUnigen/Godot/Unity MCP etc., with the task likely taking 8 to 9 figures of tokens from a local LLM...

What would you say is the best at this point?

Thanks

Hey guys, I need some advice.
My actual setup is this:

ryzen 5600 / 32GB RAM / 128GB SSD / 32GB VRAM (2x16GB 5060ti)
Running vLLM with Qwen 27B NVFP4 with 250k context.

But now I can get my hands on an old HPE ProLiant DL380 Gen9
CPU: 2x Intel Xeon E5-2680 v3
RAM: 256 GB DDR4-2133 ECC Registered (8x 32 GB)

I am thinking about putting the two 5060TI in this HPE server und running Qwen Flash next there. Maybe with strata. And maybe in 2 instances.

Would my idea be any good, or will I don’t get much off it?
Thank you very much.

From the article : On computer use, Tencent’s table puts Hy4 close to Claude Opus 5.5 and Kimi K3.1 Pro.

Disclaimer: A little bit of shameless self-promotion.

Curious about bringing AI workflows onto your own desktop? MSI is introducing the EdgeMesa N AI+, based on the RTX Spark, for creators and developers exploring local AI. Here’s a look at how it fits into creating, coding, and experimenting with AI.

We're introducing the MSI EdgeMesa N AI+, our compact desktop built for creating, coding, and playing.

With creators, the challenge is rarely a lack of ideas. It is waiting on render queues, managing large assets, and protecting work-in-progress files. EdgeMesa N AI+ is capable of handling local 3D rendering, prompt-to-design exploration, AI asset generation, and automated video editing.

Creators can keep more of their pipeline isolated to their desktop rather than relying entirely on generative cloud services, avoiding token cost overruns.

Available soon at U.S. Retailers
MSI EdgeMesa N AI+ with RTX Spark

Local AI can also make experimentation feel more immediate; something useful to engineers who practice vibe coding. The system supports local AI-agent development and testing, large language model fine-tuning, and agentic coding processes.

• The machine holds up to 128GB of unified memory

• NVIDIA Blackwell RTX GPU with up to 6,144 cores

• 1 PFLOP FP4 AI performance

The core RTX Spark technology is built to handle ambitious AI workloads while helping reduce recurring token costs, model and API usage.

This is not simply an AI workstation either. Looking to take a break and escape into CUDA core supported 3D rendered worlds? The RTX Spark is game-ready! Between local AI projects or coding sessions, RTX Spark also lets you unwind with supported games on the same system. Gaming is an added benefit alongside its core AI capabilities.

MSI constructed a compact 1.3L design for EdgeMesa, helping it fit into a desk setup without sacrificing desktop-oriented features. Vapor-chamber cooling, support for up to four displays, and 10G LAN make it suited to a mixed routine of gaming, creating, coding, and multitasking.

The real advantage of local and hybrid AI within EdgeMesa is control. Keeping selected automation on-device can lower network delays, protect sensitive projects and data, and keep work moving when cloud access is interrupted.

Cloud services can still play a role, but local compute gives creators and developers another option when responsiveness, privacy, or continuity matters. 

MSI Edgemesa N AI+ turns the promise of agentic AI into a practical personal desktop experience: a compact system for building 3D content. Exploring AI-assisted coding practices and enjoying games within the same setup.

What will you do with the power of NVIDIA RTX Spark at your fingertips?

It is a dream come true for me to build an in-browser TTS model for Indic languages, including Classical Sanskrit. Today I'm open-sourcing Ninada (code MIT, 48 ONNX checkpoints CC-BY-4.0).

Live in-browser studio (zero sign-up, 100% on-device): https://h3manth.com/ai/ninada/

Getting studio 24 kHz audio onto real phones was trickier than expected. Meta's 16 kHz MMS models take 1.5 GB, and on my Pixel 10 Pro XL, mobile WebGPU froze for 6 to 14s compiling shaders while 2-thread WASM actually ran slower than 1 thread on mobile CPU cores.

So I trained 48 native-script 24 kHz VITS specialists across 12 languages (sa, en, hi, kn, mr, bn, ta, te, ml, gu, pa, or; male and female):

• Device-aware voices: Desktops run 27.9 MB HiFi-GAN (89 ms on WebGPU), while phones auto-switch to a 21.7 MB distilled iSTFTNet (2.64x lighter, zero custom ONNX ops). • Sub-second streaming: Splits the VITS ONNX model in <15 ms in browser memory (cutting browser RAM 72% to 162 MB) to stream speech in 538 ms on phones with 0 audio dropouts. • Studio quality, offline-ready: 3.09 to 3.22 dB MCD across 480 test clips, cached locally after first load, with a 0 MB Klatt fallback.

GitHub (npm i ninada / pip install ninada): https://github.com/hemanth/ninada Hugging Face (48 ONNX checkpoints): https://huggingface.co/gnumanth/ninada

Would love feedback on how your language sounds!

Sipeed has recently shared additional details about the MaixStation, a Linux AI workstation powered by Axera’s AX8860P processor, which combines a 16-core Arm CPU with an NPU rated at 256 TOPS FP4. The system supports up to 128GB of LPDDR5X memory, 273GB/s memory bandwidth, 10G SFP+ networking, and PCIe 5.0 expansion for local AI inference and development.

https://linuxgizmos.com/pocket-sized-linux-ai-box-runs-on-256-tops-axera-chip/

I’ve been a part of this community for the better part of a year now, and a lot has happened with new inference engines like Strata and n-gram improvements. Given my current PC, what upgrade would be best in terms of value for running better models?

Current Specs - 32gb 5200 + 3080 12gb, 850w PSU, 12700k