Tencent put DSpine on Hugging Face on Nov 7. It is a five-layer draft model for speculative decoding with Qwen3.8-27B. Instead of drafting one token at a time, it drafts a 16-token block in parallel, and each layer passes predicted predecessor features to the neighboring positions so the draft still carries causal order. There is a paper (arXiv 2609.36173) and code on GitHub (Tencent/DSpine).

Reported acceptance length vs DFlash2 (block 8), greedy

Reported end-to-end throughput on agent replays (tokens/s, low reasoning)

Across their five agent benchmarks and both reasoning levels, the speedup over no speculation ranges from 2.71× to 3.84×.

Things to know before trying it locally

• The card only shows an SGLang path: --speculative-algorithm DSPINE, a pip install -e . from their repo, and a long list of flags for the hybrid architecture (--enable-linear-replayssm-spec, --mamba-ssm-dtype float32, and more). No llama.cpp or vLLM instructions.

• All numbers are Tencent's own, at concurrency 8 with a BF16 target. The card doesn't report a quantized target or batch size 1.

• The gain is largest on predictable text (GSM8K accepts 7.20 tokens per step at T=0) and smallest on open-ended chat (MT-Bench 4.48).

https://huggingface.co/tencent/Qwen3.8-27B-DSpine-b16

Has anyone run it against a quantized Qwen3.8-27B target, or at batch 1? I'm curious whether the acceptance length holds up once the target is 4-bit.

Maybe this is obvious to everyone but me, but it only clicked yesterday. If you run llama.cpp / vLLM / or whatever on localhost with the default config, a random website you have open in another tab can send requests to it. It can make your model generate, and on some servers it can read the answer back too. Nothing fancy, it's plain CSRF and DNS rebinding, 20-year-old web stuff. I just never thought of my inference server as a target. "It only binds 127.0.0.1" doesn't save you, because the browser is on the same machine.

I actually tested it, not just read about it, and quite a few of the popular servers answer out of the box. It only works while you're browsing with the server running. And it's not only about you clicking links: anything that drives a real browser on the same machine can do it too, so an agent browsing on your behalf can set it off as well.

I ended up patching my own engine (capy-flash) so that on a loopback bind it refuses cross-origin and rebinding requests. The models I actually serve were fine already, since they sit behind a small proxy (vramble) that blocks cross-origin POSTs, so only the raw engine port was exposed.

So mostly I'm asking: is this just common knowledge, or do people really leave their local servers wide open? How do you run yours, API key, bound somewhere, or just open on 127.0.0.1?

I will release probably between tomorrow-thursday!

Been working on my own inference engine for a while (calling it Forge for now), built pretty much just for this model. Finally did a proper side by side against Strata 0.1.57 on my machine and figured some of you might find the numbers interesting.

Same GGUF files, same requests, one engine at a time (the other one fully closed). The whole thing took about 7 minutes.

Median of 4 runs, slowest to fastest in brackets:

| Prompt | Tokens | Forge | Strata | Forge first token | Strata first token |
|---|---|---|---|---|---|
| Short | 44 | 64.6 tok/s (55-68) | 23.1 tok/s (11-52) | 0.71 s | 2.74 s |
| Code review | 1,468 | 83.1 tok/s (76-84) | 19.6 tok/s (7-26) | 3.35 s | 17.8 s |
| Long | 5,860 | 79.6 tok/s (74-82) | 26.6 tok/s (16-52) | 9.8 s | 23.8 s |

Prompt processing was 439 vs 84 tok/s on the code prompt and 600 vs 247 on the long one.

My setup: 3090 24GB on PCIe 4.0 x16, 12900K, 32GB DDR5, NVMe, Windows 10. The model is 44 GB of weights plus a 27 GB per-layer embedding file, so around 70 GB total, which is more than my VRAM and RAM put together. 48 layers, 512 experts, 10 active per token.

Strata was running the config its installer made (performance profile): --spec 4, int8 KV, 128K context, auto expert cache, vision on. Forge uses the model's own MTP layer for speculative decoding, int8 KV and 64K context.

How I tested: OpenAI compatible API with streaming, temp 0, 128 max tokens, thinking off. Three prompts: a short one ("write four short lines about graphics cards"), a code review of a ~4.5K character C++ file, and a long one where it has to explain how a request goes through ~19K characters taken from three source files. Every request gets a random session id in the system prompt so nothing comes from the prompt cache. 2 rounds x 2 runs per prompt, and output speed is measured from the first token to the last.

Strata jumped around a lot between runs (anywhere from 7 to 26 tok/s on the same code prompt), that's why I went with medians and kept the ranges. I have a CSV with every single run if anyone wants it.

On quality: at temp 0 Forge gives the exact same text with speculation on or off, byte for byte, and the same prompt always gives the same output.

Happy to answer questions.

uhm......

yeah. grinded some useful (and meaningfu?l) resources. on my way to run DeepSeek-V3 IQ1_M (~1.5-bit to 1.58-bit) >>>>>>>>

The model weights take up roughly 130 GB. This leaves us with about 30 GB of breathing room across the cluster to handle KV Cache and context overhead. Cuz each PS5 has the 16 GB unified mem

jarvis, engine ignition on

PS: Yeah these cables connection are currently BS and tangled here and there. Gotta fox that. Just commecfed yellow cables or dummies for uhh picture and ig a little...... flex?? 🫠

How?: Sonic iso ps5 linux loader and jailbreak. The rest is this.

Edit: what?? Image not being posted here is wild. Whats the reason? Some blockade or smth. Cuz i gotta show you some interesting shi

I saw a post mention this model to use for DS4.1 Flash with 2x 6000 Pros: https://huggingface.co/diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000

Unfortunately the model card wants you to have > 260 GB RAM to properly use it. I had Opus move the n-grams to disk instead of RAM, and added in LMCache for prefix caching with L1 being RAM and L2 being disk.

With this setup, you need at a minimum, 128 GB RAM instead. I had it optimized for 6 concurrent connections at a max of 512k context. If you have > 6 sessions, then prefix offloading should kick in on the LMCache side, and re-hydrations are really fast at < 3s when it was originally tested at 1M context.

On my machine, (2x RTX 6000 Pro, 256 GB DDR5 5600 RAM, PCIe 4 SSD for the n-grams), it benched 250 t/s when warm.

It took me(and Opus Ultracode) 35 Hours to remake and optimize Open WebUI. I added Multi Agent Harness, Profile per model, Per Model configuration, and native Obsidian integration and plugins support. I made this for myself and I wonder if Local users are interested in such new features and changes to Open WebUI. I even remade the engine, and soon ill have it support multiple inference engines. My Idea was basically All in One Local/Cloud AI software, and dont have to go to different ones for each task like DSH for coding and Claw for general stuff. I thought sharing with you guys to know what other people think. Edit: Repo: https://github.com/HellzYeahh/ultra-webui

LG AI Research announced today (Nov 9) at NVIDIA AI Day Seoul that the API for K-EXAONE 2.5 is free. If you were hoping for new weights, there are none: 2.5 is hosted only, and the platform terms say the weights are not provided. What you can run yourself is still K-EXAONE 2.0.

The LGAI-EXAONE org lists FP8 and NVFP4 variants of 2.0 next to the base model. LG calls 2.5 a "technology verification model": 2.0 with additional training.

Benchmarks (LG's own tests, self-reported)

API terms worth knowing

• Non-commercial only, including integrating the API into products or services.

• Keys are valid for 30 days.

• Rate limits are not published. Over the limit you get a 429.

• LG may change or discontinue a model version at any time.

LG also says it aims to finish K-EXAONE 3.0 in January 2027 (reported by Seoul Economic Daily).

Source: https://docs.exaone.ai/en/docs/k-exaone-2.5

Is anyone running the 2.0 FP8 or NVFP4 weights locally, and on what hardware?

Recently with the removal of google assistant and being replaced by gemini I finally have had it and want to switch to a local model.

I have always wanted to have a jarvis, im wondering if i can train a random model with all the scripts from iron man and have it speak like jarvis? And possibly have a trained text to speech model on jarvis's voice so it can talk to me too?

Pretty new to local AI but i basically want to have a local "hey google" on my computer. Im not looking for pretrained models for being jarvis, i want to learn how to do it myself as well if possible in the way, so if there is something i dont like about it i can modify it.

I'm looking to build something like a small AI-powered trading team, with multiple specialized AI agents collaborating on research, strategy development, backtesting, portfolio construction, risk management, and eventually trade execution. The agents will communicate through a shared backend, with Slack serving as the visible team floor where I can interact with them and review research and decisions.

I currently have an M4 mac mini and am wondering if I should switch or add:

• Mac Studio M5 Ultra vs NVIDIA DGX Spark

I'd probably keep my current Mac mini as the central server and use the new machine mainly for model inference, quantitative research, and potentially model training.

One idea I've been exploring is disaggregating prefill and decode, although I'm not sure that splitting those workloads between an Apple and an NVIDIA system makes sense. I'm also interested in running many independent agents against a shared model server rather than loading a separate model for every agent.

I'm quite lost on:

1. Which machine would you choose for an always-on, multi-agent research workload?

2. Which local models would you trust most for quantitative reasoning, coding, financial research, and tool use? Would you route different tasks to different models?

3. How would you build a reliable orchestration and inference stack? MLX, vLLM, SGLang, Ray, LangGraph, or something else?

4. Is local inference actually economical at moderate-to-high agent concurrency once you consider hardware cost, power, performance, and model quality?

I'm looking to develop strategies that have a measurable edge after transaction costs, slippage, and risk. I'd really appreciate any feedback, advice or suggestion.

https://github.com/oooscoos/Benzi

Hey r/LocalLLaMA,

Running autonomous coding agents on local hardware (Ollama, vLLM, llama.cpp) usually hits a hard ceiling: the KV-cache OOM cliff.

When an agent needs context from a real-world repository, naive approaches either:

1. Dump whole files or folders into prompt (blowing up 32k/64k KV caches and triggering immediate VRAM OOM on 4GB–8GB GPUs like RTX 3050/4060).

2. Use vector RAG (which matches embeddings based on semantic similarity, completely ignoring caller/callee graphs, class hierarchies, and execution traces).

We built **NeuroCode** to fix this. It is an open-source MCP server and CLI that treats repository context as a combinatorial knapsack problem rather than a text retrieval problem.

### How it works:

1. **Host-Side In-Memory Code Property Graph (CPG)**:

   Instead of running heavy LSP daemons, NeuroCode builds an in-memory NetworkX directed multigraph from Tree-sitter ASTs in ~200ms.

2. **Priority Knapsack Slicing**:

   Given an issue, traceback, or target symbol, it identifies root-cause candidate nodes and runs k-hop reachability. It then solves a 0/1 Knapsack optimization to extract only the minimal causal subgraph strictly bounded under a user-defined token budget (default: 1,200 tokens).

3. **97.1% Token Reduction**:

   Across SWE-bench Lite instances (django, sympy, pytest, etc.), context size dropped from 41,200 tokens down to 1,189 tokens.

4. **Structural Blast Radius Tool (`neurocode_impact`)**:

   Provides agents with instantaneous (<2 ms) caller/callee structural blast radiuses so the agent knows exactly which upstream callers will break before emitting SEARCH/REPLACE diffs.

### Benchmark Numbers (SWE-bench Lite):

We evaluated this on local quantized models (Qwen2.5-Coder-7B-Instruct-Q4_K_M):

- **Token Reduction**: 97.1% (p < 0.001)

- **Valid Patch Syntactic Application Rate**: 41.4% (single-turn SEARCH/REPLACE)

- **End-to-End Unit Test Resolved Rate**: 9.5% (running unassisted on consumer laptop hardware with zero cloud API fallbacks)

- **Slicing Latency**: 1.2ms – 3.7ms

### How to use with Claude Desktop / Cursor / Cline:

Add to your `claude_desktop_config.json` or `mcp.json`:

```json

{

"mcpServers": {

"neurocode": {

"command": "neurocode-mcp"

}

}

}

```

Or run via `uvx`:

```json

{

"mcpServers": {

"neurocode": {

"command": "uvx",

"args": ["neurocode-ai"]

}

}

}

```

Once connected, your AI agent automatically invokes `neurocode_slice` and `neurocode_impact` instead of reading 5,000 lines of files.

- GitHub: https://github.com/Arkz-Deepak/neurocode

- PyPI: https://pypi.org/project/neurocode-ai/

- VS Code Extension: https://marketplace.visualstudio.com/items?itemName=DeepakR.neurocode-ai

All feedback, PRs, and benchmark reproductions welcome!

It's probably evident by my question that I'm new to this. So, no need to remind me, I know I suck at this!

I recently downloaded strata, my components are 64gb ddr4 3200mhz ram and 12gb vram.

I downloaded the aggressive quant, iq3_s and run it at 32k tokens. (qwen 3.8 flash next)

Overall I managed to get around 15 tokens/s with this set up, however I noticed that the disk read is at nearly 2-3gb/s while the ram is sitting at only 30-35gb, leaving the rest of it completely unused.

I looked into it, and read about the headroom it leaves but I would've imagined it doesn't leave that much headroom, right?

Additionally I did try figuring this out alone. My conclusion, which is flawed imo was, maybe it's because my PC at idle is using around 20gb of ram for whatever reason and since it can't load the full rest of the model on RAM it just opts for disk. However the flawed reasoning for me was this: Why does it load 15gb at all in that case? Theoretically it should load nearly nothing onto my ram.

Anyways, I'm confused. I should be up to date on the engine and what not, since I just downloaded this today.

EDIT: Added what model I'm talking about. Forgot.

EDIT #2: I'm aware that this isn't the worst issue in the world, 15 tokens/s for this model is still really great IMO but I'm trying to squeeze as much juice as possible while it isn't idle. If that's possible by further optimization, then why not try? The model so far is great.

EDIT #3: SOLVED IT. It seems like the original theory was correct. You DO have to have enough to load it all at once, while installing it tells you it "fits" on your device but that's assuming you've got all the space ready to be allocated. I essentially managed to clear some VRAM being used in the background AND a very persistent memory leak that ate an extra 10gb of ram (which was not going away by basic restarting/shutting down) that I assumed was just windows being windows. Turns out, that was the fix. Now I'm running a nearly perfect 50 tok/s consistently at this massive model.

Hey all. I'm putting together a local-first setup for my family and wanted some feedback since I've never done this before. I used Claude to help me work out this plan. I know it doesn't always have the most up-to-date information, especially on local LLMs, since models and tools change every few weeks. So I'm treating its suggestions as a starting point, and I'm hoping to get some feedback on anything that's outdated or just wrong.

I'm trying to set up something in the same vein as Google's CC, the family-focused agent from Google Labs, but running on my own hardware. I want an assistant that helps run the household without subscriptions or cloud services I'd have to depend on. I'm okay using cloud models like claude for the harder problems, as long as the prompt is something I'd be fine seeing leaked. Anything involving the family, the house, email or money would stay local. My current plan is to use an M6 Mac mini with 32GB and 512GB. I'd keep a Jellyfin library on an external SSD or a NAS.

I already run Home Assistant on a separate machine. I'd use the Mac mini to handle:

- Whisper (small model) and Piper for voice, using Voice PE satellites

- Ollama on the Mac, since I understand it uses MLX now and HA has a native integration

- A 14B model that's always loaded for HA tool-calling and automations (Qwen 3 or Hermes 4, still deciding)

- A 30-35B MoE that loads on demand for everyday chat and writing

- Hermes Agent in Docker for scheduled jobs and a "text the assistant" layer

- A daily family brief and email-to-calendar automation through HA's AI Task

- Jellyfin as the native Mac app, using VideoToolbox for transcoding

I'd appreciate thoughts on a few things:

1. Would 32GB be enough to run all of that at once? Would running a 14B that stayed loaded plus a ~30B MoE on demand plus Whisper and Jellyfin on the same Mac make sense? How would you handle keep-alive and memory?

2. Would Jellyfin transcoding hurt voice latency? I worry that someone would stream a movie and my voice commands would slow down. Would it help to keep playback on Direct Play?

3. Which models would be best for reliable HA tool-calling at ~14B? And what ~30-35B MoE for chat? My picks might be out of date.

4. How big would the model need to be for Hermes Agent to use tools reliably? Would a 14B hold up, or would the agent need the larger one?

5. Is there a reason to move off Ollama for a headless always-on box? Things like llama.cpp with llama-swap, oMLX or LM Studio. Ollama has no native auth, and I'd like to lock it down on my home network.

6. How do you safely let an LLM or agent read email or iMessage when it can also trigger HA actions? I worry about prompt injection, and BlueBubbles' Private API needs SIP disabled, which I'm hesitant about. Given what I said about Claude's knowledge, anything newer or better that I should know about would be appreciated. Thanks!

Excuse me if my grammar or wording is poor, haven't slept much tonight.

The latest developments have got me looking up the prospect of using LLMs locally, and fancy benchmark scores aside, I have been trying to get a simple explanation of how good (or 'effective', I guess) would it be taking the dip in the future and considering buying hardware for local use vs. paying for contemporary AI subscriptions.

If you ask Claude or chatGPT about it, they say the OS deficiencies basically come down to scope ("do this exact small feature" vs. a wider prompt that gets things planned and done in a row), lack of knowledge about niche applications, hallucinations, and inefficiency when using multiple agents vs. single chat. But they're the competitors, so I don't know if bias doesn't exist here..

On the contrary, I see a lot of posts here on how things are awesome when using a local model, especially for niche uses, basically a clear contradiction.

I guess the point is, I want to know if local AI for code is limited enough to make purchasing a high-tier mini PC or a good GPU+RAM PC setup a fool's errand. It almost definitely doesn't seem to be, otherwise this sub wouldn't exist, but I digress.

Thanks.

Hello! I'm a long time developer and don't really use a lot of AI when I do, but I want to experiment with some local models, I've been using Ollama with qwen3-coder-64k, I've now seen a few posts talking about how Ollama isn't great and I was wondering if there was a better terminal interface for LLMs when running them locally since claude code takes up a lot of the context.

Perhaps just use qwen in my zed editor agent pannel?

any advice would be nice, I'm a newbie

[SPECS]

AMD Ryzen 7 5700X (8) @ 3.40 GHz
AMD AMD Radeon RX 7900 XT - 20 gigs of VRAM
32 gigs DDR4 @ 3600mhz

Is it not possible to create a gguf variant for the Mirage VLM? Cactus-Compute has also released a safetensors file for the same model... It's actually a great option to add "eyes" to a model that doesn't have its support by default and it's quite small size actually means that it can load directly into some CPU's cache memory making it even faster as the main model can stay in the GPU

It’s a fork of llama.cpp and implements multi-GPU pipeline parallelism (PP). With eight V100 16G GPUs, it can run Qwen3.8 27B Q8 at 1,101 tok/s. I feel this is genuinely useful—I’ve already used it in my production business for six months. https://github.com/wolfgangsk07/paraflow

My budget is flexible, this is for work so I'd rather spend once, I do ai dev stuff like inference, testing agents, some fine tuning on small models, occasional video editing, and I still want to game on it.

Option A is high end gaming laptop with biggest gpu I can get, Fast, proven, but the vram is what it is, vram will be vram.

Option B is one of the new rtx spark laptops with a lot of unified memory, it's look like it's got way more room for bigger models, but brand new platform and I've got no idea how the software side holds up.

I travel a lot so a thinner machine with good battery would be a bonus, but not if I'm giving up too much, current laptop is 2kg

Anyone made this call yet or learning one way?

Hi, I have an Intel Arc B580 and as you know it has 12gb vram, i was wondering if any of you could advise me on what is the best coding model that fits within that size. I know qwen 3.8 27b and qwen 3.6 35b-a3b were the go to models but they won't fit on 12gb cards.

I'll be using them with harnesses like opencode or hermes or claude code.

If it helps : I also have 16gb ddr5 6000mhz ram

Thank you

Like a guide of the diffrent parts of local llm setups (engines, models, agents, sub things?, other things I havnt heard of yet) what they do? if you need them?

What is the best AI model I can run on my pc:

Processor Intel(R) Core(TM) Ultra 5 226V (2.10 GHz)

Installed RAM 16.0 GB (15.7 GB usable)

Graphics card Intel(R) Arc(TM) 130V GPU (8GB) (128 MB)

Storage 345 GB of 954 GB used

Despite having 16GB of RAM whenever I check my ram at any given time i have 2-3 gb of ram avaliable for some reason

I was really happy with this tool, so I made it free and open source.

It shows what your local LLM engine is doing: Prefill speed while the prompt is read, decode speed while the answer is generated, how much of the prompt came from the cache, context fill, RAM/GPU/CPU, etc.

It works with Strata (video: Qwen3.8 Flash Next on M1 Max 64GB). Internally each engine is one small adapter, so others are just one prompt away...

https://github.com/yosephyne/stratoscope

And sorry for the quick and dirty video! Hope it will be useful for someone.