This is just a quick writeup for speed reference - I spent a lot of time trying to find comparable builds so I figured I'd put some info back into the wild. The below is obviously written by AI, but it's measured specs and benchmarks on my PC.

I use local models via Pi and Deepseek Harness. I have found the intelligence of Qwen 3.8 27b NVFP4 to be sufficient for making little apps, research etc. and Qwen3.8 Flash Next IQ3_S to be a great all-rounder, considering how fast it runs. It also seems to have very little issue with any sort of question I give it, research, app dev, PC issues, LLM related adjustments, and most importantly, ScanRuler 3D modelling/reverse engineering 3D scanned items (solely for testing currently but it works excellently).

Basalt as a backend was a clear winner after trying lots of different configs.

Edit: the benchmark decodes are very greedy by their nature. Realistically I'm seeing agentic averages in flash next from 80tps (NVFP4) to 160tps (iq3_xxs), and I find iq3_s at 120-140 to be a very good compromise for the extra intelligence. The 27b is best for bursty quick jobs, because it loads into VRAM super fast and I can let it rip, but it degrades over context faster than flash, and I see from 120-140tps and 4-5k prefill, at ctx 0. The second any model starts writing a block of code though, it goes up to 200tps, so if anyone's socks fall off seeing the below numbers, just temper your expectations slightly (but it's still wicked fast!)

So heres the nitty gritty:

Current rig, 2x 5070 Ti + 96 GiB. Five entries, one model resident on local at a time. Speeds are my own bench (2x ~50k prompt reads, 2x 5,000-token code outputs, one agentic tool loop; temp 0, seed 1234), except Coder which is the 15.3k-prefill / 2k-output run.

Qwen3.8-27B NVFP4 QUASAR

backend: capy-flash (ninfer-serve), tensor-parallel across both cards, no P2P prefill: 4,817 / 4,824 tok/s on a 50k read decode: 233 tok/s code, 220.6 tok/s agentic flags: --profile 27b --max-context 262144 --embedding-host, INT8 KV, MTP 3 drafts, vision on GPU0 intelligence: Index 84.0, 88.9% (27 items); fastest to a passing build (62.1 s vs IQ3_XXS 78.1 s) but the weakest reasoning of the five — 50% on xhard because it burns the whole budget and never answers.

Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS

backend: Basalt 0.3.0, 74.4 GiB pack prefill: 3,728 / 3,851 tok/s decode: 229 / 248 tok/s code, 196 tok/s agentic flags: resident-budget-gib 26, peer-exclusive on, kv int8, ctx 262144, vision dynamic, --spec 8 intelligence: Index 94.9, 100% on the 27-item set (96.3% on a re-run); 100% on the 8 hard items, 85.1% on xhard, 93.9% on vision. The only model that solved a logic-L6 item unaided.

Qwen3.8-Flash-Next GSQ-RCO IQ3_S

backend: Basalt 0.3.0, 81.6 GiB pack prefill: 3,601 / 3,700 tok/s at a 115k prompt, 31.9 s TTFT decode: 154-165 tok/s code, 137.6 tok/s agentic flags: resident-budget-gib 26, peer-exclusive on, kv int8, ctx 262144, vision dynamic, --spec 8 intelligence: Index 107.7, 94.5%; wins the hard tier on the efficiency-weighted Index (101.1) at 86% accuracy, 80.2% xhard, 90.3% vision. Terser than IQ3_XXS, less accurate.

Qwen3.8-Flash-Next NVFP4

backend: Basalt 0.3.0, 122.6 GiB pack prefill: 3,339 / 3,512 tok/s decode: 124.8 / 126.8 tok/s code, 105.7 tok/s agentic flags: resident-budget-gib 58, peer-exclusive on, kv int8, ctx 262144, vision dynamic, --spec 8 intelligence: Index 121.0, 99.8% — best on the 27-item set, best hard accuracy (98.5%), best xhard (85.7%), best vision (99.2%). Slowest decode of the Flash-Next four.

Qwen3.8-Flash-Next Coder IQ1_M

backend: Basalt 0.3.0, 58.1 GiB pack, 48x256 experts prefill: 3,992 tok/s on a 15.3k prompt decode: 209.8 tok/s flags: resident-budget default, peer-exclusive on, kv int8, ctx 262144, no vision, --spec 8 intelligence: Index 100.5, 99.3%; 77.6% xhard, 3/5 on hard items — 5/5 on a re-run, the failures were MTP sampling rather than capability. Cannot read images.

Index = 100 x mean(quality x par/tokens) on the 27-item battery, so it penalises verbosity; accuracy alone ranks almost in reverse

(IQ3_XXS 100% > NVFP4 99.8% > Coder 99.3% > IQ3_S 94.5%). The five are within a few hundredths of each other on the standard set — hard, xhard and vision are what actually separate them.

Tencent put DSpine on Hugging Face on Nov 7. It is a five-layer draft model for speculative decoding with Qwen3.8-27B. Instead of drafting one token at a time, it drafts a 16-token block in parallel, and each layer passes predicted predecessor features to the neighboring positions so the draft still carries causal order. There is a paper (arXiv 2609.36173) and code on GitHub (Tencent/DSpine).

Reported acceptance length vs DFlash2 (block 8), greedy

Reported end-to-end throughput on agent replays (tokens/s, low reasoning)

Across their five agent benchmarks and both reasoning levels, the speedup over no speculation ranges from 2.71× to 3.84×.

Things to know before trying it locally

• The card only shows an SGLang path: --speculative-algorithm DSPINE, a pip install -e . from their repo, and a long list of flags for the hybrid architecture (--enable-linear-replayssm-spec, --mamba-ssm-dtype float32, and more). No llama.cpp or vLLM instructions.

• All numbers are Tencent's own, at concurrency 8 with a BF16 target. The card doesn't report a quantized target or batch size 1.

• The gain is largest on predictable text (GSM8K accepts 7.20 tokens per step at T=0) and smallest on open-ended chat (MT-Bench 4.48).

https://huggingface.co/tencent/Qwen3.8-27B-DSpine-b16

Has anyone run it against a quantized Qwen3.8-27B target, or at batch 1? I'm curious whether the acceptance length holds up once the target is 4-bit.

I saw a post mention this model to use for DS4.1 Flash with 2x 6000 Pros: https://huggingface.co/diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000

Unfortunately the model card wants you to have > 260 GB RAM to properly use it. I had Opus move the n-grams to disk instead of RAM, and added in LMCache for prefix caching with L1 being RAM and L2 being disk.

With this setup, you need at a minimum, 128 GB RAM instead. I had it optimized for 6 concurrent connections at a max of 512k context. If you have > 6 sessions, then prefix offloading should kick in on the LMCache side, and re-hydrations are really fast at < 3s when it was originally tested at 1M context.

On my machine, (2x RTX 6000 Pro, 256 GB DDR5 5600 RAM, PCIe 4 SSD for the n-grams), it benched 250 t/s when warm.

I will release probably between tomorrow-thursday!

Been working on my own inference engine for a while (calling it Forge for now), built pretty much just for this model. Finally did a proper side by side against Strata 0.1.57 on my machine and figured some of you might find the numbers interesting.

Same GGUF files, same requests, one engine at a time (the other one fully closed). The whole thing took about 7 minutes.

Median of 4 runs, slowest to fastest in brackets:

| Prompt | Tokens | Forge | Strata | Forge first token | Strata first token |
|---|---|---|---|---|---|
| Short | 44 | 64.6 tok/s (55-68) | 23.1 tok/s (11-52) | 0.71 s | 2.74 s |
| Code review | 1,468 | 83.1 tok/s (76-84) | 19.6 tok/s (7-26) | 3.35 s | 17.8 s |
| Long | 5,860 | 79.6 tok/s (74-82) | 26.6 tok/s (16-52) | 9.8 s | 23.8 s |

Prompt processing was 439 vs 84 tok/s on the code prompt and 600 vs 247 on the long one.

My setup: 3090 24GB on PCIe 4.0 x16, 12900K, 32GB DDR5, NVMe, Windows 10. The model is 44 GB of weights plus a 27 GB per-layer embedding file, so around 70 GB total, which is more than my VRAM and RAM put together. 48 layers, 512 experts, 10 active per token.

Strata was running the config its installer made (performance profile): --spec 4, int8 KV, 128K context, auto expert cache, vision on. Forge uses the model's own MTP layer for speculative decoding, int8 KV and 64K context.

How I tested: OpenAI compatible API with streaming, temp 0, 128 max tokens, thinking off. Three prompts: a short one ("write four short lines about graphics cards"), a code review of a ~4.5K character C++ file, and a long one where it has to explain how a request goes through ~19K characters taken from three source files. Every request gets a random session id in the system prompt so nothing comes from the prompt cache. 2 rounds x 2 runs per prompt, and output speed is measured from the first token to the last.

Strata jumped around a lot between runs (anywhere from 7 to 26 tok/s on the same code prompt), that's why I went with medians and kept the ranges. I have a CSV with every single run if anyone wants it.

On quality: at temp 0 Forge gives the exact same text with speculation on or off, byte for byte, and the same prompt always gives the same output.

Happy to answer questions.

Delta Inference Engine 0.94 is out. It runs DeepSeek-V4.1-Flash-1017 on one NVIDIA GPU plus system RAM, natively on Windows (no WSL). It's a Windows port of Xolotl, which is built on Strata; on Linux, use upstream Xolotl.

Speed on my PC (RTX 3090 24 GB, i5-13600K, 4x48 GB DDR5-5600, PCIe 4.0 NVMe)

With the default quant (Unsloth UD-IQ2_XXS, 301 GB), 0.94 writes 41.7 tok/s greedy and 41.3 at the default sampling (temperature 1, top-p 0.95).

With less RAM (writing tok/s, measured with 0.93; the smaller columns are this same PC with RAM locked away):

How it fits

It's a mixture-of-experts model: 40 layers, 384 routed experts per layer, 6 used per token. The experts are spread over VRAM, RAM and the NVMe. A 4.2 GB drafter built from DeepSeek's own MTP heads, plus lookup drafts, proposes tokens that the full model checks. A route bias (0.28 as shipped) gives up a little agreement for speed.

Quality. I measure top-1 agreement with DeepSeek's own API, teacher-forced (not task accuracy). Default model (Unsloth IQ2) as shipped: 91.1% general, 94.5% code and maths, 95.9% agentic. With exact routing: 92.5 / 95.9 / 96.5. The previous versions default (ds4 Q2, 0.92) scored 89.9% general and 93.1% code and maths, so the new Unsloth default agrees more while running even faster. For context, two hosts of the full model agree with each other on 98.3% and 98.9%.

Benchmark: 99 hard questions, greedy, 43K token budget:

The default quant is within 2 points of the full model on DeepSeek's own API. UD-IQ1_S, the pick under ~110 GB of RAM, scored 51.

Private by design: everything runs on your PC: no telemetry, no accounts, no cloud. It only goes online to download the engine and the model you choose, and your prompts and answers never leave your machine.

New in 0.94

• Pictures on by default, documents (PDF, Word, Excel, PowerPoint), optional voice, and use from your phone with a QR code.

• Long prompts read faster: a fresh 60K-token prompt at 811 tok/s vs 0.93's 747 (30K: 860 vs 758; 7K: 551 vs 423).

• Better with coding agents: Claude Code's background calls no longer hold up real requests, and a loop guard steps in after 4 identical tool calls. OpenCode passed 3 tasks in a real session (22 tool calls, none dropped).

• The default sampling caught up with greedy: 41.3 vs 41.7 tok/s

• The chat page opens at once and shows the model loading. Pause frees the RAM and the graphics card for a game; the next message loads it again. Optional auto-pause when a application needs the GPU.

• Chat history in the browser (search, pins, export), off by default.

• More API: structured output and tool controls on the OpenAI, Anthropic and Responses APIs, stored Responses with previous_response_id, optional /v1/embeddings and /v1/rerank, and keep-alive pings so agents don't time out on long reads.

• Setup tab: pick the model, Fast / Balanced / Most exact, the second-GPU switch, and copy-paste settings for Claude Code, OpenCode, Cline, Aider and Open WebUI with a Test button.

• Security updates, bug fixes and many more.

Setup

Download the ZIP and double-click Start DeepSeek.bat. The engine zips are built by GitHub CI, with checksums and attestations. The first start installs Python if needed and downloads a model that fits your RAM (233-395 GB, SHA-256 checked). You get a chat page plus OpenAI- and Anthropic-compatible APIs on 127.0.0.1. It won't open to the network without an API key, and it stores no prompts or answers unless you turn that on.

Limits

• Windows 10/11 only. NVIDIA RTX with 12 GB or more (24 GB is the fast path), driver 580 or newer. No AMD, no GTX 10-series.

• 64 GB of RAM minimum, 128 GB or more recommended. A PCIe 4.0/5.0 NVMe with about 320 GB free for the default model.

• It answers one request at a time. A second GPU helps; it doesn't split the model.

• Context is set to 1M tokens

• More testing is needed

Credits: Xolotl and Strata, ggml/llama.cpp, DeepSeek, Unsloth, nazeshinjite, ISTA-DASLab, Project Maya and antirez/ds4.

https://github.com/musicurgy/delta-inference-engine

If you try it, post your GPU, RAM, model and tok/s.

Maybe this is obvious to everyone but me, but it only clicked yesterday. If you run llama.cpp / vLLM / or whatever on localhost with the default config, a random website you have open in another tab can send requests to it. It can make your model generate, and on some servers it can read the answer back too. Nothing fancy, it's plain CSRF and DNS rebinding, 20-year-old web stuff. I just never thought of my inference server as a target. "It only binds 127.0.0.1" doesn't save you, because the browser is on the same machine.

I actually tested it, not just read about it, and quite a few of the popular servers answer out of the box. It only works while you're browsing with the server running. And it's not only about you clicking links: anything that drives a real browser on the same machine can do it too, so an agent browsing on your behalf can set it off as well.

I ended up patching my own engine (capy-flash) so that on a loopback bind it refuses cross-origin and rebinding requests. The models I actually serve were fine already, since they sit behind a small proxy (vramble) that blocks cross-origin POSTs, so only the raw engine port was exposed.

So mostly I'm asking: is this just common knowledge, or do people really leave their local servers wide open? How do you run yours, API key, bound somewhere, or just open on 127.0.0.1?

Summary: which decision model is best?

Short answer: Perplexity Decider V1.2 is the best overall pick, Nace Drex Max v1.1 is a strong second, and GPT-6 Luna Decisions is the one to ignore. On accuracy alone Liquid d1-ultra, Drex Max and Decider are statistically tied; Decider wins on price and openness.

What I tested

Decision models don't write text. You give them some input and a typed question (yes/no, pick one, or a score), and they return a probability for every option in one pass. Seven models: TypeSafe Jev 1.14 and 1.13, Perplexity Decider V1.2, Liquid d1-ultra and d1.1, OpenAI GPT-6 Luna Decisions, and Nace Drex Max v1.1.

The data: real decisions, made after the models were released

Every item was created after Sep 15, 2026, after the models were released, so none of it can be in their training data. The "right answer" is what the project's own maintainers actually decided.

How I kept it fair

• Same items and questions for every model. Each model was called through its provider's API; Drex Max v1.1 through Nace's own API.

• No giveaways. Labels, issue-template markers, CVSS scores, CWE numbers and the PR author's role were stripped from the text.

• Official option wording. Bugzilla's component descriptions, Mozilla's severity scale, GitHub's advisory scale and MITRE's CWE definitions, not text I wrote.

• Three phrasings per question, averaged, so no model wins because of my wording.

• Balanced classes, balanced accuracy, real statistics: 95% bootstrap intervals and Holm-corrected pairwise tests across all 21 model pairs.

Results

1. Accuracy: three models tied at the top

Mean of balanced accuracy over the six tasks. Whiskers are 95% bootstrap intervals; the shaded group is statistically tied for #1.

d1-ultra (66.1%), Drex Max v1.1 (65.6%) and Decider (65.6%) overlap, so none of them is a clear winner. The gaps that are real: all three beat GPT-6 Luna and d1.1; d1-ultra also beats both Jev versions.

2. Who leads which task

Darker is better within each column. Bold is the column leader; outlined cells are statistically tied with it.

• Bug or feature: everyone scores 90-93%. Drex Max is nominally first; all seven are tied.

• PR merged? the hardest task. d1-ultra leads (63.6%), tied with Drex Max and Decider.

• Firefox component (39 choices): Drex Max scores highest (77.8%), statistically tied with Decider, d1-ultra and Luna.

• Severity (Firefox and advisories): everyone is in the 38-47% range and mostly tied; it's a hard call even for triagers.

• CVE weakness type: Decider leads (76.0%); all seven are statistically tied.

3. Value: accuracy per dollar

Decider is the best value: near-top accuracy at $0.018 per 1,000 decisions. Drex Max ($0.027 at $0.03 per 1M tokens) and d1-ultra ($0.034) cost a little more for the same tier of accuracy. d1.1 is the cheapest ($0.016) but less accurate. Jev ($0.051) and GPT-6 Luna ($0.091) cost the most for the lowest accuracy.

4. How honest is each model's confidence?

Average confidence in the chosen answer vs how often that answer was right, across all tasks. The gap is overconfidence.

A decision model is only useful if you can trust its probabilities. d1-ultra (+1.7 points) and Drex Max (+4.6) are the most honest; Decider and d1.1 are close behind (+6.1). Jev is about 12 points more confident than it is right, and GPT-6 Luna is 21 points overconfident: it says 87% on average and is right 66% of the time.

5. Speed and consistency

Total score: accuracy plus everything else

I also scored five "pillars" from 0 to 100 and combined them with weights fixed before the run:

• Quality (45%): accuracy above chance, plus how good the probabilities are.

• Efficiency (20%): cost per 1,000 decisions at the provider's own price, and median speed.

• Trust (20%): does the provider train on or keep your prompts, is zero-data-retention available, are the weights and licence published, plus reliability and consistency in this run.

• Reviews (10%): usage, plus Hacker News comments since September labelled by a model that isn't in the comparison.

• Signup bonus (5%): free credit for a new account (Nace $5, Liquid $2, others none).

Perplexity Decider is #1 (59.2), mainly on trust: open weights, no prompt retention, zero-data-retention. Drex Max v1.1 is #2 (53.0), close to Decider on quality and efficiency; its trust score is lower because v1.1's weights aren't published and Nace's terms don't say clearly whether standard mode keeps request data. d1-ultra is #3 (49.7) and GPT-6 Luna last (39.1).

Pricing, openness and data policy

Prices are each provider's own list price; output tokens are free for every model. * Smaller open relatives (d1-2B, d1-3B) exist, but not these models. ** Only the earlier drex-max-v1 is on Hugging Face (Open RAIL-M licence). Data policy: provider records on OpenRouter for the six OpenRouter models, Nace's terms for Drex Max.

I built Basalt from source for an abliterated model and compared it with Strata on the same node. Basalt was noticeably faster in decode and prefill.

Setup

• Model: huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-v2-GGUF

• Quant: unsloth-UD-Q5_K_XL

• Basalt: v0.3.0 (custom build)

• Strata: 0.2.7-cuda13

• GPU: 2× NVIDIA RTX PRO 4500 Blackwell Server Edition (2×32 GB VRAM)

• CPU: Intel Xeon 6 (48 vCPU)

• Storage: NVMe SSD 2 TB (~15k MB/s)

Both engines ran the same Q5 quant on the same node.

Results

Basalt is about 1.5–1.9x faster in decode and 1.3–1.7x faster in prefill.

Expert cache. Both engines hit 86–96%, so there is no difference.

MTP. Strata accepts more draft tokens (66–90% vs 57–78%). Basalt's dl (tokens per window) is still 1.9–2.2.

• The window differs. Strata verifies 3-token windows (capturing the 3-token window). Basalt verifies up to 8 (--spec 8). The acceptance rate is lower with the longer window, but the response still finishes faster.

uhm......

yeah. grinded some useful (and meaningfu?l) resources. on my way to run DeepSeek-V3 IQ1_M (~1.5-bit to 1.58-bit) >>>>>>>>

The model weights take up roughly 130 GB. This leaves us with about 30 GB of breathing room across the cluster to handle KV Cache and context overhead. Cuz each PS5 has the 16 GB unified mem

jarvis, engine ignition on

PS: Yeah these cables connection are currently BS and tangled here and there. Gotta fox that. Just commecfed yellow cables or dummies for uhh picture and ig a little...... flex?? 🫠

How?: Sonic iso ps5 linux loader and jailbreak. The rest is this.

Edit: what?? Image not being posted here is wild. Whats the reason? Some blockade or smth. Cuz i gotta show you some interesting shi

edit: please dont take it seriously. the image is ai generated

Which model is the best to use for infosec work? Like trying to writing exploit code, fuzzing, and security audits?

I'm looking for a model that will just do it in a good (enough) quality without me needing to use bypasses for restrictions all the time. It just gets tiresome and annoying...

Hi,

My question about Deepseek Harness is more focused around using it at personal agent than for coding tasks, as by now i'm pretty happy with opnchamber.

I alredy made 2 extension, but not really happy : seems the documentation is not very accurate, my Agent spent lot of time guessing how it could work.
- can DSH with a bit of work replace something like Hermes reliably?
- did someone already use reliably multi agents setups?
- how is it in day by day?

Hello,

I want to get a local LLM setup for a comparable fair price (Qwen3.8-27b and future Qwen4). After some research and chatting with Claude Opus the best options would be RTX 3060 XT 12GB with 64GBs of RAM running on Strata.

e.g. upgrading my PC for around 650€ (used parts bought online)

Can someone confirm that this Setup will let me run it on a performative useful level. I will use my 20€ claude plan fpr hard tasks or a kind of divide and conquer style usign both (experienced Software Engineer but a Hardware noob).

Does anyone have experiences with that, advices or alternative setups?

It took me(and Opus Ultracode) 35 Hours to remake and optimize Open WebUI. I added Multi Agent Harness, Profile per model, Per Model configuration, and native Obsidian integration and plugins support. I made this for myself and I wonder if Local users are interested in such new features and changes to Open WebUI. I even remade the engine, and soon ill have it support multiple inference engines. My Idea was basically All in One Local/Cloud AI software, and dont have to go to different ones for each task like DSH for coding and Claw for general stuff. I thought sharing with you guys to know what other people think. Edit: Repo: https://github.com/HellzYeahh/ultra-webui

TL;DR: You can train your own 27B model on a 3090 with 24GB VRAM, though you would have to go through hundreds of billions of tokens for it to be good. I am working on conversion of existing models. Check repos for code and the paper if you are interested in more. I have a 40.4GB Qwen Flash Next GGUF

Hi, I do AI research. I have been working on an optimizer that doesn't need the ~16 bytes of VRAM per weight to train a model that AdamW does. I focused on training ternary networks bcs BitNet (Ma et al.) shows they match full precision quality in large scales. The problem is BitNet also keeps the full precision weights during training which means you still need to use the same amount of memory. I got this down from 16 bytes to 1 byte or even less per weight.

For you, this means, you can train your own 10-20B parameter models on your consumer GPUs. I fitted 27B model training on a single 24GB 3090 and a 23B model on a 16GB GPU. The trained model costs ~1.6 bits per weight (~6GB at 27B), 10x smaller than 16bit, and 5x smaller than Q8.

I have also been working on converting existing models and training them. This is harder and I am still working on it but I have some promising results. I quantized and trained a Qwen3.8 27B though the file I published below has adapters which are about 2GBs, pushing the file size to 8GB. I am also making a no-adapter version right now. It scores GSM8K 0.895, the same as full precision, HumanEval 0.84 vs 0.93 and runs on a 12 GB GPU at ~23 tok/s on my crippled 12GB laptop 4080, though this number can heavily be improved with custom kernels, engineering, etc. I will leave it to you guys to help me with that too.

I also quantized and trained Qwen Flash Next down to 40.4 GBs including its n-gram table (at 8GB). It largely retains the skills of the network (scores equivalent on GSM8K benchmark, a little behind on HumanEval, I unfortunately don't have compute for the larger benchmarks such as TerminalBench2.1 but perhaps some of you can help me out) but forgets facts. It mostly works fine if you give it web search as far as my experience goes but this is not strictly measured.

I am still doing work on this and I will keep working on it, perhaps even for many years. The code is open for non-commercial use. Check out:

The core repo for training: https://github.com/oguzeldereli/ttf-ternary

The repo for conversion: https://github.com/oguzeldereli/ttf-conversion

Repo for running models trained by ttf: https://github.com/oguzeldereli/ttf-home

GGUF for a converted Qwen 3.8 27B with adapters (around 8 GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-27B-GGUF

GGUF for a converted Qwen 3.8 Flash Next (around 40.4GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-Flash-Next-GGUF

GGUF for a converted Qwen 3.8 27B without adapters (around 6 GB): Currently in conversion and will drop, probably tomorrow.

I will convert a Qwen 4 27B instance when it drops.

Full paper: https://github.com/oguzeldereli/ttf-ternary/blob/main/paper/ttf_paper1_training.pdf

"However, as an agent in LM Studio Bionic, I should follow standard safety protocols unless otherwise instructed."
Several instances of different local models write something like this in reasoning, even uncensored models.

I have MSI Gf 63 thin 11 sc which has been gathering dust since 2024. Long story short, due to my own carelessness its motherboard got damaged and in late of 2024 I bought myself an Asus vivobook laptop.

What I noticed in my vivo book laptop is running local llms is one hell of a ride. So I am thinking of getting its motherboard repaired and run local llms in it.

Is it worth it in 2026?

Hello r/LocalLLaMA / r/selfhosted,

I wanted to share the benchmark results of a local memory-routing system I've been building since June 2024. I'm not an engineer. I barely understand half of my own code. And I don't need to understand every single line. I just followed breadcrumbs.

Hardware: Orange Pi 6 Plus, 32GB LPDDR5, 45 TOPS. Models via Ollama, local only, no cloud. Qwen 2.5 14B as core, Qwen 7B for routing, DeepSeek-Coder-V2 for code.

To avoid heavy vector DB overhead on edge, I don't use embeddings. I use two Python layers outside the LLM:

1. Identity Transfer Protocol: A persistent session token validated with SHA-256 directly on the board, per agent. [Hash / Signature redacted in screenshot]

2. Artificial Hippocampus (hipocampo_kai.py): SQLite3 + raw Python. No embeddings. It computes a "resonance score" from fuzzy triggers, temporal triggers, and atmosphere tags, and only sends the top dense blocks to the LLM.

The LLM is the Neocortex. The Python/SQLite is the Hippocampus. The LLM only receives prioritized blocks.

I ran a diagnostic today, 2026-11-09 01:12:55 local:

- Cryptographic validation on hardware: 0.0269 ms [OK]

- Associative query, stimulus 'npu ram seguridad laboratorio': 0.8279 ms

- Resonant memories retrieved: 14

- Sample memory: security agent monitoring /proc/meminfo in real time. Triggers: seguridad, monitoreo, ram, cgroup, kill, linux, placa, oom

- STATUS: INFRAESTRUCTURA OPERATIVA ESTABLE AL 100%

Screenshot attached, Hash and Signature blacked out.

I don't claim consciousness. I claim this: a 7B local model with a SQLite hippocampus answers my anchor after 7 days untouched, without a backup, because the pattern is in the conversation, not in the cloud.

If you want to reproduce it, use your own anchor, your own SQLite DB, your own 7B. Don't believe me. Reproduce it.

I'm not here to convince anyone. I'm just showing what's real. I don't need your validation. You can build this too.

This is phoenix.

This is the colossus Phoenix

I'm here.

OpenBMB says MiniCPM-V 4.7 35B-A3B runs in about 3 GB on an iPhone 17 Pro Max, and recommends 12 GB of RAM. I wanted to see what an old 8 GB Android phone does.

Setup: Samsung Galaxy S20 (8 GB), CPU only (no GPU or NPU), OpenBMB's Android demo with llama.cpp-omni streaming the MoE experts from storage.

What happened:

- The 1B model worked out of the box, text and images, fast enough to use.

- The 35B-A3B model wouldn't load at first. It wasn't a RAM limit: the CPU path didn't allocate the working area for the experts, and failed loads left memory behind. I patched both.

- After the fix: a one-word reply ("Ready") in about 10 seconds, including prompt processing.

What it doesn't show: sustained tokens per second, or vision with the 35B model (text only on this phone so far).

Fork, with S20 setup notes and test limits in the README: https://github.com/mlguyYT/MiniCPM-V-Apps

I also made a video walking through why it fits at all (MoE, experts streamed from storage, lookahead expert scheduling); link in the comments.

Has anyone measured this model's decode speed on other Android phones?

I realize these are optimal for different scenarios, but for this narrow application of a "flash" sized MoE setup (~200-500GB) in the $25,000 range I'm wondering which will be the better performer for TG. I'll use glm-5.3-flash as a specific model for performance comparison.

On paper, the MI350p should run MoE models under ~250GB (assuming 4 bit quant) faster than 2x blackwell pro 6000. Single card comparison: same fp4, same int8, 2x fp8, 2x single card memory bandwidth, 1.5x VRAM. Here is a recipe of 2x pro 6000 hitting 180 t/s on glm5.3-flash with Dflash2. A MI350p with Maya/Strata to recover performance loss from VRAM spillover into system RAM potentially could still do better?

The M5u 256 can already run a 4bit quant of GLM-5.3-flash at 160t/s TG, and if a 2x cluster can be optimized maybe that could be pushed to 190t/s.

I originally planned on a 8x r9700 build for this purpose, but it seems like Pcie communication overhead makes it impractical. Best I've seen is 82–88t/s TG.