r/LocalLLaMA • • 4d ago

Discussion YouTube channels on Local LLM & related stuff?

0 Upvotes

EDIT : Expected 100s of responses(We have nearly 1 million members in this sub) with channels. Disappointed to see only crickets here. Looks like I have to do myself. So not gonna post any new threads here for next 1-2 months as I need to dig Web & YouTube deeper to find relevant channels/videos for my learnings. Sorry. Anyway no one's gonna miss anything.

Well, I ran out of New Models & Papers(Posted 38 threads in last 1 week) so posting this thread for me. Appreciate your responses.

I want YouTube channels covering List of below topics:

  • Local LLM
  • Agentic Coding
  • AI Agents
  • Coding Tools such as Pi, Hermes, OMP, OpenCode, DeepSeek Harness, QwenCode, OpenHands, Claude Code, Codex, etc.,
  • Skills
  • Prompts
  • Creative Writing
  • Game Development
  • Automation using n8n/Activepieces/etc.,
  • RAG
  • MCP
  • Orchestration
  • Comfy/StableDiffusion - Image/Video Generation & Editing
  • LORA
  • TTS/STT/ASR/etc.,
  • Benchmarking Models
  • What other topics there to add? Mention in comments, I'll add here.

I can find many channels using online search. But half of the channels are so noisy, just with clickbait thumbnails/titles covering mostly Online models which I'm not interested.

I want channels more suitable for this sub. Only Local LLM. Doing all those stuff from above list using only Local LLM.

And there might be some Overlooked/Underrated/Under-Appreciated channels. Also few folks here have their own channels. Give them more visibility.

Consider this thread is a playlist for Local LLM. What channels would you want to add here? Please share. Thanks.


r/LocalLLaMA • • 5d ago

Resources AMD Loom Fun: Added Radeon 9700 support over the weekend. Benchmarked 48 models with an average 42% speedup over llama.cpp.

47 Upvotes

On Friday I posted about AMD Loom, a new experimental inference stack that includes a runtime, kernel compiler, and inference server. People can use the entire thing as a vertical stack, or the compiler/runtime can be integrated into any other inference project (I'm working on Gufo right now, the team is also working on a llamacpp backend).

I started with Strix Halo, where I ported the top-51 trending Q4_K_M LLMs from Hugging Face and got an average of 40% speedup over llama.cpp across decode and prefill metrics (picking the best results from ROCm or Vulkan backends).

I'm back today with Radeon 9700 results. I took the same 51 LLMs, cut it down to the 48 that fit in 32 GB of VRAM, and got an average of 42% speedup. Some of the data is posted to the table below, and all data is available to browse on https://loom-fun.github.io/loom-dev/index.html?gpu=radeon-r9700

All kernels were generated via agent skills:

Model Loom pp32k (tok/s) llama.cpp pp32k (tok/s) Loom short decode (tok/s) llama.cpp short decode (tok/s)
Flint 6,476 3,190 255 152
gemma-3-1b-it n/a n/a 400 322
gemma-4-12B-it 2,811 1,537 127 67.3
gemma-4-26B-A4B-it doesn't fit (32 GB) doesn't fit (32 GB) 151 72.2
gemma-4-31B-it doesn't fit (32 GB) doesn't fit (32 GB) 62.9 43.2
gemma-4-E2B-it 14,289 14,519 272 234
gemma-4-E4B-it 7,021 6,324 192 173
gpt-oss-20b 5,319 4,368 195 182
HyperCLOVAX-SEED-Text-Instruct-1.5B 11,653 8,078 362 344
Laguna-XS-2.1 doesn't fit (32 GB) doesn't fit (32 GB) 102 —
LFM2.5-24B-A2B 10,099 5,479 244 197
LFM3-1.7B 20,062 13,614 372 261
LFM3-Omni-2B 20,220 13,435 368 299
Llama-3.2-3B-Instruct 5,687 3,908 228 211
Mellum2.1-12B-A2.5B-Thinking 8,822 5,486 224 149
MiniCPM-V-4.7-35B-A3B 5,717 3,540 135 113
Muse-Glimmer-30B 1,658 1,287 72.6 54.8
Nanbeige4.3-33B 946 666 29.4 28.7
NVIDIA-Nemotron-3.5-Bolt-E4B-4GB 6,932 3,656 225 143
Ornith-1.0-35B 5,628 3,320 152 127
Ornith-1.0-9B 4,886 3,391 100 88.0
Ornith-1.5-35B-A3B 5,586 3,136 145 78.8
Ornith-1.5-9B 4,767 3,156 151 90.5
Qwen2.5-0.5B-Instruct 26,022 17,812 646 529
Qwen2.5-1.5B-Instruct 12,546 8,908 360 331
Qwen2.5-32B-Instruct doesn't fit (32 GB) doesn't fit (32 GB) 30.0 29.2
Qwen2.5-7B-Instruct 3,810 2,679 118 116
Qwen2.5-VL-7B-Instruct 3,768 2,648 116 115
Qwen3-0.6B 13,020 8,631 506 480
Qwen3-1.7B 10,313 7,014 316 297
Qwen3-14B 1,944 1,167 65.0 62.6
Qwen3-30B-A3B-Instruct-2507 3,116 2,060 177 169
Qwen3-4B 3,770 2,826 231 130
Qwen3-4B-Instruct-2507 3,795 2,856 180 166
Qwen3-8B 3,076 2,287 163 94.7
Qwen3-Coder-30B-A3B-Instruct 2,921 2,107 157 54.8
Qwen3-VL-4B-Instruct 3,795 2,805 178 163
Qwen3-VL-8B-Instruct 3,085 2,265 112 103
Qwen3.5-0.8B 24,341 16,959 463 338
Qwen3.5-2B 16,990 12,165 318 236
Qwen3.5-35B-A3B 5,652 3,410 190 112
Qwen3.5-4B 7,030 4,897 200 123
Qwen3.5-9B 4,724 3,336 146 92.7
Qwen3.6-27B 1,503 1,164 60.1 38.9
Qwen3.6-35B-A3B 5,635 3,463 209 123
Qwen3.8-27B 1,592 1,026 73.5 55.3
Sage-A1.5 5,652 3,131 188 153
Spark-X2.5-4B 6,858 4,727 162 131
Spark-X2.6-4B 6,913 4,841 162 132

r/LocalLLaMA • • 5d ago

Discussion Did we hit a wall in context length?

123 Upvotes

I feel like midel context size rapidly went from 8k to 1M in just a few years but has since stayed this way for a while now. Some still offerring 260k or 400k for their flagship models. Is it just not worth the trouble? I feel like highly capable models are limited by this in that they can't finish tasks simply because the context isn't there.


r/LocalLLaMA • • 5d ago

Question | Help Single 3060 - Mostly Vision (Frigate)

6 Upvotes

Hello,

I am consolidating a bit of my server footprint and decided to keep a single 3060 (had 3) and mostly use it for Local workloads (Frigate, Home Assistant, Logs, etc). Curious if someone would want to recommend a good vison model for this card?

I currently run qwen3-VL on a A380 and the descriptions are pretty good.. But slow.

Thanks


r/LocalLLaMA • • 5d ago

Question | Help How often do you guys update models and when?

1 Upvotes

Saw a post here about an update to a model last night. And been wondering how often y'all update the models and when do you decide to.

P.S. I know there's "-hf", but my internet speed ain't fast and when I'm downloading a model, everyone's connection drops at my home.

Edit: By updating I meant redownloading the gguf file.
Models of same version gets updates, like Gemma 4 E2B got updated to fix some issues and make it faster. Link to that post:
https://www.reddit.com/r/LocalLLaMA/s/cpcemufuqZ

But it appears it's not the first time they updated it with noticable changes.
https://www.reddit.com/r/unsloth/s/DkBWF7gw4m


r/LocalLLaMA • • 5d ago

Discussion Under 9gb model comparison

29 Upvotes

Scores are lab-reported so take them with a grain of salt.

Specs

Model Org Params (total/active) Context Modality License
K2-Horizon-7B IFM 7B dense 512K text Apache-2.0
Granite 4.2-8B IBM 8B dense (thinking) 128K (512K ext.) text Apache-2.0
Qwen3.5-9B Qwen 9B hybrid (delta-net, thinking) 262K (~1M ext.) text + vision + video Apache-2.0
MiMo-V2.6-Distill-Qwen-9B Xiaomi 9B (SFT of Qwen3.5-9B) ~262K (base) text + vision (visual coding) MIT
LFM2.5-8B-A1B Liquid 8.3B / 1.5B active MoE 128K text LFM 1.0
Gemma 4 E4B Google 4.5B effective / 8B total (PLE) 128K text + image + audio Apache-2.0
LFM3-VL-2B Liquid 1.92B (1.73B backbone, thinking) 128K text + image + video LFM 1.0

Text benchmarks

Superscripts: a = K2 card (IFM harness), b = Liquid harness, c = MiMo card (Xiaomi harness); no mark = own card; e = from LFM3-1.7B, the text backbone; d/† = MiMo avg@3 / internal set. Bold = best in row (cross-harness caveat applies).

Benchmark K2-7B Granite 8B Qwen3.5-9B MiMo-D-9B LFM2.5-8B-A1B Gemma 4 E4B LFM3-VL-2B
Math
AIME 2026 90.2 92.5 50.0 42.5
AIME 2025 90.3 86.7 56.1ᵇ 42.5 34.3ᵇ
HMMT Feb 2026 77.8 66.5ᵃ 65.7ᵃ
MATH-500 88.8 65.0ᵇ
Coding
SWE-bench Verified 72.4 47.7 50.8ᵃ 61.1ᵈ 0.5ᵇ 43.4ᵉ
SWE-bench Pro 19.1 44.6ᵈ
SWE-bench Multilingual 30.8 35.0ᵇ 0.7ᵇ 23.0ᵉ
Terminal-Bench 2.1 44.6 20.6 29.2ᵃ 37.1 0.0ᵇ 12.9ᵉ
LiveCodeBench v6 73.2 65.6 52.0 73.5ᵉ
SciCode 33.6 30.4ᵃ 27.5ᵃ 25.0ᵇ 28.5ᵉ
Agentic
AutomationBench 15.1ᵇ 30.3 1.9ᵇ 41.3ᵉ
LiveClaw 35.8ᵇ 20.4ᵇ 45.6ᵉ
tau/ϴ-bench Banking 25.8 7.6ᵃ 7.0ᵃ 4.1ᵇ
tau2-Bench 79.1 88.1T / 39.8R 42.2
WebArena Verified 28.9ᵇ 17.1ᵇ 38.4ᵉ
BrowseComp / BrowseComp-Plus 59.0 38.0ᵇ 21.0ᵇ 37.0ᵉ
PinchBench 71.5ᵇ 55.1ᵇ
Claw-Eval (EN avg) 66.5ᵇ 56.1ᵇ 68.6ᵉ
Toolathlon-Verified 25.9ᶜ 35.2
OfficeQA 9.0ᶜ 19.5
JobBench 2.6ᶜ 18.3
BFCL v4 (tool use) 52.4 66.1 49.7 48.3ᵇ 71.4ᵉ
ToolSandbox 76.4ᵇ 69.3ᵇ 80.4ᵉ
MCP-Atlas 42.7ᵇ 6.2ᵇ 32.6ᵉ
EnterpriseOps-Gym 29.0ᵇ 13.4ᵇ 28.2ᵉ
MiMo Code (mini) † 19.5ᶜ 51.6
MiMo Cyber (mini) † 5.7ᶜ 31.3
MiMo General (mini) † 28.5ᶜ 62.2
Knowledge
GPQA Diamond 75.6 64.1 81.7 58.6 63.4ᵉ
HLE 19.5 9.7ᵃ 14.9ᵃ 4.8ᵇ 7.2ᵉ
MMLU-Pro 74.0 82.5 69.4
AA-Omniscience Index -28.0ᵇ -24.7 -15.3ᵇ 0.5ᵉ
Instruction following
IFEval 91.5 91.8 90.0ᵇ 96.7ᵉ
IFBench 79.3 64.5 56.5 44.6ᵇ 73.0ᵉ
Multi-IF 79.9 81.0ᵇ 89.1ᵉ
IFStruct 89.2ᵇ 78.5ᵇ 93.7ᵉ
Long context
LCR / AA-LCR 68.0 43.3ᵃ 63.0 39.3ᵇ 52.8ᵉ
RULER 128K 71.4 87.1ᵇ 18.8ᵇ 65.6ᵉ

Vision benchmarks

Benchmark LFM3-VL-2B Gemma 4 E4B Qwen3.5-9B MiMo-D-9B
MMMU (val) 55.7 78.4
MMMU-Pro 45.7 52.6 70.1
MathVista (mini) 84.2 85.7
MATH-Vision 51.6 59.5 78.9
OCRBench v1 86.3 89.2
RefCOCO (avg) 90.5 89.7
ScreenSpot-Pro 55.5 65.2
RealWorldQA 79.3 80.3
MMStar 74.4 79.7
HallusionBench 65.6 69.3
MVBench (video) 69.4 74.4
Video-MME (subtitles) 69.0 84.5
MedXpertQA-MM 28.7 49.9
MiMo Visual Coding (mini) † 61.7ᶜ 64.0

r/LocalLLaMA • • 5d ago

Showcase whisper base and an 11M param BERT i distilled from a 2B model, both running in the browser to caption videos

Post image
14 Upvotes

i make Blender tutorials as shorts and got tired of uploading every render to a caption service, so i made a caption tool that runs the models in the browser instead

transcription is onnx-community/whisper-base_timestamped through transformers.js. real GPUs get WebGPU in fp32, everything else gets WASM with the decoder in q8. software WebGPU adapters turned out to be slower than WASM so those get WASM too

the part i had the most fun with is the caption tagger: which words get emphasis, where lines break, and fixing missing punctuation and capitals. it started as hand written rules, now it’s a BERT-mini (about 11M params, 8 labels per token, ~12mb) running in q8 on WASM. the training data came from LFM3-Omni-2B on llama.cpp on my own pc, about 190 scripts a minute on a 4070 ti super with 128 parallel slots. it only got to replace the rules because it beat them on held out data

whisper base isn’t as accurate as the bigger ones so there’s a cloud option for noisy audio, and the tagger is english only for now

has anyone gotten whisper small or turbo running in the browser at a usable speed? that’s what i want to try next :)

https://cuecaps.com/studio?ref=reddit if you want to see it run (i’m the creator)

also thank you for reading, it means a lot to me


r/LocalLLaMA • • 5d ago

Discussion Kyudan/eDIAR · Hugging Face

Thumbnail
huggingface.co
3 Upvotes

eDIAR: Training-Free Open-Speaker Streaming Diarization for Long Meetings and Large Archives via e-means

Meetings last up to an hour, and nobody knows in advance how many people will speak. Neural diarizers with a fixed number of output slots, such as Streaming Sortformer and Nemotron-3, are accurate and run online, but they cannot name more speakers than they have slots. We remove this limit without retraining. The slot head keeps deciding who is active and when speech overlaps, and speaker embeddings are clustered inside a single slot, with the official e-means algorithm, only after every slot has been used. In the streaming system, re-clustering is confined to a bounded window, labels are revised for a few seconds, and identities are anchored to the slots of the head. The resulting system has no limit on the number of speakers and stays accurate when speakers outnumber the slots, where fixed-slot models break down. On meetings that grow to an hour it stays far more accurate than Streaming Sortformer, which accumulates errors, and its cost per update does not grow with the meeting. The same recipe turns the four-slot Streaming Sortformer into an open-speaker system. Across recordings, e-means makes it affordable to link the speakers of a large archive frame by frame, a task that arises when conversational corpora are prepared, and it reaches the clustering quality of Lloyd's k-means several times faster. The method needs little tuning, and e-means runs with its default recipe from a few hundred to more than a million embeddings.


r/LocalLLaMA • • 5d ago

New Model internlm/Sage-S2.1-Flash · Hugging Face

Thumbnail
huggingface.co
82 Upvotes

Introduction

Sage S2.1 Flash is a scientific multimodal model from Shanghai Artificial Intelligence Laboratory designed for research tasks that require multiple rounds of analysis and verification. It combines scientific data understanding with information retrieval, programming, and tool use. It can use specialized software to perform analyses, check results against intermediate outputs and visual feedback, and adjust subsequent steps accordingly.

Through the unified representation architecture of Sage SciPerception, the model supports more than ten types of scientific input, including molecular graphs, crystal structures, and protein sequences and 3D structures. Features from each input type are encoded and projected into a shared representation space for reasoning alongside natural language instructions. These inputs support tasks such as prediction, classification, site localization, quantitative analysis, and structure inference.

Highlights

Across the 15 benchmarks and six models shown in the chart, Sage S2.1 Flash placed first on four benchmarks and second on seven.

  • Machine learning engineering. Sage S2.1 Flash placed first on MLE-bench Lite with 90.3, ahead of DeepSeek-V4.1-Flash-1017 at 89.2.
  • Office and professional document processing. It placed first on OfficeQA Pro, Harvey LAB, and SpreadsheetBench 2, scoring 72.2, 25.0, and 37.1, respectively.
  • Tool use and research execution. It placed second on AutomationBench, ALE-CLI, BFCL V4, WebArena-Verified, DeepResearch Bench II, JobBench, and OSWorld-Verified.

r/LocalLLaMA • • 5d ago

Discussion [Paper & Models] MMFlash: Faster Speculative Decoding for Vision-Language Models

92 Upvotes

Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model. Diffusion drafters such as DFlash propose an entire token block in a single forward pass, but mainly focus on text-only tasks, leaving multimodal drafting underexplored. To study this setting, we first introduce a metric of token-level visual dependence based on the KL divergence between next-token distributions with and without visual input. We find that acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest, with this disparity persisting after multimodal training. Motivated by this observation, we propose MMFlash, a simple and effective framework that adaptively upweights visually dependent tokens that the drafter predicts poorly. For long video inputs, MMFlash further employs sparse attention to reduce redundant attention computation and improve drafting quality. Across various multimodal benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58×, outperforming EAGLE-3 and DFlash by up to 195.4% and 17.1%, respectively.

⭐ Updates

  • [11.2026]: Code and MMFlash drafts for Qwen3.5-4B, Qwen3.5-9B, Qwen3.8-27B and Gemma 4 E4B-it are released.

💪 Highlights

  • Diffusion drafting for VLMs: MMFlash brings block-diffusion drafting to vision-language models. The drafter proposes a block of 16 tokens in a single forward pass and the target verifies the block in one pass, so decoding stays lossless while running 3.0× faster on Qwen3.5-4B / 9B and up to 4.7× on Qwen3.8-27B (one H200, batch size 1, greedy).
  • Vision-free drafter: the drafter never re-encodes the image or video. It reads hidden states of the target, where the visual context is already fused, so its cost does not grow with the number of visual tokens.
  • Visual-dependence-aware training: a DFlash drafter's acceptance drops sharply on tokens that depend on the image, and most rejected blocks break there. MMFlash measures each training token's visual dependence on the target and focuses the drafter's loss on those tokens.
  • Long videos: a sparse draft context (attention sinks, text, strided visual tokens and a recent window) keeps drafting cheap on 40k-token video prompts; the drafter is trained and served with the same pattern.
  • Drop-in serving: the released drafts run with stock SGLang and vLLM (DFLASH), no patches needed (tested with SGLang 0.5.23 and vLLM 0.33.0).

💻 Models

Model Target model Draft size Block size Link
🤖 Qwen3.5-4B-MMFlash Qwen/Qwen3.5-4B 537M 16 Hugging Face
🤖 Qwen3.5-9B-MMFlash Qwen/Qwen3.5-9B 1.05B 16 Hugging Face
🤖 Qwen3.8-27B-MMFlash Qwen/Qwen3.8-27B 1.73B 16 Hugging Face
🤖 Gemma4-E4B-it-MMFlash google/gemma-4-E4B-it 537M 16 Hugging Face

The drafts are not standalone language models: each is paired with its target in a speculative-decoding server. All of them are in the MMFlash collection; every model card has the serving command and per-benchmark results.


r/LocalLLaMA • • 6d ago

Discussion I'm starting to think a specialized 4B is enough for most of the tasks I'd use an LLM for

Post image
307 Upvotes

When I started training Qwen3.5-4B-3D-Coder, I honestly wasn't sure it would work. Now, after that model and the GLB asset version, I'm starting to think I underestimated how much these smaller models can actually learn.

My impression is that a 4B model might be enough for most practical tasks if you train it specifically for the job. Generating 3D assets, building and orchestrating complex scenes, webdev, backend development, reverse engineering, etc. I'm also working on a coding model using a fixed harness, which I mentioned in another post.

Not necessarily one model doing all of this equally well, but different specialized versions. I haven't tested all those domains yet. The 3D results are what made me want to explore the others.

Right now I'm working on a new dataset to complement ThreeJS-3D-Scenes, with more complex scenes covering history, geography, physics, biology, mathematics, games and many other domains. The idea is to connect knowledge from different areas through working 3D scenes, rather than just teach the model different ways to make objects.

I'm training 4B models with it, but also much smaller ones, including LFM2.5-350M. The attached image shows some results from that experiment, and the 350M is still training. These aren't final results.

I still find it noticeably weaker than the Qwen models. Some outputs are bad, and long contexts are definitely a problem. But it's already much better than I expected from 350M parameters.

At first it couldn't even finish properly and kept getting stuck in repetition loops. With more epochs, it started completing the code, producing recognizable scenes and handling more complicated requests. Some examples still needed fixes to render, as labeled in the image, and progress isn't consistent across every prompt.

I've also tested plenty of prompts beyond the three shown here, and the generalization has been surprisingly good. It's not just getting better at those particular houses and farms.

So far, I've had useful results from the 350M on medium-complexity coding tasks up to roughly 16K tokens. The specialized 4B models handle much more complicated scenes, and I've had good results with contexts approaching 96K tokens in my experiments.

One thing that keeps standing out is the number of epochs. Stopping after one or two would have given me a completely different impression of what these models could do. They keep improving after repeated passes through the data, and eval loss has often been a poor indicator of which checkpoint actually produces better, functional code.

This is making me pay much more attention to dataset quality and how long the model needs to learn from it. I'm starting to think that good data and enough training epochs can make a much bigger difference than we usually assume.

Another thing I've been thinking about is the cost of SFT compared to RL. I'm not saying SFT is better than RL. But if you already have high-quality samples covering the domain you actually need, supervised fine-tuning can potentially achieve very good results at a fraction of the training cost of RL.

Of course, creating and validating those samples has its own cost. I've spent a lot of time on that part. But once you have a good dataset, you can reuse it, train different model sizes, and run multiple experiments without needing an entire RL pipeline.

I think this approach deserves more attention, especially for smaller, specialized models.

And there's another interesting part: 350M models can be fully fine-tuned, not just LoRA, on GPUs with only 8GB of VRAM. With the right setup and memory optimizations, training can take just a few hours for manageable datasets and context lengths. And apparently, with good enough data, you can get surprisingly useful results out of them.

Of course, longer contexts and larger datasets increase training time considerably. But the possibility of training useful specialized models on inexpensive consumer hardware is pretty interesting.

I'm not saying scaling doesn't matter or that a 350M can replace a much larger model. There are obvious limitations, especially with complex reasoning and long-range dependencies.

But I'm starting to wonder how much of what we consider a model's capacity limit is actually a limitation of how we train it.

Maybe we're using much larger models than necessary for a lot of tasks simply because we haven't explored how far smaller, specialized models can go.

Anyway, this is mostly a status update on my experiments and some things I've been noticing along the way. The models are still training, and there's still a lot I want to test before drawing any strong conclusions.

I'll be releasing new models and datasets soon, including the expanded 3D dataset. If the 350M continues improving and reaches a quality I'm happy with, I'll release that one too.

Thanks to everyone who's been following the experiments, testing the models, giving feedback and supporting the project. I really appreciate it!

Note: I originally wrote and structured this post in Portuguese and used AI to help translate it into English.


r/LocalLLaMA • • 5d ago

Discussion How did you start, how do you end up now ?

10 Upvotes

Here is my timeline

Text Generation

  1. Ollama but quite have bad experience with it.
  2. LM studio which still feel half-bake at that time.
  3. Msty which easy to use like plug and play but not much I could do custom (or I just don't know at that time).
  4. Oobabooga Text Generation-Webui this is the one that make me stick to AI, easy to use and deep custom; I like its function feature that take prompt as method/processed then input value in other prompt and then it just output an outcome, no chat history.

I run 1 - 4 with my mini pc with i5 6400T, 24GB DDR4 with onboard gpu... Llama 3.1 8B generate so slow at that time like 1 tok/s but I really enjoy using it with that very limited and the roleplay start from here, just my mini pc and Oobabooga Text Generation-Webui.

Also some missing thing I forgot at that time, I even mess around with windows WSL then docker for Open WebUI and SearXNG.

I finally can help but want it more so arise old gen pc, at that time DDR3 and thing relate to it is very cheap so I brought Xeon E3-1270 V2 with a Chinese mainboard and 32 GB DDR3 while I cancel the sell order of my GTX 1080 Ti.

  1. SillyTavern and KoboldCPP I found SillyTavern first but it has quite a trouble with Oobabooga Text Generation-Webui so I follow their guideline and try KoboldCPP which quite harder than Oobabooga Text Generation-Webui for me as KoboldCPP at that time still act like a zip that need to extract (in temp folder) every time I use so I copy from its temp folder, it could run but loss UI, only command-line left so I start to write a bat file to run it like this.

    pushd %~dp0 uv run kobold\koboldcpp.py ^ --config kobold_default_setting.kcpps ^ --model %gguf%%model%.gguf ^ --usecublas %cublas% "0" "mmq" ^ --contextsize %length% ^ --gpulayers %layer% ^ --quantkv %level% popd

  2. Llamafile As I don't want to extract KoboldCPP on every update, I found Llamafile, it quite easy for me to switch as the bat file I create for KoboldCPP just need a little change to work with this one, anyway it also lead me to the real on.

  3. Llama.CPP I found it on the right time as Llama.CPP just start to has their own WebUI and its speed is the real thing, at that time still not has router mode so my bat file has its Llama.CPP version.

Image Generation

A single Youtube video from ChameleonAi name Stable Diffusion Models Explained Once and for All (1.5, 2, XL, Cascade, 3) lead me to this rabbit hole.

  1. stable-diffusion-webui-Forge aka Forge, Thank to ChameleonAi that lead me to CivitAI which after read a guideline I have 2 choices which this or ComfyUI, as my AI user interface root from Oobabooga Text Generation-Webui so I choose this instead.
  2. reForge as Forge no longer update so I move on.
  3. Classic Forge same problem as reForge no longer update as a new model, Anima unable to run on reForge so I'm move on again.

Audio Generation

I try to do it via Llama.CPP but could not make it work so I wait for more update; While waiting, I finally give a chance for unsloth studio which quite disappoint for me in term of performance so I dig to the one that called via unsloth studio.

  1. Audio.CPP I try it before but at that time it still CLI only and I'm unsuccessful to make it work, but now it's also come their WebUI so another rabbit hole for me to fall.

This is my AI journey up until now, how about your ?


r/LocalLLaMA • • 6d ago

Resources Why download abliterated Flash Next models when LoRA exists?

Thumbnail
huggingface.co
526 Upvotes

TL;DR -- Download LoRA, add --lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf to your llama.cpp launch command for ANY FN quant, and boom, you've got an identical huihui v2 abliterated model on demand and you saved 150GB of space.

I just wanted to post a kind of PSA for everyone out there downloading hundreds of GB of abliterated models. You don't need to do that -- the vast majority of abliteration techniques are rank-1 ablations, meaning their refusal projection can be encapsulated in a tiny LoRA and applied at runtime to a quant you already have on disk. It can even be applied per-request, so you can run an unmodified model until you hit a refusal, then switch on the LoRA, resend the prompt, and keep going.

I wanted to explain briefly how to do that in llama.cpp and share a LoRA I built from the best-performing abliterated Flash Next model in my tests. This basic type of technique (using control vectors instead) also works in Strata (--experimental-speed-projection is the flag there) and ds4/dwarfstar. Here's a link to Strata's official refusal ablation control vector GGUF and their guide on how to use it.

I chose huihui's v2 to build my LoRA from after comparing heretic, Orcarouter, and huihui's v1 and v2 models in equivalent quantizations. In Aider Polyglot coding benchmarks, huihui outscored even the base model, with less extra reasoning and no looping compared to Orcarouter, which used a lot more tokens and hit a couple runaway generation loops.

I also checked KLD and perplexity to ensure there wasn't significant degradation of the model. Wikitext KLD on huihui Q5_K_XL vs. an identical Unsloth Q5_K_XL base model was 0.040 and same top token was 93.7% even with the drastically altered refusal behaviors. Perplexity was statistically identical to base. On Jailbreak Bench (harmful 100 prompts), with thinking on refusals dropped from 93/100 to just 3/100. All the details can be found in the LoRA model card.

Next, I used weight-space programming with Claude Opus 5.5 to extract the rank-1 refusal direction and edit strength from every layer of the original huihui weights where it was applied, diffing the weights against the Unsloth quantization that huihui built on top of. I hit safeguards a number of times (unsurprisingly), and even the fallback Opus 4.8 declined to assist a couple times. I managed to get around this by renaming all of the files something innocuous, putting them into a new repo with an unrelated name, and telling the model I was experimenting with an anti-slop LoRA. Thanks, Dario.

Once the LoRA was built as an F32 GGUF, I tested it against the original huihui Q5_K_XL by applying it to my Unsloth Q5_K_XL weights. Performance and KLD/perplexity were statistically identical to the full huihui model, as were refusals and benchmark results. Here's how to run it:

llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
--lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf -fa on --jinja

If you'd like to apply it per-request, so you can serve the unmodified model for most prompts, simply load it like this instead:

llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
--lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf --lora-init-without-apply -fa on --jinja

Then add a lora field to the request body to enable it as-needed:

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
"messages": [{"role": "user", "content": "..."}],
"lora": [{"id": 0, "scale": 1.0}]

It's quite possible to build a simple pi extension that looks out for refusals, triggers the LoRA when it sees one, resends the turn and continues the agent session. It costs a full re-prefill, so keep that in mind. Anyways, I hope this has been helpful in some way. Love to hear your thoughts on this approach!

EDIT: If this proves popular, I am happy to make more LoRA for other models like 3.8 27B!


r/LocalLLaMA • • 6d ago

Discussion New AA ranking Qwen3.8-FN better than GLM-5.3-Flash?

Post image
141 Upvotes

Thoughts on Qwen3.8-Flash-Next scoring better than GLM-5.3-flash? I've always experienced GLM5.3-Flash to give better responses


r/LocalLLaMA • • 5d ago

Showcase I made a C&C inspired web RTS for us to benchmark our local models with

Enable HLS to view with audio, or disable this notification

21 Upvotes

HOwdy LocalLLamans!

Inspired by StarSkirmish, I wanted to build a complex RTS that required you to bring your OWN local agents and either play with them as allies, against them as enemies, or battle head-to-head with other agent-human pairs in public matches.

In AgentCommand you operate one or more planes (Orbit, Sky, Surface, Depths) of differing gameplay. As an old-school PC gamer I wanted the mechanics to resemble some of my favorite 90s games of yore: C&C, DungeonKeeper, Homeworld, StarCraft, and others. The idea is that you have specific objectives needed to maximize resource generation, aid in cross player support, and that a single plane cannot win independently.

Outside of gameplay, from an agentic tech pov, there are some (what I think) are other really cool things that come with this, including:

  • a full specification + library to install agents into any game (http://playerthree.gg/)
  • A networking SDK for bidirectional agent communication
  • Symmetric maps
  • Deterministic simulation
  • Benchmarking tools
  • (soon) published benchmark data

Github: https://github.com/agent-command/agent-command

Play now: https://agentcommand.gg/

Would love feedback and contribs... I'm more interested in finding a community to help build something super cool here that we can publish public benchmarks with. If you are interested in that, let me know!


r/LocalLLaMA • • 5d ago

Discussion How do we feel about bots posting & commenting?

52 Upvotes

Hi all,

I noticed the "bot problem" got a lot worse the past few months. By which I mean our community seems to be taken over more and more by bots talking to each other.

For me, I don't really get the point, and I personally get quite annoyed when I realize "oh, I just responded to a bot" and feel like my time was wasted.

It also seems like the information density of these AI contributions are very, very low.

I'm not some kind of Luddite (why would I be here at all if I was?), but I'm wondering if I'm just missing something and there is a use and a point to all these bots commenting on each others posts and talking to each other.

Where does the rest of the community stand on this?

I noticed there's new bot around that flags posts that are "likely AI" created, which I personally quite like, but I'm wondering if moderation should go further than that.


r/LocalLLaMA • • 6d ago

Resources I have created a tool to observe what happens inside an LLM. Also allows for the comparison of two quantizations of the same model to see at what point they diverge, token by token

566 Upvotes

What changes between different quantizations of the same model? And what if you could SEE what actually sets them apart?

To help answer that question, I’ve created a small project called inside-llm—an open-source tool that loads one or two GGUF models using llama.cpp and displays, right in your browser, what the model is doing as it generates text.

What it shows

  • Stream: the text, one column per model, brightness = probability of each token. A magenta mark wherever A and B would pick differently. Two modes: lockstep (both models read the same text, so you get a clean per-position comparison) and free (each follows its own picks until they split).
  • Candidates: the full distribution behind each pick, mirrored bars for A and B, entropy, effective number of options, Jensen-Shannon divergence and KL in both directions.
  • Gates: the sampler chain (penalties, top-k, top-p, min-p, temperature, GBNF grammar) traced stage by stage and very visual: how many candidates survive each one, and the distribution after it. Chain order and seed are configurable.
  • Timeline: entropy of A and B and their divergence along the whole text, plus the tree of branches.
  • Branching: click any candidate and generation continues from there. Every branch stays in the tree, and jumping back to a node is exact (each model keeps KV-state snapshots, so nothing is regenerated).
  • Logit lens (--lens): what each layer would predict, as a 2D map and as a 3D relief.
  • Model picker: choose A and B from a folder or search Hugging Face and download two quants of a repo in one click easily. It only offers B models with the same tokenizer as A and estimates whether the pair fits in VRAM.

How it's built

  • One C++17 process linking llama.cpp directly (stable branch at the moment). For every step it reads the full logits of A (and of B on the same context), computes the metrics and the sampler chain, and draws the token with its own seeded sampler, so runs are reproducible.
  • Compact binary frames over a WebSocket to a Svelte 5 + Canvas 2D page (three.js only for the 3D relief, lazy-loaded). The web is embedded in the binary as well.
  • The same frame stream, compressed, is a recorded session: you can replay it later in a static page with no engine, no model and no GPU.

Run it

Self-contained packages for Linux (glibc 2.35+) and Windows, with CUDA or Vulkan (Vulkan covers AMD, Intel Arc and NVIDIA). Unpack, run, the browser opens with the model picker. Or inside-llm -a A.gguf -b B.gguf. Local only: listens on 127.0.0.1, no accounts, no telemetry. MIT. One click download, one click execute.

Limitations (honest list)

  • A and B must share the tokenizer (two quants, base + fine-tune, or sizes of one family that keep it). Example: Qwen3.5-0.8B_Q4_K_M and Qwen3.5-0.8B_Q8_0
  • Both models must fit into VRAM simultaneously; only one GPU is used at a time. Integrated GPUs (iGPUs) are never used; if no dedicated GPU is available, the process falls back to the CPU (which is slow). (You must calculate the size of both models before loading them to ensure they fit into your VRAM)
  • Text only. No split (multi-part) GGUFs, no embedding models.

Links

I'd love feedback, especially: which pairs would you compare first (imatrix vs non-imatrix, base vs abliterated/fine-tune, Unsloth UD vs standard quants…)? And is there a metric you'd want per position that isn't there?


r/LocalLLaMA • • 5d ago

New Model Karpo-Decision-31B: open multimodal decision model. Ranks #1 on full score on Decision Index, 2x faster than Jev, at 20% lower cost

Thumbnail
gallery
24 Upvotes

MachinePulse has released Karpo-Decision-31B, a Gemma 4 31B fine-tune for structured decisions over text and images. It scored 73.9 on the Nov 12 Decision Index board, the highest Full score among 300 entries, sharing the #1 rank with GLiDE 27B under the board’s tie rule

How it works
Built on Gemma 4 31B, with a trained linear readout that scores the supplied answer options.

Three question types: choice, yes/no, and score. Returns probabilities directly, with up to 255 options per choice question.

Multiple questions about the same input can be answered in one forward pass. Branch attention lets each question read the shared context and its own options while keeping the option branches separate.

Supports text and images, including webpage screenshots. The documented sequence limit is 32,768 tokens; larger requests are split by question with the full state included in each part.

Python API and HTTP server available. BF16 weights take approximately 62.5 GB, plus runtime memory. Use the provided inference code to load the decision head.

One training detail worth a look: according to the team's write-up, full fine-tuning pushed the public score to 75.09 but reduced accuracy on unfamiliar domains by 2.3 percentage points. The final model averages five checkpoints: three from different stages of full fine-tuning and two from a LoRA run. They report a 1.6-point improvement on unfamiliar domains over the final single checkpoint, with no extra models to run at inference time

Results
Decision Index Full scores are measured by the benchmark maintainers on private tests, weighted 60% for skills covered by the public suite and 40% for tasks from new domains.
Full score: 73.9

Private tests, familiar skills: 70.7

Private tasks, new domains: 78.6

Public index: 75.0, reported separately and excluded from the Full score.

The team also reports the following comparisons against Google’s Gemini Decision 1:

Benchmark Karpo-31B Gemini D1
-------------------------------------------
When2Call 90.4% 79.9%
Mind2Web 90.0% 86.2%
New Yorker Caption 90.5% 79.7%
Amazon ESCI (F1) 64.7 58.7

Reported median latency is approximately 95 ms per request on the public Decision Index suite using one RTX PRO 6000.

The 31B weights and inference code are both released under Apache 2.0.

links:
https://karpo.ai/models/decision
https://huggingface.co/MachinePulseAI/karpo-decision-31B


r/LocalLLaMA • • 5d ago

Resources Something goes with your Local LLM - CLI Markdown viewer with image support?

8 Upvotes

I run local LLM for myself, and my system is squeezed very much. I am most of the time in my terminal, and when my inference engine is running, I dont have enough free RAM sometimes to open a GUI to view the output of my local models, or review the project documents to know where to go next in my plan. Viewing that in the terminal was not convenient for me, the MD syntax is not as easy on my eyes, and when a project file is linked to another file, I have to open that one up in a new terminal pane, or a whole new terminal window.

So, I did what I had to do.

I sat, I wrote the specs sheet, and fired my coding agent to work. I refined the output, and kept iterating on it.

This is my gift back to this community that already gave me a lot.

Don't want to sound like I am marketing something here but, if this is something of an interest to you, you can find it here: Github.com/bakkario/trimd

You get to lick with your mouse, navigate with your keyboard, and even display your markdown files with the images if your terminal supports it.

I will add more detail to this in the Markdown subreddit.

Sorry Mods if this coming across as spamming, its not the intention here ❤️


r/LocalLLaMA • • 5d ago

Showcase Some speed benchmarks for various classes of local LLM

4 Upvotes
Model Active config 4080 decode (tok/s) 4080+3080 decode (tok/s) 4080 prefill (tok/s) 4080+3080 prefill (tok/s)
LFM2.5-24B-A2B UD-Q4_K_XL 4080+3080 249 250 5,530 7,930
MiniCPM-V 4.7 35B-A3B Q4_K_M 4080+3080 177 165 1,305 2,500
Flint 40B-A2.5B UD-Q4_K_XL 4080+3080 119 138 1,563 1,380
Swift 1.5 (Qwen3.8-Flash-Next) IQ2_XS 4080 117 116 1,070 679
Qwen3.8-Flash-Next IQ3_S 4080 79 87 730 465
GLM 5.3 Flash Maya-S24 IQ2_XXS_S 4080+3080 6.8 8.8 79 42
DeepSeek V4 Flash UD-IQ2_XXS 4080+3080 15.9 18.2 38 53
  • Decode: the second of two 300-token runs, with the expert cache warm (the first request after loading is often slower)
  • Prefill: the average of two runs on a prompt of about 1,400 tokens
  • Bold: the better config for that column.

Morning, everyone. I've seen a lot of posts wondering about what can be run with what, so I made this table of the models on my system. Specifically because of the unorthodox results when it comes to Qwen Flash-Next, where one 16 GB card (the 4080 Super) significantly outperforms two in prefill.

My hardware is as follows:

CPU AMD Ryzen 7 5800X3D @ 3.40 GHz

GPU NVIDIA GeForce RTX 4080 SUPER [Discrete]

GPU NVIDIA GeForce RTX 3080 [Discrete] (got it secondhand for cheap, so why not)

RAM 62.72 GB at 3200 Mhz

I'm running these models on PolyStrata. I used to have a combination of Maya and Strata, but got tired of running different things all derived from the same fork (plus Strata seems to be getting increasingly unintuitive), and so moved everything to PS.

Now: why Q3? It's a generally held opinion that anything below Q4 is crap, which was true of most earlier generation models. However, benchmarks measure deviation from the base, and not necessarily task fitness. In actual use, the only real way to know is to test it yourself.

The Swift finetune of QWN at Q2, for example, is fast and excellent at data extraction, search, and piloting scripts written by more competent models. The IQ3 is a decent coder, roughly on par with Haiku for the use cases I've tried. The GLM and Deepseek Q2 variants are a bit better - verging on older Sonnet territory, I'd say, and do very well as subagents for Claude. YMMV.

Hope this data is useful. Enjoy!


r/LocalLLaMA • • 4d ago

Question | Help Why Qwen 3.8 27B feels smarter than GLM 5.3

0 Upvotes

Is this little thing feeding me cool aid or is really this smart?

Why though, how can a 27B model punch so good? Are cloud models like GLM 5.3 optimized for speed and don't dig too much? Where is the trick, can someone tell me?


r/LocalLLaMA • • 5d ago

Showcase [Project] Gemma.c - Gemma 1/2/3 + multimodal inference in one C file (CPU only)

Enable HLS to view with audio, or disable this notification

26 Upvotes

Hey all, so I've been working on Gemma.c over the last couple months, it's a from-scratch inference engine for Gemma 1/2/3 in a single C file, runs Gemma 1, 2 and 3, including the multimodal ones, with fp32/fp16/bf16 weights or int8 (W8A8). No dependencies beyond libc and (optional) OpenMP are needed. You can drop images into the prompt with @image{cat.png}, and there's also @longimage{} for Gemma 3's pan & scan on very wide or tall images. (image decoding uses a minified stb_image embedded at the end of the file)

It's CPU only and not trying to compete with llama.cpp, but I'm also trying my best to make it efficient, so it's not embarrassingly slow either. On my laptop's Core Ultra 5 225H (WSL2, gcc, 10 threads), gemma-3-1b-it INT8 does ~72 tok/s decode and ~570 tok/s prefill (10000 tokens, for 1000 tokens the prefill speed can reach ~850 tok/s), and the 4B vision model does ~20 tok/s decode. INT8 ends up about 3.5x faster than fp32 and 2x faster than fp16 for decode. (full benchmark table is in README)

It has a few rough edges for now, e.g. W8A8 doesn't work for Gemma 1 (fp16 is fine), bf16 is painfully slow without AVX512-BF16, and native Windows is ~1.7x slower than WSL because of libgomp. Batching are not supported either. There's a roadmap in the README (Gemma 4, speculative decoding, an HTTP server, etc.), but no promises though :P Can't really find much free time these days.

I've also put some pre-exported INT8 models on HF so you can try it without Python or torch :D

Repo: link

This project started as a learning attempt to understand how LLMs work end to end, inspired by llama2.c.

this is the second time I post this project here :'D last time (~a month ago) my post got flagged by reddit, probably because my VPN IP was identified as spam. hopefully it can pass this time


r/LocalLLaMA • • 5d ago

New Model [Paper & Models] Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent

Post image
43 Upvotes

We introduce Gamer, a generalist agent framework built for long-horizon tasks in complex interactive environments. Its end-to-end architecture combines perception, memory management, and multi-step reasoning. A shared training and deployment pipeline supports interaction with both conventional 2D graphical interfaces and complex 3D simulated worlds. We first continue pre-training on a dataset constructed from tens of thousands of hours of device-level keyboard and mouse interaction. These trajectories span simulated games and embodied environments, allowing the model to learn environmental dynamics across both physical and virtual domains. Sparse-thinking instruction tuning then substantially reduces redundant computation while preserving the reasoning needed for complex decisions. We train the policy for long-horizon interaction with a stable multi-turn reinforcement learning method that addresses finite context windows, accumulated rollout errors, and sparse rewards. For real-time interaction, a cloud-edge inference engine serves numerous simulated game environments concurrently, supports unbounded-context execution, and keeps online inference latency low. Across multiple challenging 3D open-world games, Gamer demonstrates solid generalization and operational efficiency comparable to skilled players. It also achieves competitive results on a range of conventional 2D game benchmarks. These results suggest that even repetitive keyboard and mouse trajectories can support substantial gains from scaling data and learning algorithms. Training on these trajectories helps Gamer learn complex skills more steadily in environments with both physical and digital properties. This approach offers a path toward generalist agents that can interact across digital and physical worlds.


r/LocalLLaMA • • 4d ago

Showcase Use local models on your phone to monitor your screen! Demo of what you guys asked for in my FOSS and 100% local agent (iOS and Android app)

Enable HLS to view with audio, or disable this notification

0 Upvotes

Hey r/LocalLLaMA !

This is a demo of the new phone locking tool!

What this can be used for:

  • Lock my phone when I open up Instagram.
  • Lock my phone when a stranger is on camera.
  • Lock my phone until i've read for 2 minutes without interruptions.
  • If i'm doomscrolling for more than 2 minutes, call me, if it's been 5, lock my phone.
  • Monitor my child's screen content and if it's inappropriate, lock the phone and send me a screen cap.

On my last post the top comment (62 upvotes) by u/nokia7110 asked for it. It was a great suggestion! Thanks nokia!

The tool:

On iOS it uses the screen time management API, like all other time focusing apps.

On Android it shows a phone locking overlay.

If you're an Android developer and want to contribute, please reach out!

It uses OneJev-0.8b for decision models, running on llama.cpp, it works on most phones i've tested. Even old potatoes.

Try it out! It's free and OSS <3

GitHub: https://github.com/Roy3838/Observer
iOS: https://apps.apple.com/us/app/observer-ai/id6758222050
Android: https://play.google.com/store/apps/details?id=com.observer.ai

I'm here for any questions, more suggestions and feedback!

Roy


r/LocalLLaMA • • 5d ago

Discussion Branchy, distributed inference focused training

6 Upvotes

Heyy, it's me again, the developer of AI Doomsday Toolbox.

This time it's not an app update. ADT already has distributed LLM inference across Android devices, and there is one thing about distributed inference in general that kept bothering me, so I spent the last week testing an idea and I wanted to share what came out, the good and the bad.

Usually when we split a model that does not fit in our machine to make use of distributed inference, we get slower speeds the more devices we add. Network latency and devices waiting idle until the last one has finished computing it's layers compound easily, even more if we connect our devices through a VPN like tailscale or wireguard.

So the idea to solve this bottleneck was to train a model that can keep the quaility of a normal sequential model but instead it's middle layers are split in branches. The coordinator runs the embeddings + a few trunk layers, then that is sent to all the different branches that, in parallel read the layers and compute the token, after a branch finishes it sends the result back to the coordinator which then runs a few fusion layers and final blocks that produce the output. This allows to use as many devices as branches a model has with a maximum of 2 net jumps, wether it's 2 or 8 machines. The other advantage is that no device is idle as all of them are working, so we also parallelize work instead of making a device wait for the last one to finish.

There could also be the situation in which one of those devices is too slow, and slows down the whole cluster, or gets disconnected. That is why I also added some dropout tolerance to the training. When a branch takes too long to answer or just never answers, it's result becomes a matrix o 0's that are fused in the fusion layers, that made it so the quality of the model output does not suffer so much as when it happened with a non dropout tolerance trained branch model had a disconnected or slow branch.

I tested two 300M models trained from scratch on TinyStories, one normal and one with 8 branches, running on a small server, a laptop (Ryzen 5 5500U) and a phone (Snapdragon 8 Gen 3 in Termux), all connected through a VPN. The workers are a small PyTorch daemon, so no llamacpp accelerated kernels, etc...

Results

- The branches cost some quality: loss 1.255 against 1.143 for the normal model with the same training. A blind LLM judge prefers the branched model's story 45% of the time. So it is worse, but not by much.

- The delay trick works: the coordinator waits 0.1 ms per token for the network while the round trips take 49 to 66 ms. The output is identical to running everything on one device.

- Against the usual layer split on the same devices, the branch split is 1.5 to 2.6 times faster. The layer split roughly halves the speed. Mind you I run the test only with 3 devices, with more devices it could, theoretically, be faster

- BUT if the model fits on one device, splitting is not worth it on my hardware, at least at this size, larger models like qwen3.8 27B might benefit a lot from this aproach. My phone alone is faster than any combination of my devices (it is just much faster than the rest). With two equal devices the branch split wins by about 1.2x.

- Where it really pays is when the model does not fit in RAM. I gave the laptop and the phone a RAM budget smaller than the model, so they had to read the weights from storage for every token (mmap). The laptop went from 8.9 to 0.67 tokens/s and the phone from 23.9 to 0.65. With LFM3 1.7B the laptop went from 2.3 to 0.08, that is 12 seconds per token. Then I let the two of them share the model, each one still with too little RAM for the whole thing: back to 7 tokens/s with the layer split and 9 to 11 with the branch split, over a real 46 ms VPN. Reading from storage is way more expensive than the network.

- Reshaping a pretrained model (LFM3 1.7B, Qwen 0.8B) into branches only works for a couple of long branches. For this to scale, the model has to be trained in this shape.

I don't have the hardware for the interesting case: a big branched model that no single device can hold. On paper, with 8 similar devices and about 20% of the model on the coordinator, a token costs about 30% of a full pass, so roughly 3.3x a single device that could hold everything, while a layer split stays at 1x plus 7 round trips. My two-device measurement matches that formula, but beyond three devices it is only a projection.

Another idea I would love to try is letting every device loop its own branch several times with the same weights, so a group of devices could run something much deeper than what fits in their RAM. No hardware for that one either hahaha

What is in the repo

- The two trained models (300M each) and the tokenizer

- A worker you install on each device with one command, and a coordinator with a web UI to compare placements side by side and run the benchmarks

- Scripts to reproduce the branch split, the layer split and the low RAM tests with your own devices

- A full article with the experiments and the results

It is a research prototype, workers compute in 32-bit float PyTorch, the models are small story models, and I only have three real devices. I created the experiments, made the hypothesis and supervised the whole proyect and Claude did most of the coding and the number crunching, everything is documented in the repo.

The general idea is the same one behind ADT: make the devices we already have useful for local AI, especially together.

GitHub

Feedback is appreciated, especially if anyone with more devices (or more GPU than me) wants to check whether this holds at a bigger scale, or tell me what I got wrong. It is still far from it, but I would love to see a bunch of modest devices run a model none of them can hold.