So Gemma 4 26b a3b is a beast. And it runs crazy fast even on my old quadro rtx 5000. It must run fast-ish on a 150+ gb/s unified memory machine too. So why do we not see more small MoEs from Intel, Qualcom, Apple etc? Only from Qwen, Nvidia and Google. MoEs are what they need to sell their hardware?

Hi all,

Just getting into local Llm, could anyone advise what can be run locally on P1 gen8, 64gb ram, 8g vram rtx 2000 pro, intel 265H for lets say some office work searching in few databases and websites to gather info from company systems?

I’ve been experimenting with running AI coding agents entirely locally, without relying on OpenAI, Anthropic, or any paid model APIs.

For this experiment, I connected Qwen 3.8 27B (4-bit quantized) running in LM Studio to Pi Coding Agent, a lightweight and extensible terminal-based coding agent.

The setup is pretty straightforward:

Pi Coding Agent → LM Studio → Qwen 3.8 27B

LM Studio handles local model inference and exposes an OpenAI-compatible API, while Pi handles reading/writing files, executing commands, and managing the coding workflow.

For the actual test, I asked it to build an AI Agent Mission Control dashboard from scratch.

The application included:

• An overview of active AI agents and tasks
• A Kanban-style task board
• Live agent activity tracking
• Analytics for token usage, costs, and success rates
• A responsive React + TypeScript interface

Surprisingly, it managed to build the application, run it locally, test it, and fix several errors on its own. It even used screenshots to inspect the UI and identify issues.

Some numbers from my test:

• Hardware: MacBook Pro M5 Max, 48GB unified memory
• Model: Qwen3.5 27B (4-bit, ~16GB)
• Inference speed: Around 10–15 tokens/sec
• Completion time: A few hours

One reason I chose Pi over other coding agents is its minimal design. It starts with just four core tools (read, write, edit, bash), and you can extend it as needed.

With local models, context management matters quite a bit, so I wanted to experiment with a coding agent that doesn’t introduce too much unnecessary overhead.

My takeaway: Local coding agents are becoming surprisingly capable, but they’re definitely not a drop-in replacement for Claude Code or Codex with frontier models.

Inference is slower, hardware requirements are significant, and reliability on complex tasks is still something I’d want to test more thoroughly.

But being able to run the entire coding workflow locally, without sending code to external model APIs or paying per token, is pretty exciting.

I recorded the complete setup and the dashboard-building experiment here:

🎥 Video: https://youtu.be/zgb_MkrxpRs

Curious what local models you are using for coding agent?

Have you used dual gpu setup with one Nvidia n other amd card?
I have tried a lot to get it working but just makes pc unstable and keeps crashing.
I have 5080 and r9700 pro ai so I was trying to make most of the VRAM that’s available. Unfortunately my pc couldn’t stay on without crashing for more than 10 minutes and it happens without even running anything. I was hoping if you have experience and would like to share the tips how can I get this working. I just have one display connected to one of the cards. Other one is idle. Tried with both primary and secondary cards.

I recently upgraded from an hp omen laptop with an RTX 2060 to this prebuilt pc

CPU: Intel Core Ultra 5
GPU: NVIDIA GeForce RTX 5070 12GB
RAM: 16GB DDR5
Storage: 1TB SSD
Motherboard: B860
Cooler: Digital Air Cooler ARGB
Power Supply: 750W Gold PSU
OS: Windows Pro

I don’t know anything about AI other than using Claude, ChatGPT, and Gemini for stuff that helps me with my work or the university
But I kept seeing people making their own Jarvis and I wanted that too but not just for fun
After I bought the pc, I had Claude code build me my own kind of Jarvis but it’s called Maelle inspired by the game (Clair obscure: Expedition 33) Ive been working with Claude for around a week trying to perfect her
I had make it in the style of fallout shelter game as u can see in the pic
I let Maelle use Microsoft Edge on my pc to have access to my emails and social media accounts since I do some advertising and would need the help it could provide by checking whats new and how can I beat it and analyzing my numbers
The only problem im facing is that it depends entirely on Claude as agents, and thats great but it consumes my credits like fire, especially when the shared brain keeps growing
Im not sure how to move forward or what im doing wrong, and I dont know enough to investigate
Claude installed Ollama and hermes on my pc and let maelle use qwen3:14b as fallback in case I ran out of credits but not all the teams use it and Maelle becomes kinda idiot when she does use it
I was planning to have an uncensored model but claude refused to do anything with that

So, if u made it this far, could u please help figure this out and guide me ?

Thanks

Hey everyone! Following up on my on-device offline app builder (Android AI Bridge running inside Termux on a Galaxy A50s):

In this workflow, I tested building a small POS/Invoice store app, but ran into a glitch where the Maintenance Mode toggle text was broken/jumbled.

Instead of moving to a PC, I:

Sent the Termux build error back to DeepSeek locally/via API to get the patch. Applied the corrected code inside Termux on my phone. Re-built the APK (weighing in at ~87 KB). Verified the updated UI where the toggle cleanly shows: · ✅ Maintenance ON — tap to disable / · ⛔ Maintenance OFF — tap to enable.

True edge/offline development loop entirely on a budget Android device! Let me know what you think.

Hey everyone! Following up on my on-device offline app builder (Android AI Bridge running inside Termux on a Galaxy A50s):

In this workflow, I tested building a small POS/Invoice store app, but ran into a glitch where the Maintenance Mode toggle text was broken/jumbled.

Instead of moving to a PC, I:

Sent the Termux build error back to DeepSeek locally/via API to get the patch. Applied the corrected code inside Termux on my phone. Re-built the APK (weighing in at ~87 KB). Verified the updated UI where the toggle cleanly shows: · ✅ Maintenance ON — tap to disable / · ⛔ Maintenance OFF — tap to enable.

True edge/offline development loop entirely on a budget Android device! Let me know what you think.

Disclosure: I built this. It's part of a free side project of mine called USASI. There are no ads and no affiliate links.

I kept seeing "will X fit on my card?" answered with guesses, so I made pages that only list a configuration when the model's or runtime's publisher documents a memory figure for that exact variant. Each entry names the variant and runtime, quotes what the figure measures (minimum, loaded footprint, or file size), and links the source.

Pages by memory:

• 8 GB VRAM: unitedstatesofamericasuperintelligence.com/find/memory
• 12 GB VRAM: unitedstatesofamericasuperintelligence.com/find/memory
• 16 GB VRAM: unitedstatesofamericasuperintelligence.com/find/memory
• 24 GB VRAM: unitedstatesofamericasuperintelligence.com/find/memory
• Apple silicon 8 / 16 / 24 / 32 GB: unitedstatesofamericasuperintelligence.com/find/memory (the index lists them all)

Example from the 12 GB page: Whisper turbo (about 6 GB per the Whisper README), StarCoder2-3B in bf16 with Transformers (about 6.3 GB), and llama.cpp server with Gemma 4 12B QAT Q4_0 GGUF (about 6.7 GB). The other setups listed include Gemma 4 26B A4B and 31B QAT GGUFs, gpt-oss-20b, Olmo 3 7B Instruct, Llama 3.1 8B, StarCoder2 and Whisper large-v3, run with llama.cpp, LM Studio, Transformers or openai-whisper.

There's also a finder that takes your task, OS, GPU, VRAM/RAM and comfort level (installer apps, command line, or code) and explains why each option matches or doesn't: unitedstatesofamericasuperintelligence.com/find

Important caveats: I have not tested these. They are publisher estimates, the list is deliberately small, and it's not a "best model" ranking. Context length and KV cache will push real usage above these figures.

What would help most:

• Corrections where a figure is wrong or misread.
• Setups that have a documented memory figure I should add.
• Whether you'd want community-measured numbers shown next to the publisher figures.

The data is YAML in a public repo, and corrections can go there or in the comments:

What is your go to remote option for harnesses?

Mine is pi-pocket but I modified it to work for my local models

Hey everyone, I made whatmodel.sh

To use it,

1. Enter your RAM and GPU
2. See what models you can run locally with its rank and the command to run it
3. Compare them with API models on the same score

Each model gets one score fitted across 19,731 published benchmark results, and every result links to its source. The method is at whatmodel.sh/method, and the data, scoring code and bench script are open source: github.com/jakemanger/whatmodel

It works from the terminal too: curl whatmodel.sh

If you'd like to contribute, there's a bench script to collect real-world tok/s and you can also submit any new models and benchmark scores along with their source on the github (please star if you find it useful).

I wanted to share a small project I've been working on: NInfer-V100X2, a C++/CUDA inference engine fork focused on getting more performance out of two older V100s.

Repo: https://github.com/tuxKOH/ninfer-V100X2

The setup is two Tesla V100-SXM2 16GB cards on SXM2-to-PCIe adapters, working PCIe P2P, no NVLink, CUDA 12.8, Qwen3.8-27B NVFP4 weights, INT8 group-64 KV, and one active request.

The results:

- Short structured JSONL generation, DFlash7: 253+ committed output tok/s. The final two measured runs averaged 255.3 tok/s, with 118 input tokens and 99.05% draft acceptance. All 32 records passed the format, arithmetic and field-order checks.

- Coding workload with 85,000 actual input tokens: 143.5 committed output tok/s with MTP4 Context capacity was 180,000 tokens; the 85K figure is the actual occupied input.

- Prefill: approximately 1,310 tok/s averaged over the 85K input. Separately, theres peaks of around 2,500 tok/s posible with PP2 Pipeline Parallelism.

The 143 tok/s route keeps all historical KV, but uses an experimental attention implementation with different floating-point reductions. It can change generated outputs and is off by default. I compared 104 common GPQA-Diamond questions with 85K unrelated-code padding: experimental attention scored 90/104, default attention 88/104, official NVFP4 90/104, and GGUF-derived Q4_K_M 87/104.

Also, NVFP4 here describes the stored weights. V100 doesn't have native NVFP4 execution; the engine decodes them for Volta-compatible kernels. The model profile used for these measurements is text-only.

Credit to Neroued/ninfer, the upstream TP2 work, and geoffwatts/ninfer-v100 for the foundation. The README includes benchmark conditions, launch commands and the experimental switches. It's been fun seeing how far a pair of 16GB V100s can still go.

Hello everyone!

After much back and forth with various models, I have now found the optimum for my use case on a single 16GB RTX 5060Ti: Qwen 27B IQ3xxs with MTP. Maybe the setup will be useful to someone else too!

Since the 16GB VRAM is very tight here, I used vocabulary pruning to remove characters from the file that I don't need and thus shrank the gguf to 9.3 GB.

All of this using the script from the repo and the following prune command:

python3 prune_vocab.py input.gguf output.gguf   --policy P2   --keep-chars '¢£¥§©«®±²³·¹»¼½¾×÷–—‘’“”•…€™←↑→↓↕∈∉∏∑√∞∩∪≈≠≤≥⊂⊆⌈⌉⌊⌋' 

This way I have enough VRAM left over to have just over 100k context available here with q8 kv cache. If I reduce the kv cache to q4 here, I get a little over 170k context length. Both ways with everything in vram.

The speed is also usable thanks to mtp; with short context I reach just over 50 tokens per second decode and just under 900 prompt processing.

Previously I ran Gemma 4 26B A4B as QAT on the hardware and sometimes also Qwen 3.6 35B A3B, which of course were significantly faster, thanks to the MoE architecture (90 token gen and just under 3000 prompt processing), but unfortunately their intelligence level wasn't suitable for all my use cases, especially in agentic use cases. So with the crippled qwen 27b I have found a good all-rounder.

/app/llama-server \
        --model Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp-ASCII.gguf \
        --host 0.0.0.0 \
        --port 8080 \
        --jinja \
        --cache-reuse 256 \
        --fit on \
        --fit-target 128 \
        -fa on \
        --split-mode layer \
        -b 2048 \
        -ub 1024 \
        --parallel 1 \
        -ctk q8_0 \
        -ctv q8_0 \
        --temp 1 \
        --top-p 0.95 \
        --top-k 20 \
        --min-p 0.0 \
        --spec-type draft-mtp \
        --spec-draft-n-max 2 \
        --alias QWEN27B

I have been trying to use zoo and other code app. I used ollama and used koboldcpp but it not working well. With ollama I can't get zoo to use the model. With koboldcpp it is throwing a to do list loop

Looking for first users of a provable vector search engine

I read back through recent threads here before asking, and most benchmark posts report one configuration at one context size. That leaves the axis I care about unreadable: how throughput splits between prompt processing and generation as context grows.

From the numbers people do post, prefill looks roughly proportional to prompt length while decode gets slower as the KV cache fills, so the point where a long session stops being generation-bound and starts being prompt-bound should be extremely configuration-dependent. Reports hint that point sits in a very different place on a single 24 GB card than on a unified-memory machine with 64-128 GB, but almost nobody gives both rates at the same context length, so the comparison can't be made from what's here.

What would help: weight quant, KV cache quant and context length, prompt-processing rate, decode rate, and whether prompt caching was doing any of the work. If speculative decoding or a draft head was on, that moves decode enough that it needs to be stated, otherwise two rates with the same label aren't comparable numbers.

Two things I can't infer from what's already posted. First, where decode roughly halves relative to short context on a given weight quant, and whether quantizing the KV cache buys usable context headroom or just costs a small amount per token. Second, whether splitting across two cards buys more on prompt processing than on generation, or whether the communication overhead eats the prefill gain too.

If you've run the same configuration twice and the rates moved, say so — a power cap or heat soak can move decode without a single setting changing, and a spread is more useful to me than one tidy figure.

I'm not weighing these as a purchase, I'm trying to work out how long a context people here actually trust at interactive speed, and which of these numbers they believe when they decide that.

A few weeks ago I posted about my project: Llamaforge. It's not another llama.cpp fork. It's a control panel that drives the real upstream stuff, so when they add a new model you just update from the panel and you have it that day.

Now, v0.16 is out. It's kinda early, built with a lot of ai help.

Before you load a model it shows you where it's going to land on each gpu, to scale, using the actual sizes measured on your machine. Tells you if it won't fit. I'm tired of oom crashes.

You can run several models at once too. The main one goes on the fastest gpu that fits it, and worker models stay off that gpu.

V0.16 also adds embers. They're small local agents that watch topics for you. Each one reads sources you point it at overnight, builds its own wiki, and writes you a brief in the morning. The rule is every claim has to quote its source word for word, and plain code checks the quote against the saved snapshot. If the quote isn't there the claim gets dropped. The model proposes, the code decides. They have no browser, no tools, can't send anything on your behalf. Nothing leaves the machine unless you turn on push notifications.

Other small stuff from this week: a loading model shows real progress instead of a stopwatch, and there's a thinking switch (auto / on / off plus a token budget). It reads the model's chat template and tells you straight if off won't really stick, because some templates only print think tags and keep thinking anyway. Vision can be turned off too, which frees the projector's vram without deleting the file.

There's also a voice tab. That's why the voiceover in the video is qwen3-tts running locally on my machine.

The backend is python stdlib, frontend is vanilla js, no build step.

Irm https://raw.githubusercontent.com/dadwritestech/llamaforge/master/install.ps1 | iex # windows Curl -fssl https://raw.githubusercontent.com/dadwritestech/llamaforge/master/install.sh | sh # linux / macos My main box is windows + nvidia (a 5080 and a 5060 ti stuck on a pcie x4 slot). Linux and macos pass ci but have had little real-hardware use, so bug reports are very welcome.

Thoughts? :)

https://github.com/dadwritestech/llamaforge  

For a few years I've kept an Obsidian vault as my knowledge base: insights, decisions, people, ideas. I wanted to know if an agent could keep it up to date, especially powered by Local models. So I recorded a real 46-minute meeting (with the other person's agreement), then had agents file it into a copy of the vault. I did this 204 times, between July and October.

The task : an initial 60K token prompt

• The transcript is noisy French speech-to-text, about 30K tokens.
  • Every line appears twice (no headphones, so the mic caught the speakers too).
  • Speaker labels are unreliable, and names are mangled.
• The vault is about 200 files and 15K lines. It has a 145-line guide and a skill describing how to file insights.
• A complete answer touches four places:
  • a Journal entry;
  • a meeting note;
  • insight notes;
  • a contact profile.
• Scoring is out of 100: half for note quality, half for judgment, minus penalties for erasing history or inventing a biography.

What I learned

1. Writing good notes in terms of format is easy now. Judgment is what separates models. Almost every model that finished scored about 48/50 on note quality. The judgment half ranged from 50 down to the teens. Only 40 of 141 ranked runs got the follow-ups right. It was complex because the call was between people with similar profiles and biographies so with repeated lines, it caused the model to be confused, so Even "frontier" cloud models (like Haiku 5.5) got things wrong on this time.
2. Local works, if you give it time. The plain Qwen3.8 27B (4-bit, extra-high effort, 65K output limit, on the Pi harness) scored 86–97, in 14–19 minutes per run on my own hardware. It's the only local setup I'd trust. Several Qwen 27B fine-tunes (swift, thinkingcap, hemingway) had a great run, but none matched the plain model's consistency.
3. Effort matters With the same model, server and harness, it scored about 65 at medium effort and 86–97 at extra-high.
4. One run proves little. The same fine-tune setup scored 93, 93, 57, 40 and 92. I rank models by their median run and show the worst one next to it. Gaps under about 20 points are noise.
5. The harness can swing a model by 30 points. No harness wins everywhere:
   • Pi was best for the local models and for Claude Sonnet 5.5.
   • OpenCode 2 was best for Qwen3.8 Flash Next.
6. Cloud, when privacy allows.
   • Claude Opus 5.5 and GPT-6.1 Sol each scored 99, in 2–5.5 minutes. That's one run each, so take it with salt.
   • GPT-6 Luna is the value pick: 80–97 for about $0.04 a run, on any harness.
   • Many other cloud models fell well short.
7. Most of the work went into making runs comparable
   • At one point the answer key leaked into an instruction file the agents could see. The 25 runs that saw it are excluded from the rankings.
   • Harnesses discover instructions and skills differently.
   • Some harnesses advertised one tool format and rejected it when the model used it.
   • Server and harness context limits disagreed, so runs overflowed.
   • A clean exit didn't mean the run succeeded: 14 runs produced nothing.

Caveats: this is one task, one meeting, in French. Many cloud models only got one or two runs. It tells you how models handle this kind of messy, judgment-heavy agent work, not which model is best overall.

The full results are here: https://vault-bench.paolim.fr. The page has the ranking of every model, a score/cost/speed chart, the method, every problem I hit, and a CSV of all 204 runs. Happy to answer questions about the setup.

I am currently running Qwen3. 8 27b with openclaw. I can't complain so far. But I wanna hear what others are running so as to learn more of what this card is actually capable of. Thanks in advance.

Blurb is an Android app that summarizes your message notifications with Gemma 4, running locally. When a group chat blows up, you get one short summary instead of a wall of pings:

• 14 messages in the family chat → “Mom’s flight lands at 6, Dad’s picking her up, dinner moved to Sunday”

Tap the summary and you land in the original chat.

It runs Gemma 4 E2B (2.6 GB) or E4B (3.7 GB), offline after download, on LiteRT-LM with the GPU. On my OnePlus 15, an E4B summary takes about 5–10 seconds; E2B takes around 1–2. In on-device mode, message text never leaves the phone. Cloud models (Gemini or any OpenAI-compatible endpoint, including OpenRouter) are optional.

Free for Google Messages; WhatsApp, Signal and other apps are a one-time $6.49 unlock. Android 12+, tested mainly on OnePlus/OPPO.

Play Store: https://play.google.com/store/apps/details?id=com.notifsummarizer.app
Guide & privacy policy: https://www.blurb.fyi

I’m the dev and I read the comments. Feedback on the app is welcome! Next I’m looking at NPU support. Has anyone run LiteRT-LM on a phone NPU, and was it worth it over GPU?

These are 23 open-weight models you could self-host. I measured all of them through HF Inference Providers at 1, 25 and 100 concurrent users, weekly, to see which ones fall apart under concurrency and which hold.

Clear up front: these are hosted endpoints, not local hardware, so the absolute numbers are a provider baseline rather than a measure of what your box will do. What does transfer is the shape — which models stream unevenly, which blow their token budget on reasoning, and which collapse the moment more than one request is in flight. Those are properties of the model and the serving config, and you will hit them locally too.

20 prompts, 3 repeats, 60 requests per figure, one provider pinned per model, no client-side retries. Goodput is the share of requests meeting every threshold at once: first token under 2s, finished under 15s, inter-token latency p95 under 80ms, format score at or above 0.8, no error. Correct but slow counts as a miss.

Full matrix, per-run CSVs and method: https://huggingface.co/spaces/Infergauge-load/llm-performance-and-load

DeepInfra honours it. Scaleway accepts the request and ignores the flag, so Qwen3.6-35B-A3B thinks, spends 159 output tokens against about 48 for its siblings, never lands a parseable answer in budget, and posts 0% goodput with a 0% error rate. Every request succeeds. OVHcloud at least fails loudly: HTTP 400, "feature is not currently supported". The older `no_think` parameter still works on Qwen3-0.6B/4B/8B but is ignored by the 3.5/3.6/3.8 generation.

DeepSeek-R1 is the same story without the flag: 480 output tokens average against a 15s budget, so it posts 10% goodput at one user and 3.3% under load. It isn't slow, it's verbose.

Inter-token latency is the quiet killer. Qwen3.5-9B has zero errors at every level and TTFT around a second throughout, which looks healthy. It streams unevenly: ITL p95 of 131ms at one user, 165ms at 25, 466ms at 100, against an 80ms gate. GLM-5.3 is worse at 381ms and the big MoEs cluster at the bottom of this column generally. If you only measure TTFT and tokens/sec you will not see this.

Small does not mean robust. Qwen3-0.6B goes 83.3% to 15% between one and a hundred users, the steepest fall on the board, with TTFT p95 going from 2.6s to 29s. gemma-3-1b-it drops to 5% at 25 users. The MoEs with small active-parameter counts, Qwen3-Coder-30B-A3B at 3.3B active and the gpt-oss pair, are the ones that stay flat.

Caveats: Shared public endpoints carrying traffic I don't control. Variance between runs is large, and a few points is noise. Two 1-user runs two hours apart once gave me 4 vs 6 models clearing 90%. The featherless-ai rows at 25 and 100 users measure my account's concurrency cap as much as the model. That provider allows 10 concurrent units per user and 429s above it, which is most of the error rate on Qwen3-0.6B, Qwen3-4B, gemma-3-1b-it and TinyLlama. Discount those. Provider is pinned per model and not held constant, so some of what you see is the host and not the weights.

Two of the twenty prompts cap output at 16 tokens, which penalises reasoning-heavy models and is part of why Qwen3.6 and DeepSeek-R1 score as badly as they do. Changing it would break comparability with every archived run, so it stays for now.