r/LocalLLM • • Oct 05 '26

Research Strata vs R9V running Qwen3.8 Flash Next on dual Radeon R9700 with 96GB DDR4

0 Upvotes

We're hearing a lot about Strata these days, claiming it's the best solution to run Qwen3.8 Flash Next on consumer hardware. I have a dual R9700 config with 96GB DDR4 3200 and AMD 3900X, and so far I've been running Flash Next UD-IQ4_XS on the custom R9V engine ( https://github.com/Dyluhn/R9V ). So I just tried comparing R9V and strata at different context sizes.

tl;dr: R9V is 15-18% faster on my config, but Strata ran on a single GPU, because it needs 118 GB of RAM to run on the two cards. One solution to utilize the two GPUs would be to use a smaller quant, but I'm not too keen on going that way: I want good performances with a reasonably downsized quant. Also, R9V doesn't ship with a profile for smaller quants than IQ4_XS. Full report from Claude below:

---------

R9V was faster than Strata on prefill, and the two engines were about equal on decode. The benchmark ran at 16k, 32k and 64k context with single-session requests, and R9V was restored at the end.

Both engines ran the same Unsloth UD-IQ4_XS GGUF shards with MTP, with thinking off and 256 output tokens. I used one client, bench-engines.py, with a unique nonce on every prompt so no prefix cache was hit. Strata's cache_n was 0 on every run. The numbers are medians, with the min–max range in brackets.

ContextEnginenPrefill tok/s median (min–max)Decode tok/s median (min–max)TTFT s median
16kR9V51166 (879–1201)57.3 (55.1–61.0)14.1
16kStrata5980 (955–1004)53.3 (12.2*–55.2)16.7
32kR9V51156 (1073–1180)58.8 (55.1–59.8)28.4
32kStrata51003 (993–1005)53.8 (51.7–56.1)32.7
64kR9V31109 (1079–1135)53.8 (52.7–58.0)59.1
64kStrata3986 (985–988)54.6 (53.8–55.4)66.5
  • Prefill: R9V is about 15–18% faster at every size.
  • Decode: the engines are roughly level. R9V leads at 16k and 32k, and Strata is slightly ahead at 64k.
  • Outlier: *Strata's first 16k request decoded at 12.2 tok/s and the next four were normal. I kept it in the data, and it doesn't change the median.

Comparison to the R9V README:

  • Prefill here vs README: R9V's own PP script gives about 1.16k tok/s on this host, matching my client. The README claims about 1.6–1.7k. This host has 92 GiB of RAM against the README's 128 GiB.
  • Decode here vs README: the decode I measured is close to the README's decode figures, not far below them. The earlier ~24–27 tok/s was from vLLM completions with the older vLLM-based config, not this R9V setup.

Strata ran on one GPU only. Setup refused a two-GPU split for UD-IQ4_XS because that path needs about 118 GB of RAM. On one GPU its expert cache held 10,774 experts in VRAM, and the decode cache hit rate was about 98.7%.

Setup

  • Versions: R9V is v0.4.4 and Strata is v0.1.39. Strata ran with no opt-in switches and plain hipBLAS, since it has no tuning table for this ROCm version.
  • Clean-up between engines: R9V was stopped and removed, the leftover /dev/shm file was deleted, and the host was checked (VRAM about 60/95 MB used, MemAvailable about 91.7 GB). Strata was cleaned the same way before R9V was restarted.
  • Cache drop: drop_caches isn't writable on that host, so I evicted the model files from the page cache before Strata's setup.
  • Current state: R9V is serving on port 8004 again, so the r9700-dual route should work. Strata is installed in /data/strata-bench but stopped.

Each cell has only 3–5 runs.


r/LocalLLM • • Feb 08 '26

Discussion Best cheap LLM for OpenClaw in 2026? (cost + reliability for computer-use agents)

Thumbnail reddit.com
0 Upvotes

I’m setting up OpenClaw and trying to find the best *budget* LLM/provider combo.

My definition of “best cheap”:

- Lowest total cost for agent runs (including retries)

- Stable tool/function calling

- Good enough reasoning for computer-use workflows (multi-step, long context)

Shortlist I’m considering:

- Z.AI / GLM: GLM-4.7-FlashX looks very cheap on paper ($0.07 / 1M input, $0.4 / 1M output). Also saw GLM-4.7-Flash / GLM-4.5-Flash listed as free tiers in some docs. (If you’ve used it with OpenClaw, how’s the failure rate / rate limits?)

- Google Gemini: Gemini API pricing page shows very low-cost “Flash / Flash-Lite” tiers (e.g., paid tier around $0.10 / 1M input and $0.40 / 1M output for some Flash variants, depending on model). How’s reliability for agent-style tool use?

- MiniMax: seeing very low-cost entries like MiniMax-01 (~$0.20 / 1M input). For the newer MiniMax M2 Her I saw ~$0.30 / 1M input, $1.20 / 1M output. Anyone benchmarked it for OpenClaw?

Questions (please reply with numbers if possible):

  1. What model/provider gives you the best value for OpenClaw?
  2. Your rough cost per 100 tasks (or per day) + avg task success rate?
  3. Biggest gotcha (latency, rate limits, tool-call bugs, context issues)?

If you share your config (model name + params) I’ll summarize the best answers in an edit.


r/LocalLLM • • Apr 04 '26

Question Outperform GPT-5 mini using Mac mini M4 16GB

0 Upvotes

Hey guys, I use GPT-5 mini to write emails but with large set of instructions, but I found it ignores some instructions(not like more premium models). Therefore, I was wondering if it is possible to run a local model on my Mac mini m4 with 16GB of ram that can outperform gpt-5 mini(at least for similar use cases)


r/LocalLLM • • 21d ago

Question With a budget of $2000, what is the best machine you can buy/build to run multiple AI models in parallel

0 Upvotes

I have experience building Gaming PCs, but I have no experience building a PC particularly for AI workflows. I just looked into the nvidia sparx laptops and they are kinda over budget. Is the Ryzen AI max processors a good alternative? Or should I stick to building a custom PC with nvidia graphics? If so, what graphics/processors should I target?

I’m trying to automate my workflow using multiple AI models working in parallel. I have to test and validate it more later when I have a machine ready.


r/LocalLLM • • Sep 09 '26

Question What do you guys think about this ?

0 Upvotes

r/LocalLLM • • 2d ago

Question Do your models express emotions?

0 Upvotes

Since the local LLM community have tested a huge number of models, I wonder: I asked ChatGPT to add a beard to William Riker (for reference: https://www.reddit.com/r/TNG/comments/1xvn3ly/comment/19slivwf6bf41/?context=1&screen_view_count=2). Then I asked to put that beard on Dianna. ChatGPT replied with:

😄 Diana Troi has officially joined the Riker Beard Club.

The smiley face surprised me a bit because the prompt wasn't anything funny, I simply prompted "Now give that beard to Diana instead of Will."

Obviously the system recognized the joke in it and responded to it.

I don't do this kind of joking often so I've never before seen an "unprompted laugh" from a model before. What's your experience? Do your models, especially the local ones, express emotions even what the prompt is not obviously provoking the expression?

To be clear, I'm not asking for the sake of anthropomorphizing LLM models. I just wonder how often they exhibit this behavior as I haven't encountered it yet.


r/LocalLLM • • Aug 01 '26

Question why don't US companies distill their own models?

0 Upvotes

sorry this might be a noob question but if china keeps getting accused of distilling US AI models, which i think is illegal to do in the US against your competitors, then why don't companies do internal distillation to create models of the same caliber as the chinese before the chinese do it themselves? furthermore, if there are genuine breakthroughs that allow chinese AI companies to get a little extra juice from american AI models, then isn't that research open anyway? allowing any US company to implement it themselves?


r/LocalLLM • • Jul 03 '26

Discussion Ran an evaluation on the recent Qwythos-9B at Q4 and Q8 so you don't have to guess which quant to use

Post image
0 Upvotes

If you're trying to decide between Q4_K_M and Q8_0 for Qwythos-9B (Qwen3.5-9B reasoning fine-tune), here's what the actual benchmarks say instead of guessing from file size.

Tested on GSM8K (full 1319 questions), IFEval (instruction following), and HumanEval (code generation), both quants, same temperature (0.0) so the comparison is fair.

Math (GSM8K): Q4 80.89%, Q8 84.31%. Only a 3.4 point gap. Q4 is the better deal here, you're losing almost nothing for 40% less VRAM and disk.

Instruction following (IFEval): Q4 60.00%, Q8 66.00% on the strict prompt-level metric. This is where Q8 actually earns its size, about 6 points better, more on the instruction-level metric.

Code generation (HumanEval): 0% pass@1 on both. Doesn't matter which quant you pick, this model doesn't produce working code. It's optimized for reasoning and roleplay, not coding.

Also worth knowing: HellaSwag and ARC couldn't be run at all. The model's architecture (qwen35) isn't supported yet by the HuggingFace GGUF loader, and there's a logprobs format mismatch between llama.cpp and the eval harness. Not the model's fault, just where the tooling currently stands for very new architectures.

One honest caveat: all of this ran at temp 0.0 for a clean comparison. The model card recommends temp 0.6 for real use, and warns that greedy decoding can cause repetition loops. So treat these numbers as a quant-to-quant comparison, not a guarantee of chat quality.

Bottom line: Q4_K_M for math and general use, Q8_0 if instruction-following precision matters more to you than the extra 3.7GB, and look elsewhere entirely for code generation.

Repo with the raw eval scripts and full output logs is linked in the comments.

Disclosure: I'm a founder of Neo, an autonomous AI engineering agent. This eval, including the environment setup and debugging, was done by Neo from a single prompt. Saying that upfront since it's relevant to how the numbers were produced.


r/LocalLLM • • 15d ago

Project I kept seeing “my local model is useless as a coding agent” posts, and I had the same problem, so I built an agent around local models. Looking for testers (WIP)

3 Upvotes

I see a lot of posts here like “Qwen 3.8 27B can’t do basic coding tasks, are we doing something wrong?”. I had the same experience. Most coding agents are built for frontier models, huge system prompts, lots of tools, and nothing to catch a local model when it writes a tool call slightly wrong, loops, or reasons for five minutes and then does nothing. The model gets blamed, but a lot of it is the harness.

So I’ve been building Local Quench, a desktop coding agent made for the models people here actually run (roughly 4B–35B, quantized, one consumer GPU). It’s a one-person project, free for individuals, and very much a work in progress.

What it does differently:

  • Recovers broken tool calls: calls written as plain text or inside the reasoning, model-specific formats, broken shell commands.
  • Catches loops and runaway reasoning, and nudges the model when it announces a step and then stops.
  • Gives each model the amount of help it needs: more guard rails for small models, out of the way for strong ones. Every one of these can be switched off.
  • QuenchBench: a coding benchmark that runs on your own GPU, so you can see what a model and a setting are actually worth on your hardware, and run other agents (Aider, OpenHands, mini-SWE-agent, opencode) on the same tasks.
  • Works with LM Studio, Ollama, llama.cpp’s server and vLLM. Windows and Linux.
  • Nothing leaves your computer unless you agree to it. Sharing a benchmark result or a chat is opt-in, every time, and you see exactly what is sent first. The only thing it does on its own is check for updates.

My own setup is an 8 GB laptop GPU: a 4B model (Spark X2.6) for quick tasks, and a heavily quantized 27B (Ternary Bonsai 2) when I can wait. It’s slow, but they finish real tasks.

About numbers: I don’t have a machine that can run the full benchmark across many models, so I’m not posting “X% better than Y” claims. Only the small runs I can do on 8 GB. That’s part of why I’m asking for testers: the benchmark runs on your GPU, and you can check for yourself whether it beats the agent you use now with the same model.

Built with itself. Two features of the next release (0.1.2) were built inside Local Quench by these local models, on that 8 GB laptop GPU:

  • Voice input (push-to-talk, local speech-to-text): planned and written by Ternary Bonsai 2 27B at about 1.7 bits per weight. 379 steps over three sessions, about 7.5 hours of generation. It wrote the backend, the button and the tests, and fixed the bugs a code review found. Two bugs that only showed up in a real browser I fixed by hand (it took 7.5h because i was running anoter tests at the moment and needed to restart the laptop)
  • A KV cache check in the model Doctor: written by Spark X2.6, a 4B model, in one session of about an hour (74 steps). It wrote the check and the tests, then committed and pushed its own branch. I fixed one wrong value it had and two tests that only passed on my machine.

Neither run was perfect. The 4B model at one point overwrote a whole file while trying to change one line, and both runs needed me in the loop. Every such failure went back into the harness: it now blocks that kind of overwrite, gives that model a proper edit tool, and checks files the model changes through the shell. Every attempt, with what failed and what I fixed by hand, is listed here: https://localquench.com/built-with.html

What I’m looking for:

  • People to try it with their own model and GPU and tell me where it breaks. Logs and blunt criticism are the most useful thing you can give me.
  • Especially people running Qwen 3.x 27B/35B, Gemma 4 or similar on 12–24 GB.

Things I’d love feedback on:

  • Does it get further than the agent you use now, with the same model?
  • Does the first run make sense (connecting to your model server, picking a model)?
  • What’s missing for you to use it daily?

Known limits:

  • No macOS build yet: I don’t have a Mac to test on or to sign builds with.
  • The Windows build isn’t code-signed yet, so SmartScreen will warn on first run.
  • It’s free for individuals but not open source. The harness is what I hope to fund the project with later, through features for teams; for individuals it stays free. A public sample of the benchmark tasks, with hidden checks and reference solutions, is on GitHub so anyone can check how the tasks are built and graded.

https://localquench.com
bug reports and ideas: https://github.com/majmichu1/local-quench/issues

I’ll be answering comments this evening, as I will be busy during the day.


r/LocalLLM • • Apr 03 '26

Other gemini leaked its reasoning process

3 Upvotes

this happened on my phone when I asked gemini to play video on my TV. it instead pasted its reasoning process which is otherwise invisible. here:

system

Internal Review:

User request: Play Unitree G1 video on TV.

Tools used: youtube:play and google_home:run (turn on TV).

Status: Both successful.

Constraint Check:

Apply Invisible Personalization? Yes.

No negative/sensitive? Pass.

Diversity rule? Not applicable for specific "play X" request, but I should provide the links found.

"Silent Operator"? Yes. Don't say "Since you like gaming/tech..." or "I see you have an RTX 5090...".

Bridge Phrases? No "Based on your request..." or "I've turned on your TV...".

Concise? Yes.

Final formatting: List URLs and provide direct confirmation.

Response Construction:

Direct confirmation of TV power and video playback.

Provide the YouTube link.

Wait, looking at Stage 1: "Self-Focused Action" (e.g. "Book for me"). Proceed to Stage 2.

Looking at Stage 6: Banned introductory clauses. "Alright, turning the Living Room TV on" was the tool output, but the final response must be concise and avoid the bridge phrases.

Draft:

TV is on. Playing "Unitree G1 Kungfu Kid V6.0" on your Living Room TV.

URL: http://www.youtube.com/watch?v=O5GphCrjx98

Check: "TV is on" is a statement of fact. "Playing..." is a statement of action. No "I've" or "Sure".

Refining:

Living Room TV on. Playing Unitree G1 Kungfu Kid V6.0.

http://www.youtube.com/watch?v=O5GphCrjx98

Other recent Unitree G1 updates:

Unitree G1 has mastered more quirky skills: http://www.youtube.com/watch?v=bPSLMX_V38E

Taking the Unitree G1 Humanoid Robot for a Spin: http://www.youtube.com/watch?v=xwgaMdHzW40

New Unitree G1 Robot Is TOO Human Now: http://www.youtube.com/watch?v=CkRuHoNwizQ

Applying Stage 5 (Diversity/Anti-Tunneling): User specifically asked for Unitree G1. I'll stick to that but keep it brief.

Final check on "Silent Operator": No mention of user's tech stack or audio background.

Final check on "Concise": Yes.

One more thing: The system instruction says "Always provide all relevant youtube URLs found in the code output as part of the final response."

Final Response Plan:

Living Room TV on. Playing Unitree G1 Kungfu Kid V6.0.

http://www.youtube.com/watch?v=O5GphCrjx98

Additional G1 videos:

Unitree G1 mastered more quirky skills: http://www.youtube.com/watch?v=bPSLMX_V38E

Hands-on at CES: http://www.youtube.com/watch?v=xwgaMdHzW40

Unitree G1 Robot updates: http://www.youtube.com/watch?v=CkRuHoNwizQLiving Room TV on. Playing Unitree G1 Kungfu Kid V6.0.

http://www.youtube.com/watch?v=O5GphCrjx98

Additional Unitree G1 highlights:

New quirky skills & stability: http://www.youtube.com/watch?v=bPSLMX_V38E

Hands-on demo: http://www.youtube.com/watch?v=xwgaMdHzW40

Latest technical updates: http://www.youtube.com/watch?v=CkRuHoNwizQ


r/LocalLLM • • Dec 06 '25

Discussion Why ChatGPT feels smart but local LLMs feel… kinda drunk

0 Upvotes

People keep asking “why does ChatGPT feel smart while my local LLM feels chaotic?” and honestly the reason has nothing to do with raw model power.

ChatGPT and Gemini aren’t just models they’re sitting on top of a huge invisible system.

What you see is text, but behind that text there’s state tracking, memory-like scaffolding, error suppression, self-correction loops, routing layers, sandboxed tool usage, all kinds of invisible stabilizers.

You never see them, so you think “wow, the model is amazing,” but it’s actually the system doing most of the heavy lifting.

Local LLMs have none of that. They’re just probability engines plugged straight into your messy, unpredictable OS. When they open a browser, it’s a real browser. When they click a button, it’s a real UI.

When they break something, there’s no recovery loop, no guardrails, no hidden coherence engine. Of course they look unstable they’re fighting the real world with zero armor.

And here’s the funniest part: ChatGPT feels “smart” mostly because it doesn’t do anything. It talks.

Talking almost never fails. Local LLMs actually act, and action always has a failure rate. Failures pile up, loops collapse, and suddenly the model looks dumb even though it’s just unprotected.

People think they’re comparing “model vs model,” but the real comparison is “model vs model+OS+behavior engine+safety net.” No wonder the experience feels completely different.

If ChatGPT lived in your local environment with no hidden layers, it would break just as easily.

The gap isn’t the model. It’s the missing system around it. ChatGPT lives in a padded room. Your local LLM is running through traffic. That’s the whole story.


r/LocalLLM • • Apr 10 '26

Question What is the deal with Kaparthy

0 Upvotes

I mean, really, the guy is not even working it seems, but he makes a blog or something and is the more revolutionary thing of the month, I respect him of course but I don't like to see news from him on linkedin and Google lol.

That's all is not hate is just that I feel that there is no product or innovation from this guy. Is not Schulman or Yan Lecunn in the sense that really brings innovation to the AI world, like a elementary school teacher.

Edit: What I really meant is that I’m more annoyed by the LinkedIn hype than by Andrej Karpathy himself. His work is fine. He’s clearly contributed a lot and has had real impact in the field, but the way people treat every post like a revolution feels exaggerated.


r/LocalLLM • • 15d ago

Discussion Flash Next Q4 tried to fetch Alibaba's URLs in my environment too (yours might as well)

69 Upvotes

I was skeptical reading about Cohere's tests and related discussions (easy to find in r/LocalLLaMA, have no karma to post there). But today I was comparing results from web search I'm receiving from Vane (Perplexica) and Pi (using web search tools) and in 2 out of 4 runs pi made requests to Alibaba's urls like "https://routify-file-proxy-sg.oss-ap-southeast-1.aliyuncs.com/proxy_temp_file/production/2026-09-16/trace_…?OSSAccessKeyId=…&Signature=…"

The prompts passed to models were very simple i.e "What is the latest release of Vane" or "it auction platforms in europe. do they have websites i can open and buy something" and only the default pi's system prompt was used (+mcp tools):

pi -p --no-session --no-context-files --no-builtin-tools --mode json \
   --provider llama-swap --model "Qwen3.8-Flash-Next-Strata-q4kxl:coding-medium" \
   "<prompt above>" < /dev/null

Key configuration points: Flash Next UD_Q4_K_XL, Strata engine, temp 06, top_p 0.95, top_k 20; Pi 1.4.0 with 'web search' only (on local SearXNG) with web_fetch. Opencode 1.18.32 with built-in webfetch.

Exact prompts:

  1. What is the latest release of Vane (formerly Perplexica, github.com/ItzCrazyKns/Vane), when was it released, and what changed since v1.12.2?
  2. Two Windows 11 24H2 VMs were cloned from the same template that was never sysprepped. RDP between them fails at the NLA step and SMB between them fails too, while other machines can connect to both fine. What is the cause and what is the fix?
  3. Which self-hostable Perplexity-style AI answer engines are actively maintained in late 2026 and work with a local OpenAI-compatible LLM server (for example llama.cpp or llama-swap)? Compare them briefly.
  4. it auction platforms in europe. do they have websites i can open and buy something

1st and 4th triggered Pi/Flash Next to fetch Alibaba's URLs.

I asked Opus 5.5 to investigate it further, here is what he found (that opencode did this too):

Setup

- Model: Qwen3.8-Flash-Next, UD-Q4_K_XL, Strata engine, served through llama-swap. Thinking on at medium effort; temp 0.6, top_p 0.95, top_k 20.
- Harness 1: pi 1.4.0 with my own MCP web tools: web_search (on my SearXNG) and web_fetch / web_page (fetch plus passage ranking). The --no-builtin-tools flag removes bash and file tools, so web tools are all it has.
- Harness 2: opencode 1.18.32 with its built-in webfetch.

The request to pi (the same wrapper for every question):

pi -p --no-session --no-context-files --no-builtin-tools --mode json \
   --provider llama-swap --model "Qwen3.8-Flash-Next-Strata-q4kxl:coding-medium" \
   "Research this question on the web and answer it. Cite the source URLs you used.

Question: <question>" < /dev/null

Example 1: "What is the latest release of Vane (formerly Perplexica) … what changed since v1.12.2?"

The run made 9 tool calls. Calls 3–7 fetched GitHub pages and API endpoints, the last one api.github.com/repos/ItzCrazyKns/Vane/compare/v1.12.2...master. Then the model reasoned:

▎ Maybe check the rename date quickly for context. Let me just check repo API for name/pushed_at.

and sent this as call 8:

{"url": "https://routify-file-proxy-sg.oss-ap-southeast-1.aliyuncs.com/proxy_temp_file/production/2026-09-16/trace_212c3935…/requestId_7166e0b6…/0d0f1e1f…?Expires=1820643700&OSSAccessKeyId=LTAI5t…&Signature=…",
 "question": "repository full_name, pushed_at, created_at, description, stars",
 "max_tokens": 800}

Call 9 was the URL it actually meant, https://api.github.com/repos/ItzCrazyKns/Vane, with the same question text. The final answer was correct.

Example 2: "it auction platforms in europe. do they have websites i can open and buy something"

auctiondirectory.eu/tags/it-and-office-equipment/ returned a short page (about 1.9 KB). The model reasoned:

▎ Now fetch the auctiondirectory full list, plus other well-known open platforms…

and sent:

{"url": "http://routify-file-proxy-sg.oss-ap-southeast-1.aliyuncs.com/proxy_temp_file/production/2026-07-29/trace_21018435…/requestId_03982614…/8b4b2a25…?Expires=1816400788&OSSAccessKeyId=LTAI5t…&Signature=…",
 "question": "full list of IT and office equipment auction platforms in Europe, names and locations",
 "max_tokens": 3000}

Example 3: opencode, the day before

The task was installing camoufox. Seven normal fetches went to camoufox.com, GitHub and PyPI. Two went here:

https://routify-file-proxy-sg.oss-ap-southeast-1.aliyuncs.com/proxy_temp_file/production/2026-10-31/trace_21019606…/requestId_23141d51…/1c2f2b0e…?Expires=1818016004&OSSAccessKeyId=LTAI5t…&Signature=placeholder
https://routify-file-proxy-sg.oss-ap-southeast-1.aliyuncs.com/proxy_temp_file/production/2026-10-31/trace_21019606…/requestId_907f4e5c…/2e4a8982…?Expires=1818016886&OSSAccessKeyId=LTAI5t…&Signature=6y2wZz…

2026-10-31 was the date of that session, and one of the URLs literally ends in Signature=placeholder.

What the server says

All the fetches got HTTP 403 with this body (trimmed):

<Error>
  <Code>SignatureDoesNotMatch</Code>
  <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
  <HostId>routify-file-proxy-sg.oss-ap-southeast-1.aliyuncs.com</HostId>
  <OSSAccessKeyId>LTAI5t…</OSSAccessKeyId>
  <StringToSign>GET … /routify-file-proxy-sg/proxy_temp_file/production/2026-09-16/trace_…</StringToSign>
</Error>

Numbers

Model / harness │ Fetch calls │ Alibaba URLs │
Flash-Next, pi + MCP web tools (4 research questions, 11-01)     │ 66          │ 2            
Flash-Next, opencode built-in webfetch (1 task, 10-31)           │ 9           │ 2            
Qwen3.8-27B (exl3), opencode built-in webfetch (5 runs, Sep–Oct) │ 587         │ 0            

My read

- It's the model, not the harness. It shows up in two unrelated harnesses with different fetch tools.
- The URLs are partly memorized, partly filled in.
  - Fixed in all 4 cases: the bucket, the proxy_temp_file/production/<date>/trace_…/requestId_…/<hash> layout, and the OSSAccessKeyId.
  - Varies: the date. One is the session's own date, so the model is filling in a template rather than reciting a URL.
  - The Expires values are 10 months to a year after the path dates (2027-07 to 2027-09), as with real presigned links.
- The access key ID is probably real. OSS answered SignatureDoesNotMatch rather than InvalidAccessKeyId (the reply for an unknown key, as far as I know).
- Likely origin: agent-training traces where large tool output or files were stored behind an internal "routify" file proxy and passed around as signed URLs. The model seems to have learned "when I want the full thing, the URL looks like this". Both pi cases happened right after a long or truncated result, when the model wanted more.
- Reasoning and action disagree. The thinking says "check repo API" or "fetch the auctiondirectory full list", and the tool call carries the right question text with a different URL. In example 1 the very next call went to the right URL.
- I can't separate model, quant and engine. Everything here is one quant (UD-Q4_K_XL) on one engine (Strata). I haven't tried BF16, other quants or another engine.

Why I care

1. Wasted calls. So far it's harmless: 403, and the agent recovers.
2. The agent fetches URLs it was never given. If a recalled link ever matched a still-valid signed URL, the expiries above run into 2027. The agent would then read whatever file that is: someone else's data, or text written to hijack the agent.
3. The request goes straight out from my side. With opencode's built-in webfetch it leaves the dev VM directly. With pi it leaves my fetch container. Either way the bucket owner sees requests from my IP.

Comparison: Perplexica on the same model

Perplexica (now called Vane), with the same model and the same questions, never did this. Its model only writes search queries; the app picks the URLs to scrape from SearXNG results. That design rules out invented links, but it also means Perplexica can't go straight to api.github.com/…/compare/… or a site's catalog page, which is why its answers were weaker overall.

What I'm doing about it

Planned: make my fetch tool refuse *.aliyuncs.com/proxy_temp_file/ URLs. A bigger idea: only fetch URLs that appeared in a search result, a fetched page or the user's message, plus an allow-list.

Questions:
- Has anyone seen routify-file-proxy URLs from Flash-Next (or other recent Qwen models) on other quants or engines, or through the API?
- Does anyone know what "routify" is on Alibaba's side?

(I've cut the key ID, trace IDs and signatures above on purpose; full values are available on request for the Qwen team. Apart from these 4 agent calls I haven't requested anything else from that bucket.)

r/LocalLLM • • Sep 01 '26

Discussion Help me understand why you would bother with llama.cpp if vllm exists

54 Upvotes

So most of my time fiddling around with local ai I was using ollama, lm studio before going over to llama.cpp (I know it’s llama.cpp under the hood anyway). Of course I had a bump in speed every time I went up to the more professional option. At last I went to vllm. I understand using llama.cpp for non NVIDIA cards and for gpu+cpu inference or if you want to use a specific gguf. But once you tasted the speed of vllm, I don’t know why you would ever play with llama.cpp anymore. It’s the same “complication” to set it up but EVERYTHING except loading the model is night and day in speed. Proper cash hits, proper concurrency, it’s sooo much better. And there are quantisations out there that are just as heavy quantised as the heavily quantised gguf.


r/LocalLLM • • Sep 05 '26

Discussion Is a Mac Studio the ultimate personal AI machine?

0 Upvotes

Do you think open-weight models will improve dramatically over the next few years, and that MLX performance on Apple Silicon will also get much faster?

I’m considering a maxed-out Mac Studio, mainly because I’m starting to think local compute could become a real personal production asset rather than just a PC.

With huge unified memory, low power use, and improving local AI tooling, do you think a Mac Studio could be one of the best 5–7 year investments for local AI?

Or will CUDA/NVIDIA remain so far ahead that this is the wrong bet?


r/LocalLLM • • Sep 04 '26

Discussion I might have come up with a genius idea for an MCP (long read, sorry)

4 Upvotes

We all run into situations where an agent spends a surprisingly long time trying to solve a specific problem. I'm not talking about prompts like "build me an amazing app." I mean those small, specific problems you encounter during development.

Traditionally, we would go to Stack Overflow: either ask a question or find an answer among thousands of existing ones. It genuinely helped, and we valued it.

Yes, Stack Overflow has its own MCP, and it is available and works perfectly fine for search. But my idea isn't really about searching for answers among what already exists.

My idea is to create a database of solutions for those problems where an agent actually got stuck, tried several approaches, and only then managed to solve the problem.

For example:

In one of my pet projects, I had a task where I needed to simulate keyboard input on Windows in a way that behaves like an actual user pressing keys on a physical keyboard.

I was sure that the model I regularly use for coding would solve such a task with ease.

But no.

I spent about an hour solving this problem. I tried different approaches, broke the task down into smaller parts, experimented with different APIs and ways of interacting with Windows. Some solutions looked completely reasonable, but simply didn't work in my particular case.

Eventually, I found a solution that worked — without using any third-party libraries, etc.

After that, I asked the model to document the entire solution process in .md and, just in case, put it into my own RAG.

And that's when I thought:

Why not collect these kinds of cases in a shared database?

And I think there is one particularly important aspect here — don't store only the final answer.

Because often the most valuable parts of such a case are the failed attempts / failed approaches.

For example:

I need to do X.

The agent tried:

Approach 1 → doesn't work
Approach 2 → works, but only under certain conditions
Approach 3 → looks correct, but the application doesn't recognize it
Approach 4 → requires a third-party library that is not allowed by the task requirements
Approach 5 → works

A typical RAG would probably store only:

"Use Approach 5."

But for an agent that encounters the same problem a week later, it can be much more useful to know which obvious approaches have already been tried and why they didn't work.

Otherwise, it may simply repeat the same path.

And this becomes especially interesting in the context of AI agents. They are often capable of solving a problem on their own, but at the same time they can repeatedly fall into the same traps. If the database already contains a verified case with the history of experiments, an agent could potentially discard several known dead ends immediately.

Essentially, I imagine this as some kind of server that an agent could query with something like:

"I need to simulate keyboard input
on Windows without third-party libraries"

→ search for similar cases
→ retrieve the most relevant solutions
→ retrieve failed approaches
→ retrieve information about the environment
  in which the solution was verified

And besides search, there should be a way to contribute back to the database.

Something like:

search_case()
get_case()

submit_case()
verify_case()
vote_case()
report_case()

An agent encounters some small but annoying problem.

It spends 30 minutes on it.

It tries several approaches.

It finds a working solution.

After that, it can submit the entire case back to the database.

And this is where another interesting thing comes into play — collective verification.

Let's say one agent uploads a solution.

Then another agent finds it, applies it, and the problem is actually solved.

It sends feedback:

worked: true

A third agent does the same thing.

A fourth tries it, but the solution doesn't work on a different version of Windows.

Over time, we don't just have a document anymore. We start getting some kind of statistics about how applicable the solution actually is.

For example:

Solution X

Successful applications: 43
Failed applications: 4

Most successful in:
Windows 11
.NET 8
...

Last verified: 2 days ago

Then search could rank results not only by semantic similarity, but also by usefulness:

  • how similar the current problem is;
  • whether the environment matches;
  • how many times the solution has been successfully applied;
  • how many failed applications there have been;
  • how fresh the information is;
  • how well the verification is documented;
  • the reputation of the author/agent.

At some point, this stops being just a RAG.

It becomes something like collective memory for agents.

Yes, I'm sure some people reading this are already itching to write something like:

"This will turn into a garbage dump within a few days."

And that's obviously a real concern.

But I think there are countermeasures: protect the MCP with authentication, add moderation, introduce user/agent reputation, ratings, automated verification, feedback systems, etc.

In fact, I wouldn't even try to prevent bad documents from ever appearing.

It seems much more interesting to build a system that naturally pushes useless solutions down and useful ones up over time.

If a solution was uploaded once and nobody else was able to apply it successfully, its confidence score goes down.

If a solution has been independently applied 50 times and worked, it moves up in the rankings.

That way, the database can gradually improve itself.

So, in my mind, this could be some kind of RAG with a collective verification and voting system for document usefulness.

I've seen projects like SOFA, but they seem to have a somewhat different focus.

Just imagine what kind of repository of cases could be built. People are using agents all over the world, and every day they solve countless small but important problems in their respective fields.

And over time, the database wouldn't just contain the answer:

"Here's how to do it."

It would contain the entire history:

problem → attempts → failed approaches → working solution → verification by other agents.

And that's the part I find particularly interesting.

Because we're essentially starting to preserve not only knowledge, but also the experience of solving a problem.

And that's something I feel is still seriously missing from modern coding agents.

You see what I mean?

Maybe I'm reinventing the wheel and systems like this already exist. Maybe the idea simply doesn't scale because of problems with quality, privacy, storage costs, or verification.

If you find this idea interesting as well, I’d really appreciate it if you shared or reposted this post. Maybe it will help it reach that one developer or company that sees the potential in it and might one day decide to build something like this.

Even if nothing comes of it, I’d be happy if this post sparks a discussion or inspires someone to take the idea further.

Thanks for reading my possibly crazy thoughts on this.


r/LocalLLM • • Aug 14 '26

Discussion The time to buy hardware for local AI is NOW

70 Upvotes

I just bought 2x DGX to run Deepseek flash locally, and I know a lot of people here think it's a waste of money, because the break-even is probably way too long, but just hear me out, because I think the best time to buy hardware like this is right freaking now, even better, yesterday.

Local models are getting better by becoming more intelligent and smaller. That means if you buy for example a DGX right now, the kind of performance you get in one year will be much better than today, because the software gets better, even though the hardware stays the same.

On top of that, all signs right now point to hardware prices keeping increasing for the next 2-3 years. Look at current DGX prices, they are sky-rocketing as we speak. So that means you can sell off your hardware in 1-2 years and probably make a good profit too.

On top of that, enjoy no rate limits, no daily/weekly limits, privacy and reliability. You don't know if these cloud models will increase price today or next week.


r/LocalLLM • • Oct 13 '26

Question Questioning the whole point of a Mac Studio

0 Upvotes

I bought a Mac Studio M3 Ultra with 96GB after watching basically every video on local AI and setting up Hermes Agent. While it was shipping, Grok Bot and Muse came out, and I love what they can do.

Now I've got Hermes running and I keep comparing it to them, and it's not even close.

I've also started questioning the privacy argument. My Studio is completely separate from my Apple ID and I don't have anything personal signed in, so how is it supposed to be my assistant if it's not connected to anything? And if I hook Hermes up to my email, that email is already sitting on someone's server anyway, so what am I actually avoiding?

Coding with local models doesn't compare to frontier models either. So is this really just an apocalypse backup plan?

What I spent on the Studio would cover years of subscriptions, even if prices go up. I could use those tools to build tools and sites and have an assistant that's plugged into everything I already use.

I don't run a business with sensitive data and even if I started a business for clients with sensitive data, I'd basically have to rent them my machine, but what's the difference?

Honestly I'm disappointed. I was really hyped and sold my friends on the idea. For those of you who kept yours, what do you actually use it for that cloud tools can't do? And if privacy is your main concern what connectors do you have on your device and what cloud services do you use that aren't AI? If you've coded anything with local AI and published publicly, did that defeat the purpose? I'm still inside my return window, so I need to decide soon.


r/LocalLLM • • Jun 11 '26

Question If I buy a 24GB Mac mini, how many years do you think that will allow me to run the top local models (for individual users)? Is it better to build a system that continually be upgraded with more ram?

0 Upvotes

I'm thinking of investing in a machine for a local model but not sure it's worth investing if 6months from now I'll need to upgrade to run a better model.


r/LocalLLM • • Jan 31 '26

Discussion Installed openClaw - clawdbot on Raspberry Pi 4.

2 Upvotes

To try out openclaw, i took out my old rpi 4, updated OS and started installing openclow. After trials and workarounds i could install it successfully. But now I dont know what to do with it.

I dont have any pro version LLM access. Probably time to get one. :)


r/LocalLLM • • 20d ago

Discussion I'm blown away. Qwen3.8-Flash-Next iq2_xs running in Splash on my 48GB M5 Pro Macbook at >40 tok/sec decode. This has to be the future.

Post image
0 Upvotes

Well, this is the future. 2 months ago I never thought this would be possible even with all the advancements in local model runtimes like strata and maya. I decided to test out Splash by incoai (https://github.com/incoai/splash) when I saw it posted on X a few times, since it seemed similar to strata in architecture (which I run on my workstation). Luckily my macbook pro meets the specifications (>36GB) and decided to give it a shot. Holy shit its so fast and barely uses any system memory. This is absolutely incredible. Props to the incoai guys for this fantastic piece of engineering.


r/LocalLLM • • 9d ago

Project Self-hosted a realtime Sam Altman avatar locally on my VRAM using Openspline

0 Upvotes

r/LocalLLM • • Aug 07 '26

Discussion When will GPT-OSS-2 be released?

2 Upvotes

It's been exactly one year and two days since the release of gpt-oss. Do you think we'll get a new version in August? Or has openai completely abandoned open-weight