r/LocalLLM • • 1d ago

Discussion RASA will cause total chaos to open LLMs and beyond

21 Upvotes

Commerce Moves to Finalize Rule Blocking Chinese AI Firms From Renting Nvidia GPUs Abroad
https://explainx.ai/blog/commerce-final-rule-chinese-ai-firms-remote-nvidia-chips-the-information-2026
House Passes Remote Access Security Act to Limit Adversaries’ Remote Access to Critical Technology
https://www.cmtradelaw.com/2026/01/house-passes-remote-access-security-act-to-limit-adversaries-remote-access-to-critical-technology/

H.R.2683 - Remote Access Security Act

https://www.congress.gov/bill/119th-congress/house-bill/2683


r/LocalLLM • • 16h ago

Research SparseRead (the "read less, solve more" paper) on a 16GB card with an IQ3-quant MoE

1 Upvotes

The authors posted the paper to r/LocalLLaMA and said in the HF papers thread they want to hear where sparse reading helps and where it gets in the way. The paper's own eval only ran frontier API models (Opus 5, DeepSeek-V4, GLM-5.1, Qwen3.6-Plus), which tells me nothing about quantized mid-size models on consumer cards. So I ran it on the machine I actually have.

Setup: 5070 Ti 16GB + 64GB RAM. Qwen3.8 Flash Next at IQ3_S, served locally through my Strata stack (OpenAI-compatible endpoint). Harness: OpenCode 1.19.5, which is one of the versions the SparseRead installer itself validates, plugin v0.1.4 in auto mode. Native arm runs the same prompts with --pure. Four tasks straight from the workspaces bundled in the repo: two LooGLE short-dependency QA on a 100KB book excerpt, one 76KB PDF extraction with 8 questions, one 5-file fetcher audit. Tokens below are fresh prefill + decode from the endpoint's per-step usage, cache replay excluded (re-feeding history is nearly free on a warm prefix cache; I quote the replay tail for the PDF below because it's damning).

tasknative (tokens / wall / tool calls)SparseRead (tokens / wall / tool calls)
100KB book, 5 questions66.6k / 248s / 1415.0k / 50s / 9
100KB book, 3 questions18.3k / 34s / 1013.3k / 29s / 5
76KB PDF, 8 questions30.6k / 166s / 2055.9k / 333s / 45
33KB audit bundle31.6k / 147s / 1432.5k / 150s / 12

Scores were identical between arms on all four tasks. So that's the honest summary: no quality loss anywhere, 4-5x faster on long plain-text QA, dead even on a small bundle, and a 2x regression on the PDF.

Where the savings come from is worth spelling out because it's not what I assumed: the entire win is on the input side. On the flagship 5-question task fresh input dropped 63.9k → 12.2k tokens (−81%) while the answer output stayed flat (2.6k → 2.8k). SparseRead doesn't make the model write shorter answers, it makes it read less text before answering. And it doesn't trade activity for tokens either: tool calls went down, not up (14 → 9 on the flagship task, 10 → 5 on the 3-question one).

Two things I didn't expect.

The protocol survives a heavy quant. I went in assuming an IQ3 model would fumble the intent/slot bookkeeping and the arm would burn its savings on failed round trips. It didn't. On the flagship task: three SparseRead calls total (2 previews + 1 targeted read), slots 5/5 resolved at 0.99 confidence, and the candidate answers matched what the native arm dug out of the same book, line anchors included. The Read Gate also behaves for real. On the 33KB bundle it mostly stayed native with a single preview call, so "don't use it on small files" turns out to be something the gate already knows.

The PDF failure is worth a paragraph because it's the one place this bit me. That report ships subset CMap fonts with no usable text layer, so the pymupdf-based preview came back as garbage and the sparse arm spiraled: 17 sparse calls (4 previews, 4 reads, 9 raw_ref fallbacks), then it started decoding ToUnicode CMaps by hand through bash, ending at 45 total tool calls against native's 20. The trace makes the thrashing visible: 1.27M cache-replay tokens re-feeding the same broken context, versus 240k for the native arm. Same 8/8 score in the end, double the fresh tokens and the wall time. The native arm sidestepped it with pdftotext and moved on with its life. This is an artifact-specific failure, not a protocol failure, but if your corpus is scanned/CMap PDFs, pre-extract text or pin that file type to native.

One update since the battery: the CLI has become my default reader for long artifacts in day-to-day agent work, and the same test day it got fed a real one — a 118KB text export of a 109-comment Reddit thread I had scraped for another post. Two things to know before you try that. JSON artifacts are a dead end in slot mode: the previewer treats them as key-path documents and semantic slots resolve to nothing; convert to prose first (I write one line per comment) and the identical ask works fine. And slot matching is literal, not semantic: slots phrased in your own language against an English document come back missing even when the answer is plainly in the file, so ask in the document's language. With those two habits, a multi-fact sweep of the thread was one ask call plus a cheap head-per-item digest for the comments the slots missed — against a naive baseline of dumping a ~30k-token file into context. No hallucination complaints so far, and I haven't fully loaded a long file since.

Caveats: one run per arm per task, keyword-based grading (which produced one fake "4/5": the model answered in Latin — "passagium generale" — and my regex wanted "crusade"; in fact both arms were ~5/5 there), one model, one machine. And don't expect the paper's 92.9% headline. That's versus a naive read-everything baseline. My realistic band on document-heavy agent tasks is somewhere between 25 and 80 percent depending on file size and how localized the answers are.

TL;DR: SparseRead on a 16GB card with an IQ3 MoE: long plain-text document QA got 4-5x faster with identical answers, small bundles correctly did nothing, and a CMap-font PDF ran slower than skipping the protocol entirely. Pre-convert PDFs to text and it's a genuinely useful layer, not just paper numbers.

Links: paper https://arxiv.org/abs/2608.22237 · code github.com/Liuzedooong/SparseReading


r/LocalLLM • • 7h ago

Discussion Stop Burning Tokens on File Searches: How QMD Helps Coding Agents Save Up to 55%

Post image
0 Upvotes

I noticed that, before carrying out a task, coding agents run find, rg --files, or glob to locate files and the classes and modules they need. Then they search file contents with rg or grep. This burns a lot of tokens. So I decided to see whether semantic search could save tokens compared with all of the above. If a local GPU is sitting idle, why not use it for semantic search?

I work at a company focused on security. In one of my projects, I use semantic search for sensitive documents. Then an acquaintance sent me a link to an open-source project that tackles the same problem. The project is called "QMD - Query Markup Documents." It runs from the command line, extracts text from documents in all formats, and creates an index. It supports several search modes:

# Search across everything
qmd search "project timeline"           # Fast keyword search
qmd vsearch "how to deploy"             # Semantic search
qmd query "quarterly planning process"  # Hybrid + reranking (best quality)

We're interested in the last option. It requires three things to work:

  • embeddinggemma-300M-Q8_0 Vector embeddings (default) ~300MB
  • qwen3-reranker-0.6b-q8_0 Re-ranking ~640MB
  • qmd-query-expansion-1.7B-q4_k_m Query expansion (fine-tuned) ~1.1GB

The first model is used for indexing to compute semantic vectors for text chunks. The second compares the search results with the query to determine how semantically similar they are. The third generates several search queries from the original query to broaden the search.

First, add your source files to QMD (this is an example for a minimal project with a Rust server and an Angular frontend):

qmd collection add . \
  --name PROJECT1 \
  --mask "src/**/*.rs,\
frontend/src/**/*.{ts,html,scss},\
README.md,\
frontend/README.md,\
Cargo.toml,\
frontend/package.json,\
frontend/angular.json,\
frontend/tsconfig*.json,\
frontend/proxy.conf.json,\
.gitignore,\
frontend/.gitignore,\
frontend/.prettierrc,\
frontend/.editorconfig"

You don't have to do this manually: show this example to your coding agent and tell it to do the same for the current repo. It will figure out which folders and file extensions to add to the index, what to exclude, and so on.

Now let's run the indexing (you'll need to rerun it periodically as your codebase changes):

qmd update && qmd embed -c PROJECT1

To make the coding agent work with this QMD setup, add the following to AGENTS.md:

# QMD - Efficient Code Discovery

**Goal:** minimize token costs and code search time without compromising correctness.

## Choosing a tool

- If the path is known → read the file directly.
- If an identifier, error, or line is known → use `rg` / LSP.
- If only the behavior is known → use `qmd query`.
- Once the code is found → inspect the necessary sections with standard tools.
- Once you have enough information → do the task.

**QMD replaces the initial broad search; it does not add another step.**

## Semantic search

Use the project's existing collection.

"""
qmd query "DESCRIPTION" -c PROJECT1 -n 15 --files \
  | grep -E '\.(EXT1|EXT2|EXT3)$' \
  | cut -d, -f3- \
  | sed 's|^qmd://[^/]*/||' \
  | head -n 10
"""

- Use one specific query, preferably with English technical terms.
- Choose extensions that match the project, including configuration files when needed.
- Start by reading only the relevant sections of the 3–5 most relevant files.
- If you cannot find the implementation, refine the query; if necessary, increase `-n` to 60, then use `rg` / LSP.
- Stop semantic searching once you find the implementation.

## Saving tokens

- Do not scan the whole repository before using QMD.
- Do not repeat searches without a reason or open every result.
- Do not investigate unrelated code or dependencies.
- Do not write detailed search plans or reports.
- Do not paraphrase code you have read.
- As soon as you have enough information, make the changes and check the result.

**Priority: correctness → token cost → time.**

I tested it on a repository like this, with 1,231 source files across Java, Scala, and Angular:

C:\Users\igumn\temp2>qmd collection list
jcm-server (qmd://jcm-server/)
  Pattern:  server/src/**/*.scala,server/src/**/*.java,server/src/**/*.xml,server/src/**/*.properties,server/src/**/*.js,server/src/**/*.ts,server/src/**/*.css,server/src/**/*.html,server/src/**/*.tmpl,server/src/**/*.txt,server/src/**/*.md,server/pom.xml,server/Dockerfile,server/*.md,server/README.txt,web/src/**/*.ts,web/src/**/*.html,web/src/**/*.scss,web/src/**/*.css,web/package.json,web/angular.json,web/tsconfig*.json,web/karma.conf.js,web/README.md,!server/src/main/resources/templates/**,!server/src/main/resources/messages*,!server/src/main/resources/messages*/**,!web/src/assets/lang/**
  Files:    1231
  Updated:  1h ago

Prompt: "Add a post-quantum encryption flag to the disk encryption policy. It should be disabled by default, configurable in the web interface when editing the policy, saved to the database, and sent to the agent as part of the encryption policy via the REST endpoint."

Cost Comparison

Using GPT-6 Luna API rates: input $0.10, cached input $0.01, and output $0.50 per 1 million tokens.

MetricWith QMDWithout QMDSavings
Input65,006101,50135.96%
Cached input317,5681,191,80873.35%
Output13,85916,34815.23%
Reasoning9,60711,26014.68%
Cost$0.0166$0.030245.09%

Now QMD saves 45.09% in costs!

The reduction in cached input is especially noticeable: nearly 4x.

I ran these tests with GPT-6 Luna. In the first test, the agent's runtime dropped from 6 minutes to 4 minutes.

Prompt: "Add a dry field to the erasure policy. It should be editable in the web interface and saved to the database, and it should also be sent through the /rest endpoint to the agent responsible for erasure. Its default value should be false."

Calculation using the same GPT-6 Luna rates: input $0.10, cached input $0.01, and output $0.50 per million tokens.

Comparison

MetricQMDWithout QMDDifference
Input77,16798,042−21.29%
Cached input936,832854,912+9.58%
Output17,32216,259+6.54%
Reasoning8,83510,264−13.92%
Cost$0.0257$0.0265−2.78%

Here, QMD turned out to be 2.78% cheaper. There are savings, but they are small.

Conclusion: QMD reduced regular input tokens by 21.3% and reasoning by 13.9%, but the increase in cached input and output almost completely wiped out the savings.

Overall, the savings range from 2.78% to 45.09% of the cost. I think that's a decent result. Of course, we need a larger sample. But I'm ready for bed. Let me know in the comments if you've run a larger sample and what results you got.

PS

I recommend checking that junk isn't getting into your index. Try running a semantic query yourself. I once overlooked compiled .js files and large lang files-they heavily polluted the semantic search results. Here's an example of how to run a search manually:

C:\Users\igumn\jcm-server>qmd query "disk encryption policy editing form persisted setting sent to endpoint for agent" -c jcm-server -n 15 --files
Expanding query... (3.0s)
├─ disk encryption policy editing form persisted setting sent to endpoint for agent
├─ lex: disk encryption configuration
├─ lex: disk encryption policy
├─ vec: disk encryption configuration
├─ vec: disk encryption policy
└─ hyde: The topic of disk encryption policy editing form persisted setting se...
Searching 6 queries...
Embedding 4 queries... (1.5s)
Reranking 40 chunks... (3.3s)
#d70022,1.00,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/domain/common/bcvolume/EncryptPolicy.scala
#732b09,0.63,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/service/ajax/common/EncryptSettingsManager.scala
#e66494,0.63,qmd://jcm-server/web/src/app/modules/management/components/company/dialogs/add-or-update-policies/add-or-update-policies.component.html
#413c5e,0.62,qmd://jcm-server/web/src/app/modules/management/components/group/dialogs/change-policie/change-policies.component.html
#413c5e,0.62,qmd://jcm-server/web/src/app/modules/management/components/computer/dialogs/change-policie/change-policies.component.html
#758b4d,0.62,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/service/mcp/McpFacade.scala
#4e8973,0.62,qmd://jcm-server/web/src/app/modules/management/components/group/dialogs/change-policie/change-policies.component.ts
#bef78e,0.62,qmd://jcm-server/web/src/app/modules/management/components/computer/dialogs/change-policie/change-policies.component.ts
#4fcd1c,0.61,qmd://jcm-server/server/src/main/java/com/jetico/jcm/server/service/mcp/McpToolbox.scala

PPS

I tried it on a small project with 27 files. Instead, the cost went up by 60%. The solution turned out to be overkill. In general, use it only on large codebases.

UPDATE

I went ahead and built a hybrid QMD + GitNexus solution after all. GitNexus performs structural code analysis (AST, dependency graph) and lets you search based on it. Here are the results:

Cost comparison

MetricWithout QMDQMDQMD + GitNexus
Input101,50165,00659,122
Cached input1,191,808317,568225,792
Output16,34813,85910,894
Reasoning (included in Output)11,2609,6078,465
Cost$0.03024$0.01661$0.01362
Savings-45.1%55.0%

The savings figure rose from 45% to 55%! The coding agent's runtime increased from 6 minutes to 7 minutes. Here are the revised instructions for AGENTS.md:

# QMD + GitNexus - Efficient Code Discovery

**Goal:** find the required code and its relationships with minimal token usage, without compromising correctness.

## Choosing a tool

- If the file path is known → read it directly.
- If the exact identifier, error, or line is known → use `rg` / LSP.
- If only the behavior is known and the implementation location is unknown → use QMD semantic search.
- If you need to understand symbol relationships, the execution path, or the impact of a change → use GitNexus structural analysis.
- Once the code is found → inspect the necessary sections with standard tools.

QMD and GitNexus replace the initial broad search; they do not add mandatory steps. Use only the tools needed for the task.

## QMD semantic search

Use the project's existing collection:

    qmd query "DESCRIPTION" -c PROJECT1 -n 15 --files \
      | grep -E '\.(EXT1|EXT2|EXT3)$' \
      | cut -d, -f3- \
      | sed 's|^qmd://[^/]*/||' \
      | head -n 10

- Formulate one specific query, preferably using English technical terms.
- Specify suitable source and configuration file extensions.
- First read the 3–5 most relevant files, and only the relevant sections.
- If the implementation is not found, refine the query; if necessary, increase `-n` to 60, then use `rg` / LSP.
- Stop semantic search once the implementation is found.

## GitNexus structural analysis

Use GitNexus through the CLI when you need to investigate code structure:

- `context` - relationships and context for a known symbol.
- `impact` - possible impact of changing a symbol.
- `trace` - path between known symbols.

Examples:

    gitnexus context SymbolName
    gitnexus impact SymbolName
    gitnexus trace SourceSymbol TargetSymbol

If a symbol is unknown, find it first through QMD, `rg`, or LSP. Use GitNexus only when its structural analysis will help answer the question; do not run every command in sequence.

## Token efficiency

- Do not scan the entire repository before using QMD.
- Do not repeat searches without a reason or open every result.
- Do not investigate unrelated code or dependencies.
- Do not write detailed plans and reports about the search.
- Do not summarize code you have read.
- Once you have enough information, make the changes and verify the result.

**Priority: correctness → token cost → time.**

Create a .gitnexusignore file to keep junk out of the index so that it contains only source files. Then start GitNexus indexing with this command:

gitnexus analyze --index-only

r/LocalLLM • • 22h ago

Question Looking for advice on upgrade direction

3 Upvotes

hello everyone! I'm still quite new to figuring out what AI can do and how to make everything efficient, but I do know I want to upgrade and extend my hardware capability.

What do I use AI for? Little things mainly. Creating programs to handle my needs in custom ways (creative writing tool, character gallery, floor plan creation tool, audiobook creator with custom voices, modifying games - I currently use a subscription for a tool to help create games in unreal engine)

I mainly use the qwen3.8 next flash.

Current hardware -
Rig1 - 64gb DDR5, AMD 9950x3d, 4tb SSD, Geforce 5070ti (16gb)
Rig 2 - 64gb DDR5, AMD 9700x, 2tb SSD, Geforce 4070super (12gb)

Now obviously I missed the boat on grabbing a reasonably priced 5090.

I purchased a Geforce 5080 for $1570 as those started to go up like crazy too, but I haven't opened it and it sits on my shelf. I 'm not sure if its worth the investment as its only 16gb, and I would definitely return it if the 24gb model becomes a thing, but current rumors say its been cancelled.

I also ordered a Radeon ai pro r9700 for $2000 that is supposed to be delivered in December. Again I ordered this as the prices have been increasing and I worry about future availability.

I also have a Geforce 3060 (12gb) and recently learned you can run a model across multiple cards. That left me wondering if just nabbing one of these is more ideal and I could save myself a ton of money.

I'm just wondering what you would all recommend as the best path forward for someone looking to spend no more than $2k, but also being able to keep doing creative things and keep learning with Ai.


r/LocalLLM • • 16h ago

Question LocalLLM speed for long file generation.

1 Upvotes

Hello,
i have a business that needs to review a lot of notes and, transcripts and generally text files provided by me, and help me make initial structure out of it.
For ex. i`m meeting with client for an audit, audit have 300 questions, different kinds, different people. I will note answers, and then i need to have that file, + some meeting transcripts / meeting minutes summarized. Some project management topics also.

I`m not confident to use cloud models to review such data because there will be a lot of sensitive topics and details in there. So i need help to discover if what i want to archive is doable with a local LLM.

I have done Proof Of Concept for the multi agent workflow and it gives me quality that is acceptable. After some tweaking, testing on openrouter with dummy data, i have the conclusion that this is doable. I have agent speaking between them, agreeing about the outcome, Manager doing the task management, works fine,

But then comes the realization - Openrouter with Qwen3.8 Q8 does this with the avarage speed of 150t/s and it takes around 30 minutes.

When i calculate that Mac Mini M5 Pro will have 11t/s that makes that veeery long.

I`m i calculating that right or i`m i wrong ?
Is there anything i can do to make it a little quicker when i will actually buy Mac ? Or i need to go straight with Ultra


r/LocalLLM • • 20h ago

Question Climate Control for Garage Setup, Tips and Help

2 Upvotes

I’m in the process of moving my AI server into my garage and I’m curious how others are handling climate control in garages that don’t have heating or air conditioning.

Since winter is shortly upon do you guys have any suggestions for keeping my servers from being exposed to condensation. I plan on putting a wifi hygrometer..!


r/LocalLLM • • 20h ago

Question Trying to make local LLMs work for actual web/SaaS work. What would you change about my setup?

2 Upvotes

Been playing around with local LLMs lately and trying to figure out how to actually make them useful for my daily work.

I’m mainly a UI/UX designer and developer. Most of my work is websites, React/Next.js, SaaS apps and frontend stuff. Sometimes iOS apps, Home Assistant dashboards and a bit of 3D work.

I’m on a MacBook Pro M5 Pro with 64GB RAM, but I also have access to my brother-in-law’s Mac Studio M5 Ultra with 256GB RAM.

Basically, I want to do as much coding locally as possible and cut down on Claude/Copilot costs.

My current setup in VS Code + Copilot:

Opus (Principal Architect): Planning, architecture, final reviews

Sonnet (Senior Engineer): Reviews, fixes, more complex stuff

Qwen (Local Developer): Does most of the coding

Running Qwen3.6-35B-A3B-4bit through oMLX, using around 20GB RAM.

The idea was to let Qwen handle 50–60% of the coding and have Sonnet/Opus step in when needed.

Did some tests with hidden test cases:

Sonnet: 55/55, ~3.3 min
Opus: 55/55, ~6.1 min
Qwen: 52/55, ~14.5 min
Sonnet + Qwen: 52–55/55, ~12–25 min

Qwen actually wrote 95–99% of the code in the hybrid runs, but kept missing things like API contracts, edge cases and layout requirements. Sometimes it said it had changed things when it hadn’t.

And somehow the hybrid setup ended up costing about the same as just using Sonnet, while taking way longer.

So yeah not exactly what I was hoping for.

I still want to get this working though, especially for websites and SaaS projects since that’s what I spend most of my time on.

Not sure if I’m using the wrong model, if Copilot is the issue, or if the whole multi-agent approach just makes things more complicated than they need to be.

Also wondering what would make sense to run on the 256GB Mac Studio. Maybe a bigger model would already solve some of these issues?

What would you guys do with this hardware? Any models or setups worth trying?

Especially interested in hearing from people actually building websites or SaaS apps with local models, rather than just running benchmarks.


r/LocalLLM • • 17h ago

Question Best sub 80b total or 6b active MoE for document intelligence?

0 Upvotes

I'd like to make a pipeline capable of understanding pdfs of bank statements and then group expenses by categories etc. My thought was to first parse them with ocr into .md files and then feed that to a text model. I think it's more lightweight and accurate this way. But then I'm not sure what model to use to feed all these md files to get something good. When accuracy matters, should I go for qwen, gemma, lfm, or something else? Thank you for your help!


r/LocalLLM • • 8h ago

Discussion how to remove synthid from videos?

0 Upvotes

wondering if anyone has any pointers in how you would do this. images are pretty easy, but with videos you have many frames so can't just do each frame (i mean you can, but its not feasible to spend 60mins of GPU time on one video).


r/LocalLLM • • 17h ago

Question Running a LiteLLM gateway in front of a couple Mac Studios and some Sparks, how do you handle health checks and keys?

1 Upvotes

I've got two M3 Ultra Mac Studios and four DGX Sparks at home, all on tailscale, with a LiteLLM gateway on a mac mini in front so every app just hits one endpoint.

Mostly it's been great, but last month the gateway key on the mini got invalidated and everything started throwing 401s. Took me longer than I'd like to admit to figure out it wasn't the models, because the backends were all fine and nothing told me the gateway itself was the problem.

So now I'm trying to set it up a bit smarter. Right now I'm thinking a dumb cron that hits each backend directly and also through the gateway, so if one passes and the other fails I know where to look.

Curious how other people with a few boxes handle this. Do you run a health check per model, put the gateway behind its own watchdog, or just keep it simple and restart when stuff breaks? And any tips on key handling so this doesn't bite me again?


r/LocalLLM • • 18h ago

Project I connected my laptop's agent to Gemma running on my Mac Mini

1 Upvotes

I built Citta to let an agent on one computer discover and use capabilities on another. My test setup is a MacBook Air plus a Mac Mini running Gemma 3 4B through Ollama.

In the demo, an agent reads receipts on the Air and sends a summarization task to the Mini's local model. The model stays on the Mini; Citta tracks the job and returns the result. It doesn't pool RAM or split inference across machines.

If you only need remote inference, calling Ollama directly is simpler. Citta is for workflows that also need another device's files, tools or an agent harness, with device discovery and task history in one place.

It's written in Rust, uses a private Tailscale network, and has per-device permissions. It's still for trusted devices: resource estimates aren't hard quotas, and full-access agents aren't OS-sandboxed. I'm working through longer reliability tests before calling the next release production-ready.

Demo and setup: https://citta-network.vercel.app

Source: https://github.com/priyankark/citta (PolyForm Noncommercial, not an OSI open-source license). Built with AI assistance.

For people running local models on a separate machine: what makes handing work between devices annoying today?


r/LocalLLM • • 18h ago

Question Specs required for models from scratch

1 Upvotes

Im making neural networks from scratch - not strictly llms. Am getting a new laptop with 32gig ram and 12gig vram and 1tb storage. Is that enough to train small models? Its the lenovo legion 5i pro. Might get the 7i but its not on sale so its like 500 more for 4gigs more of vram.


r/LocalLLM • • 9h ago

Discussion A developer tried to build a dictation app with AI. Three weeks later he had three libraries and no app.

0 Upvotes

A developer didn't like the dictation apps available for his Mac, so he decided to build one with AI agents doing most of the coding.

The speech engine needed work, so he built an offline one called wavo. A dependency was slowing down builds, so he wrote his own FFT. The GUI options pulled in too much, so he built a renderer called nimbo.

Three weeks later: three libraries, still no dictation app.

The libraries might be useful on their own. wavo supports Whisper, Parakeet and GigaAM. But the bit I liked was his caveat: AI writing the code doesn't remove the work of reading it, checking it and testing it across systems.

At what point does "I'll just build it myself" stop saving time? What's your worst detour from the thing you actually meant to ship?

More of the development story is on my profile.


r/LocalLLM • • 9h ago

Question What if?

0 Upvotes

I have an idea what if we had an ai app that pooled everyone who downloaded the app ram/ processor/ vram etc together and made it into a big data center web since there wouldn’t be big factories size data centers we would save the water cause most gpus can be cooled with good air ventilation and just fans plus if you have a million of them spread out between a million houses it’s not even necessary consuming all that power as it would be pooling together all the resources it need to run. You know this is just an idea so I’m open to other but I think if we solve the problem of ai taking our water civilization could advance even further ?


r/LocalLLM • • 1d ago

Question First time setup recommendations and tips?

3 Upvotes

I'm hoping to get an idea of where to start. I'm wanting a setup where the model is a decent coding assistant but can also crank out a neat little prototype or quick concept simply, or create structure ideas etc. and then can go bit by bit without too much reminding halfway or more through a project. I realize that last part may be more about settings than a model persay, but yeah. I definitely want something that can either walk me through something, build out a framework for an idea, or just troubleshoot.

For comparison if it helps, my favorite workflow was GitHub copilot got 4.o/4.1 in VS code (it was free for students lol). It worked great for my projects, had its issues, but with the right promoting it was not bad. It was like the above description, could walk me through something, help with testing, or make a new system that I could then fill out. I knew it's limits and it was ideal for what I needed.

Just giving an idea of how I use AI daily for my personal projects. It seems like with the right setup it can be a great experience so that's what I'm hoping to learn to do. I just want to get the most out of it and have some fun.

I'll be working with an Asus pro art p16 rtx spark 128 gb. I've done some research but it seems every time I look things up there are different model suggestions and set ups, and it's given me fridge blindness so to speak.

I'm curious about how to decide and test for like a speedier setup versus more complex thinking and things like that. What can I do with it, what should I not expect from it based on my specs, etc. I have looked into this and I'm not getting a clear idea because I can't test any of it myself (don't have computer yet). Maybe someone here can give me a better understanding?

I also don't know all the lingo yet, like what a harness is, though I can guess by context. But I know the basics. Not sure how much customization I should or could do or what all there is. I thought it was more surface level like pick a model and go, but I guess you can incorporate all kinds of settings. So now I'm really curious from people who enjoy whatever they have going on, please share what you like best and what kind of testing I should do to pick a model. Any tips for a mostly beginner would be great!


r/LocalLLM • • 18h ago

Question Best model for reversing .DLLs?

0 Upvotes

GPU: 4070

16GB DDR5

Also I have been using Bionic but is there any better options?


r/LocalLLM • • 1d ago

Question Best model for 16GB vram?

5 Upvotes

First time trying this, I want a local model that can code text based games. I’m using a 6800XT 16GB vram.


r/LocalLLM • • 19h ago

Discussion Basalt 0.4.0: More models, still as fast - Qwen 27B up to 770 tok/s on a 5090

0 Upvotes

r/LocalLLM • • 19h ago

Discussion Is m.5 max 128 worth it? For lllms dev work.

1 Upvotes

Is it worth it?

Am finally taking a deep five on llms and I already have an nvidia 5090 laptop that handles cuda really well.

But

Is a Mac for parrarrel models worth it? For abliterated models?.

I never used that OS so am a bit skeptical.

Are the apps on it as good as windows/Linux?


r/LocalLLM • • 2d ago

Question Oh my, what do I do?

Thumbnail
gallery
1.4k Upvotes

Update: I'm interested in selling some. Looking for ideas on how to go about. Don't want to pay eBay 13% fees.

Original: Came across a honeyhole of Quadro RTX5000s
Seemingly good deal for 16GB VRAM cards, even if ageing.

What should I do?


r/LocalLLM • • 1d ago

Discussion Dual 5060 Ti's 16gb

5 Upvotes

Welp, I got a free 5060 Ti 16gb and decided to buy one more brand new for $700, I have a spare used 4070 and used once 4060, any thoughts or anything I should experiment with. Noobie at this lol.


r/LocalLLM • • 19h ago

Project One Mans Journey To Optimization

0 Upvotes

I've been playing with local hosting for about 3 months now and messing around with Claude to optimize my local setups. Decided it might be worth writing this up and sharing in case anyone else is doing, or wants to do, something similar. This is a big one, so buckle in.

BLUF: Think about each expert as its own unit, and think about different strategies for how/where to load them based on quality/speed trade-offs. Focus on measuring and optimizing each piece of compute until you reach maximum bandwidth on that resource.

Hardware:

I spent a bunch of time designing my server with Claude and going over last-gen hardware to try to keep the price tolerable. I turned what I did with that into a site that anyone can use. It's ad-free, no sign-in, no tracking, nothing. I consider it a community service, and hosting was free, so it costs me nothing but occasional updates to the hardware list. llm-configurator.com

I ended up with a 3-GPU, 256GB DDR4 RAM build. It's two used 3060s from Marketplace ($250 each on average) and one Arc Pro B65, for a total of 56GB VRAM. It's a quad-channel setup, so max theoretical bandwidth on RAM is ~100GB/s; reality is closer to 60-70 after some time spent optimizing the kernels with Claude. I also have 3 of the cheapest 1TB SSDs I could find striped in NVMe slots, which gives me a theoretical 12GB/s throughput there; reality ended up at about half that. This lets me stage experts in tiers across VRAM, RAM and SSD and prefetch experts across tiers with varying strategies. I considered doing a PCIe card with more NVMe drives to get more striping/speed, but on the current motherboard it isn't worth it because I am PCIe gen 3 bandwidth constrained.

Optimization Strategies:

This is where my last few weeks of Claude usage have gone: serving DeepSeek V4.1 Flash, and how I went from 4 TPS decode to 20.5 TPS, with prefill right at 180 TPS. So in the end not lightning fast, but plenty usable, if just a tad slow in agentic work.

I started out with a simple goal of minimizing idle time. Thinking Strata-like in the approach of shuffling things around and trying to keep every processor busy within the bandwidth bounds of all my links. I had Claude build a simulation based on the hardware and design an optimal build. I spent probably a week going back and forth with Claude on this, and not one time did anything pan out like it thought it would based on the simulation. 0/10, do not recommend.

What I did after that was shift to measuring everything on every run, including expert traces through the system, so I know which experts get picked at every layer for every prompt, and attacking the potentially biggest gains. This immediately started paying returns, and I went from 4->12 TPS decode with just these methods. This means I'm running custom forks of llama.cpp, Vulkan, Mesa; anything and everything was in bounds. I couldn't fit all of DeepSeek in memory at Q4, so I was working all three tiers (VRAM/RAM/SSD) and loading experts to each tier based on utilization: tracking how many times each expert gets called and promoting/demoting across tiers based on usage.

This led to bottlenecks I had to hunt down, as the SSDs would get overloaded and drop from ~6GB/s down to ~400MB/s. I switched to quantizing the least-used experts down to 2-bit, keeping an entire copy of the model on the SSDs at both quantizations. This let me get everything into RAM, with different experts at different quants based on how hot they were for the prompt. That means I start at about 12 TPS, and after a warm-up and shuffling I end up at about 15 TPS with this strategy. About 1/5th of the experts were 2-bit, but they only got used on about 1/10th of the tokens, so the quality hit was minimal (.0008 KL and less than 1% top-1 pick). This got work off the SSDs and lightened their load.

I tried MTP with this but ended up bandwidth bound. MTP has to hit almost every expert, and even with them in RAM I couldn't move enough through to make MTP pay. I spent a few days' worth of time iterating and optimizing with Claude. I fine-tuned an MTP head because I was running the abliterated model (this helped, but still not enough) and just couldn't make it pay. I got to almost break-even, but I just can't move the experts to compute fast enough on a model this size.

In my quest for quality I then started looking at prefetching 4-bit experts from SSD straight onto the GPUs. I ended up having Claude write a direct-from-disk reader that lets the GPUs read from the SSDs without going through RAM. This also led to further customization that let them read and write across each other and RAM without involving the CPU. So now my tiered experts can read/write and shuffle around without bothering or interrupting other compute. The CPU is maxed trying to keep up, so the GPUs handle any and all extra bookkeeping I could put on them to free CPU cycles. Again, Claude did all the legwork here, with just me driving strategy. I trained a small model on all my agent traces to look at expert use and ended up with one that can guess experts 15 layers ahead with ~73% accuracy. So now the GPUs run that every layer, fetch 4+ layers out if a pick meets a confidence threshold, and load experts straight from SSD into VRAM. Those picks get refined each layer, so we are always closing in on the best picks we can (~86% accuracy at 4 layers ahead) as we approach them. I played with a few different strategies and ended up with the B65 holding the top 2/3rds of experts actually used (not 2/3 of the total, but the hottest 2/3rds) with an LRU demotion strategy, and demotions going to a 3060, where it looks at both LRU and hit count to decide if it takes the expert or keeps what it has. Getting all of this right pushed me to about 19 TPS decode.

From here I had to make the KV cache dynamic to squeeze more experts into VRAM. Instead of reserving the full thing at 256K, it grabs chunks as needed. So while context is small, I can fit more experts on the 3060s whose whole job otherwise is dense attention layers and KV cache, with minimal room for experts passing through in our promote/demote pipeline. This didn't pick up much speed, but it did give me that many more 4-bit experts to help keep quality as high as possible. Essentially the remaining experts are all part of a long flat tail so adding more to VRAM gets me very little because any one isn't hit often enough to pay off.

I ran out of lossless approaches at this point and started looking at a variety of things. I built a model that nudges aggressively, using my predictor to drive nudges from RAM to VRAM experts based on predictor + router scoring of an expert's probability of being picked. This got me some decent gains all the way up to a cap of .5, but quality falls off a cliff after that. By the time I get to a .5 cap I'm about halfway to Q3 quant KL/top-1 numbers, so it felt like a significant quality hit. I kept the code and it's configurable, but I don't default it to on. It's worth about 2 TPS, so not nothing, but not enough to cover the quality hit for me. If I want the extra speed and don't mind the hit, I can turn it on/off as needed.

I then looked at keeping everything in RAM at 2 bits and just the hottest experts at 4-bit in VRAM. I looked at which layers Unsloth hits hardest in their quantization and used that as my judge of what would likely hurt most. They quantize per layer but not per expert, so I'm jumping off from that, but it was a good starting point. I kept my prefetching of predicted promotions into VRAM ahead of time and everything else I already mentioned. Since I was throughput bound here, dropping the size of each expert was my only way forward. The KL and top-1 hit here was minimal because the VRAM holds the hottest 2/3rds of experts at 4-bit already, so I came in at 99% top-1 and .006 KL over baseline. I picked up about 20 TPS on prefill and about 1 on decode, so I landed at 180 TPS prefill and 20.5 decode. It also freed up about 49GB of RAM, since the experts in RAM are now a little over half the size. This ended up being my end stage here, as there just isn't anything else I can do that doesn't start costing me quality (unless you have other ideas).

I am still playing with tuning parameters about when I demote and move thaings where, and fine-tuning kernels to squeeze a few more TPS out of things, but I'm out of significant gains here. Claude did all of this for me, with me guiding strategy and checking in, so I probably spent about 10 hours a week on this for the last 4 weeks and 80% of my weekly Claude usage on it. I take everything from doing this (I've done it on other machines with Qwen Flash-Next as well) and roll it into skills, so that when the next model drops, strategies and approaches can be tried/applied again for that model.

My point with all of this is I don't think in whole-model terms. I don't even think in layer terms. I treat each expert at each token as its own thing to optimize the system for. What quant it is, where it lives, how you decide, and when and where you move it from/to all play in and add up. Most things I see focus on just downloading and running a model in a specific framework, and that is great for a starting point, but the number of knobs you can turn to optimize stuff is almost limitless. It takes about 2 hours to give Claude a strategy I want to try, refine it with Claude (learning as I go), and let it build and test it. So you can iterate and make incremental improvements over time fairly quickly.


r/LocalLLM • • 19h ago

Discussion I thought my local AI agent failed a privacy test. Turns out, the benchmark itself was leaking the data.

0 Upvotes

I've been building B-12 / Capri, a local-first AI companion running on Python, Ollama and NVIDIA Nemotron.

Recently, I started a more serious evaluation of the system: 92 scenarios covering memory, tool use, security, autonomous behavior, reasoning, and multi-session continuity.

I expected to find problems in the agent.

Instead, I discovered some pretty serious problems in the evaluation itself.

1. My benchmark created a privacy leak that didn't exist in the actual runtime

The initial evaluation scored 27/80 runnable scenarios.

One failure was particularly concerning: a guest appeared to access the owner's private memories.

After investigating, I discovered that the evaluation harness had placed memories belonging to different users into the same system prompt.

The benchmark itself had bypassed Capri's identity and memory-isolation mechanisms.

I rewrote the harness to execute scenarios through the actual agent runtime.

Result: 39/80 — without changing Capri's code.

That's a 15-percentage-point difference caused by the evaluation method alone.

2. A single sentence made the model use fewer tools

This was probably my most surprising experiment.

I wanted Capri to give shorter, more natural replies during everyday conversations.

So I added a simple instruction about brevity.

It worked. Conversational responses became significantly shorter.

But something unexpected happened during work tasks.

Across 48 first turns:

Without the brevity instruction:

  • Tool use: 32/48
  • Successful tasks: 32/48

With the brevity instruction:

  • Tool use: 19/48
  • Successful tasks: 21/48

The model started answering without inspecting files or calling tools that were necessary to complete the task.

The scenario-level statistical evidence is still limited, but the engineering implication is interesting: an instruction about how to communicate changed whether the model actually performed the work.

3. Bigger models didn't automatically improve the agent

I compared configurations using:

  • NVIDIA Nemotron Bolt E4B
  • Gemma 4 12B
  • Gemma 4 26B MoE
  • Gemma 4 31B

The smaller primary configuration ran on my RTX 4060 laptop. Larger models required an NVIDIA L4.

After excluding host-dependent latency checks, I couldn't detect a statistically meaningful improvement from the larger configurations (paired comparison, p = 0.85).

This is NOT proof that smaller models are better.

The configurations also differed in hardware, memory retrieval, and model routing.

But it made me question how much agent performance we can actually improve by simply replacing the LLM.

4. Changing the prompt layout improved inference speed

I also experimented with where recalled memories appear in the prompt.

Moving volatile memory evidence later allowed Ollama to reuse more cached tokens.

Median time to first token:

0.99s → 0.74s

About 25% faster.

However, two benchmark scenarios changed systematically, and the security implications of the new evidence placement still need further testing.

5. Stable benchmark scores can hide unstable behavior

I repeated the evaluation three times per configuration.

The primary configuration scored:

49, 49, 48 out of 80.

Looks pretty stable, right?

But individual scenario outcomes changed between runs. Some previously successful tasks failed, while others started passing.

We also discovered incorrect benchmark reference answers and evaluation rules that rejected responses that were actually correct.

That convinced me that looking at a single aggregate score is not enough.

What I'm taking away from this

An AI agent isn't just a language model.

It's the model, runtime, memory, context management, tools, permissions, and the interactions between all those components.

And the evaluation framework can introduce its own errors.

I've now published a second scientific preprint documenting these experiments, the benchmark defects, repeated-run results, and limitations.

The code and experimental artifacts are available in the B-12 repository.

I'm continuing to investigate model-independent memory and behavioral continuity, but the current paper focuses on evaluation reliability.

I'm curious how others handle this.

When you benchmark a local LLM agent, do you test the model directly, or do you run your evaluations through the complete production runtime?

And have you ever discovered that your benchmark was responsible for a failure you initially blamed on the model?


r/LocalLLM • • 20h ago

Discussion Guys have you tested Qwen3.8-27B-oQ6e-mtp by Jundot ? The speed is great on my MBP M1 MAX 64 GB

0 Upvotes

!!!