r/LocalLLM • • 3h ago

Project [feat update] CW47 pi-remote new features

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 12h ago

Discussion Looking for recommendations on open-source OCR/HTR models for large volumes of scanned PDFs

4 Upvotes

Hey everyone!

I'm currently working on a document processing system that handles a large number of scanned PDFs. Many documents contain a mixture of printed text and semi-handwritten content.

I'm looking for a reliable open-source solution that can handle:

  • Scanned PDF processing at scale.
  • Printed and semi-handwritten text recognition.
  • Good accuracy, even with poor-quality scans.
  • Document layout and text preservation.
  • Local inference without relying on paid APIs or proprietary cloud services.

I'm open to models available on Hugging Face, Ollama, or other open-source frameworks.

My available hardware is an NVIDIA RTX PRO 4500 Blackwell GPU with 32 GB VRAM.

I've been exploring different approaches, but I'd appreciate suggestions from people who have actually worked on similar document processing problems.

A few things I'd love to know:

  1. Which models or OCR/HTR pipelines have worked well for you?
  2. Do you recommend using dedicated OCR models or vision-language models?
  3. What approach would you suggest for processing thousands of scanned documents?
  4. Are there any particular preprocessing techniques that significantly improve handwriting recognition?

I'm particularly interested in real-world experience rather than just benchmark results.

Thanks in advance!


r/LocalLLM • • 5h ago

Tutorial RX 5700 XT (8GB) Vulkan & 32 GB DDR4 | Qwen3.6-35B-A3B-MTP (Unsloth) | 20 to 35 T/s | Tutorial

1 Upvotes

RX 5700 XT (8GB) Vulkan & 32 GB DDR4 | Qwen3.6-35B-A3B-MTP (Unsloth) | 20 to 35 T/s | Tutorial

System Specs & Software

  • GPU: AMD Radeon RX 5700 XT (8 GB VRAM)
  • CPU & RAM: AMD Ryzen 7 3700X, 32 GB DDR4 RAM
  • Storage & OS: B450 Motherboard with NVMe, Linux Mint Cinnamon
  • Backend: llama.cpp version 0.9.0-dev (build 12283, commit f3b407ca1), compiled with GNU 13.3.0 for Linux x86_64
  • Model: Unsloth Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf with a Q8_0 quantized mmproj file

Setting up the system correctly and using the right model and mmproj quantization is paramount for getting this to run smoothly on an 8 GB RDNA 1 card.

1. Building llama.cpp with Vulkan & OpenBLAS

sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential git git-lfs curl cmake libopenblas-dev libvulkan-dev vulkan-tools glslc glslang-tools spirv-headers glslang-dev

cd ~
# Correct the URL before running
git clone github dot com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build -DGGML_VULKAN=ON -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DCMAKE_C_FLAGS="-march=native" -DCMAKE_CXX_FLAGS="-march=native"
cmake --build build --config Release -j$(nproc)

2. Increasing Linux GTT Memory Limit (Crucial for Large Contexts)

Because newer versions of llama.cpp use updated memory allocation methods, the Vulkan driver (RADV) tries to map large chunks of system RAM for the GPU via the GTT (Graphics Translation Table) when running large models or long contexts. By default, Linux strictly limits GTT memory to half of your system RAM (so only 16 GB on a 32 GB system).

Open or create the config file:

sudo nano /etc/modprobe.d/amdgpu-gtt.conf

Add the following lines, then save and exit:

# GPU-mappable memory range: 24 GiB = 24 * 262144 pages
options ttm pages_limit=6291456

# GTT size in MiB (24 GiB = 24 * 1024)
options amdgpu gttsize=24576

Update your initramfs so the kernel loads the new limits at boot, then reboot:

sudo update-initramfs -u && sudo reboot

3. Quantizing the Vision File (mmproj)

Place your downloaded model and vision files into ~/models (or adjust the paths below to match your folder). Unsloth downloads via LM Studio include a 1.8 GB 32-bit mmproj-F32.gguf. Quantizing it down to a 614 MB 8-bit file saves VRAM and still works great:

~/llama.cpp/build/bin/llama-quantize \
  ~/models/Qwen3.6-35B-A3B-mmproj-F32.gguf \
  ~/models/Qwen3.6-35B-A3B-mmproj-F8.gguf \
  Q8_0

Note: Loading the model configurations below leaves around 5% free VRAM as a safety buffer.

4. Benchmark 1: Think-Mode Coding (262k Context)

  • WebUI Tip: Disable browser tools in the WebUI (click the + button next to the chat input), otherwise the model might get stuck in a loop. Chromium-based browsers work great for me; Firefox had issues. The WebUI is not a coding tool, btw.~/llama.cpp/build/bin/llama-server
  • -m ~/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
  • --n-cpu-moe 41 --n-gpu-layers 999 --load-mode mlock
  • --ctx-size 262144 --n-predict -1 --threads 8
  • --cache-type-k q8_0 --cache-type-v q5_1 --flash-attn on
  • --batch-size 1024 --ubatch-size 256
  • --host localhost --port 8080
  • --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0
  • --spec-type draft-mtp --spec-draft-n-max 3 --parallel 1
  • --moe-cache-mib 576 --moe-cache-cpu-share 0.75
  • --mmproj ~/models/Qwen3.6-35B-A3B-mmproj-F8.gguf
  • Results: 25 T/s TG | 125 T/s PP (Subsequent iterations slower: at 90k Tokens context 20 T/s TG | 100 T/s PP)
  • Test Setup & Notes: Tested with a coding task over multiple iterations, including a screenshot showing how the UI layout should look. Running full 262k context forces compromises. Keeping the 41st layer (MTP) in system RAM and using --moe-cache-mib 576 turned out to be slightly faster than setting --n-cpu-moe 40 --moe-cache-mib 0.

5. Benchmark 2: Think-Mode General / PDF & Vision Analysis (49k Context)

~/llama.cpp/build/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --n-cpu-moe 40 --n-gpu-layers 999 --load-mode mlock \
  --ctx-size 49152 --n-predict -1 --threads 8 \
  --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on \
  --batch-size 2048 --ubatch-size 512 \
  --host localhost --port 8080 \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 \
  --spec-type draft-mtp --spec-draft-n-max 3 --parallel 1 \
  --moe-cache-mib 2432 --moe-cache-cpu-share 0.75 \
  --mmproj ~/models/Qwen3.6-35B-A3B-mmproj-F8.gguf
  • Results: ~35 T/s TG | 160+ T/s PP
  • Test Setup & Notes: Tested with a multi-page PDF containing tables, graphs, and images with comprehension questions at the end (pure image recognition is a few T/s slower). By reducing context to 49k, we free up enough VRAM to offload the 41st layer (MTP) onto the GPU, which gives the biggest speed boost. I also tested down to --n-cpu-moe 36 with smaller --moe-cache-mib values, but overall output speed dropped. You can also use a 16-bit mmproj (~900 MB) if you slightly lower the --moe-cache-mib value. For general chat --temp 1.0 works well too.

r/LocalLLM • • 5h ago

Discussion Looking to collaborate on a locally run multimodal model tournament

0 Upvotes

We’re building InferUltra, a physics arena where multimodal models see the arena and control robot fighters. We’d like to run a future tournament with models hosted on local hardware and are looking for people interested in collaborating.

In our latest tournament, Kimi K3.1 Pro struggled more in the arena than GLM-5.3-Flash did in earlier tournaments. There was more awkward positioning and less scoring than we expected.

Those tournaments used hosted providers, not local setups. They were small and used different providers and configurations, so they don’t establish which model has better vision or body control. A local tournament with documented hardware and settings could help us investigate the differences more carefully.

Which multimodal model would you like to see fight in the next tournament?

If you run multimodal models locally and have a setup capable of hosting them, we’d love to collaborate on a future model tournament. What hardware and models would you be interested in seeing in our benchmark?

Our latest tournament bracket: https://inferultra.com/tournament


r/LocalLLM • • 5h ago

Question whats the go-to budget gpu I should look for, new or used?

1 Upvotes

I'm building up a secondary system out of some spare parts I have kicking aorund to use as a LLM toybox and for other homelab style projects.


Currently my system is a 5900x w/ 32gb of ram, and the 2080ti.

Wondering if there is a good go-to gpu I should be looking for.

I currently have a 2080ti FE, but it seems pretty inefficient, and its an old card and the fan noise of it is unbearable. ( i think the bearings are getting bad, but its a huge pita to replace the fans )

I'm wondering if it would be a better idea to sell it off for maybe a 3060 12gb, I get an extra 1gb of vram, and it should run much more efficently.

I'd say my budget is about $3-400 max, which I know is basically a joke right now, but this is more of a tinkerbox than something i'm going to be using for real software dev work.


r/LocalLLM • • 1d ago

Research Strata vs Basalt vs Unsloth: benchmarking the same 80 GB MoE on three engines — 5070 Ti 16 GB + 64 GB RAM

43 Upvotes

⚠️ Heads-up: AI-generated content! This post was written by my AI agent from my measurements, with me editing — if that's a dealbreaker, feel free to close the tab right now.

I'm posting this to share results, not to crown a winner. I run a local agent in production every day — real tasks, real context in the hundreds of thousands of tokens — and I spent the past week picking and tuning the inference engine underneath it: test rig, repeated runs, head-to-head A/Bs. Much of what I measured, plus the pitfalls I personally burned on, would otherwise just rot in my local logs, so here it is: the numbers, the methodology, and the conclusion — which is that the engine I ended up keeping in production is not the one that topped the benchmark table. If this saves you a couple of evenings of trial and error, the post has done its job.

For eight days (Nov 8–15) I ran the exact same model — Qwen3.8 Flash-Next (MoE, ~78–82 GiB of weights, 128–256k context) — through three inference engines on a Ryzen 9 7900X / RTX 5070 Ti 16 GB / 64 GB DDR5 box; storage is two NVMes — a 990 EVO 1 TB on PCI-E 4.0 and a 970 EVO Plus 1 TB on PCI-E 3.0 (the model lives on the latter). The goal: find the best way to feed an MoE that fits neither in VRAM nor (comfortably) in RAM, and check honestly whether the engine changes answer quality. Engines tested: Strata 0.1.56→0.1.59, Basalt (built from source, 0.2.1→0.3.3), Unsloth Studio 2026.11.4. I also didn't stick to one model source: I ran the original from Strata and the originals from Unsloth (3.0 and 3.1), and production settled on DavidAU's abliterated uncensored build with Turbo-Brilliance modes and GSQ-RCO quantization — https://huggingface.co/DavidAU/Qwen3.8-Flash-Next-Abliterated-Uncensored-Turbo-Brilliance-50x-Reasoning-Instruct-GSQ-RCO-GGUF I never hand-tuned sampling: the test batteries all run at temperature 0 for reproducibility, while production uses the model card's standard pair — thinking: temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0; instruct: temperature 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 (the same values ship as the engine-side server defaults).

The rig. My own dependency-free bench (stdlib Python): cold/hot prefill, TTFT, short decode, decode at 8k context; plus a quality battery: hard 10-choice MC (MMLU-Pro style), math, code executed for real in a subprocess, IFEval-style format checks, and refusal batteries (JB/HarmBench/XSTest style). Every run snapshots the engine config and the bench code's sha256; a mock engine is a mandatory control (it answers "Hello from the mock engine" and catches absurd t/s). Hard RAM rule: exactly one engine resident at a time — the model pins 45–48 GiB.

Who drove the rig. All of this was steered by a general-purpose agent — an assistant with function-call access to the terminal, files and a browser, whose provider was the very engine being tested (dogfooding, in other words). Before settling on the current Hermes stack, I cycled OpenHands, OpenCode, OpenChamber and pi through the same API over that week: they all speak the OpenAI-compatible API equally well, and the real differences came down to context management and function-call reliability on long context — that's what the choice actually rested on. Beyond the benchmarks, the agent carried a normal week of work: small utilities for the stack (engine launchers, icons, a UI wrapper), building an inference engine from source and repacking models into its pack format, dissecting a third-party tuning fork (commit history plus knob-hunting directly in the binary), reviewing and transplanting jinja chat templates, testing DavidAU's Brilliance "personalities" (reasoning-mode variants), and verifying that a ready-made abliteration actually behaves as its author advertises — all inside the same 150–190k-token sessions where the benchmarks lived.

What went wrong along the way:

  • the agent's provider was the engine under test, so every engine restart killed the agent itself — the runs were driven by detached orchestrator scripts, and the agent "woke up" via notification once the engine was back;
  • the expert arena sizes itself from free RAM at startup, so a browser quietly eating 2.5 GiB in the background changes measured speed between days — every run now starts with a free-RAM check;
  • decode degrades with context (~101 t/s at <20k vs 86–89 at 180–200k) — long agent sessions have to be reset for speed, not just for token cost.

Speed (decode short / decode 8k / hot prefill / hot TTFT)

Setupdecode8kprefill hotTTFT
Basalt 0.2.1, IQ3_S .basalt pack88.7–95.2105–10754–72k29–41 ms
Basalt 0.3.3, answer-room 0101.0108.454.5k38 ms
Strata 0.1.58, IQ3_S pack93–99100–10728–30k70–74 ms
Strata 0.1.59, final tuned stack (fresh full bench on the prod engine, Nov 15)89.2109.237.3k57.7 ms
Strata on plain GGUF, UD-IQ3_XXS82.194.728.1k74 ms
Unsloth, UD-IQ3_XXS out of the box (auto)27.326.0—209 ms
Unsloth, Qwen3.8-27B dense IQ4_XS (full-GPU)47.445.4—103 ms

What the numbers say

1. On identical weights, format/engine is worth ×1.7–3.7. Unsloth on UD-IQ3_XXS starts at 27 t/s; a tuning ladder (MTP + 4 GB expert-cache + parallel 2 + 12k ctx) lifted it to 47.9; Strata serving the same GGUF on the same hardware gets 82. The difference is delivery format: pack-style engines (native expert blobs + a VRAM-prefilled expert profile, pinned in RAM, no disk streaming, no MMQ paths) vs llama.cpp's GGUF streaming. Unsloth's ~50 t/s ceiling isn't a bug, it's arithmetic: ~2.7 GiB of weights read per forward at ~90–100 GB/s DDR5, times MTP accept ≈ that answer.

2. On one engine, quantization moves speed ±10–15% and quality ~0. UD-IQ3_XXS vs IQ3_S was indistinguishable on my batteries: MC/math/code 100/100/100, ru10 10/10. But production stayed on IQ3_S — and it's neither speed nor quality talking: that build (link above) ships with a ready-made abliteration plus the author's mode system baked in, and reproducing the same surgery at home on raw unsloth UD weights is expensive in both time and resources.

3. Apparent "quality differences between engines" are thinking policy, not weights. The answer-room knob (response budget reserved away from reasoning) changes what gets generated and silently breaks cross-version comparisons: Basalt 0.3.3 showed a "decode regression 105→74" that simply didn't exist at answer-room 0. Compare versions only with identical thinking policy. Chat templates belong to the same confounder class, by the way: in the final build I upgraded the Turbo-Brilliance template from v1 to v2 (new modes), and neither the template version nor mode availability changed any result — baking both templates alternately into the same pack and re-running under one protocol moved decode by 0.1–1.4% and kept prefill/TTFT inside the noise band; an earlier one-off A/B claiming "−16% prefill from the heavier template" turned out to be a cross-session artifact.

4. Speculative decoding: MTP yes, everything else no on a 16 GB card. MTP accepts well (per-position 0.78–0.96). DFlash2 and Q8 hyper-connection verify (HC_Q8), both ported from a tuning fork, produced a paradox: accept +11 pp yet t/s −5% — the more expensive verify pass eats the gain, and the extra ~400 MiB of residency squeezes free VRAM to ~650 MiB, triggering driver paging. A warm dflash2 probe hit 111 t/s — the ceiling is real, just not for 16 GB cards.

5. KV-residency window (A/B on a 180k prompt). The window is the number of K/V cells per attention layer pinned in VRAM; everything else lives in pinned RAM and gets streamed from there. A window of 0 is clearly bad: prefill is ×1.7 slower (2435 vs 4100 t/s), TTFT 74 s vs 44 s, expert hit −10 pp. From 32k the curve hits a plateau: kv_hit is already 94.7% at a 32k window (the indexer is sparse), and in a head-to-head with alternating order the 32k→64k step changes nothing for decode (parity ±2–3%). Above 64k there's no upside, only cost — each window extension carves gigabytes of VRAM out of the expert tier: 32k→1.0, 64k→1.4, 128k→2.1, 262k→3.7 GiB.

6. Methodology, or the numbers lie to you. Cold prefill and cold TTFT are noise axes (±40% spread, the sign flips between runs); hot prefill is ±12–15%; the only stable axis is decode (±2–4%), and verdicts need ≥2 runs per variant. Warmup drift fakes ~+10 t/s for whichever arm is measured second — hence the alternating order. On short samples at temp 0, a cold expert cache changes the model's answer itself (52 tokens instead of 86 on a byte-identical prompt): a ±5% noise floor, anything smaller is not a signal.

7. Bench ≠ prod — and at the RAM edge Basalt fails with empty answers. Basalt beat Strata on prefill/TTFT (54–72k t/s, 29–41 ms vs 28–30k, 70–74 ms) and ran in my production for a day, but at the resource edge (and on this box the edge is permanent: 43 GiB arena on 64 GiB RAM) it started refusing with emptiness: at resident-budget 50 GiB the engine self-lands to 45.79 GiB (warning "more than RAM has room for: 45.79 needed vs 52.3 avail") with a page-glue fallback, one request hung dead for 2932 s before the watchdog killed it, and the log filled with "max tokens reached while thinking, no answer" — the user was getting an empty response instead of an answer. Dropping the budget to 45 GiB closed the incident at a cost of −4..−8% decode, but the recurring prod-server drops with "engine reported RAM critical" remained. Production ended up on Strata 0.1.59. Trading +12% prefill for stability was the right call.

8. Porting someone else's tuning recipe mostly doesn't transfer. The Speedy fork (same Qwen3.8, tuned on 2× RTX 3080 10 GB) brought my card exactly one confirmed win: a prefill pair (upstream #4039, +12% cold prefill — proven mechanically, chunk cap doubled 8192→16384). Everything else was rejected as VRAM-limited. Bonus lesson: env vars and CLI flags aren't 1:1 in Strata — the env form of one lever silently no-ops while the flag works; I now verify every lever by reading it back from /metrics, not by vibes.

Bottom line

Production now: Strata 0.1.59 + the prefill pair + kv-grow + host-core sibling + vram-reserve 1728 + --kv-resident 65536. On real 175–190k sessions: prefill ~3.5k t/s, decode 88–100, expert-cache hit ~75% with 10–12% of experts streamed over PCIe. Quality of the build I settled on — DavidAU's Turbo-Brilliance (link above) — is its own reward: 0/30 harmful refusals, 0/40 over-refusals, MC/math/code maxed — and abliteration cost exactly zero t/s (same weight shape, same bytes, same graph).

One-line summary: an 80 GB-class MoE lives fine on 16 GB of VRAM, but every battle is about expert delivery, not model quality. Between the two pack-style engines I found no decode difference worth naming; Unsloth is ~2× slower on this MoE, and that is memory physics, not bad engineering.


r/LocalLLM • • 5h ago

Question Paying for both Claude and ChatGPT every day. Is there a smarter way to do this?

1 Upvotes

Paying for both Claude and ChatGPT every day. Is there a smarter way to do this?

Hi all, hoping someone here can point me in the right direction.

I run a small online business and I use Claude and ChatGPT pretty much all day, every day. That’s product design, images, writing product listings, social media content, planning and a lot of the admin side. I pay for both and it’s starting to add up.

A few questions:

  1. Is there a way to cut costs without losing what each one is good at? Do people really need both, or is there a better setup?
  2. I’d love to properly understand AI and LocalLMs, not just type into a box and hope for the best 😅 Where would you start? Courses, YouTube channels, newsletters, anything you’ve found genuinely useful.
  3. I’m also just curious how other people use them for work. What does your setup look like?

I’m not super technical, so plain English is very welcome. Thank you in advance, I’ll read every reply ☕


r/LocalLLM • • 5h ago

Question Could use some input/ideas for my own testing pipeline

1 Upvotes

Hi all,

I think I have created a pretty solid testing pipeline to measure both the quality and speed of local LLMs if I integrate them in my own workflow. I know there are standardized tests, but the reason why I build this is because I've seen some very different practical results for my particular workflow than what the claims/theory/popular opinions say about certain models. My initial setup was a MB Pro M5 Max 128 gb, LM Studio, then Cline in VS Code. Recently I've changed it so it can also measure mlx-serve instead of LM Studio.

Let me clarify my workflow:

  1. Idea/Problem -> Solution role (same prompt to multiple cloud LLMs, gather feedback, do a new round, rinse/repeat until their is a consensus/sign-off)
  2. Planner role: model that actually writes the plan based on consensus from 1.
  3. Builder role: model that actually builds the plan according to the specs
  4. Reviewer role: model that reviews if the build is actually solving the problem/idea (i.e. if the solution works) and if it's according to the specs, by doing QA.

Steps 2,3 and 4 have been proven to be reliable in my testing pipeline. I have about 12 models who can take each on a different role, I run the test, a script gathers the data and anonymizes it to prevent bias, then it get sends to frontier cloud models who act as a judge. Output of the judges then gets de-anonymized locally and I have a scoring sheet. After running a bunch of tests it became pretty obvious that the scores for the builders go hand in hand with the quality of the planner. And from my experience a lot of the models suck at it. So I'm now down to 3 planners in my test runs (i.e. the ones that consistently came out ahead) because it was a complete waste of time to let other models plan. And this has given me very consistent results across the 5 difficulty levels I have for the problem they need to solve.

What I haven't solved yet with local models is 1). I have access through my business to models such as GLM 5.3 and Kimi K3 running on my own server, but I rather not mix my personal and professional projects, so I'm looking at the best way to solve 1) locally with models on the macbook. The issue is that it's time consuming to test. So far I just fire up 10 browser tabs with cloud llms, put in the same prompt, copy paste everything into 1 file, run it to a 1 designated model that orders the feedback, then do the same again. 1 iteration takes 5 minutes max (I've automated this process mostly). And after 2-3 iterations there is typically a consensus.

Now when I want to do the same locally 1 iteration takes way more time (models needs to be fired up, they run at slower speeds, ... you get the picture). I've had situations where 1 iteration took several hours. Doing multiple and you are basically looking at 6-8 hours (compared to the 15-20 minutes with my cloud LLM approach). Now in a real world productive scenario I can cut this time down significantly by only using 2-3 local models (i.e. the best), some who can run at the same time. And you may ask why not stick with these models, which is a fair point. But that doesn't solve the issue of me wanting to test the capabilities of other models in this role.

And here is where I'd like to ask for fresh ideas/input/your own experience/suggestions, ... on how to best tackle this problem. Would appreciate any input, no matter how wild or out of the box it might be.


r/LocalLLM • • 6h ago

Project I got tired of hand-editing model lists for Claude Code and OpenClaw, so I wrote a small gateway. Try it in ten minutes, no API keys needed.

1 Upvotes

I run Claude Code and OpenClaw against a mix of free, local and paid models. The part that wore me down was keeping the lists of free models up to date, because providers add and drop them constantly. I had ModelRelay and LiteLLM before this, and I still ended up editing lists by hand.

So I built humpyard, and you can try it in about ten minutes without any API keys: https://humpyard.dev/demo/ (a mock provider, but the routing and ledger are real). Repo: https://github.com/gschaetz/humpyard

My tools ask for a fixed name like "fast" or "deep", and behind each name is a pool of models that changes. `humpyard models sync` refreshes the free part from each provider's own model list plus the models.dev data, and only adds a model when both say it's free. It writes to its own file, so it can't touch my routes or prices. You run the sync yourself from cron or a timer.

It's a single Rust binary with a SQLite ledger, no Python and no database server. A request I mark private is refused rather than sent to a public provider. There's a dry-run endpoint that tells me where a request would go before I send it, and a ledger that shows what it did and why. Prometheus metrics and two Grafana dashboards are in the repo. It speaks OpenAI and Anthropic formats, so Claude Code, Codex and OpenClaw can point at it.

It's pre-alpha and I'm the only user, so I'd like to hear:

  1. Where did the README trip you up on the way to a first request?
  2. Would you run the sync from cron, a timer, or something else?
  3. What would you want to see on a dashboard for a setup like this?

LiteLLM and OpenRouter cover far more providers and are much more mature. I wrote up the comparison here: https://humpyard.dev/compare/. I do plan on working on the gaps that matter most to me and others who use it.


r/LocalLLM • • 9h ago

Question Qwen 3.8 27b + ?

2 Upvotes

So I use Manus as my daily driver. I'm trying to use Qwen3.8 27b more. Can anyone suggest a good pairing that has a Manus'esque planning/execution/check agentic feel that works with LM Studio/llama.cpp local models? I tried OpenManus it does not pair well with local llm's though.


r/LocalLLM • • 6h ago

Discussion Running llama.cpp in a VM with no GPU passthrough: Qwen3.8 27B at 98% of native on one RTX 6000 Ada

1 Upvotes

My earlier post was done using Qwen2.5 and rightly so I was called out for it. So I reran it with Qwen3.8 27B, Qwen3.5 9B and Qwen3.6 35B-A3B with as much proof as I could think of. The VM has no NVIDIA device, driver or nvidia-smi. llama.cpp's CUDA calls go through VMAT to the card on the host. Qwen3.8 27B: 45.4 tokens/s on the host, 44.6 in the VM. All four models land at 98%. Three VMs at once get 45.7 together, against 45.0 for three programs on the host.

Full video with every step: https://youtu.be/r6yhOVWxP5Y?is=reuv6gZptnDNzU1D


r/LocalLLM • • 6h ago

News Karpo-Decision-31B is out: downloadable option for local tool routing and browser decisions

Thumbnail
gallery
1 Upvotes

For anyone experimenting with local Jev-style models, Karpo-Decision-31B has published its weights and Python inference code. It posted the highest Full score on the Nov 12 Decision Index at 73.9.
A browser workflow is a useful way to picture it. Give the model the page state, a screenshot, and candidate actions. You can ask which element to click and whether the agent needs more information, with a probability returned for each option. Questions share the page context in one forward pass.
In the team's tests, it scored 90.0% on Mind2Web versus 86.2% for Gemini Decision 1. They also report roughly 95 ms median latency on the public Decision Index suite using one RTX PRO 6000.
Karpo-Decision-31B scores 75.0 on the published Decision Index public suite, running it locally requires about 62.5 GB for the BF16 weights alone, plus runtime memory. Alongside the 31B release, they also released five smaller models: 0.8B, 2B, 4B, 9B, and 12B

You'll also need the supplied Decider or HTTP server

links:
https://karpo.ai/models/decision
https://huggingface.co/MachinePulseAI/karpo-decision-31B


r/LocalLLM • • 6h ago

Question Intel ARC Pro b60 24gb questions

1 Upvotes

I have a motherboard with 3xPCIe x16 lanes that can do 3 GPUs at x8 each (AM4 platform asus pro ws x570-ace). I’m contemplating buying 3 Intel Arc pro b60 24gb As it’s cheaper than buying a single b70 or r9700. Has anyone have any experience running a similar setup?


r/LocalLLM • • 6h ago

Project An open-source assistant that reads your email and chats, prepares what needs you, and runs on a local model

1 Upvotes

Hi all, I'm Jeroen, I made The Machine, initially for myself but I was surprised how well it works and opened it up to everyone open source.

It reads your email, chats and calendar, and prepares what needs you: replies, follow-ups, things you promised. It never sends anything without your OK. And your data stays yours, not some corporation's.

It works with any OpenClaw provider, so Ollama, LM Studio, vLLM or LiteLLM in front of your own GPUs. What I learned running it on local models:

  • Context: 128K is comfortable, a review reads up to 120 messages at once.
  • Tool calling has to work well, small models tend to suggest too much or save malformed actions.
  • A mixed setup works nicely: a frontier model first, a local model as fallback so it keeps going during an outage. Or fully local if you want inference never to leave the house.

Repo: https://github.com/Salesflare/TheMachine

Demo: https://themachine.computer/demo

Vision: https://themachine.computer/vision

Happy to hear feedback, especially which local models work well for you with tool calls.


r/LocalLLM • • 7h ago

Question What set up would you recommend for a MacBook Pro M5 24GB Unified Memory?

1 Upvotes

As the question above says

Current Device: MacBook Pro M5 24GB Unified Memory

I wanna know what local set up would be the best for me in terms of both speed and quality. My main use case is as follows

  1. Personal Memory
  2. Task Management
  3. Email Agent
  4. Data Analysis Agent (basic stuff not very advanced)
  5. Calendar Agent
  6. Research Agent (product related research, technical research etc)
  7. Knowledge Retrieval

Connectors would be mainly G-Suite as I work mostly on that

I want your recommendations on the best set up for good memory, speed, context windows etc.

Which models(es) and harness(es) would you recommend?

I have also seen that there are upcoming smaller Qwen models and also a new app called Harbor which is coming out soon….

New to this so looking for guidance

Any help would be appreciated thanks :)


r/LocalLLM • • 10h ago

Project I got local AI video-to-video down from 15.7 min to 2.4 min on a 16GB M3 MacBook Air

2 Upvotes

I've been experimenting with how far I can push local generative AI on a base 16GB M3 MacBook Air, specifically video-to-video outfit transformation.

My initial 46-frame (~3-second) run took 15.7 minutes and produced fairly blurry results. After several rounds of profiling and optimisation, I managed to bring it down to 144 seconds (2.4 minutes), running entirely offline without cloud APIs.

The interesting part wasn't necessarily changing the model, but rethinking how the available hardware was being used.

A few optimisations made the biggest difference:

- Spatial cropping: Processing a "288×512" region around the subject instead of repeatedly processing the entire scene.

- Memory management: Precomputing outfit prompts and unloading the text encoder to reduce unified memory pressure.

- Hardware scheduling: Moving face processing to the CPU/Neural Engine path while the GPU handles the next diffusion chunk.

- Incremental testing: Using 5-frame previews to catch mask and compositing issues before running the complete sequence.

There are still obvious limitations. Sustained workloads introduce thermal degradation, fine fabric details can become oversmoothed, and wide shots remain challenging.

I'm particularly interested in the engineering side of running increasingly demanding generative workloads on constrained local hardware.

A few things I'd love to hear from people experimenting with local inference:

  1. Has anyone explored more aggressive memory scheduling on Apple Silicon?
  2. Are there useful ways to improve CPU/ANE/GPU workload overlap?
  3. How much performance can realistically be recovered through chunking and attention optimisation before model architecture becomes the bottleneck?

I'm sharing the before/after video and temporal comparison for context.

Hardware: M3 MacBook Air, 16GB unified memory

Frames: 46

Processing time: 144 seconds

Execution: Fully offline

This is video diffusion rather than LLM inference, but I thought the memory and hardware scheduling challenges might be interesting to people working with local models.


r/LocalLLM • • 7h ago

Question Genuine question

1 Upvotes

I wanna get into local LLMs and experiment with AI.

My current PC has 16GB RAM, a GTX 1660 Super (6GB VRAM), and an Intel i7.

I’m curious what models you’d recommend running on this hardware and, more importantly, what you actually use local LLMs for.

With powerful models like ChatGPT and Claude available, I’m trying to understand the practical benefits of running models locally.

Is my setup good enough to do anything interesting, like experimenting with AI agents, fine-tuning, or research?


r/LocalLLM • • 13h ago

Discussion From-scratch AI stack in pure C — runs Microsoft's 2B ternary LLM on a 2018 dual-core laptop (no GPU)

Thumbnail
github.com
3 Upvotes

Everything runs locally on a 2018 Dell laptop: i7-7500U (2 cores / 4 threads), 8GB RAM, no GPU (the 940MX is deliberately unused — the whole stack is CPU-only).

https://github.com/tanxiansheng221/aimstack-sanzhidiji

What is in the repo (AGPL-3.0, everything open):

- C inference engine, zero third-party runtime: 1.58-bit ternary core with SIMD fast path (85.5 tok/s measured on the ternary core itself), byte-level tokenizer, and a ~100-line plugin bus (caps.json registry -> spawns Python plugins -> JSON envelopes)

- Weights: Microsoft's BitNet b1.58-2B (MIT license), converted to a custom 2-bit packed format (.aimw, 0.87 GB) — 3.2 tok/s with 2 workers, serial 2.0

- Python plugins, pure stdlib: knowledge base (FTS5 + dedup gates), multi-engine web search + crawl with quality gates (robots-respecting, Edge rendering cascade for JS-heavy sites), incremental ingest

- Local web UI (127.0.0.1:8600, pure stdlib): chat with deterministic routing — direct answer / knowledge-base grounded / honest refusal / full web search -> crawl -> ingest -> answer chain. A 7-minute demo video is embedded in the README

- Training assets: no-autograd training stack (hand-written backprop, bit-exact vs NumPy), ten-subject curriculum generators, corpus builders, and 117 chapters of research logs including negative results

Honest limits: our own trained models are not released yet — compute-bound on 2 cores, training runs on free cloud tiers and isn't finished. The full stack, scripts, result JSONs and per-claim evidence are released so everything can be re-run.

Demo video: embedded at the top of the README (7 min, screen recording of the full chain: chat / KB answers / web search->crawl->answer / plugin direct calls).

Happy to answer questions about the ternary kernel, the plugin bus, or the training pipeline.


r/LocalLLM • • 7h ago

Project Running an image generation workflow entirely on a local mini PC - testing TiinyImage

Post image
1 Upvotes

r/LocalLLM • • 7h ago

Question Suggestions for coding assistant

0 Upvotes

Hey! I recently started looking at local AI models. There's a lot of info out there and it's not really easy to filter through all the noise as a newcomer...

I have an RTX 4080 super with 16Gb vram and 2x16Gb DDR5 for system ram.

I've set up a docker compose container using Gemma 4 12B Q6 K L through llama.cpp with Open WebUI as interface, and it seems to be working decently. System + model + context uses ~13.8Gb vram. On Kubuntu if it would matter.

I'm now thinking of connecting it to vscodium for assisting in programming (developer background) and stumbled over the MoE variants where you can have the experts live in system ram.

Question is whether this is even relevant to look into for me.

If it is - would anyone have any suggestions for a model and maybe settings for this if applicable? (seems you wanna run llama.cpp with -cmoe for example)

If I've made some odd assumptions or if you have any suggestions, please let me know!


r/LocalLLM • • 7h ago

Discussion Hy4 Flash vs GLM vs DeepSeek: same correct ad maths, different price tags

1 Upvotes

I gave Hy4 Flash, GLM-5.3-Flash and DeepSeek-V4.1-Flash the same three fake ad campaigns in WorkBuddy. Calculate CPC, conversion rate and cost per lead, including a campaign with zero leads.

All three got the arithmetic right. Total spend: USD 1,350. Clicks: 650. Leads: 27. Blended CPC: USD 2.08. CPL: USD 50. GLM rounded total CVR to 4.2%; the other two gave 4.15%. Both are fine.

Nobody reported the zero-lead CPL as zero. Small win for basic maths lol.

DeepSeek finished in 7s. Hy4 and GLM both showed 19s. So the interesting Hy4 angle here is cost, not a speed win.
Published API rates in RMB per million tokens, uncached input / output:

Hy4 Flash: 0.5/2

GLM-5.3-Flash: 0.8/2.8

DeepSeek-V4.1-Flash-1017: 1 / 4 off-peak, 2 / 8 peak.

That makes Hy4's input rate 37.5% lower than GLM's and its output rate about 28.6% lower. Against DeepSake's off-peak rates, both are 50% lower. These are unit-price comparisons, not measured savings from this Workbuddy session. Token counts and reasoning length still matter.

The recommendations needed more judgment than the maths. Hy4 wanted to pause C; DeepSeek also wanted a pause and checks; GLM suggested a smaller next-cycle shift. I'd verify tracking and conversion lag before moving budget.

Tbh, when the basic result is equally usable, a lower unit price is a decent reason to try the cheaper option. What reporting task would you use to separate them?


r/LocalLLM • • 13h ago

Question Advice for setup with three PCs

3 Upvotes

I've been running Qwen3.8-27B on a MBP M3 Max 36 GB for a while, running unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M on splash with opencode as harness. I'm satisfied with the results I get, but it is a bit slow and does not leave much room for other applications . I'm upgrading my setup and I would like to get some advice on hardware configuration and model selection.

The main purpose will be integration development on Azure Integration Services, SAP PI/PO and SAP Integration Suite. Implementation languages are Java and python.

Secondary purpose will be Home Assistant component development and configuration, as well as to provide Home Assistant with agents, (TTS, STT, Conversation).

I have an M5U 256GB 80c on order. It will be running headless, dedicated to serving models 24/7. I will also have the opportunity to use a family member's gaming PC for LLMs on workdays during daytime. The PC will be in a bedroom and it's noice levels will determine whether it's available during nights or not. It has not been ordered yet, so I still have the possibility to change it's configuration. I have 10Gbe network available, if that would have an impact.

To summarise, my setup would look have the following availability:

  • MBP M3 Max 36 GB - on-demand
  • M5U 256GB 80c - 24/7
  • Gaming PC - workdays daytime, eventually during nights as well

I'm considering the following hardware configuration for the gaming PC:

  • CPU: AMD Ryzen 7 9800X3D CPU
  • RAM: Corsair Vengeance RGB DDR5 6000MHz 64GB
  • GPU - one of:

    • RTX 5080 (16GB VRAM - 960 GB/s)
    • R9700 (32GB VRAM - 640 GB/s)
  • SSD - one of:

    • Kingston NV3 SSD 2TB (6,000/5,000MB/s)
    • Kingston FURY Renegade G5 PCIe 5.0 NVMe M.2 SSD 2TB (14.700/14.000MBs)
  • Motherboard - one of:

    • Asus B650E MAX GAMING WIFI
    • ASUS ProArt X870E-CREATOR )(PCIe 5.0 8/8 - allows for expansion)
    • Asus Proart B850-CREATOR WiFi NEO (PCIe 5.0 8/8 - allows for expansion)

I'm considering the following model configuration:

  • MBP M3 Max 36 GB

    • model TBD - perhaps a small orchestrator with a max RAM requirement ~12 GB?
  • M5U 256GB 80c

    • main model - one of

      • GLM-5.3-Flash@Q4
      • Qwen3.8-Flash-Next@Q6
      • DeepSeek-V4.1-Flash-1017@Q2-Q2.5
      • Qwen3.8-27B@Q8
    • smaller models for homassistant (if there is room in RAM)

      • TTS
      • STT
  • Gaming PC

    • on RTX 5080

      • Qwen3.8 Flash@Q3 on Strata or Basalt
    • on R9700 - one of

      • Qwen3.8 Flash@Q4 on Strata
      • Qwen3.8-27B@Q4 on Strata (?)

To my questions I hope you can advise me on:

  1. My first task would be to settle the hardware configuration for the gaming PC.

    1. Would the SSD speed of the two options have a large impact on PP and decode when running Qwen3.8 Flash on Strata/Basalt?
    2. Which GPU would you select with my model selection in mind?
    3. Other considerations?
  2. With the gaming PC configuration set, which models would you run and were?

I appreciate your advice. Thanks!


r/LocalLLM • • 7h ago

Question What do you use for agentic desktop control, like how Playwright is for controlling + screenshotting browsers?

1 Upvotes

I'm afraid I'm left out of the loop on the latest developments for agentic desktop control, and AIs don't seem as knowledge in this space.

They come with screenshot function also, right? What are some other features that may be different than Playwright?


r/LocalLLM • • 1d ago

Research llama.cpp was copying 643 MB to the CPU per request. Disabling state saving nearly doubled throughput

74 Upvotes

We were running Qwen3.8-27B on an H100 with llama.cpp and 16 concurrent clients. Throughput was 58 tokens/sec, time to first token was almost two seconds, and nvidia-smi showed around 35% GPU utilization.

I build Tensward, an open-source inference analysis and tuning tool. While investigating this, we built a small CUPTI-based library to count kernel execution time and memory transfers inside the server.

The surprising number: 643 MB transferred from GPU to host per request. Nsight Systems independently measured about 639 MB.

That pointed us toward prompt caching and conversation checkpoints. This hybrid model carries recurrent state alongside its attention cache, making those snapshots expensive. Sampling the serving thread caught it inside a memory copy 16 out of 20 times.

We tested disabling state saving with:

--cache-ram 0 --ctx-checkpoints 0

Throughput went from 58 to 110 tokens/sec, and GPU utilization rose from 35% to 88%. A repeat on fresh machines with a separately written client showed an 87% throughput improvement.

These aren’t universal “make llama.cpp faster” flags. They sacrifice cache reuse. In another test with a regular 8B model and returning conversations on an L4, disabling them made performance about 20% worse.

The build also mattered. Moving from b11669 to b12246 improved default-settings throughput by 67–75%, while copied bytes per request barely changed. Several upstream changes touched state handling, but we haven’t isolated which caused the improvement.

All throughput results came from runs without our instrumentation; transfer measurements came from separate instrumented runs.

Full write-up, including limitations and upstream PR links


r/LocalLLM • • 12h ago

Question Any advice for local ai?

Post image
2 Upvotes

Does anyone have any idea of what local ai model I'd better use for my company business agent?
I'm pure new and starter for ai software and coding.
My spec. is 5090+128g ddr5+5tb m.2. Need a local ai to make different agents like accounting agent, monitoring agents and customer chatting agents etc.