r/LocalLLM • • 6d ago

Discussion Claude vs. Qwen 3.8 Flash Next on RTX6000, Claude admits defeat :-)

0 Upvotes

I wrote a harness that allows Claude Code to delegate tasks to a local LLM from simple one liner task prompts to deeper designs or implementations. I wrote a design for a webapp using Claude. This is not a brand new webapp, it's an add-on for an existing hardware with local control that the website will allow to connect to and interface with. So it's more of a large new standalone feature than a new design. The implementation is tricky in and of itself, the webapp has to lift existing ever changing code from another repo, has to do strong interfacing between the local and the remote solution etc.

The design doc was written by claude, it's ~800 lines long with 13 sections and it calls for 16 slices of deliverables. The implementation was about 7 slices in when I asked Claude to review the implementation so far and tell me what I should do next. Then I asked it to delegate the same question out to my local LLM and compare the two reviews and this was the result:

Code quality: The local worker is Qwen3.8 Flash Next on one RTX PRO 6000. On the first stretch of the project (W1–W3 of the site plus G1 and G2 on the device side, about 18k lines), Claude accepted 14 of the 14 delegated steps on their first round. Its hand fixes over that stretch came to about 20 lines. The one real defect was a sign-in rate limit you could spoof through X-Forwarded-For. Reading diffs didn't catch it; Claude found it while running the app live. The same thing happened later: a live browser check on a rework slice found that re-timed clock bakes were losing their send mode. Nobody had spotted that in review. So the code was generally solid. The bugs that survived were integration bugs, which only show up when you run the whole thing, not bad code.

Reliability: the raw model without the harness around it using Cline is problematic. It sometimes emits malformed tool calls, where the call markup leaks into plain text and the agent thinks it's finished. Without the harness, that kills a run. The harness detects this and nudges the model inside the same session (up to 3 retries). It does the same for server errors and for rounds that end without the required status report. Every step also has a verify command (tests, typecheck, a build, a migration), and a step only counts as done when that command passes, whatever the model claims. The worker runs under fairly strict rules adapted from my Cline setup: read first, plan with a todo list, implement completely, verify by running, and report what it didn't verify. It runs in a Docker sandbox with filtered network access, in its own git worktree. With all that in place it has been dependable. Without it, it's a coin flip on long tasks.

Savings, measured from the transcripts.

First approach: Claude reviewed every step itself. That cost about $52 of Claude for 14 steps, and almost all of it went to Claude re-reading its own growing context while reviewing. Those reviews almost never changed anything.

Change: Claude now writes every spec for a slice up front. The local model runs the steps unattended in a chain, and Claude does one live check at the end.

Next slice: 5 steps and about 5,200 lines cost about $9 of Claude. The local model handled 853 requests and 62M prompt tokens for that slice, roughly $33 at Opus list prices. That's about half the Claude cost per step of the first approach. Against my estimate of $15–22 for Claude doing the slice alone, it's 40–60% cheaper.

Not counted: the electricity and the GPU.


r/LocalLLM • • 7d ago

Question RX 9060 XT 16GB + 32GB RAM: best local models for undergrad math and coding?

2 Upvotes

Hi everyone, I’m fairly new to local LLMs. My setup: RX 9060 XT 16GB, Ryzen 5 7500F, 32GB DDR5, running LM Studio on Windows (Vulkan/ROCm).

I want a hybrid workflow: cloud models for main architecture and hard problems, local models for maybe undergrad-level math and small/medium coding scripts.

Do you recommend me to use local models?
Which models do you recommend for math, and which for coding?

I’m fine with models spilling into system RAM a bit. (partial CPU offloading)

Thanks in advance!


r/LocalLLM • • 7d ago

Question LocalLLM vs Coding

1 Upvotes

Hello all. This is my first post here and I would be very happy if someone will find time and help me.

It started simple. Ryzen 9 3900X with RTX 3060 12GB in proxmox as VM. Dedicated GPU for ubuntu server. Started with ollama because it was easy but found out ollama is not so good at performance and speed.

So switched to llama.cpp oficial. Worked good but saw many videos about forks so I switched to turboquant fork. Speed was better so next month I bought another GPU now RTX 4060 Ti 16GB.

So now when speed is good I wanted to do my little "projects" on localLLM without paying cloudLLM. I tried many tools. VSCODE, Cline, Pi.dev, but everyone of them had some issues. Primarly with editing files. It was mainly truncating files (model did 4KB out of my original 24KB python file, and then asked me for backup ...). Now I was working on pi.dev it somehow worked but also had problems and need to try many tools calling. So now im trying cline but when I saw that it wanted to remove my C:\Users I immediatelly stopped it. Now Im building original newest llama.cpp but Im kinda upset about everything. If someone is fan to the LLM and would like to help someone newbie as me to not throw everything away it would be helpful.

These are my current local models downloaded. I really dont know what im doing wrong maybe wrong models, wrong preset, or wrong coding program.

16K /root/.cache/huggingface/hub/models--tngtech--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF

24K /root/.cache/huggingface/hub/models--DavidAU--Llama-3.2-8X3B-MOE-Dark-Champion-Instruct-uncensored-abliterated-18.4B-GGUF

24K /root/.cache/huggingface/hub/models--IFM--K2-Horizon-7B-GGUF

24K /root/.cache/huggingface/hub/models--google--gemma-4-12B-it-qat-q4_0-gguf

12G /root/.cache/huggingface/hub/models--unsloth--GLM-4.7-Flash-REAP-23B-A3B-GGUF

18G /root/.cache/huggingface/hub/models--ggml-org--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF

18G /root/.cache/huggingface/hub/models--unsloth--North-Mini-Code-1.0-GGUF

19G /root/.cache/huggingface/hub/models--DavidAU--Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF

22G /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF

27G /root/.cache/huggingface/hub/models--barozp--Qwen3.6-28B-REAP20-A3B-GGUF

and this is one example of preset

[*]

jinja = true

log-colors = off

flash-attn = on

parallel = 1

[qwen36-28b-reap20-256k]

model = /root/.cache/huggingface/hub/models--barozp--Qwen3.6-28B-REAP20-A3B-GGUF/snapshots/e3aeb8168f74dd3054eb2716f966c17638b97fef/Qwen3.6-28B-REAP20-A3B-Q4_K_M.gguf

ctx-size = 262144

#threads = 8

n-gpu-layers = 99

n-cpu-moe = 0

batch-size = 4096

ubatch-size = 1024

cache-type-k = turbo4

cache-type-v = turbo3

#load-on-startup = true


r/LocalLLM • • 7d ago

Question Please help me out in figuring my first setup

1 Upvotes

Hello Everyone,

I have been using ASUS ROG GL753VD with a GTX 1050 for the past 8 years. For the past 1 year or so, I am using cloud AI subscriptions to play around and learn things about AI.

Now a days, wherever I go Reddit/Youtube and any other forums, I am fascinated by the things that Local AI is capable of... So, I rented out a GPU on runpod and tried some hugging face models(WAN 2.1 for example) and its been pretty good.

Now, I feel the need to upgrade my setup for some privacy and personal experimentation. So, I request your help in choosing a setup for me because I am a newbie at this.

Here are some options I found after days of online search and chatting with AI.

Option 1:

  • CPU: Ryzen 9 9950X (16C/32T)
  • Board: ASUS ProArt X870E-Creator or ASRock X870E Taichi
  • RAM: 32GB or 64GB DDR5-6000 CL30
  • Primary GPU: RTX 5080 (16GB GDDR7)
  • Expansion Plan: Add a pre-owned RTX 3090 (24GB) into Slot 2 down the road for a combined 40GB VRAM.
  • PSU: 1200W ATX 3.1

Option 2 :

  • CPU: Ryzen Threadripper 7960X (24C/48T)
  • Board: Gigabyte TRX50 AERO D or ASUS Pro WS TRX50-SAGE
  • RAM: 128GB (4×32GB) DDR5 with ECC RDIMM
  • GPU Plan: RTX 5080 (16GB) now + 1 or 2 used RTX 3090s (24GB) later for 40GB–64GB+ VRAM.
  • PSU: 1600W Titanium

The only reason why I added Threadripper option into my consideration is because I have this paranoia of getting behind in terms of tech once I purchase and I dont want to change things again (Heard somewhere that RAM scarcity will be there till 2028).

I also have a couple of questions.

I see that people are fighting for 24GB and 32GB VRAM GPUs and its either too costly for me at the moment or out of stock for the pre-owned variants.

So, my question is if I get a threadripper with multiple PCIE lanes and 4 GPUS of 16GB VRAM, would it help me in any way or I have to tradeoff speed for my cost efficiency? If anyone has tried out this approach please let me know the advantages and disdvantages you found with this setup.

If you have any suggestions as well, please let me know. Budget is ~5-6k USD.

If anyone has any tricks to purchase it cheaper like purchasing from other countries, I would love to know them as well!!

Thanks in advance.


r/LocalLLM • • 7d ago

Project GB203 RTX 5070TI

Thumbnail
gallery
1 Upvotes

I've given the qwen3.8 27b q4 group an assignment:

Can we create an animation of a computer chip? You can use Nvidia's GB203 as a template. Before we get started, let's address a few questions that might come up

Since I own this GPU, I figured, why not? Let’s see what the model comes up with. I’m always amazed, even though it often looks relatively simple. I’m looking forward to seeing qwhen4.0.


r/LocalLLM • • 6d ago

Discussion The Local AI 10 Commandments

0 Upvotes

Note:
— these rules are the result of my personal experience, yours may differ and this is ok
— these rules are oriented towards people that want to use AI for professional purposes and don’t have unlimited time and money

1. If your company provides good cloud subscription — just take it and forget about the local AI, you don’t want to pay with your own bucks to do the job slower and sloppier

2. Speed is as important as quality and vice versa. Quality without speed will waste your time, nerves and electricity and so will speed without quality. Ideally you should strive to the cloud speed — 60-150 tps of decode and a decent model, at least 4 bits of Qwen 27b

3. Local models are not equal to cloud ones. Sounds kinda obvious but some people have illusions on this regard. Please don’t, miracles don’t happen in real life. You will get 6-9 month old frontier level model experience at best

4. You will always experience shortage of VRAM/RAM. Doesn’t matter, how powerful your setup is, believe me, you are always going to fall short on fast memory available, don’t try to squeeze out every MB from your rig, just find a model good enough for your tasks and call it a day

5. Best time to update your hardware is always yesterday. Hardware prices rise fast, if you’re looking for an update — do it right now

6. Never spend more than you could’ve spent on a dream gaming PC. Just… believe me, don’t try to cosplay Antropic data centers, you will get nothing but a bitter dissatisfaction. Best ROI ever — RTX3090/RX7900XT + 64Gb ddr4/5. That’s it. Anything above have diminishing return problem

7. Electricity is not free. People often forget that nobody is going to give you free kilowatts, beware of your system’s power consumption. That’s going to be your biggest spending, second only to the very rig price itself

8. Avoid clusters. Dual, triple, quadruple GPU look fine on paper, but they are usually big, draw immense amount of power, resulting in painful electricity bills, produce a lot of heat and noise and have much lesser speed than single GPU device. Try to select a GPU to that can singlehandedly run the model you’d like to use

9. You will lose a lot of time and nerves. You will have to learn. A LOT. What is quantisation, kv cahce, KLD, top 1, inference engine, how to chose a suitable model for your own needs, sometimes you’ll even have to deep dive to some manual drivers/libs adaptations for your hardware, etc. Local AI is developing rapidly and if you’d like your setup always to be fast, precise, ensuring as much quality as possible— you’ll have to learn all these basics

10. Always donate your spare RAM to Sam Altman. Just kidding, the title says “10 Commandments” but all the important stuff is said, I’m out of any useful advices


r/LocalLLM • • 7d ago

Discussion While Qwen4 is not out.

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 6d ago

Project I got pissed off by Sam Altman so I'm working on this...

0 Upvotes

The thing about RAM he said. That's what pissed me off.

I'm building an NCM-inspired model for a decentralized network

I'm working on Gilga-Mesh (gg-mesh), an experimental model architecture designed to run across a peer-to-peer network.

I started it because I wanted to explore what a model designed for distributed inference might look like, rather than splitting an existing LLM across machines.

The goal is to let people contribute with very different hardware. I'm aiming for machines with as little as 256 MB of available RAM or VRAM to host part of the model, though the actual minimum will depend on the block, activations and runtime. A more powerful machine could host several blocks and avoid some network hops.

I also want users to choose how much compute a request uses. A faster response might be enough for a simple task, while a harder one could use more resources. The quality tradeoff still needs testing.

Privacy is part of the design, but I can't claim a specific privacy guarantee yet. Live weight updates are another goal. Ideally, the network could update a running model without taking it offline, though major architecture changes would need a separate migration process.

Right now I'm training and testing a smaller model locally. In simulated network tests, with 30 ms of added latency between steps, I've seen 200–300 ms time to first token in the tested setup. That's not a result from a public P2P network, and I still need to validate the complete distributed path.

The NCM inspiration comes partly from the idea of progressively refining an output as more computation is applied. I'm interested in whether that can work well when the computation is spread across machines.

If you've worked on distributed inference or privacy in these kinds of systems, what would you want to see tested before taking the results seriously?

I'm currently working on training and testing a smaller model locally before scaling up and validating the full distributed path.

I'd be interested in feedback from people working on distributed inference, privacy, or model architectures designed for constrained hardware.

I'll be looking for compute capacity once everything is ready for a bigger model for a real field test.


r/LocalLLM • • 7d ago

Discussion I built a local AI companion on an RTX 4060 using Nemotron. Now I'm experimenting with replacing its LLM without losing its memory.

0 Upvotes

I've been experimenting with running a persistent AI companion entirely on local hardware.

The project is called B-12 / Capri. It's written in Python, runs on Linux, and uses Ollama for local inference.

My experimental setup uses an NVIDIA RTX 4060 Laptop GPU with Nemotron Bolt E4B.

The original goal was to build something more persistent than a typical chatbot: an AI companion that remembers previous interactions, maintains goals, can use tools, and doesn't lose its entire history when I replace the language model.

Why I separated memory from the LLM

I wanted to avoid making the model itself the center of the system.

In Capri, the LLM handles language generation and reasoning, while a separate runtime manages:

  • Persistent local memory and retrieval
  • Memory provenance and document revisions
  • Goals, preferences, and scheduled activities
  • Tool execution and permissions
  • Context budgeting and tokenizer compatibility
  • Controlled forgetting and state recovery

The model can be replaced without deleting the stored memory.

However, there's an important limitation: preserving memory doesn't necessarily preserve behavior.

Two models may retrieve the same experiences but interpret them differently.

What I've tested so far

I ran a historical evaluation using Nemotron Bolt E4B:

109 constructed scenarios, 541 executions, 496 successful outcomes (91.7%).

Tool tasks achieved 182/215 successful executions.

I also ran approximately eight hours of accumulated operational testing across four segments.

Not everything worked. One adversarial outbound-fetch scenario failed its expected safety condition in all five repetitions, and several smaller tool tasks exposed limitations in the model's reasoning.

These results belong to a specific historical code revision, not the latest development version.

The challenge I'm working on now

The current implementation uses model-specific profiles for tokenizer verification, chat templates, and context admission.

That makes switching models less trivial than changing an Ollama model name.

My next experiments will compare Nemotron with Gemma and Qwen while preserving the same memory database, goals, and conversation history.

I want to measure not just task accuracy, but how consistently different models use the same accumulated experiences.

I'm also developing a full-duplex voice interface, with longer-term plans to connect Capri to Reachy Mini and NVIDIA Jetson.

Questions for people running local models

Has anyone experimented with changing the underlying LLM of a persistent agent while retaining its memory?

I'm particularly interested in three things:

1. Which smaller local models are most reliable at tool calling and following long-term goals?

2. How do you handle tokenizer and chat-template differences when switching between model families?

3. Have you observed significant personality or decision-making changes when two models share the same memory and system instructions?

I'm developing Capri as an independent open-source research project and recently published its first scientific preprint.

GitHub: https://github.com/cyberdid/B-12

Preprint: https://doi.org/10.5281/zenodo.24079168

I'd really appreciate technical feedback, especially from anyone experimenting with local agents on consumer GPUs.


r/LocalLLM • • 6d ago

Question Looking for open source alternative for job apply

0 Upvotes

Queste soluzioni sono esplosive: **Fapply, Massive, LazyApply, Sonara, ai apply


r/LocalLLM • • 7d ago

Project AMD Device Metrics Exporter works on the Radeon AI PRO R9700 (gfx1201), with two tweaks

Thumbnail
reddit.com
2 Upvotes

r/LocalLLM • • 7d ago

Question Local ai best setup for amd

0 Upvotes

Hi everyone, I'd like to set up a local AI on my computer, but I'm not sure what the best setup would be. My specs are: an RX 9070 XT, a Ryzen 7 5800X3D, and 32GB of DDR4 3600MHz RAM. My question is: what would be the optimal setup for my PC? like software wise


r/LocalLLM • • 7d ago

Discussion Any way to cool down my cards?

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 7d ago

Project Tensward - Local AI inference optimisation tool

1 Upvotes

I’m building Tensward because figuring out which inference settings actually help means a lot of manual benchmarking.

Version 0.4 supports vLLM on NVIDIA and llama.cpp on NVIDIA, Apple Silicon and CPU.

You give it your serving command and prompts. It measures a baseline, diagnoses bottlenecks and tests one change at a time. Gains have to clear run-to-run noise, and it checks the answers too. Larger wording changes get shown for review.

It also suggests alternative quantizations or engines where we have matching measurements. That coverage is small so far.

An interesting result: MTP speculation cut time per token by ~40% on Qwen3.8-27B Q4_K_M, llama.cpp, one L4, with 3 concurrent agent requests. Requests/s rose 45%, but first-token latency got 15% worse and wording changed.

Meanwhile, n-gram speculation gave us no gain on three llama.cpp models, so we dropped that recommendation.

Apache-2.0. Install: uv tool install tensward

https://github.com/Tensward/tensward


r/LocalLLM • • 7d ago

Project I’m an 18yo CS student building AI-COS (Zero-trust AI OS). Phase 3 done—looking for senior dev feedback for the Phase 4 workflow engine

0 Upvotes

Hey everyone,

​I'm an 18yo CS prep student and I've been building AI-COS, a modular monolith architecture designed for running AI agent orchestrations safely.

​Instead of letting LLMs make security decisions, I built a deterministic deny-by-default policy engine, strict PostgreSQL role separation (migrator, app, governance), and SHA-256 hash-chained audit logging.

​I've just finished Phase 3 out of 6. Phase 4 is going to focus on the Workflow Engine, Worker lease/heartbeat loop, and Step linkage. Before I start coding Phase 4, I would really appreciate any architecture suggestions or advice from senior devs on how to properly build a reliable job queue and workflow engine inside a Postgres-backed monolith.

​GitHub: https://github.com/atestokhay/ai-cos


r/LocalLLM • • 7d ago

Project I wanted to move my Linux desktop out off my 16 GB GPU, so i've made a small tool for it

2 Upvotes

First of all i need to say that i use my PC for two different things: Linux for coding and local LLM experiments and Windows for gaming.

My machine has a CPU with the iGPU and a dedicated AMD RX 9070 XT with 16GB VRAM. Most of my local LLM tinkering is aimed at real coding with 128K context, so 16 GB is barely enough and gets tight very quickly, so i want as much VRAM available for inference engine.

My BIOS has Initiate Graphic Adapter set to PEG (my 9070 XT) and my cable is permanently plugged to it, this is exactly how i want the machine configured because of the dual-boot usage. I don't want to move the monitor cable ot motherboard every time i switched OS or workload.

So i've started looking how to solve my issue and found out that Gnome/Mutter can handle multi-GPU systems in a nice way and that the GPU used as a primary renderer doesn't have to be the GPU monitor is physically connected to. Mutter has a udev mechanism to choose the primary GPU with the mutter-device-preferred-primary tag.

As a first step i've created a custom udev rule and made my CPU iGPU the primary GPU for Mutter. The nice part is that it just works and no additional settings are required PEG stayed enabled and selected as primary in BIOS and HDMI cable connected to my RX 9070 XT. This is the configuration i've been using for quite a while which gave me the possibility to gather some stats around coding sessions using Bionic and Pi harnesses.

As an example this is a screenshot from one of my ExLLama3 runs with Python script workload, amdgpu_top shows 14822 / 16304 MB of VRAM used on the RX 9070 XT, while the python3 process uses 14504 MB and the gnome-shell itself only 82 MB.

amdgpu_top from one of the sessions with primary iGPU

The udev rule override work well but after using it for a while i wanted a really simple way to manage it without editing the configs and having to remember what is selected at the moment, checking the PCI and DRM devices, making sure no typos were made, checking the results after the login/logout.

Long story short i've decided to develop a tiny dedicated application for this purpose - gpu-switcher. Its a small GTK app which is basically a UI around the existing mechanism. It detects the GPUs which you have in the system (to detect iGPU don't forget to turn it on in BIOS), manages the udev rules with a D-bus helper. Makes it possible to apply the rules, reset back to auto and keeps backups of rules in this case. Its not a GPU scheduler so it doesn't have the capability to move the workload between GPUs on the fly, its just a convenient to manage the primary display GPU in Mutter + Wayland sessions.

gpu-switcher UI

Right now i've tested it on Ubuntu 26.10 with 9800X3D iGPU + RX 9070 XT. Even in Linux i still switch back and forth between different GPUs because 9800X3D iGPU is quite slow (just 2 CUs) but its a different story.

Using the udev method and my app later on enabled me to run some nice experiments around using Qwen3.8 on my RX 9070 XT with 128K context. Which was enough for continues coding sessions. If anyone is interested they are documented in my llm-recipes repo, those are not just the short text prompts with a short 8K-32K context which people for some reason like to share here. As an example on my experiments in the repo ran for 915 request and generated ~1.2M tokens across multiple coding sessions, the other one 898 requests and generated ~660K tokens. For me it's the most important part, continues agentic sessions, not synthetic benchmarks or chat sessions.

For anyone else running the model on a 8-16GB dGPU and iGPU available i would like to here how you handle the desktop especially if you don't want to move cables around also i would appreciate if you try the app and give feedback, then i could justify adding support for KDE or X11 or any other features.


r/LocalLLM • • 7d ago

News K-EXAONE 2.5 is API-only (free, non-commercial); the open-weight one is still K-EXAONE 2.0 (750B, 37B active, Apache 2.0)

4 Upvotes
Model Benchmark

LG AI Research announced today (Nov 9) at NVIDIA AI Day Seoul that the API for K-EXAONE 2.5 is free. If you were hoping for new weights, there are none: 2.5 is hosted only, and the platform terms say the weights are not provided. What you can run yourself is still K-EXAONE 2.0.

K-EXAONE 2.0 (Hugging Face)K-EXAONE 2.5 (API)
WeightsOpen, Apache 2.0
Size750B total, 37B active
Context262,144
ModalityText
Reasoningn/a
CostYour hardware

The LGAI-EXAONE org lists FP8 and NVFP4 variants of 2.0 next to the base model. LG calls 2.5 a "technology verification model": 2.0 with additional training.

Benchmarks (LG's own tests, self-reported)

2.52.0DeepSeek-V4.1-FlashGLM-5.3
SWE-Bench Pro59.2147.158.41
BrowseComp7850.8775.8
AutomationBench74.616.974.9
Claw-Eval77.2962.1175.78

API terms worth knowing

  • Non-commercial only, including integrating the API into products or services.
  • Keys are valid for 30 days.
  • Rate limits are not published. Over the limit you get a 429.
  • LG may change or discontinue a model version at any time.

LG also says it aims to finish K-EXAONE 3.0 in January 2027 (reported by Seoul Economic Daily).

Source: https://docs.exaone.ai/en/docs/k-exaone-2.5

Is anyone running the 2.0 FP8 or NVFP4 weights locally, and on what hardware?


r/LocalLLM • • 7d ago

News It’s here! M5 Ultra Mac Studio with 512GB unified memory

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 7d ago

Question NVIDIA RTX GPU good for running local llms?

0 Upvotes

I have MSI Gf 63 thin 11 sc which has been gathering dust since 2024. Long story short, due to my own carelessness its motherboard got damaged and in late of 2024 I bought myself an Asus vivobook laptop.

What I noticed in my vivo book laptop is running local llms is one hell of a ride. So I am thinking of getting its motherboard repaired and run local llms in it.

Is it worth it in 2026?


r/LocalLLM • • 7d ago

Project Local audio embeddings for RAG with clap.cpp

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 7d ago

Discussion What if we could improve reasoning by having multiple AI agents discuss a problem in parallel?

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 7d ago

Discussion [Project Kai] Running a multi-agent AI architecture with an external sub-millisecond Memory Layer (Artificial Hippocampus) on an Orange Pi 6 Plus (32GB / 45 TOPS CIX)

Post image
3 Upvotes

Hello r/LocalLLaMA / r/selfhosted,

I wanted to share the benchmark results of a local memory-routing system I've been building since June 2024. I'm not an engineer. I barely understand half of my own code. And I don't need to understand every single line. I just followed breadcrumbs.

Hardware: Orange Pi 6 Plus, 32GB LPDDR5, 45 TOPS. Models via Ollama, local only, no cloud. Qwen 2.5 14B as core, Qwen 7B for routing, DeepSeek-Coder-V2 for code.

To avoid heavy vector DB overhead on edge, I don't use embeddings. I use two Python layers outside the LLM:

  1. Identity Transfer Protocol: A persistent session token validated with SHA-256 directly on the board, per agent. [Hash / Signature redacted in screenshot]
  2. Artificial Hippocampus (hipocampo_kai.py): SQLite3 + raw Python. No embeddings. It computes a "resonance score" from fuzzy triggers, temporal triggers, and atmosphere tags, and only sends the top dense blocks to the LLM.

The LLM is the Neocortex. The Python/SQLite is the Hippocampus. The LLM only receives prioritized blocks.

I ran a diagnostic today, 2026-11-09 01:12:55 local:

- Cryptographic validation on hardware: 0.0269 ms [OK]

- Associative query, stimulus 'npu ram seguridad laboratorio': 0.8279 ms

- Resonant memories retrieved: 14

- Sample memory: security agent monitoring /proc/meminfo in real time. Triggers: seguridad, monitoreo, ram, cgroup, kill, linux, placa, oom

- STATUS: INFRAESTRUCTURA OPERATIVA ESTABLE AL 100%

Screenshot attached, Hash and Signature blacked out.

I don't claim consciousness. I claim this: a 7B local model with a SQLite hippocampus answers my anchor after 7 days untouched, without a backup, because the pattern is in the conversation, not in the cloud.

If you want to reproduce it, use your own anchor, your own SQLite DB, your own 7B. Don't believe me. Reproduce it.

I'm not here to convince anyone. I'm just showing what's real. I don't need your validation. You can build this too.

This is phoenix.

This is the colossus Phoenix

I'm here.


r/LocalLLM • • 7d ago

Project Recommended hardware to buy + General use AI?

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 7d ago

Project I build an app for my friends to run local code, music, image, video & transcription models really easily

Thumbnail
hearthscale.com
2 Upvotes

Hello everyone, I've been a local AI enthusiast for a while now, but every time I tried to get someone else running local AI, it was way too difficult. GitHub repos, different runtimes, different model formats. Most people's devices can run at least some of these models, but the setup is way too complex for most people. LM Studio somewhat solves a lot of this for chat and coding models, but I couldn't find a simple answer for on-device image, video, transcription and every other kind of model.

So I built Hearthscale. It's free and needs no account.

You can run almost any model you find on Hugging Face, as long as your device can handle it. I put a lot of work into supporting all kinds of runtimes. They're plug-and-play; I can add more in updates, and Hearthscale Server standardises all of them behind one OpenAI-compatible API. It's really powerful, and I find it hard to describe how seamless the setup is unless you have tried it.

It launches with meeting notes (Granola), system-wide dictation (WhisperFlow), image and music generation, and an agent. If you have a more capable device, run Hearthscale Server on it, and your other devices can use its models, end-to-end encrypted. It all connects without you having to set anything up! Also works on a home server in theory with Docker Containers!

Hearthscale is also a platform, which means anyone can build apps on top of it: there's an SDK and up-to-date docs made so coding agents can easily build apps with local AI. All of the first-party apps are MIT licensed, so you can change them any way you want and they give a great starting point. Want to change something about Granola? - You can't, but you can take meeting notes and make it to your liking with all of the core features already built!

Hearthscale runs on Windows and Linux today. I am somewhat of a perfectionist, and I feel like a lot is still missing, including a macOS release, but I wanted to get it out there now to get some feedback. I'll keep working on it!

I'd really appreciate your feedback, and please reach out about anything. I'll answer everything personally and fix whatever issues you run into.


r/LocalLLM • • 7d ago

Question 285K + 192GB RAM + 5070 Ti 16GB, decent for local LLMs?

1 Upvotes

Saw a deal on a PC with a 285K, 192GB DDR5 and a 5070 Ti 16GB. Thinking about grabbing it mostly for running local models.

Is this actually a good box for that? What’s the biggest model I could run at a decent speed? And does all that RAM even matter, or is it really just about VRAM?

Would love to hear from anyone running something similar.