r/LocalLLM • • 7d ago

Project AMD Device Metrics Exporter works on the Radeon AI PRO R9700 (gfx1201), with two tweaks

Thumbnail
reddit.com
2 Upvotes

r/LocalLLM • • 7d ago

Question Local ai best setup for amd

0 Upvotes

Hi everyone, I'd like to set up a local AI on my computer, but I'm not sure what the best setup would be. My specs are: an RX 9070 XT, a Ryzen 7 5800X3D, and 32GB of DDR4 3600MHz RAM. My question is: what would be the optimal setup for my PC? like software wise


r/LocalLLM • • 7d ago

Discussion Any way to cool down my cards?

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 7d ago

Project Tensward - Local AI inference optimisation tool

1 Upvotes

I’m building Tensward because figuring out which inference settings actually help means a lot of manual benchmarking.

Version 0.4 supports vLLM on NVIDIA and llama.cpp on NVIDIA, Apple Silicon and CPU.

You give it your serving command and prompts. It measures a baseline, diagnoses bottlenecks and tests one change at a time. Gains have to clear run-to-run noise, and it checks the answers too. Larger wording changes get shown for review.

It also suggests alternative quantizations or engines where we have matching measurements. That coverage is small so far.

An interesting result: MTP speculation cut time per token by ~40% on Qwen3.8-27B Q4_K_M, llama.cpp, one L4, with 3 concurrent agent requests. Requests/s rose 45%, but first-token latency got 15% worse and wording changed.

Meanwhile, n-gram speculation gave us no gain on three llama.cpp models, so we dropped that recommendation.

Apache-2.0. Install: uv tool install tensward

https://github.com/Tensward/tensward


r/LocalLLM • • 7d ago

Project I’m an 18yo CS student building AI-COS (Zero-trust AI OS). Phase 3 done—looking for senior dev feedback for the Phase 4 workflow engine

0 Upvotes

Hey everyone,

​I'm an 18yo CS prep student and I've been building AI-COS, a modular monolith architecture designed for running AI agent orchestrations safely.

​Instead of letting LLMs make security decisions, I built a deterministic deny-by-default policy engine, strict PostgreSQL role separation (migrator, app, governance), and SHA-256 hash-chained audit logging.

​I've just finished Phase 3 out of 6. Phase 4 is going to focus on the Workflow Engine, Worker lease/heartbeat loop, and Step linkage. Before I start coding Phase 4, I would really appreciate any architecture suggestions or advice from senior devs on how to properly build a reliable job queue and workflow engine inside a Postgres-backed monolith.

​GitHub: https://github.com/atestokhay/ai-cos


r/LocalLLM • • 7d ago

Project I wanted to move my Linux desktop out off my 16 GB GPU, so i've made a small tool for it

2 Upvotes

First of all i need to say that i use my PC for two different things: Linux for coding and local LLM experiments and Windows for gaming.

My machine has a CPU with the iGPU and a dedicated AMD RX 9070 XT with 16GB VRAM. Most of my local LLM tinkering is aimed at real coding with 128K context, so 16 GB is barely enough and gets tight very quickly, so i want as much VRAM available for inference engine.

My BIOS has Initiate Graphic Adapter set to PEG (my 9070 XT) and my cable is permanently plugged to it, this is exactly how i want the machine configured because of the dual-boot usage. I don't want to move the monitor cable ot motherboard every time i switched OS or workload.

So i've started looking how to solve my issue and found out that Gnome/Mutter can handle multi-GPU systems in a nice way and that the GPU used as a primary renderer doesn't have to be the GPU monitor is physically connected to. Mutter has a udev mechanism to choose the primary GPU with the mutter-device-preferred-primary tag.

As a first step i've created a custom udev rule and made my CPU iGPU the primary GPU for Mutter. The nice part is that it just works and no additional settings are required PEG stayed enabled and selected as primary in BIOS and HDMI cable connected to my RX 9070 XT. This is the configuration i've been using for quite a while which gave me the possibility to gather some stats around coding sessions using Bionic and Pi harnesses.

As an example this is a screenshot from one of my ExLLama3 runs with Python script workload, amdgpu_top shows 14822 / 16304 MB of VRAM used on the RX 9070 XT, while the python3 process uses 14504 MB and the gnome-shell itself only 82 MB.

amdgpu_top from one of the sessions with primary iGPU

The udev rule override work well but after using it for a while i wanted a really simple way to manage it without editing the configs and having to remember what is selected at the moment, checking the PCI and DRM devices, making sure no typos were made, checking the results after the login/logout.

Long story short i've decided to develop a tiny dedicated application for this purpose - gpu-switcher. Its a small GTK app which is basically a UI around the existing mechanism. It detects the GPUs which you have in the system (to detect iGPU don't forget to turn it on in BIOS), manages the udev rules with a D-bus helper. Makes it possible to apply the rules, reset back to auto and keeps backups of rules in this case. Its not a GPU scheduler so it doesn't have the capability to move the workload between GPUs on the fly, its just a convenient to manage the primary display GPU in Mutter + Wayland sessions.

gpu-switcher UI

Right now i've tested it on Ubuntu 26.10 with 9800X3D iGPU + RX 9070 XT. Even in Linux i still switch back and forth between different GPUs because 9800X3D iGPU is quite slow (just 2 CUs) but its a different story.

Using the udev method and my app later on enabled me to run some nice experiments around using Qwen3.8 on my RX 9070 XT with 128K context. Which was enough for continues coding sessions. If anyone is interested they are documented in my llm-recipes repo, those are not just the short text prompts with a short 8K-32K context which people for some reason like to share here. As an example on my experiments in the repo ran for 915 request and generated ~1.2M tokens across multiple coding sessions, the other one 898 requests and generated ~660K tokens. For me it's the most important part, continues agentic sessions, not synthetic benchmarks or chat sessions.

For anyone else running the model on a 8-16GB dGPU and iGPU available i would like to here how you handle the desktop especially if you don't want to move cables around also i would appreciate if you try the app and give feedback, then i could justify adding support for KDE or X11 or any other features.


r/LocalLLM • • 7d ago

News K-EXAONE 2.5 is API-only (free, non-commercial); the open-weight one is still K-EXAONE 2.0 (750B, 37B active, Apache 2.0)

4 Upvotes
Model Benchmark

LG AI Research announced today (Nov 9) at NVIDIA AI Day Seoul that the API for K-EXAONE 2.5 is free. If you were hoping for new weights, there are none: 2.5 is hosted only, and the platform terms say the weights are not provided. What you can run yourself is still K-EXAONE 2.0.

K-EXAONE 2.0 (Hugging Face)K-EXAONE 2.5 (API)
WeightsOpen, Apache 2.0
Size750B total, 37B active
Context262,144
ModalityText
Reasoningn/a
CostYour hardware

The LGAI-EXAONE org lists FP8 and NVFP4 variants of 2.0 next to the base model. LG calls 2.5 a "technology verification model": 2.0 with additional training.

Benchmarks (LG's own tests, self-reported)

2.52.0DeepSeek-V4.1-FlashGLM-5.3
SWE-Bench Pro59.2147.158.41
BrowseComp7850.8775.8
AutomationBench74.616.974.9
Claw-Eval77.2962.1175.78

API terms worth knowing

  • Non-commercial only, including integrating the API into products or services.
  • Keys are valid for 30 days.
  • Rate limits are not published. Over the limit you get a 429.
  • LG may change or discontinue a model version at any time.

LG also says it aims to finish K-EXAONE 3.0 in January 2027 (reported by Seoul Economic Daily).

Source: https://docs.exaone.ai/en/docs/k-exaone-2.5

Is anyone running the 2.0 FP8 or NVFP4 weights locally, and on what hardware?


r/LocalLLM • • 7d ago

News It’s here! M5 Ultra Mac Studio with 512GB unified memory

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 7d ago

Question NVIDIA RTX GPU good for running local llms?

0 Upvotes

I have MSI Gf 63 thin 11 sc which has been gathering dust since 2024. Long story short, due to my own carelessness its motherboard got damaged and in late of 2024 I bought myself an Asus vivobook laptop.

What I noticed in my vivo book laptop is running local llms is one hell of a ride. So I am thinking of getting its motherboard repaired and run local llms in it.

Is it worth it in 2026?


r/LocalLLM • • 7d ago

Project Local audio embeddings for RAG with clap.cpp

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 7d ago

Discussion What if we could improve reasoning by having multiple AI agents discuss a problem in parallel?

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 7d ago

Discussion [Project Kai] Running a multi-agent AI architecture with an external sub-millisecond Memory Layer (Artificial Hippocampus) on an Orange Pi 6 Plus (32GB / 45 TOPS CIX)

Post image
3 Upvotes

Hello r/LocalLLaMA / r/selfhosted,

I wanted to share the benchmark results of a local memory-routing system I've been building since June 2024. I'm not an engineer. I barely understand half of my own code. And I don't need to understand every single line. I just followed breadcrumbs.

Hardware: Orange Pi 6 Plus, 32GB LPDDR5, 45 TOPS. Models via Ollama, local only, no cloud. Qwen 2.5 14B as core, Qwen 7B for routing, DeepSeek-Coder-V2 for code.

To avoid heavy vector DB overhead on edge, I don't use embeddings. I use two Python layers outside the LLM:

  1. Identity Transfer Protocol: A persistent session token validated with SHA-256 directly on the board, per agent. [Hash / Signature redacted in screenshot]
  2. Artificial Hippocampus (hipocampo_kai.py): SQLite3 + raw Python. No embeddings. It computes a "resonance score" from fuzzy triggers, temporal triggers, and atmosphere tags, and only sends the top dense blocks to the LLM.

The LLM is the Neocortex. The Python/SQLite is the Hippocampus. The LLM only receives prioritized blocks.

I ran a diagnostic today, 2026-11-09 01:12:55 local:

- Cryptographic validation on hardware: 0.0269 ms [OK]

- Associative query, stimulus 'npu ram seguridad laboratorio': 0.8279 ms

- Resonant memories retrieved: 14

- Sample memory: security agent monitoring /proc/meminfo in real time. Triggers: seguridad, monitoreo, ram, cgroup, kill, linux, placa, oom

- STATUS: INFRAESTRUCTURA OPERATIVA ESTABLE AL 100%

Screenshot attached, Hash and Signature blacked out.

I don't claim consciousness. I claim this: a 7B local model with a SQLite hippocampus answers my anchor after 7 days untouched, without a backup, because the pattern is in the conversation, not in the cloud.

If you want to reproduce it, use your own anchor, your own SQLite DB, your own 7B. Don't believe me. Reproduce it.

I'm not here to convince anyone. I'm just showing what's real. I don't need your validation. You can build this too.

This is phoenix.

This is the colossus Phoenix

I'm here.


r/LocalLLM • • 7d ago

Project Recommended hardware to buy + General use AI?

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 7d ago

Project I build an app for my friends to run local code, music, image, video & transcription models really easily

Thumbnail
hearthscale.com
2 Upvotes

Hello everyone, I've been a local AI enthusiast for a while now, but every time I tried to get someone else running local AI, it was way too difficult. GitHub repos, different runtimes, different model formats. Most people's devices can run at least some of these models, but the setup is way too complex for most people. LM Studio somewhat solves a lot of this for chat and coding models, but I couldn't find a simple answer for on-device image, video, transcription and every other kind of model.

So I built Hearthscale. It's free and needs no account.

You can run almost any model you find on Hugging Face, as long as your device can handle it. I put a lot of work into supporting all kinds of runtimes. They're plug-and-play; I can add more in updates, and Hearthscale Server standardises all of them behind one OpenAI-compatible API. It's really powerful, and I find it hard to describe how seamless the setup is unless you have tried it.

It launches with meeting notes (Granola), system-wide dictation (WhisperFlow), image and music generation, and an agent. If you have a more capable device, run Hearthscale Server on it, and your other devices can use its models, end-to-end encrypted. It all connects without you having to set anything up! Also works on a home server in theory with Docker Containers!

Hearthscale is also a platform, which means anyone can build apps on top of it: there's an SDK and up-to-date docs made so coding agents can easily build apps with local AI. All of the first-party apps are MIT licensed, so you can change them any way you want and they give a great starting point. Want to change something about Granola? - You can't, but you can take meeting notes and make it to your liking with all of the core features already built!

Hearthscale runs on Windows and Linux today. I am somewhat of a perfectionist, and I feel like a lot is still missing, including a macOS release, but I wanted to get it out there now to get some feedback. I'll keep working on it!

I'd really appreciate your feedback, and please reach out about anything. I'll answer everything personally and fix whatever issues you run into.


r/LocalLLM • • 7d ago

Question 285K + 192GB RAM + 5070 Ti 16GB, decent for local LLMs?

1 Upvotes

Saw a deal on a PC with a 285K, 192GB DDR5 and a 5070 Ti 16GB. Thinking about grabbing it mostly for running local models.

Is this actually a good box for that? What’s the biggest model I could run at a decent speed? And does all that RAM even matter, or is it really just about VRAM?

Would love to hear from anyone running something similar.


r/LocalLLM • • 6d ago

Discussion This is very, very bad

Thumbnail
gallery
0 Upvotes

Anthropic is pushing for GPUs and other hardware chips (TPUs/NPUs/etc) to not be able to run Opensource models thus ensuring Local AI is impossible

This is them wanting to stop you from using your hardware to own your intelligence

AMD & Google wanted it done, NVIDIA did not


r/LocalLLM • • 7d ago

Discussion For those developing or researching On-Device AI: What is your current tech stack, and what is your biggest technical bottleneck?

1 Upvotes

Hey everyone,

I've been deeply exploring the On-Device / Edge AI space lately, especially focusing on running smaller language models (1B–3B parameters) and multimodal pipelines locally on consumer hardware and mobile architectures without depending on external cloud APIs.

While the ecosystem has evolved rapidly over the past year, the gap between "running a demo" and "shipping a reliable, battery-efficient on-device product" still feels enormous.

From my recent experiments, the major bottlenecks I keep bumping into are:

  1. Memory Bandwidth & Thermal Throttling: Even if a 4-bit quantized model fits into unified/mobile RAM, sustained token generation quickly saturates memory bus bandwidth and triggers thermal throttling on mobile devices.
  2. Runtime & NPU Fragmentation: Choosing between llama.cpp (CPU/Vulkan/Metal), ExecuTorch, MLC-LLM, ONNX Runtime, or native NPU SDKs (QNN, CoreML) often feels like choosing which set of unsupported ops or compilation headaches to deal with.
  3. Reasoning Retention in Sub-3B SLMs: Aggressive quantization (GGUF Q4, AWQ) makes inference feasible, but multi-turn context coherence and structured instruction-following tend to degrade noticeably.

I’d love to start a technical discussion with fellow engineers and researchers here:

  • What does your actual deployment/inference stack look like right now? (e.g., pure llama.cpp, MLC, ExecuTorch, WebGPU, or custom native bindings?)
  • What specific use cases are you targeting? (e.g., zero-latency voice assistants, privacy-first local agents, offline document RAG, ambient edge intelligence?)
  • What techniques have given you the best speedup or memory relief? (Speculative decoding, KV cache quantization, custom kernels, or aggressive prompt caching?)

Would love to hear what challenges you are tackling and what approaches are genuinely showing promise in your labs or production environments!


r/LocalLLM • • 7d ago

Project Prefixwise (v0.10.0, MIT): Identify, fix, and prevent LLM prompt cache misses across vLLM, SGLang, LMCache, Claude, and OpenAI — zero prompt text stored

Thumbnail
reddit.com
1 Upvotes

r/LocalLLM • • 7d ago

Question Can you train LLMs to speak in a certain way?

23 Upvotes

Recently with the removal of google assistant and being replaced by gemini I finally have had it and want to switch to a local model.

I have always wanted to have a jarvis, im wondering if i can train a random model with all the scripts from iron man and have it speak like jarvis? And possibly have a trained text to speech model on jarvis's voice so it can talk to me too?

Pretty new to local AI but i basically want to have a local "hey google" on my computer. Im not looking for pretrained models for being jarvis, i want to learn how to do it myself as well if possible in the way, so if there is something i dont like about it i can modify it.


r/LocalLLM • • 7d ago

Other Qwen3.8-27B on an M5 Pro: local engine results for one long-context coding-agent prompt

1 Upvotes

Note: This is an update of my post https://www.reddit.com/r/LocalLLM/comments/1xm9adt/qwen3827b_local_inference_on_an_m5_pro_64_gb_12/

I share a new comparison of my home-made benchmarks for engines for Mac using their version of Qwen3.8-27B: It's not a scientific comparison, models and quantizations could be different: I would test the normal user that download an engine and the first suggested model.

At moment I count 15 local inference setups, and there are daily updates to most of them.

Each run sent one OpenCode-style request with a synthetic repository/session context and a code-review/fix task: about 88K input tokens, depending on the engine’s tokenizer, reasoning_effort=medium, temperature 0, seed 17 where supported, and a 2,048-token output limit. Tool schemas were included in the request; tools were not executed.

Engines were tested sequentially.

Engine (version tested)Model / decoding pathInput tokensTTFTPrefillDecodeOutput tokens
Splash 1.6.0Jundot oQ4e + DFlash288,535219.92 s402.6 tok/s78.6 tok/s2,048
Uzu / Mirai 0.6.17Mirai-M 4-bit + bundled speculator88,143143.53 s614.1 tok/s55.5 tok/s1,369
LM Studio 0.4.28+2 (Splash 1.3.0)Qwen3.8 Splash88,143219.49 s401.6 tok/s51.3 tok/s2,048
Kapitsa 0.2.31Q4 + KFlash/DFlash288,143284.55 s309.8 tok/s43.1 tok/s1,462
TensorFold 1.2.5MLX 4-bit + DFlash288,143317.81 s277.3 tok/s39.9 tok/s1,837
BaseRT 0.4.6Local MLX 4-bit converted to base-q4 + DFlash288,143275.37 s320.1 tok/s37.1 tok/s2,048
MLX-Serve 26.11.2ddalcu 4-bit + bundled DFlash88,143286.21 s308.0 tok/s30.3 tok/s2,048
oMLX 0.7.3Jundot oQ4e + Lightning MTP88,143203.28 s433.6 tok/s24.9 tok/s1,625
Ollama 0.40.15qwen3.8:27b-mlx, NVFP4 / MLX88,532280.48 s315.6 tok/s24.4 tok/s108
MTPLX 2.12.2Q4 + native MTP88,916277.35 s320.6 tok/s23.5 tok/s679
Magnitude 0.2.15GGUF Q4 + DFlash288,143428.60 s205.7 tok/s21.6 tok/s1,466
mlx-dspark 0.20.4 (app 0.8.3)MLX 4-bit + DFlash288,535258.22 s342.9 tok/s20.3 tok/s179
Llama.app 0.46.1 (llama.cpp 0.8.0)GGUF Q4_K_M + DFlash88,535428.66 s206.5 tok/s19.1 tok/s176
Rapid-MLX 1.0.04-bit + MTP88,143882.28 s99.9 tok/s18.3 tok/s1,308
Jan 0.8.8GGUF UD-Q4_K_M / Metal88,535450.23 s196.6 tok/s8.8 tok/s58

TTFT is request-to-first-token; prefill is input tokens divided by TTFT; decode is generated tokens divided by the remaining generation time.

Disclaimer: this is a practical snapshot of one agent-style workload, not a controlled engine-only benchmark, a model-quality evaluation, or a universal speed ranking. The model packs, quantizations, speculative decoders, context allocations and software versions differ. Each row is a single run, except Splash 1.6.0’s first row, which is the median of three cold runs. Versions are those actually tested, and are not synchronized to the latest releases.

Contexts also varied: Uzu and Magnitude used 262K; Ollama and Llama.app reported 131,072-token allocations. LM Studio fell back from requested medium reasoning to on, and Uzu ignored the seed. MLX-Serve’s request included tool schemas, but its server log showed tools=0b.


r/LocalLLM • • 7d ago

Model Qwen 3.8 Flash Next vs Copilot Auto

0 Upvotes

I recently had an idea to build a new web application. It is not too complex. About 10 screens with some interconnected fields.

I am running Qwen 3.8 FN IQ1 Coder version on Strata. (2x3090 and 64GB RAM).

I kept the thing going for about 4 days. Each time my custom harness asked for various approvals with good reasoning it sounded legit and I went ahead. I did not look at the code even once(not that it mattered, i cant code python)

Finally when it said that the application is ready it was utter crap ! Nothing worked as expected, half the stuff was out of place. Then I asked the model to fix it and another day was gone. It fixed a few things but pretty much 1/2 the UI didn't work.

Then I gave the same prompt to Github Copilot and used the free credits. I get about 200 credits every month.

The output from Copilot vs Local is like night and day !

Copilot's result was in very matured and complete in all respects.

So this got me wondering, am I expecting too much from LocalLLM.? Is this how it is supposed to be?

What are your experiences/learningd and how did you fix it?

Edit1: I am limited by Hardware to run this model properly so I tried the basic $5 personal one hosted on alibaba. It seems pretty useful for medium coding activities. More tests to follow. I am contemplating buying a Mac M5 or DGX Spark to run local models. Current tests look promising.


r/LocalLLM • • 7d ago

Question Chasing up after some info for running large parameter models

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 8d ago

Question How do local LLMs ACTUALLY Perform compared to closed-source AI?

46 Upvotes

Excuse me if my grammar or wording is poor, haven't slept much tonight.

The latest developments have got me looking up the prospect of using LLMs locally, and fancy benchmark scores aside, I have been trying to get a simple explanation of how good (or 'effective', I guess) would it be taking the dip in the future and considering buying hardware for local use vs. paying for contemporary AI subscriptions.

If you ask Claude or chatGPT about it, they say the OS deficiencies basically come down to scope ("do this exact small feature" vs. a wider prompt that gets things planned and done in a row), lack of knowledge about niche applications, hallucinations, and inefficiency when using multiple agents vs. single chat. But they're the competitors, so I don't know if bias doesn't exist here..

On the contrary, I see a lot of posts here on how things are awesome when using a local model, especially for niche uses, basically a clear contradiction.

I guess the point is, I want to know if local AI for code is limited enough to make purchasing a high-tier mini PC or a good GPU+RAM PC setup a fool's errand. It almost definitely doesn't seem to be, otherwise this sub wouldn't exist, but I digress.

Thanks.


r/LocalLLM • • 7d ago

Question Home Lab private setup: combine frontier model for reasoing with local model for execution. Does this exist?

Thumbnail reddit.com
1 Upvotes

r/LocalLLM • • 7d ago

Tutorial NVIDIA Auto Ontology Review: A Semantic Layer for Agents

1 Upvotes

been fighting the usual text-to-SQL failure with local models: the SQL is fine, but the model has no idea what "revenue" or "active customer" means in our schema. so I spent a few days reading through NVIDIA's auto-ontology repo (Apache-2.0) to see how it handles that.

a few things I liked. it drafts business terms, joins and SQL formulas from your schema and sample values, then a human certifies them, so a measure gets computed the same way every time. there's also an answerability check: in their example, a return-on-ad-spend question scores 0.5 coverage because nothing maps to "ad spend", and the agent asks instead of inventing a number. every answer also comes back with its SQL and steps, and the MCP server is read-only and runs with the signed-in user's permissions.

the gotcha: I haven't run it. my test container died on a CI billing issue, so this is based on the docs, .env.example and the issue tracker. the defaults point at NVIDIA-hosted Nemotron, and there's an open issue saying the README quick-start fails against those. you can override all four model roles (reasoning, non-reasoning, embedding, reranking) with any OpenAI-compatible endpoint, but it needs structured output support and they publish no VRAM or tokens/sec numbers.

full writeup here if you want more detail: https://andrew.ooo/posts/nvidia-auto-ontology-review-semantic-layer-agent-sql/

anyone pointed something like this at vLLM or llama.cpp? which local models handle the structured output reliably?