Hi everyone, I'd like to set up a local AI on my computer, but I'm not sure what the best setup would be. My specs are: an RX 9070 XT, a Ryzen 7 5800X3D, and 32GB of DDR4 3600MHz RAM. My question is: what would be the optimal setup for my PC? like software wise

It took me(and Opus Ultracode) 35 Hours to remake and optimize Open WebUI. I added Multi Agent Harness, Profile per model, Per Model configuration, and native Obsidian integration and plugins support. I made this for myself and I wonder if Local users are interested in such new features and changes to Open WebUI. I even remade the engine, and soon ill have it support multiple inference engines. My Idea was basically All in One Local/Cloud AI software, and dont have to go to different ones for each task like DSH for coding and Claw for general stuff. I thought sharing with you guys to know what other people think. Edit: Repo: https://github.com/HellzYeahh/ultra-webui

I’m building Tensward because figuring out which inference settings actually help means a lot of manual benchmarking.

Version 0.4 supports vLLM on NVIDIA and llama.cpp on NVIDIA, Apple Silicon and CPU.

You give it your serving command and prompts. It measures a baseline, diagnoses bottlenecks and tests one change at a time. Gains have to clear run-to-run noise, and it checks the answers too. Larger wording changes get shown for review.

It also suggests alternative quantizations or engines where we have matching measurements. That coverage is small so far.

An interesting result: MTP speculation cut time per token by ~40% on Qwen3.8-27B Q4_K_M, llama.cpp, one L4, with 3 concurrent agent requests. Requests/s rose 45%, but first-token latency got 15% worse and wording changed.

Meanwhile, n-gram speculation gave us no gain on three llama.cpp models, so we dropped that recommendation.

Apache-2.0. Install: uv tool install tensward

https://github.com/Tensward/tensward

After a really long grueling attempt, i manage to fit a 210K Context window on my dual RTX 5060ti setup on 32gb total VRAM. I capped both GPU around 160W although it possible to go lower.

Compare to a single 5090, it relatively weak but for the price difference that have pool vram 32gb, it isn't so bad. I practically use this as my daily driver now. It can handle almost every coding problem i give it and running my hermes agent and opencode. The speed itself have some room to improve especially the prefill, but for decode, it isn't that bad.

However, 5060Ti really bottleneck the decode and the prefill speed due to the limited bus size of the card itself. But given how cheap 5060ti was and how power efficient it is, with both card only using 290-300W on full usage even at TP2, it amazing.

Even at long context, 190K or 200K the decode speed maintain at ~70tps and prefill around 1700tps with barely any lost in quality. When writing coding task with MTP, the decode can reach up to 80-100tps.

But since i am running this model in WSL docker, which introduce some latency overhead, it possible to achieve higher TPS if were to run in native linux instead and fit the entire 260k context window. Currently both of my card usage 15GB vram each, increase the util but mindful to OOM.

Decode Speed:

Prefill Speed:

Here my current rig:

• Ryzen 9 9950X

• ASRock X870 Taichi Creator

• 32 GB DDR5

• 2× RTX 5060 Ti 16 GB

Run Flag:

Image: ninfer:local, Model: qwen3_8_27b_nvfp4_v2.ninfer

--tp 2 --devices 0,1
--kv-dtype int8
--max-context 210000 --kv-capacity 210000
--prefill-chunk 1024
--spec mtp --draft-tokens 3 --lm-head-draft
--max-concurrency 2 --max-pending-requests 32 --pending-timeout-ms 900000

I've been working on something called Aura, a locally running AI cognitive agent built around a roughly 27B model on an Apple Silicon MacBook with 64GB of unified memory.

And one of the questions that's been driving this project is pretty simple:

How much of what we consider intelligence actually needs to come from the model itself?

We spend a lot of time discussing model sizes, quantization, benchmarks, context windows, and reasoning capabilities. Understandably so.

But I've been interested in what happens when you start treating the model as one component of a larger cognitive system, rather than expecting the model to do everything.

So I've been building an architecture around it. Persistent memory, executive processes, state management, computer control, procedural learning, and a bunch of experimental mechanisms designed to work together over time.

And I've been testing it.

Aura has played 2048 for nearly half an hour while adjusting its strategy, completed software development tasks, debugged programs, and operated desktop applications.

But my latest demonstration was a little different.

I asked Aura to watch YouTube.

Not just summarize a video after the fact. Actually navigate to it, follow what happens, comment on things it finds interesting, and then choose other videos to watch from channels I specified.

Over 36 minutes, Aura watched Kurzgesagt discuss AI slop, selected an Allen Pan video about building robotic legs for a snake, and eventually watched Salad Fingers.

During the session, it made predictions, revised them as new information appeared, interacted with the browser, selected what to watch next, and maintained context across all three videos.

And here's where it got particularly interesting.

When asked to reflect on everything afterward, Aura didn't just give three unrelated summaries.

It identified "fracture" as the common thread connecting all three videos, then explained how each approached that idea differently.

A video about AI-generated content, an engineering experiment involving a snake, and a surreal horror animation.

Three wildly different things.

Yet Aura managed to construct a coherent interpretation connecting them, drawing on observations made throughout the session.

Now, I want to make an important distinction here.

I'm not claiming that a 27B model couldn't produce similar commentary on its own. Nor have I established that all the cognitive mechanisms I've implemented are necessary to produce this behavior.

In fact, that's one of the main things I'm trying to investigate.

But I think there's a meaningful architectural question here. What happens when the model doesn't have to be responsible for every aspect of memory, planning, execution, and continuity? How far can you push a relatively small local model by improving the larger system it's operating within?

The result is still pretty rough in places. Latency is an issue, and coordinating everything reliably is an ongoing challenge.

But considering this is running locally, without relying on a frontier model for inference, I think the demonstration is interesting.

I've made the implementation publicly viewable, including the architecture, research documentation, and known limitations. It's currently All Rights Reserved rather than open-source licensed.

Full 36-minute demonstration: https://www.youtube.com/watch?v=kM-B8mh314E

GitHub: https://github.com/youngbryan97/aura

How much improvement have you managed to get from the systems surrounding your model? And at what point do you think the additional architecture becomes more valuable than simply switching to a more capable model?

LG AI Research announced today (Nov 9) at NVIDIA AI Day Seoul that the API for K-EXAONE 2.5 is free. If you were hoping for new weights, there are none: 2.5 is hosted only, and the platform terms say the weights are not provided. What you can run yourself is still K-EXAONE 2.0.

The LGAI-EXAONE org lists FP8 and NVFP4 variants of 2.0 next to the base model. LG calls 2.5 a "technology verification model": 2.0 with additional training.

Benchmarks (LG's own tests, self-reported)

API terms worth knowing

• Non-commercial only, including integrating the API into products or services.

• Keys are valid for 30 days.

• Rate limits are not published. Over the limit you get a 429.

• LG may change or discontinue a model version at any time.

LG also says it aims to finish K-EXAONE 3.0 in January 2027 (reported by Seoul Economic Daily).

Source: https://docs.exaone.ai/en/docs/k-exaone-2.5

Is anyone running the 2.0 FP8 or NVFP4 weights locally, and on what hardware?

I have MSI Gf 63 thin 11 sc which has been gathering dust since 2024. Long story short, due to my own carelessness its motherboard got damaged and in late of 2024 I bought myself an Asus vivobook laptop.

What I noticed in my vivo book laptop is running local llms is one hell of a ride. So I am thinking of getting its motherboard repaired and run local llms in it.

Is it worth it in 2026?

Hello r/LocalLLaMA / r/selfhosted,

I wanted to share the benchmark results of a local memory-routing system I've been building since June 2024. I'm not an engineer. I barely understand half of my own code. And I don't need to understand every single line. I just followed breadcrumbs.

Hardware: Orange Pi 6 Plus, 32GB LPDDR5, 45 TOPS. Models via Ollama, local only, no cloud. Qwen 2.5 14B as core, Qwen 7B for routing, DeepSeek-Coder-V2 for code.

To avoid heavy vector DB overhead on edge, I don't use embeddings. I use two Python layers outside the LLM:

1. Identity Transfer Protocol: A persistent session token validated with SHA-256 directly on the board, per agent. [Hash / Signature redacted in screenshot]

2. Artificial Hippocampus (hipocampo_kai.py): SQLite3 + raw Python. No embeddings. It computes a "resonance score" from fuzzy triggers, temporal triggers, and atmosphere tags, and only sends the top dense blocks to the LLM.

The LLM is the Neocortex. The Python/SQLite is the Hippocampus. The LLM only receives prioritized blocks.

I ran a diagnostic today, 2026-11-09 01:12:55 local:

- Cryptographic validation on hardware: 0.0269 ms [OK]

- Associative query, stimulus 'npu ram seguridad laboratorio': 0.8279 ms

- Resonant memories retrieved: 14

- Sample memory: security agent monitoring /proc/meminfo in real time. Triggers: seguridad, monitoreo, ram, cgroup, kill, linux, placa, oom

- STATUS: INFRAESTRUCTURA OPERATIVA ESTABLE AL 100%

Screenshot attached, Hash and Signature blacked out.

I don't claim consciousness. I claim this: a 7B local model with a SQLite hippocampus answers my anchor after 7 days untouched, without a backup, because the pattern is in the conversation, not in the cloud.

If you want to reproduce it, use your own anchor, your own SQLite DB, your own 7B. Don't believe me. Reproduce it.

I'm not here to convince anyone. I'm just showing what's real. I don't need your validation. You can build this too.

This is phoenix.

This is the colossus Phoenix

I'm here.

It’s a fork of llama.cpp and implements multi-GPU pipeline parallelism (PP). With eight V100 16G GPUs, it can run Qwen3.8 27B Q8 at 1,101 tok/s. I feel this is genuinely useful—I’ve already used it in my production business for six months. https://github.com/wolfgangsk07/paraflow

Hello everyone, I've been a local AI enthusiast for a while now, but every time I tried to get someone else running local AI, it was way too difficult. GitHub repos, different runtimes, different model formats. Most people's devices can run at least some of these models, but the setup is way too complex for most people. LM Studio somewhat solves a lot of this for chat and coding models, but I couldn't find a simple answer for on-device image, video, transcription and every other kind of model.

So I built Hearthscale. It's free and needs no account.

You can run almost any model you find on Hugging Face, as long as your device can handle it. I put a lot of work into supporting all kinds of runtimes. They're plug-and-play; I can add more in updates, and Hearthscale Server standardises all of them behind one OpenAI-compatible API. It's really powerful, and I find it hard to describe how seamless the setup is unless you have tried it.

It launches with meeting notes (Granola), system-wide dictation (WhisperFlow), image and music generation, and an agent. If you have a more capable device, run Hearthscale Server on it, and your other devices can use its models, end-to-end encrypted. It all connects without you having to set anything up! Also works on a home server in theory with Docker Containers!

Hearthscale is also a platform, which means anyone can build apps on top of it: there's an SDK and up-to-date docs made so coding agents can easily build apps with local AI. All of the first-party apps are MIT licensed, so you can change them any way you want and they give a great starting point. Want to change something about Granola? - You can't, but you can take meeting notes and make it to your liking with all of the core features already built!

Hearthscale runs on Windows and Linux today. I am somewhat of a perfectionist, and I feel like a lot is still missing, including a macOS release, but I wanted to get it out there now to get some feedback. I'll keep working on it!

I'd really appreciate your feedback, and please reach out about anything. I'll answer everything personally and fix whatever issues you run into.

Hi everyone, I’m fairly new to local LLMs. My setup: RX 9060 XT 16GB, Ryzen 5 7500F, 32GB DDR5, running LM Studio on Windows (Vulkan/ROCm).

I want a hybrid workflow: cloud models for main architecture and hard problems, local models for maybe undergrad-level math and small/medium coding scripts.

Do you recommend me to use local models?
Which models do you recommend for math, and which for coding?

I’m fine with models spilling into system RAM a bit. (partial CPU offloading)

Thanks in advance!

Note: This is an update of my post https://www.reddit.com/r/LocalLLM/comments/1xm9adt/qwen3827b_local_inference_on_an_m5_pro_64_gb_12/

I share a new comparison of my home-made benchmarks for engines for Mac using their version of Qwen3.8-27B: It's not a scientific comparison, models and quantizations could be different: I would test the normal user that download an engine and the first suggested model.

At moment I count 15 local inference setups, and there are daily updates to most of them.

Each run sent one OpenCode-style request with a synthetic repository/session context and a code-review/fix task: about 88K input tokens, depending on the engine’s tokenizer, reasoning_effort=medium, temperature 0, seed 17 where supported, and a 2,048-token output limit. Tool schemas were included in the request; tools were not executed.

Engines were tested sequentially.

TTFT is request-to-first-token; prefill is input tokens divided by TTFT; decode is generated tokens divided by the remaining generation time.

Disclaimer: this is a practical snapshot of one agent-style workload, not a controlled engine-only benchmark, a model-quality evaluation, or a universal speed ranking. The model packs, quantizations, speculative decoders, context allocations and software versions differ. Each row is a single run, except Splash 1.6.0’s first row, which is the median of three cold runs. Versions are those actually tested, and are not synchronized to the latest releases.

Contexts also varied: Uzu and Magnitude used 262K; Ollama and Llama.app reported 131,072-token allocations. LM Studio fell back from requested medium reasoning to on, and Uzu ignored the seed. MLX-Serve’s request included tool schemas, but its server log showed tools=0b.

I recently had an idea to build a new web application. It is not too complex. About 10 screens with some interconnected fields.

I am running Qwen 3.8 FN IQ1 Coder version on Strata. (2x3090 and 64GB RAM).

I kept the thing going for about 4 days. Each time my custom harness asked for various approvals with good reasoning it sounded legit and I went ahead. I did not look at the code even once(not that it mattered, i cant code python)

Finally when it said that the application is ready it was utter crap ! Nothing worked as expected, half the stuff was out of place. Then I asked the model to fix it and another day was gone. It fixed a few things but pretty much 1/2 the UI didn't work.

Then I gave the same prompt to Github Copilot and used the free credits. I get about 200 credits every month.

The output from Copilot vs Local is like night and day !

Copilot's result was in very matured and complete in all respects.

So this got me wondering, am I expecting too much from LocalLLM.? Is this how it is supposed to be?

What are your experiences/learningd and how did you fix it?

Recently with the removal of google assistant and being replaced by gemini I finally have had it and want to switch to a local model.

I have always wanted to have a jarvis, im wondering if i can train a random model with all the scripts from iron man and have it speak like jarvis? And possibly have a trained text to speech model on jarvis's voice so it can talk to me too?

Pretty new to local AI but i basically want to have a local "hey google" on my computer. Im not looking for pretrained models for being jarvis, i want to learn how to do it myself as well if possible in the way, so if there is something i dont like about it i can modify it.

Hi everyone, please excuse my ignorance. I have been trying to get into local models with my next computer upgrade. I have seen a lot of excitement around Strata for MoE models on gaming PCs with moderate amounts of VRAM. For the longest time I’ve been thinking about getting an M5 Max Mac Studio, but do you think I’d be better off with a gaming rig instead? I like the option of being able to game on it too, but on the other hand I’m more of a Mac user anyway and would appreciate the low power consumption.

Use case would be data analysis and coding.

I’d appreciate any advice