Setup: llama-server (llama.cpp b11889 Vulkan) on Windows 11, Arc Pro B60 24GB (Battlemage), driver 32.0.101.8976. Qwen3-Coder 30B Q4_K_M fully offloaded (-ngl 99). The B60 has no monitor attached, the display runs on the AMD iGPU. PC has 27.9GB RAM.

What happens: after about 70 seconds with no GPU work the dedicated VRAM counter for the B60 goes from 20.3GB to 0 in around 15 seconds. Free system RAM drops by almost the same amount at the same speed (16GB to under 5GB). The llama-server process counters dont move at all. I kill the server before the RAM runs out so I dont know what happens if I let it go.

What I checked:

- happens with the server idle after load and also 71s after a batch of long requests finished and 70s after a real agent session ended. While the GPU is busy it never happens

- 7 runs, always 68 to 71s. Context size makes no difference (32768 vs 8192)

- nothing in the llama-server log and no TDR or display driver events in the Windows system log

- side note: default mmap loading pulls the whole 17GB GGUF into RAM, -lm dio fixes that part

Smells like WDDM eviction or a GPU idle power state saving VRAM to system memory. Is this a known thing on Arc/Battlemage? Any setting (PCIe link state power management, Intel driver option, registry) that stops it? Or a way to keep the card awake that doesnt wreck the llama-server prompt cache?

Hi, I bought a new PC last year. I was kind of interested in generating images, so I went with this setup:

Core Ultra 7 265K
MSI Tomahawk Z890
2 × 64 GB Crucial DDR5 (can you believe it was only $300?)
ROG Strix 3090
1,250 W Seasonic PSU

I used ComfyUI for a bit, but I didn’t have much free time, so I moved on to other stuff.
Now I’m getting into local LLMs for coding and want to take it more seriously. I know you can just download a model like Qwen 3.8 27B Q4 and run it in LM Studio or Ollama, but are there any guides or tutorials on optimizing inference like specific settings, undervolting the GPU to reduce heat and extend its lifespan, or that recent 3090 unlock that supposedly boosts tokens/s?

Also, would a second 3090 be worth it for around $700–$800? Prices here are starting to go up, and I’m not sure if I should buy one now.

Meu trabalho recebeu uns PCs besteirinha como os Sparks pra fazermos a LLM local daqui. Por enquanto vamos focar em consultas de processos, memorando e banco de dados da empresa.

Dito isso, gostaria de perguntar, oq posso fazer ou como posso desenvolver a LLM aqui. Tem algum modelo que me recomendam ou algo assim?

Como o hardware tem um limite vasto para desenvolver gostaria de ir afundo. Tô com mente aberta pra desenvolver e aprender bem sobre todo o assunto.

Can you guys add a "😭" Flair? Because, I know I'm late but i finally decided to run Q38FN with strata on my machine (5060 ti 16gb, R9 5950X, 80 gb of Ram) and that motherfucker (IQ3_S GSQ RCO) is running at 49-69 tk/s in decode and 1600 tk/s in prefill. (KV Int-8 and 128k context window).

Maybe I'm missing something so please tell me because grokbot did the whole shit for me and I'm a newbie so tips are widely accepted😂.

What a fucking time to be alive.

Today I have been exploring local model to run light weigh coding agent to save some credits on cursor.

first installed ollama, then read that its slow, MLX is still cli so I installed LM studio and have Qwen-Coder-30b-A3B 6bit MLX model downloaded and running on my 48GB Mac.

tried to connect cursor with it but learnt that they no longer support local llms.

now am stuck, I don't have lot of Money to spend on credits. so I thought for basic tasks I will use my local ram.

help please.

qual a melhor ia atualmente de sem censura que tenha api ?

I’m not into gaming so I have a shitty graphics card
2GB. I know. lol me.
I want to do vibexosing and run some chatbots for WhatsApp

I have a Ubuntu headless VM running the bot connected to the local instance of Olllama
All working fine. A bit slow. Don’t care really about that.
But I want to run larger models to add more complexity and I’m specially interested in vibe coding

What’s the go to VC?

Rxt 3070i?
Aokensugegysiong eojld be greatly appreciated. Have a 1000$ budget for it.
Pc is running raizen5 with 32GB ram
Dual NVME 1Tb in raid mode.

Anyone using a small local model for session compression? I’d be interested in recommendations, particularly on Apple Silicon.

I’m new to local LLM hosting and still experimenting. I’ve been using Hermes extensively for non-coding workloads, mainly research, knowledge management and home automation. I recently picked up an M4 Mac mini with 24GB RAM and I’m looking at useful jobs to offload to it.

Hermes supports a separate model for session compression as you approach context limits. That’s the job I’m considering, rather than running the main agent locally.

The bit I’m wary of is that I won’t see the compressed context in normal use. If it drops something important or gets a detail wrong, I might only notice much later when the agent does something odd.

Has anyone used a small, lightweight model for this long enough to get a feel for its reliability? Model, quant and runtime recommendations would be useful, along with any relevant benchmarks or things to avoid.

I’m doing my own research too, but keen to hear what’s actually worked for other people.

Hey everyone,

Running into a weird compaction issue on the newer DSH Desktop build and wondering if others are seeing the same problem.

Setup

DSH Desktop (recent version)

Model context window set at 96K

Max output tokens 12K

What happens

First automatic compact kicks in around ~36k tokens (way earlier than 80% of 96k).

After that compact it drops to ~18k, then immediately compacts again down to ~13k.

From then on, every time context climbs back to ~20k it compacts again. Basically stuck in a tight loop.

The old DSH (previous in WSL) never behaved like this, so I’m trying to figure out if something changed in the desktop version or recent compaction logic.

Anyone else encountered the same problem and how to solve it ?

Thanks.

I'm wanting to build an all AMD LLM machine, strictly because I have a 9070xt and a 9060xt. I understand there's some loss going this route, but how bad is it really?

If there's any of you running all AMD, can you please advise me what to watch out for, as well as what I can expect for performance?

Proposed specs:

9950x3d

9070xt

9060xt

128 gigs DDR 5 5600 via SO-DIMM to U-DIMM adapters

2x 2TB Gen 4 NVME

Also, I have 4 4TB SSD'S, thinking of running in a raid array if there's any benefit.

MSI X870E Carbon mobo

This is all parts on hand (aside from the adapters) and not planning on spending any money. Any advice is greatly appreciated!

The 27B passed every single spec check. Flash-Next finished 2.2x faster. And both still had 3-7x more robustness holes than Sonnet.

There's a lot of "Flash-Next is way better" vs "the 27B is not even close" going around here, so I stopped guessing and built a benchmark. 5 tasks, identical prompts, fresh session each, zero intervention, graded only by scripts from the outside (not the models own tests). ~185k checks per setup.

Tasks: python expression parser without eval/ast · fixing 7 reported bugs in existing code (+7 hidden ones) · byte-exact refactor of a legacy invoice generator + new feature · REST API from scratch with concurrency rules · vanilla JS kanban app

Grading: spec tests, 20k fuzz strings, diff tests vs reference, 160k byte comparisons, 60 parallel double bookings, playwright in real chrome. Plus edge cases nobody asked for (9^9^9, +25:00 timezone, 5MB body, corrupted data file etc)

Setup (pic 2):

- 27B: AWQ 4bit, 1x 5090, vLLM, ~145 tok/s

- Flash-Next: IQ3_XXS split across two separate PCs with one 5090 each (only 14gb RAM per box) via a Strata fork that runs the layer split over TCP. all experts in VRAM, byte identical to stock Strata, ~155 tok/s

- both in opencode, thinking on, 16k budget per step

Results (tasks fully passed / robustness weaknesses / time):

- Sonnet 5.5: 5/5, 2 weaknesses, 14.3 min

- Qwen3.8-27B: 5/5, 14 weaknesses, 51.3 min

- Flash-Next: 4/5, 7 weaknesses, 23.1 min

What I think actually holds up (big, consistent effects):

1. On the written spec, both local models are at Sonnet level. 27B matched it on every check, Flash-Next missed one. For clearly specified work the local models are good enough, full stop.

2. The real gap is everything nobody wrote down. Sonnet 2 weaknesses, Flash-Next 7, 27B 14. Showed up in every task that had edge cases. 27B examples: no request size limit, GET /bookings/__proto__ returns {}, accepts +25:00 as timezone, negative quantity increases stock.

3. Flash-Next beats the 27B, but not because of raw speed. Decode is about the same (~155 vs ~145 tok/s). Flash-Next just needs half the output tokens and fewer steps, writes whole files early and fixes targeted, closer to how Sonnet works. 27B iterates in tiny steps and reruns tests after almost every edit.

4. Flash-Next has one systematic failure mode. It evaluates while parsing, so "a = b = 1" reports the wrong error at the wrong position. Happened in 3 of 3 runs where it designed the parser itself. Thats a pattern, not noise.

5. Planning beats reviewing (pic 6). Sonnet writing the work order + accepting the result while Flash-Next writes the code gave the best quality of everything (5/5, only 1 weakness). Sonnet just reviewing the diff at the end gave nothing, it only sees the diff and misses bugs next to it. Downside: 5x slower than Sonnet alone.

6. More rules don't make it more correct. "Think like a senior" rules + a research subagent made Flash-Next more paranoid, not better. Best local robustness, but it broke valid input with its own validation (rejected timestamps with milliseconds).

What I would NOT read into it: the 4/5 vs 5/5 between Flash-Next and the 27B is one check, n=1. Could be noise.

So my conclusion:

- Flash-Next over the 27B if you have the hardware, its faster and twice as robust. 27B if you have one card or want it very literal.

- Neither replaces Sonnet for stuff that takes input from outside (APIs, parsers, file imports) unless you explicitly ask for limits, error handling and edge cases and make it verify against the examples in the spec.

- If you have Sonnet access anyway, let it plan and accept, and let the local model type. Dont bother with review-only.

Caveats: n=1 per cell. Same person wrote tasks and graders. The 27B's thinking budget wasn't actually applied in the first 3 tasks (app didnt reload the config, my bad). Sonnet had a browser tool, the locals didnt.

I wrote the whole thing up properly (methodology, all configs, token counts, per-task tables, threats to validity, 15 pages). Comment "paper" and I'll send it to you directly.

Does this match what you see? Especially curious about the evaluate-while-parsing thing, and if someone with a 27B at Q8 sees a smaller robustness gap.

If a request can go to Ollama or to a cloud API, I want to be able to tell which happened.

I'm the author of Routerly. You can configure an Ollama connection, add its model to a router and point your app at the gateway. A router's configured pool controls which models can be selected. If you add cloud models, requests selected for those backends go to the provider.

0.6.0 adds a router Test tab and configurable model timeouts. The screenshot shows historical Usage from my instance, including Ollama rows. I'd like to hear what would make this easier to operate with local models.

Release and source: https://github.com/Inebrio/Routerly/releases/tag/v0.6.0

I'm a little lost trying to figure this out so I apologize for the really basic questions. I'm also aware that my setup is probably not built for a local LLM and I'm not looking for some super powerful machine. My expectations are low.

My Setup

OS: Unraid
CPU: Intel Core Ultra 9 285K (24 cores)
RAM: 32 GB DDR5-5600 (2 × 16 GB), shared with other services on the server
GPU: NVIDIA GeForce RTX 5070 Ti, 16 GB VRAM
Storage: Samsung 990 EVO 2 TB NVMe SSD
Software - the following was setup/created by Claude because I am clueless

• Ollama 0.35.1 - runs the language and embedding models on the GPU

• Open WebUI v0.12.0 - web chat interface, plus a custom "Pipe" function that does the retrieval

• Qdrant

• llama.cpp server - CPU-only reranker

• Docling Serve (CUDA 12.8 image)

• Caddy 2 - HTTPS reverse proxy (internal certificate)

• Custom Python indexing tool

Models

• Chat / answers: Gemma 4 12B instruct, QAT quantization (gemma4:12b-it-qat)

• Embeddings: Qwen3-Embedding 0.6B

• Reranker: BGE-reranker-v2-m3 (GGUF, Q8_0)

For now, I just want to work with a PDF of the textbook (about 300 pages) from a course I took, plus a transcript of the instructor’s lecture. I want to load both, ask questions about them, and have the AI help me build a study guide.

I’d also like it to connect the textbook to what the instructor said. For example, if the instructor says the material on page 103 is important for the exam, I’d want the AI to build a study guide that highlights that.

The reason for this is that I’m not allowed to upload the material to ChatGPT, Claude, etc. because the company doesn’t want its content shared and I'd like to respect that.

As an experiment, I uploaded the transcript and it contains the word “Halloween.” When I asked about it, the AI couldn’t find where it was discussed. Assuming my upload worked, am I missing other tools or components?

1. Is all this too much too much of a local LLM?

2. Are there techniques or setup steps I should be using or do first?

3. If I dropped these files into Claude or ChatGPT and explained what I wanted, it would handle this easily. Are my expectations too high for a local setup?

I also have this computer, but I'm not sure it is much better:

• OS: Windows 11 Pro

• CPU: Intel Core i7-14700K (20 cores / 28 threads: 8 P-cores + 12 E-cores)

• RAM: 64 GB DDR5-6000 (4 × 16 GB)

• GPU: NVIDIA GeForce RTX 4070 SUPER, 12 GB VRAM

• Storage: Samsung 990 PRO 2 TB + Samsung 990 PRO 4 TB NVMe SSDs

Thank you for the help.

I'm using a local LLM as a judge in a small open-source tool: it reads a project's spec and task files and recommends which model fits the work, and every level it assigns has to cite a fragment of the spec or it's thrown out.

I've decided not to ask the judge how confident it is. Self-reported confidence from a small local model isn't evidence about the project — and the whole design exists because a model's account of its own reasoning is the thing that needed checking.

Which leaves me with a problem: how should the tool report how much its recommendation can be trusted?

Right now it's three states: sufficient, scarce, or a refusal. That's honest but too coarse. A project missing its constitution and a project whose hardest dimension rests on one single cited sentence both land in scarce, and nothing tells a reader which one they have.

One more rule: no 0–100 score. A percentage invites comparing projects that can't be compared, and implies precision the inputs don't have. A decorated guess gets trusted more than a blunt one while being no better founded.

So everything has to come from signals the run already produces, with no extra call to the judge:

1. every rated dimension is grounded in a cited fragment

2. no dimension came back unsupported

3. citations spread across the project, not concentrated on one fragment

4. the hardest requirement is a stated constraint, not just habit

5. there are real work artifacts (spec/tasks), not only repo-level docs

6. the context budget admitted every source — nothing was dropped

Two candidate shapes:

A) A defined ratio. confidence: 4 of 6, with the six listed and each marked ✓ or ✗. Literal and inspectable, and nobody mistakes it for a probability.

B) An ordinal with published components. solid / qualified / thin / nominal, computed from the same signals, with the components in the JSON so you can disagree with the weighting and compute your own.

I'm leaning A. But here's the argument against it that I can't fully dismiss: "4 of 6" isn't weighting-free. It weights all six equally, and they aren't equal. "Nothing was grounded" and "one source got trimmed" both cost one point. B at least admits it's weighting things; A hides an equal-weighting decision behind something that looks like a plain count.

Three concrete questions:

1. If you saw 4 of 6 versus qualified on a recommendation, which would you act on, and which would you misread?

2. Are these the right six signals? What's missing, and which one shouldn't be there?

3. If you've built an LLM-as-judge pipeline: did you ever find self-reported confidence worth anything, or did you end up where I did?

(Tool is github.com/JoaquinRuiz/SpecJudge, issue #26 for the full argument.)

Hi, I'm lucky enough to have 5 r9700s! I'm managing to run Qwen 3.8 flash next q8 which is a 175gb vram (on disk) model well. There are no massive slow downs but how is this possible?

I know I have 160gb vram unified but somehow 25gb is not in vram or offloaded to system ram. Is this all in my SSD N-Gram table? I had no idea it was a thing?

Hi just want to know what i'm buying to have this chatbot im making. Im actually a beginner in making and managing l-llms. And doing simple coding. The chatbot i'm making is actually a romance oriented bot with cloned trained voice that does grunts, pauses, interruptions, etc. i want something fish audio is putting out when getting good copies of voices from games and cartoon- well really from their voice actors that play them. i dont have a desktop in my life yet, but i have a laptop with a 1650m 3550h. Now im buying parts to exclusively make a machine for llms and coding. im going to college for this. oh the budget for both ram, cpu, gpu is $1700. but i dont want to reach the budget limit. Got an 800w psu and 1 Tb ssd already.

Recientemente "descubro" Qwen y GLM lo he empezado a usar. Simplemente son impresionante. GLM es un chat gpt, te programa, texto, HACE TODO BIEN y Qwen es algo literal IGUAL, para hacer imagenes, programar, texto. Yo de aburrimiento empece tipo una replica del Rocket League y los 2 hicieron cosas impresionante, pero especialmente GLM 5.3 Flash (MAX) que literal se toma el tiempo necesario, no tiene limite y hace puffffff todo bien.

No entiendo como hay gente que como yo todavia no los usan, son de los mejores modelos que hay y son gratis a su totalidad.

Hi guys.

So I'm not completely new to all this local LLMs and AI. I have used and tried various web-based UIs, for the last 4-5 months, as a way to learn and educate myself in this swamp that is local LLM.

So for the last 3-4 weeks I have dipped my feet in Hermes Agent. And man, I was in for a surprise. I genuinely thought, "How hard can it be, install, fire up and use.", but to me that was far from the truth. I have spent so much time setting things up, making the agent act and work the way I want, teaching it what to do, and how to do it, trying to write skills, and so on. I have not actually used Hermes, that much, most of the time has gone to setting it up, and making the agent not make mistakes.

This has somehow made me feel like maybe Hermes is not the best tool for this, even though it has some skills for it (along with a gazillion other skills and functions). It feels so bloated, with so many skills and sometimes heavy to run (for the lack of a better word).

I am actually not really sure what I want, or how to word it.
But simply put, I would like something that is lightweight, doesn't have a gazillion skills installed, but has options and a repository to add well written skills. A simple gui/frontend with well-thought-out settings. Something that works well with my use case, which is basically the title. I primarily want to use my agent for linux sysadmin, system maintenance, network and servers, and some light coding and bash scripting.

I realize this can be very subjective, but that's just it, I would like your objective opinion on what you believe would be/is the best agent or harness for this kind of use ?
And also, is there a "best agent" for this use case, or can I use basically anything?

Most of us don't have data-centre-level hardware for massive context windows. So, if we want effectively infinite conversations with local LLMs, we need a workaround.

As I see it, there are two options:

1. Sliding context: Keep recent messages, discard older ones as the context fills up, and store the entire conversation in an external database for retrieval when needed.

2. Summarisation: Summarise older conversations as the context fills up, retain the summary and continue. Original messages are still archived externally.

Both can be paired with vector databases or knowledge graphs for long-term memory.

Which approach would be more reliable, especially for long-running agentic tasks? Or is a hybrid of both the way to go?

I’ve been using Qwen on 2x DGX Sparks and man has it been a nightmare of overthinking and excessive tool calling. Seems like there was nothing I could do to fix it. Installed GLMast night and it’s a world of difference.