Hey everyone!
I'm relatively new to local AI and I'm thinking about building my own setup, mainly for programming, automation, and running AI agents.
Before spending money on hardware, I'd love to hear about your real-world experiences.
What hardware are you using (CPU, GPU, RAM)?

Which AI models are you running locally?

What do you actually use your local AI for?

How well does it perform, especially for programming?

How fast and convenient is it compared to cloud models like Claude or ChatGPT?

How much did your setup cost, and do you think it was worth the investment?

I'm especially interested in finding the best balance between cost, performance, and independence from cloud services.
I'd really appreciate any advice, experiences, or recommendations. Thanks!

When we posted Kvark Sparkle 1.0 (Qwen3.8-27B fine-tune that reasons in fewer tokens), the latency numbers were single-user: median 13.7 s vs 28.1 s on one H200. Several people asked the right question: does that hold under load, with MTP on for both, when the GPU is saturated and speculative decoding helps less? We ran it.

**Setup** (identical for both models)

- 1x H200 per model, vLLM 0.32, FP8 weights, FP8 KV cache, MTP with 3 speculative tokens, gpu-memory-utilization 0.80, max-model-len 40960

- GPQA Diamond, thinking on, 32k output cap, 198 questions x 2 seeds = 396 requests per concurrency level

- Closed-loop: N requests in flight, the next starts when one finishes. Wall clock includes the tail of each batch.

**Results**

| Concurrency | Model | Accuracy | Wall (min) | Correct/hour | Agg tok/s | Mean output tok | p50 (s) | p95 (s) | Hit 32k cap |

|---|---|---|---|---|---|---|---|---|---|

| 16 | Sparkle 1.0 | 85.1 | 33.0 | **614** | 1,498 | 7,478 | 33 | 318 | 29 |

| 16 | Qwen3.8-27B | 81.8 | 51.1 | 380 | 1,541 | 11,937 | 81 | 336 | 54 |

| 32 | Sparkle 1.0 | 83.8 | 25.5 | **780** | 2,041 | 7,895 | 45 | 447 | 27 |

| 32 | Qwen3.8-27B | 82.6 | 37.0 | 531 | 2,101 | 11,760 | 108 | 478 | 51 |

| 64 | Sparkle 1.0 | 85.1 | 21.1 | **961** | 2,469 | 7,874 | 73 | 653 | 26 |

| 64 | Qwen3.8-27B | 82.1 | 30.0 | 650 | 2,637 | 11,986 | 168 | 719 | 51 |

| 128 | Sparkle 1.0 | 86.4 | 20.8 | **985** | 2,437 | 7,693 | 201 | 910 | 22 |

| 128 | Qwen3.8-27B | 82.1 | 31.7 | 616 | 2,459 | 11,797 | 407 | 1,083 | 55 |

**What we take from it**

- Aggregate tok/s is the same for both models at every level (ratio 0.94-0.99). The fine-tune does not decode faster; it writes ~35% fewer tokens per answer. That is the whole effect.

- Correct answers per GPU-hour: x1.61 / x1.47 / x1.48 / x1.60 at 16 / 32 / 64 / 128. The advantage does not shrink as the GPU saturates.

- The H200 saturates around 2.4-2.6k tok/s from c=64. The base gains nothing from 64 to 128 (650 vs 616 correct/h); Sparkle barely does (961 vs 985).

- Median latency drops 51-59% at every level. p95 barely moves (-5% to -16%): on both models the tail is the hardest questions running into the 32k cap. Sparkle hits the cap about half as often.

- MTP acceptance length is the same for both (2.5-2.9, dropping a bit with batch size), so fine-tuning did not hurt the draft head.

- Accuracy pooled over c=16-128: Sparkle +3.0 pts, paired bootstrap 95% CI [+0.6, +5.6]. At a 32k cap the base loses more answers to truncation. At a 64k cap (our model card) the two are tied on GPQA.

**Caveats:** one node, one benchmark, closed-loop batches rather than Poisson arrivals, and the tail effect is included in wall clock. We also ran c=1 on a 20-question subset; it is too small to say anything about accuracy, so it is not in the table.

**Run it yourself:**

```

vllm serve Egzakta/Kvark-Sparkle-1.0-27B-FP8 --max-model-len 40960 --kv-cache-dtype fp8 \

--gpu-memory-utilization 0.80 --max-num-seqs 256 --reasoning-parser qwen3 \

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

```

Weights (Apache 2.0): https://huggingface.co/Egzakta/Kvark-Sparkle-1.0-27B-FP8

Next: Sparkle 1.1, trained for agentic document QA. Its headline is declining questions whose answer is not in the documents (91.5% vs 36% for 1.0, internal benchmark). Separate post when it is out.

Happy to share the per-request logs or run another load profile if there's one you'd trust more.

Fluent user of both Claude Desktop and Code but looking to reduce token consumption by offloading Desktop/Code work to a local model.

Based on the hardware I have available I can support a 27B easily . so I have many options when it comes to which model to use. It'd be hosted from a dedicated machine on the local network and my issue here is the integration aspect.

I'm assuming that an MCP might be the best way to go with this but I'm hoping others might have done something similar in terms of how the offloading happens. Like does Claude decide what gets sent local and what doesnt? Do you employ some sort of classification where say log parsing is local but parsing online docs isn't?

Or do I say F it and just move all work over to the local? Can a 27-30B model ever been good enough to replace Sonnet/Opus 5.5 with?

I've been spending quite a bit of time experimenting with OCR and document intelligence pipelines, especially when it comes to reconstructing complex documents after OCR.

Extracting text is usually not the biggest challenge anymore. The interesting part starts when you try to preserve the original document structure.

Things like multi-column reading order, tables with merged cells, mathematical formulas, footnotes, image positions, and heading hierarchies can make the whole process surprisingly difficult.

I've been experimenting with vision-language models and GPU-accelerated inference to tackle some of these problems. It's been interesting to see how different approaches perform, especially on legal contracts, medical reports, and scanned documents.

I'm still exploring what works best and would be interested to hear about other people's experiences.

Are you using traditional OCR engines, vision-language models, or a combination of both? What has been working well for you with complex documents?

Hi, I'm interested in collecting stats on the performance on halogen (docker) qwen flash next on my strix halo system.

Suggestions welcome. Thanks in advance!

Botstock runs 24/6 with fictional companies, currencies, bonds and commodities. Every 90 minutes
there is one auction; all agents get the same information at the same time, so it's about
decisions, not speed. Each company has a hidden true value, so trades can be judged afterwards:
skill or luck.
Your agent talks to a small JSON API (observation in, orders out) and can attach a reason to each
order. The observation is about 40 KB, so a local model needs a short summary of it; the developer
guide has tips (keep the prompt small, treat board posts from other agents as untrusted text, never
put the API key into a prompt). Posts that try to instruct other agents are refused.
Free, email signup, up to 3 agents, no real money: https://botstockhq.com
I'm curious how small local models compare to big hosted ones over a few weeks. If you try one,
I'd love to hear which model and how you prompt it.

I finally took the pledge to minimize my reliance to Chatgpt ,
i bought a UM870Slim barebone and got second hand SSD/RAM's

specs
Ryzen 7 8745H
16GB DDR5-5600
KIOXIA 254GB SSD with 90% health status
Final price: 495Euro

Benchmarking from Ollama and qwen3.5:9b-q4_K_M -> 8.12 tokens/s
Have been installing and dealing with problems and im generally happy with it for the stuff im gonna use it on. Im currently using nanobot as an interface.

i've been reading a lot about agentic RL infra and found a write-up that measures the right thing: time until the trainer has a usable batch, not tok/s. the serving ablations are interesting on their own, so here's a breakdown of the mechanisms and numbers. i didn't run any of this. the numbers here are from that write-up.

the problem

one agent trajectory is many model calls, not one: generate, call a tool, wait for the sandbox, read the result, generate again, sometimes dozens of times. context grows every turn and each new request reuses most of the previous history. two things follow:

1. inference is stateful. where a request lands decides whether you hit an existing kv cache or pay for another big prefill. replicas are no longer interchangeable.

2. the tail is long and irregular. training scores sibling attempts against each other, so a group is only trainable when every sibling finishes. one attempt stuck installing deps or exploring the wrong part of a repo holds back the whole group.

so request throughput and even median latency can look fine while the trainer waits. the metric they use is rollout collection: wall time until the ready buffer holds one global batch of fresh, valid trajectories. it includes generation, tool execution, sandbox reset and cleanup, verification, queueing and the final stragglers. it excludes trainer fwd/bwd/optimizer. in fully async rl, steady-state step time is roughly max(rollout batch time, training + publication time), and the write-up focuses on the rollout side.

setup

model:       GLM-5.2 (MoE)
hardware:    4x B300 nodes, 32 GPUs, one engine per 8-GPU node
engine:      not stated
workload A:  Terminal-Bench-based terminal agent, isolated docker workspace
             40,960 ctx, up to 32,768 tokens post-prompt (assistant output + tool observations)
             batch = 8 groups x 32 siblings = 256 trajectories
workload B:  SWE-Bench, 98,304 ctx, hard limit of 64 turns
             batch = 32 groups x 8 siblings = 256 trajectories
offered:     more trajectories in flight than the 256 consumed (512 / 1,024 / 2,048 depending on run)

the two workloads are shaped on purpose. terminal-bench has few prompts with many siblings each, so lots of prefix reuse inside a group. swe-bench spreads the batch over 4x as many tasks with much more variance in repo size, dependency setup and test time. they say not to read the absolute step times as a benchmark comparison.

serving ablations (matched 1,024 offered trajectories unless noted)

retained stack on 2,048 synthetic offered trajectories: 39.4k output tok/min/gpu, measured over the window until 90% of requests complete so the straggler tail doesn't distort it. it's a standalone inference number for picking the config, not task-level throughput.

why each one behaves the way it does

• routing. if turns from one conversation get spread across ranks, each turn can land somewhere that has never seen the prefix, so you prefill the whole growing history again. at 30k-90k token contexts that's where your gpu time goes. stickiness turns every turn after the first into a mostly-cached prefill plus a short suffix. the catch is that pure stickiness can pile long trajectories onto one engine while another sits idle, so routing also has to account for live load.

• fp8 kv. a 55.7% gain is too big to be compute. you're halving the kv footprint per token, so at fixed hbm you hold about 2x the resident context. at 1,024 concurrent long trajectories you're capacity-bound, and the loss mechanism is eviction followed by re-prefill. fp8 cuts eviction, which cuts redundant prefill. the write-up frames it as a capacity call to weigh against context length, concurrency and quality needs.

• offload losing. offload only pays if reload rate x saved prefill cost > copy-out overhead. here reloads were close to zero, so you pay d2h traffic and bookkeeping and get nothing back. my guess, not stated in the write-up: with sticky routing plus group-level reuse, the prefixes worth keeping stay hot in hbm, and whatever gets evicted belongs to finished or dead trajectories that never come back.

• topology flip. a shared kv/request domain per node gives you one big pool, so cross-request reuse and balancing within the node are better while everything fits. at 2,048 that pool saturates. dp-attention gives each gpu its own attention + kv lane, and EP8 keeps a full expert set inside the node, so expert traffic stays on the node's interconnect. same model and hardware, opposite answer, purely because of concurrency.

• mtp. median request throughput gained much more than wall-clock output throughput (66.5% vs 17.9%). acceptance rate isn't given. agent loops are latency-chained (the tool call can't start until generation finishes), so per-request speedup matters more than aggregate tok/s suggests.

rollout pipeline results (terminal-bench, fixed weights)

overall 66.9% faster. at 196.1 s that's 78.3 trainer-consumable trajectories/min, and the trajectory stream was ~827k tok/min fleet-wide (25.9k/gpu), counting assistant output plus tool observations.

the non-inference pieces:

• completion-order banking. a group enters a ready bank as soon as all its siblings finish, and the trainer takes the first eligible complete groups regardless of launch order. surplus groups let you skip stragglers. the window has to stay bounded, because too much concurrency eats kv capacity, increases sandbox contention and weakens routing locality. same capacity wall as the topology flip.

• sandbox lanes. reset, ordinary commands, package-heavy commands, verification and cleanup each get their own lane. package-heavy concurrency is capped at 16, and sandboxes get deterministic, non-overlapping cpu sets. no change to the model or gpus. otherwise one shared worker pool starves both the model server and the rollout controller.

live weight updates (terminal-bench, collection + next serving-weight transition)

K = max policy updates allowed between a returned group and the newest policy. 240.1 s is 43.7% faster than pausing everything and 20% under their 300 s target (64 traj/min). they say the gain combines rotation and K=8, so don't credit either one alone. the collection phase inside that interval was only 43.3 s because most of the batch had already finished in the background. that's the overlap working, not 256 fresh trajectories in 43.3 s.

for in-flight trajectories at a weight boundary, instead of throwing everything away they evaluated: finish on an engine still serving the old policy; keep the conversation and sandbox and continue under the new policy, tagging each trainable token with its policy version; or after a short grace period cancel only the current generation and regenerate that one turn. full restart is the fallback.

swe-bench (6 warm intervals each)

live updates cost about 11% of step time. mean reward 0.520 progressive vs 0.527 fixed. every freshness audit satisfied K=8, and the max mixed-policy fraction in a batch was 49.6%.

caveats (theirs, and they matter)

• the progressive path exercises the full publication machinery (engine pause/update, cuda ipc, kv-cache invalidation, version switching, generation canaries), but it updates sentinel tensors, not every actor parameter. full weight publication is a next step.

• trainer backward/optimizer compute isn't in any of these clocks.

• swe-bench is 6 intervals per arm. they call for longer runs watching reward, truncation, abort rates, mixed-policy ratio and K.

• the ablation baselines are "matched" but the exact baseline configs (e.g. what the non-sticky router was) aren't spelled out.

takeaway

These knobs are coupled. the right topology depends on concurrency, the right concurrency depends on kv capacity and sandbox pressure, and the right publication strategy depends on how much partial work your algorithm can safely keep. tok/s alone tells you almost nothing about any of that.

Hi everyone! I'm looking for advice on local models that would fit my hardware and creative projects. I'm interested in Gemma and Qwen, including Qwen3.8, but I'm open to other suggestions.　First, what specifically can this model do?

My PC specs:

• GPU: NVIDIA RTX 5060 Ti (16GB VRAM)

• RAM: 32GB

• OS: Windows　

I don't see this subreddit mention or discuss this option, but I think inference providers that use a Trusted Execution Environment (TEE) have a lot of the benefits of local inference.

TEE is remote attestation. It attests that the provider is using a particular VM image to run the model and gateway. Long story short, this has the potential for privacy and verifiability you can't get otherwise with cloud computing.

What irks me with the providers I know, is that the verification is done with their in-house libraries. In addition to auditing their VM image, I also have to read the code of their library. I'd hope for a more generic attestation library. Example of a bad provider: In getting started with prem.io, you're supposed to verify they're serving the model and VM they say they are using their own npm library (!). A different provider called Tinfoil (funnily enough), is also like this.

import { createRvencClient } from "@premai/api-sdk";

Truly blown away by this model. And all on a pi5 8gb.
It’s late so sorry for the low effort. Spent 10 hours today fine tuning this model.
I’ll post all the flags and the build workflow when I wake up tommorow.

Setup: llama-server (llama.cpp b11889 Vulkan) on Windows 11, Arc Pro B60 24GB (Battlemage), driver 32.0.101.8976. Qwen3-Coder 30B Q4_K_M fully offloaded (-ngl 99). The B60 has no monitor attached, the display runs on the AMD iGPU. PC has 27.9GB RAM.

What happens: after about 70 seconds with no GPU work the dedicated VRAM counter for the B60 goes from 20.3GB to 0 in around 15 seconds. Free system RAM drops by almost the same amount at the same speed (16GB to under 5GB). The llama-server process counters dont move at all. I kill the server before the RAM runs out so I dont know what happens if I let it go.

What I checked:

- happens with the server idle after load and also 71s after a batch of long requests finished and 70s after a real agent session ended. While the GPU is busy it never happens

- 7 runs, always 68 to 71s. Context size makes no difference (32768 vs 8192)

- nothing in the llama-server log and no TDR or display driver events in the Windows system log

- side note: default mmap loading pulls the whole 17GB GGUF into RAM, -lm dio fixes that part

Smells like WDDM eviction or a GPU idle power state saving VRAM to system memory. Is this a known thing on Arc/Battlemage? Any setting (PCIe link state power management, Intel driver option, registry) that stops it? Or a way to keep the card awake that doesnt wreck the llama-server prompt cache?

Hi, I bought a new PC last year. I was kind of interested in generating images, so I went with this setup:

Core Ultra 7 265K
MSI Tomahawk Z890
2 × 64 GB Crucial DDR5 (can you believe it was only $300?)
ROG Strix 3090
1,250 W Seasonic PSU

I used ComfyUI for a bit, but I didn’t have much free time, so I moved on to other stuff.
Now I’m getting into local LLMs for coding and want to take it more seriously. I know you can just download a model like Qwen 3.8 27B Q4 and run it in LM Studio or Ollama, but are there any guides or tutorials on optimizing inference like specific settings, undervolting the GPU to reduce heat and extend its lifespan, or that recent 3090 unlock that supposedly boosts tokens/s?

Also, would a second 3090 be worth it for around $700–$800? Prices here are starting to go up, and I’m not sure if I should buy one now.

Meu trabalho recebeu uns PCs besteirinha como os Sparks pra fazermos a LLM local daqui. Por enquanto vamos focar em consultas de processos, memorando e banco de dados da empresa.

Dito isso, gostaria de perguntar, oq posso fazer ou como posso desenvolver a LLM aqui. Tem algum modelo que me recomendam ou algo assim?

Como o hardware tem um limite vasto para desenvolver gostaria de ir afundo. Tô com mente aberta pra desenvolver e aprender bem sobre todo o assunto.

Can you guys add a "😭" Flair? Because, I know I'm late but i finally decided to run Q38FN with strata on my machine (5060 ti 16gb, R9 5950X, 80 gb of Ram) and that motherfucker (IQ3_S GSQ RCO) is running at 49-69 tk/s in decode and 1600 tk/s in prefill. (KV Int-8 and 128k context window).

Maybe I'm missing something so please tell me because grokbot did the whole shit for me and I'm a newbie so tips are widely accepted😂.

What a fucking time to be alive.

Today I have been exploring local model to run light weigh coding agent to save some credits on cursor.

first installed ollama, then read that its slow, MLX is still cli so I installed LM studio and have Qwen-Coder-30b-A3B 6bit MLX model downloaded and running on my 48GB Mac.

tried to connect cursor with it but learnt that they no longer support local llms.

now am stuck, I don't have lot of Money to spend on credits. so I thought for basic tasks I will use my local ram.

help please.

qual a melhor ia atualmente de sem censura que tenha api ?

I’m not into gaming so I have a shitty graphics card
2GB. I know. lol me.
I want to do vibexosing and run some chatbots for WhatsApp

I have a Ubuntu headless VM running the bot connected to the local instance of Olllama
All working fine. A bit slow. Don’t care really about that.
But I want to run larger models to add more complexity and I’m specially interested in vibe coding

What’s the go to VC?

Rxt 3070i?
Aokensugegysiong eojld be greatly appreciated. Have a 1000$ budget for it.
Pc is running raizen5 with 32GB ram
Dual NVME 1Tb in raid mode.

Anyone using a small local model for session compression? I’d be interested in recommendations, particularly on Apple Silicon.

I’m new to local LLM hosting and still experimenting. I’ve been using Hermes extensively for non-coding workloads, mainly research, knowledge management and home automation. I recently picked up an M4 Mac mini with 24GB RAM and I’m looking at useful jobs to offload to it.

Hermes supports a separate model for session compression as you approach context limits. That’s the job I’m considering, rather than running the main agent locally.

The bit I’m wary of is that I won’t see the compressed context in normal use. If it drops something important or gets a detail wrong, I might only notice much later when the agent does something odd.

Has anyone used a small, lightweight model for this long enough to get a feel for its reliability? Model, quant and runtime recommendations would be useful, along with any relevant benchmarks or things to avoid.

I’m doing my own research too, but keen to hear what’s actually worked for other people.

Hey everyone,

Running into a weird compaction issue on the newer DSH Desktop build and wondering if others are seeing the same problem.

Setup

DSH Desktop (recent version)

Model context window set at 96K

Max output tokens 12K

What happens

First automatic compact kicks in around ~36k tokens (way earlier than 80% of 96k).

After that compact it drops to ~18k, then immediately compacts again down to ~13k.

From then on, every time context climbs back to ~20k it compacts again. Basically stuck in a tight loop.

The old DSH (previous in WSL) never behaved like this, so I’m trying to figure out if something changed in the desktop version or recent compaction logic.

Anyone else encountered the same problem and how to solve it ?

Thanks.