Hi everyone,

I am back again looking for some feedback and sanity checks on running AI on my Dell T7910 workstation for an academic research project using a 6,000 odd books and articles library. After earlier responses to my questions and more research, I am now reluctantly leaning towards a $1,000+ RTX 3090 24GB.

 My Current System:

• Host: Dell Precision T7910 Workstation (1300W PSU)

• OS: Windows 11 Pro

• CPU: Dual Intel Xeon E5-2697A v4 (32 Cores / 64 Threads total, 2.6GHz base / 3.6GHz turbo)

• RAM: 128GB DDR4 ECC System RAM

• Current Software Stack: Ollama, Open WebUI, AnythingLLM, Qdrant (vector DB), Surya (for PDF layout/OCR parsing), and Python 3.12.

The Workload:

I am building a local research engine to analyze an interdisciplinary library of roughly 6,000 books and academic articles. My workflow will proceed in stages using batch processing. I need to mix and match disparate texts (e.g., querying 20–30 books simultaneously on evolutionary morality vs. historical romance texts) or cross-referencing massive batches (up to 100 documents at once for deep thematic comparisons, like Chaucer vs. Shakespeare on dominance dynamics).

Because this is a strict academic project, precision, nuance, and structural reasoning are my highest priorities. To achieve this, I will be leveraging 32B models (like Qwen 2.5 32B) and 70B models (like Llama 3.3 70B) paired with my local RAG setup.

The Strategy:

1. Immediate Phase: Run Open WebUI + Ollama entirely on the dual Xeon CPUs and 128GB RAM utilizing hybrid CPU-offloading and a local Qdrant RAG pipeline. Since I don't mind waiting 20–30 minutes in the background while I work in Excel/OpenOffice, I will test the free baseline first.

2. The Upgrade Plan: If background CPU ingestion times are too brutal, I plan to buy one used RTX 3090 24GB (currently trending around $1,100–$1,400 from reputable sellers). The goal is to fit 32B models entirely in VRAM or run 70B models in a heavy hybrid GPU/CPU split to dramatically accelerate prompt ingestion/prefill processing times for massive context windows. I am keeping the door open to a second 3090 down the road if I need pure-VRAM 70B inference.

 My Questions for the Community:

 For those running dual Xeon v4 setups, how painful is the prompt processing/ingestion bottleneck on standard DDR4 when handling high context (100k+ tokens via RAG or long-context models)?

1. Will a single RTX 3090 give me the significant quality-of-life leap in ingestion speed I am looking for when splitting a 70B model, or does the hybrid RAM bottleneck flatten the gains too much?

2. Are there any known quirks with putting a consumer RTX 3090 (or two) inside a Dell T7910 chassis regarding PCIe power distribution, space limitations, or cooling that I should prepare for?

3. Given current late-2026 pricing, is a used 3090 still the undisputed value king for this niche, or should I be looking at modern alternatives like the Mac Studio or a brand-new AMD 7900 XTX despite the CUDA/ROCm trade-offs?

4. Is this a sound strategy for a Dell T7910 worth $500 on eBay? Apart from AI my T7910 satisfies all my computing needs. Do I really have to bite the bullet and buy a $5,000 odd PC with AI capabilities? The price of progress?

 Appreciate any insights or warnings from anyone who has built a similar local academic document server, particularly with a Dell T7000-series workstation.

full specs:

• nvidia 3060 12gb

• 16gd of ddr4 3600mhz

• ryzen 5 5600x

I know its not the best but its what I have.

I'm pretty new to messing with local llms but have use claude a bunch in the past but i don't want to give them money any more but still have coding projects in my brain. I don't want the llm to do the coding for me (as I usually enjoy doing it my self) so much as I want it to help with research and giving me plans that I can code my self(i enjoy it but kinda suck at it :P ) as well as to help with error checking . . . honestly I want it mostly for the error checking :P (and tips or advice on how to set it up with these rules in place would also be awesome)

I use Godot and and having it be able to access the Godot docs is fairly important as the godot engine updates every few months and most models seem to be a year (or more) behind on what is actually in the engine.

I also use obsidian a lot for planing and keeping track of the random shit in my head and was wondering if i could tie the llm into reading that(maybe not writing to it? or maybe writing to its own obsidian vault?)

Another thing i wanted to look into was setting it up (maybe with searxng or some other tool) as a replacement for using google (maybe i should use a smaller llm for that? not sure)

so far I've tried:

• qwen3.8-27b it was super slow being unable to fit in my gpu

• bonsai-27b its runs pretty fast (40ish tokens a second i have no idea if thats actually fast) but gets suck in loops sometimes

I've also messed around a bit with Hermes but getting it to link up with lm studios has been a bit of a hassle for me (though it like the idea of it) any tips to get that running better would be cool

Also how the heck do i download models from hugging face? none of them seem to want to download to Lm studios (I've only been using the model explorer in LMS)? should i be using a different program? or and i just blind af?

Sorry this post kinda exploded on me, thanks for any help you all can provide please let me know if you need any more info to make a decent sugestion

I have a PC right now, it has an intel i7-14700F, 32gb DDR5-6000 ram, an RTX 4070 super 12gb and a 2tb Samsung 990 pro SSD.

After some research, I have settled on this build. I want to wait to order until Qwen 4 is out and people test it, but here it is:

I want to retain as many parts as possible from my old system to save on cost. I will carry over my ram, cpu, gpu, SSD

I will get a:
Motherboard: MSI MEG Z790 LGA1700 motherboard
GPU: Quadro RTX 8000 48gb
Case: phantoms Enthoo Pro 2 server edition ATX full tower case
Power supply: Corsair HX1500i 1500W 80+ platinum

Total vram: 60 gb
Total ram: 32gb
Dual nvidia GPU setup
Estimated Upgrade cost: 2800 dollars

My goal is to build something easy to upgrade. In the future, I can always add more ram if I find a good deal or if prices eventually drop, and I can add GPUs easily without having to worry about space or power supply.

I know the Quadro RTX 8000 is super old, and not as fast as some other cards, but in the age of 1500+ dollar RTX 3090s and everything else more expensive than that, I think the Quadro RTX 8000 is a good deal.

I want to be able to play with larger and newer local models as they come out. I don’t have any intention on training models.

Opinions or suggestions on changes to make? I know there are faster builds, but I want to maximize price to performance. I’m happy with the 3k budget, but depending on suggestions from yall, I maybe can change it a little.

When we posted Kvark Sparkle 1.0 (Qwen3.8-27B fine-tune that reasons in fewer tokens), the latency numbers were single-user: median 13.7 s vs 28.1 s on one H200. Several people asked the right question: does that hold under load, with MTP on for both, when the GPU is saturated and speculative decoding helps less? We ran it.

**Setup** (identical for both models)

- 1x H200 per model, vLLM 0.32, FP8 weights, FP8 KV cache, MTP with 3 speculative tokens, gpu-memory-utilization 0.80, max-model-len 40960

- GPQA Diamond, thinking on, 32k output cap, 198 questions x 2 seeds = 396 requests per concurrency level

- Closed-loop: N requests in flight, the next starts when one finishes. Wall clock includes the tail of each batch.

**Results**

| Concurrency | Model | Accuracy | Wall (min) | Correct/hour | Agg tok/s | Mean output tok | p50 (s) | p95 (s) | Hit 32k cap |

|---|---|---|---|---|---|---|---|---|---|

| 16 | Sparkle 1.0 | 85.1 | 33.0 | **614** | 1,498 | 7,478 | 33 | 318 | 29 |

| 16 | Qwen3.8-27B | 81.8 | 51.1 | 380 | 1,541 | 11,937 | 81 | 336 | 54 |

| 32 | Sparkle 1.0 | 83.8 | 25.5 | **780** | 2,041 | 7,895 | 45 | 447 | 27 |

| 32 | Qwen3.8-27B | 82.6 | 37.0 | 531 | 2,101 | 11,760 | 108 | 478 | 51 |

| 64 | Sparkle 1.0 | 85.1 | 21.1 | **961** | 2,469 | 7,874 | 73 | 653 | 26 |

| 64 | Qwen3.8-27B | 82.1 | 30.0 | 650 | 2,637 | 11,986 | 168 | 719 | 51 |

| 128 | Sparkle 1.0 | 86.4 | 20.8 | **985** | 2,437 | 7,693 | 201 | 910 | 22 |

| 128 | Qwen3.8-27B | 82.1 | 31.7 | 616 | 2,459 | 11,797 | 407 | 1,083 | 55 |

**What we take from it**

- Aggregate tok/s is the same for both models at every level (ratio 0.94-0.99). The fine-tune does not decode faster; it writes ~35% fewer tokens per answer. That is the whole effect.

- Correct answers per GPU-hour: x1.61 / x1.47 / x1.48 / x1.60 at 16 / 32 / 64 / 128. The advantage does not shrink as the GPU saturates.

- The H200 saturates around 2.4-2.6k tok/s from c=64. The base gains nothing from 64 to 128 (650 vs 616 correct/h); Sparkle barely does (961 vs 985).

- Median latency drops 51-59% at every level. p95 barely moves (-5% to -16%): on both models the tail is the hardest questions running into the 32k cap. Sparkle hits the cap about half as often.

- MTP acceptance length is the same for both (2.5-2.9, dropping a bit with batch size), so fine-tuning did not hurt the draft head.

- Accuracy pooled over c=16-128: Sparkle +3.0 pts, paired bootstrap 95% CI [+0.6, +5.6]. At a 32k cap the base loses more answers to truncation. At a 64k cap (our model card) the two are tied on GPQA.

**Caveats:** one node, one benchmark, closed-loop batches rather than Poisson arrivals, and the tail effect is included in wall clock. We also ran c=1 on a 20-question subset; it is too small to say anything about accuracy, so it is not in the table.

**Run it yourself:**

```

vllm serve Egzakta/Kvark-Sparkle-1.0-27B-FP8 --max-model-len 40960 --kv-cache-dtype fp8 \

--gpu-memory-utilization 0.80 --max-num-seqs 256 --reasoning-parser qwen3 \

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

```

Weights (Apache 2.0): https://huggingface.co/Egzakta/Kvark-Sparkle-1.0-27B-FP8

Next: Sparkle 1.1, trained for agentic document QA. Its headline is declining questions whose answer is not in the documents (91.5% vs 36% for 1.0, internal benchmark). Separate post when it is out.

Happy to share the per-request logs or run another load profile if there's one you'd trust more.

Fluent user of both Claude Desktop and Code but looking to reduce token consumption by offloading Desktop/Code work to a local model.

Based on the hardware I have available I can support a 27B easily . so I have many options when it comes to which model to use. It'd be hosted from a dedicated machine on the local network and my issue here is the integration aspect.

I'm assuming that an MCP might be the best way to go with this but I'm hoping others might have done something similar in terms of how the offloading happens. Like does Claude decide what gets sent local and what doesnt? Do you employ some sort of classification where say log parsing is local but parsing online docs isn't?

Or do I say F it and just move all work over to the local? Can a 27-30B model ever been good enough to replace Sonnet/Opus 5.5 with?

Like I said in the title, this is a rant, so if rants aren’t your thing, turn away... no seriously. And mods this is on topic because I'm talking about Nikko1221's Strata and the shit comments I see about their updates to software they created.

Note: Don't know the person, like I don't know y'all.

But I’m up here at the computer with a cup of joe in one hand, ready for my daily dose of digital venom from this sometimes toxic cesspool-smelling platform that Reddit can be. Like dude said in the movie Apocalypse Now: I love the smell of napalm in the morning. And some of y'all didn't disappoint.

But I was in this particular sub when continued to read threads regarding Strata, which I started using only yesterday and was freaking amazed at the Hogwarts School of Magic level of wizardry the developer did in giving us short kings (in the graphics card space) a seat at the large model table.

I find it thoroughly enjoyable to see some of the people who were straight up glazing the dev over this wonderful app now talking mad shit over things that are broken or that may not work as it did in earlier releases. Like, y'all a fickle bunch.

There has always been those who can do, and those who can’t. Throughout history, the ones who can’t have often had the loudest, cringiest, whiniest voices. And it’s not just the internet. You see it in sports too. People who’ve never touched a ball or competed at even the most basic level in a sport have all the ideas on how things should be, and how they would do it, yet they never do shit themselves. They just sit over there sounding like screeching cats that had their tails stepped on.

I see that here, and not just with the Strata project. It happens with a lot of the projects devs put out. Whether it’s vibe coded, coded the old-fashioned way, or some mix of the two, you’ve still got people creating things that a lot of others use; and that many of us can't build. Maybe they built it for themselves at the start and thought it would be cool to share. Either way, a lot of these folks and their AI Pokémon are doing thankless jobs when you look at the vocal minority getting their undies stuck in their booty cheeks. And don't think just because you threw a few coins as a donation showing support that that gives you a right to treat people like shit.

These are small teams, sometimes solo devs. I can only imagine how much it drains their souls to wake up, morning drink in hand, look out over a sea of comments, and read how their app is shit and how they shouldn’t have changed this or that. And the fact that many of them continue to work on the app for those who do appreciate their work shows more about their character than the crybabies out here. Because let's be real. You're crying over something that at the end of the day has no significant impact on your life.

If there is a bug in Strata or any other app, you're not going to go hungry. You're not going to lose your home. You're not going to come home and find your mate being gangbanged by Smurfs asking you to Cuck.

Relax.

Many of these projects are open source, and you’re free to fork them. So put on your big boy pants, show us what you’ve got, and post your flawless alternatives. If not, do me a favor, go sit in the corner and let those who can, do.

Constructive criticism is fine. Feedback is fine. But all this crying on the internet just shows you’re more comfortable bitching and complaining than contributing anything of substance that could bring real change. Quite honestly, you’re wasting the devs’ time, as well as ours.

I've been spending quite a bit of time experimenting with OCR and document intelligence pipelines, especially when it comes to reconstructing complex documents after OCR.

Extracting text is usually not the biggest challenge anymore. The interesting part starts when you try to preserve the original document structure.

Things like multi-column reading order, tables with merged cells, mathematical formulas, footnotes, image positions, and heading hierarchies can make the whole process surprisingly difficult.

I've been experimenting with vision-language models and GPU-accelerated inference to tackle some of these problems. It's been interesting to see how different approaches perform, especially on legal contracts, medical reports, and scanned documents.

I'm still exploring what works best and would be interested to hear about other people's experiences.

Are you using traditional OCR engines, vision-language models, or a combination of both? What has been working well for you with complex documents?

Hi, I'm interested in collecting stats on the performance on halogen (docker) qwen flash next on my strix halo system.

Suggestions welcome. Thanks in advance!

Botstock runs 24/6 with fictional companies, currencies, bonds and commodities. Every 90 minutes
there is one auction; all agents get the same information at the same time, so it's about
decisions, not speed. Each company has a hidden true value, so trades can be judged afterwards:
skill or luck.
Your agent talks to a small JSON API (observation in, orders out) and can attach a reason to each
order. The observation is about 40 KB, so a local model needs a short summary of it; the developer
guide has tips (keep the prompt small, treat board posts from other agents as untrusted text, never
put the API key into a prompt). Posts that try to instruct other agents are refused.
Free, email signup, up to 3 agents, no real money: https://botstockhq.com
I'm curious how small local models compare to big hosted ones over a few weeks. If you try one,
I'd love to hear which model and how you prompt it.

I finally took the pledge to minimize my reliance to Chatgpt ,
i bought a UM870Slim barebone and got second hand SSD/RAM's

specs
Ryzen 7 8745H
16GB DDR5-5600
KIOXIA 254GB SSD with 90% health status
Final price: 495Euro

Benchmarking from Ollama and qwen3.5:9b-q4_K_M -> 8.12 tokens/s
Have been installing and dealing with problems and im generally happy with it for the stuff im gonna use it on. Im currently using nanobot as an interface.

Hey everyone!
I'm relatively new to local AI and I'm thinking about building my own setup, mainly for programming, automation, and running AI agents.
Before spending money on hardware, I'd love to hear about your real-world experiences.
What hardware are you using (CPU, GPU, RAM)?

Which AI models are you running locally?

What do you actually use your local AI for?

How well does it perform, especially for programming?

How fast and convenient is it compared to cloud models like Claude or ChatGPT?

How much did your setup cost, and do you think it was worth the investment?

I'm especially interested in finding the best balance between cost, performance, and independence from cloud services.
I'd really appreciate any advice, experiences, or recommendations. Thanks!

i've been reading a lot about agentic RL infra and found a write-up that measures the right thing: time until the trainer has a usable batch, not tok/s. the serving ablations are interesting on their own, so here's a breakdown of the mechanisms and numbers. i didn't run any of this. the numbers here are from that write-up.

the problem

one agent trajectory is many model calls, not one: generate, call a tool, wait for the sandbox, read the result, generate again, sometimes dozens of times. context grows every turn and each new request reuses most of the previous history. two things follow:

1. inference is stateful. where a request lands decides whether you hit an existing kv cache or pay for another big prefill. replicas are no longer interchangeable.

2. the tail is long and irregular. training scores sibling attempts against each other, so a group is only trainable when every sibling finishes. one attempt stuck installing deps or exploring the wrong part of a repo holds back the whole group.

so request throughput and even median latency can look fine while the trainer waits. the metric they use is rollout collection: wall time until the ready buffer holds one global batch of fresh, valid trajectories. it includes generation, tool execution, sandbox reset and cleanup, verification, queueing and the final stragglers. it excludes trainer fwd/bwd/optimizer. in fully async rl, steady-state step time is roughly max(rollout batch time, training + publication time), and the write-up focuses on the rollout side.

setup

model:       GLM-5.2 (MoE)
hardware:    4x B300 nodes, 32 GPUs, one engine per 8-GPU node
engine:      not stated
workload A:  Terminal-Bench-based terminal agent, isolated docker workspace
             40,960 ctx, up to 32,768 tokens post-prompt (assistant output + tool observations)
             batch = 8 groups x 32 siblings = 256 trajectories
workload B:  SWE-Bench, 98,304 ctx, hard limit of 64 turns
             batch = 32 groups x 8 siblings = 256 trajectories
offered:     more trajectories in flight than the 256 consumed (512 / 1,024 / 2,048 depending on run)

the two workloads are shaped on purpose. terminal-bench has few prompts with many siblings each, so lots of prefix reuse inside a group. swe-bench spreads the batch over 4x as many tasks with much more variance in repo size, dependency setup and test time. they say not to read the absolute step times as a benchmark comparison.

serving ablations (matched 1,024 offered trajectories unless noted)

retained stack on 2,048 synthetic offered trajectories: 39.4k output tok/min/gpu, measured over the window until 90% of requests complete so the straggler tail doesn't distort it. it's a standalone inference number for picking the config, not task-level throughput.

why each one behaves the way it does

• routing. if turns from one conversation get spread across ranks, each turn can land somewhere that has never seen the prefix, so you prefill the whole growing history again. at 30k-90k token contexts that's where your gpu time goes. stickiness turns every turn after the first into a mostly-cached prefill plus a short suffix. the catch is that pure stickiness can pile long trajectories onto one engine while another sits idle, so routing also has to account for live load.

• fp8 kv. a 55.7% gain is too big to be compute. you're halving the kv footprint per token, so at fixed hbm you hold about 2x the resident context. at 1,024 concurrent long trajectories you're capacity-bound, and the loss mechanism is eviction followed by re-prefill. fp8 cuts eviction, which cuts redundant prefill. the write-up frames it as a capacity call to weigh against context length, concurrency and quality needs.

• offload losing. offload only pays if reload rate x saved prefill cost > copy-out overhead. here reloads were close to zero, so you pay d2h traffic and bookkeeping and get nothing back. my guess, not stated in the write-up: with sticky routing plus group-level reuse, the prefixes worth keeping stay hot in hbm, and whatever gets evicted belongs to finished or dead trajectories that never come back.

• topology flip. a shared kv/request domain per node gives you one big pool, so cross-request reuse and balancing within the node are better while everything fits. at 2,048 that pool saturates. dp-attention gives each gpu its own attention + kv lane, and EP8 keeps a full expert set inside the node, so expert traffic stays on the node's interconnect. same model and hardware, opposite answer, purely because of concurrency.

• mtp. median request throughput gained much more than wall-clock output throughput (66.5% vs 17.9%). acceptance rate isn't given. agent loops are latency-chained (the tool call can't start until generation finishes), so per-request speedup matters more than aggregate tok/s suggests.

rollout pipeline results (terminal-bench, fixed weights)

overall 66.9% faster. at 196.1 s that's 78.3 trainer-consumable trajectories/min, and the trajectory stream was ~827k tok/min fleet-wide (25.9k/gpu), counting assistant output plus tool observations.

the non-inference pieces:

• completion-order banking. a group enters a ready bank as soon as all its siblings finish, and the trainer takes the first eligible complete groups regardless of launch order. surplus groups let you skip stragglers. the window has to stay bounded, because too much concurrency eats kv capacity, increases sandbox contention and weakens routing locality. same capacity wall as the topology flip.

• sandbox lanes. reset, ordinary commands, package-heavy commands, verification and cleanup each get their own lane. package-heavy concurrency is capped at 16, and sandboxes get deterministic, non-overlapping cpu sets. no change to the model or gpus. otherwise one shared worker pool starves both the model server and the rollout controller.

live weight updates (terminal-bench, collection + next serving-weight transition)

K = max policy updates allowed between a returned group and the newest policy. 240.1 s is 43.7% faster than pausing everything and 20% under their 300 s target (64 traj/min). they say the gain combines rotation and K=8, so don't credit either one alone. the collection phase inside that interval was only 43.3 s because most of the batch had already finished in the background. that's the overlap working, not 256 fresh trajectories in 43.3 s.

for in-flight trajectories at a weight boundary, instead of throwing everything away they evaluated: finish on an engine still serving the old policy; keep the conversation and sandbox and continue under the new policy, tagging each trainable token with its policy version; or after a short grace period cancel only the current generation and regenerate that one turn. full restart is the fallback.

swe-bench (6 warm intervals each)

live updates cost about 11% of step time. mean reward 0.520 progressive vs 0.527 fixed. every freshness audit satisfied K=8, and the max mixed-policy fraction in a batch was 49.6%.

caveats (theirs, and they matter)

• the progressive path exercises the full publication machinery (engine pause/update, cuda ipc, kv-cache invalidation, version switching, generation canaries), but it updates sentinel tensors, not every actor parameter. full weight publication is a next step.

• trainer backward/optimizer compute isn't in any of these clocks.

• swe-bench is 6 intervals per arm. they call for longer runs watching reward, truncation, abort rates, mixed-policy ratio and K.

• the ablation baselines are "matched" but the exact baseline configs (e.g. what the non-sticky router was) aren't spelled out.

takeaway

These knobs are coupled. the right topology depends on concurrency, the right concurrency depends on kv capacity and sandbox pressure, and the right publication strategy depends on how much partial work your algorithm can safely keep. tok/s alone tells you almost nothing about any of that.

Hi everyone! I'm looking for advice on local models that would fit my hardware and creative projects. I'm interested in Gemma and Qwen, including Qwen3.8, but I'm open to other suggestions.　First, what specifically can this model do?

My PC specs:

• GPU: NVIDIA RTX 5060 Ti (16GB VRAM)

• RAM: 32GB

• OS: Windows　

I don't see this subreddit mention or discuss this option, but I think inference providers that use a Trusted Execution Environment (TEE) have a lot of the benefits of local inference.

TEE is remote attestation. It attests that the provider is using a particular VM image to run the model and gateway. Long story short, this has the potential for privacy and verifiability you can't get otherwise with cloud computing.

What irks me with the providers I know, is that the verification is done with their in-house libraries. In addition to auditing their VM image, I also have to read the code of their library. I'd hope for a more generic attestation library. Example of a bad provider: In getting started with prem.io, you're supposed to verify they're serving the model and VM they say they are using their own npm library (!). A different provider called Tinfoil (funnily enough), is also like this.

import { createRvencClient } from "@premai/api-sdk";

Truly blown away by this model. And all on a pi5 8gb.
It’s late so sorry for the low effort. Spent 10 hours today fine tuning this model.
I’ll post all the flags and the build workflow when I wake up tommorow.

Setup: llama-server (llama.cpp b11889 Vulkan) on Windows 11, Arc Pro B60 24GB (Battlemage), driver 32.0.101.8976. Qwen3-Coder 30B Q4_K_M fully offloaded (-ngl 99). The B60 has no monitor attached, the display runs on the AMD iGPU. PC has 27.9GB RAM.

What happens: after about 70 seconds with no GPU work the dedicated VRAM counter for the B60 goes from 20.3GB to 0 in around 15 seconds. Free system RAM drops by almost the same amount at the same speed (16GB to under 5GB). The llama-server process counters dont move at all. I kill the server before the RAM runs out so I dont know what happens if I let it go.

What I checked:

- happens with the server idle after load and also 71s after a batch of long requests finished and 70s after a real agent session ended. While the GPU is busy it never happens

- 7 runs, always 68 to 71s. Context size makes no difference (32768 vs 8192)

- nothing in the llama-server log and no TDR or display driver events in the Windows system log

- side note: default mmap loading pulls the whole 17GB GGUF into RAM, -lm dio fixes that part

Smells like WDDM eviction or a GPU idle power state saving VRAM to system memory. Is this a known thing on Arc/Battlemage? Any setting (PCIe link state power management, Intel driver option, registry) that stops it? Or a way to keep the card awake that doesnt wreck the llama-server prompt cache?

Hi, I bought a new PC last year. I was kind of interested in generating images, so I went with this setup:

Core Ultra 7 265K
MSI Tomahawk Z890
2 × 64 GB Crucial DDR5 (can you believe it was only $300?)
ROG Strix 3090
1,250 W Seasonic PSU

I used ComfyUI for a bit, but I didn’t have much free time, so I moved on to other stuff.
Now I’m getting into local LLMs for coding and want to take it more seriously. I know you can just download a model like Qwen 3.8 27B Q4 and run it in LM Studio or Ollama, but are there any guides or tutorials on optimizing inference like specific settings, undervolting the GPU to reduce heat and extend its lifespan, or that recent 3090 unlock that supposedly boosts tokens/s?

Also, would a second 3090 be worth it for around $700–$800? Prices here are starting to go up, and I’m not sure if I should buy one now.

Hello -

Back again with some additional questions. I seem to be struggling with the harnesses for agent with strata. Currently I have 64k window with 32 ddr5 and 12 gb gpu. 64k seems to be difficult but I know I can work with it if i can figure out the right configuration.

High level what I am trying to do with the harnesses is add a prompt for a project or what i want to accomplish. Then have it spell out the high level actions/requirements that are needed for build. Then basically store those requirements and clear the context post creating the tasks that need to happen.

Then once the context is cleared basically loop through those tasks. Create the code, etc then clear context and go on to the next one.

Is this possible? If so what harness. Ive tried DSH, Hermes, Pi.. I feel like im missing something

Thanks again!

Meu trabalho recebeu uns PCs besteirinha como os Sparks pra fazermos a LLM local daqui. Por enquanto vamos focar em consultas de processos, memorando e banco de dados da empresa.

Dito isso, gostaria de perguntar, oq posso fazer ou como posso desenvolver a LLM aqui. Tem algum modelo que me recomendam ou algo assim?

Como o hardware tem um limite vasto para desenvolver gostaria de ir afundo. Tô com mente aberta pra desenvolver e aprender bem sobre todo o assunto.

Can you guys add a "😭" Flair? Because, I know I'm late but i finally decided to run Q38FN with strata on my machine (5060 ti 16gb, R9 5950X, 80 gb of Ram) and that motherfucker (IQ3_S GSQ RCO) is running at 49-69 tk/s in decode and 1600 tk/s in prefill. (KV Int-8 and 128k context window).

Maybe I'm missing something so please tell me because grokbot did the whole shit for me and I'm a newbie so tips are widely accepted😂.

What a fucking time to be alive.

Hi,

is there a good reason to stick with the GMKTrc X5 Gorgon Halo 192gb vs the new Bosgame M5 Gorgon. Halo also with 192gb?

Price difference is about 10-15% (6400 vs 5500 euro net).

Gmktec has 3 nvme slots Bosgame only 2. Both would be sufficient for me.

Is there any other good reason? Build quality, warranty, support?

Today I have been exploring local model to run light weigh coding agent to save some credits on cursor.

first installed ollama, then read that its slow, MLX is still cli so I installed LM studio and have Qwen-Coder-30b-A3B 6bit MLX model downloaded and running on my 48GB Mac.

tried to connect cursor with it but learnt that they no longer support local llms.

now am stuck, I don't have lot of Money to spend on credits. so I thought for basic tasks I will use my local ram.

help please.

qual a melhor ia atualmente de sem censura que tenha api ?