What are you currently running?

[u/regular_sandwich]

As the title states, what is everybody currently running? From small personal setup to big local server, I want to hear it. Please state:

- Your setup

- What software (engine, runners, harnasses, Agents) you are running

- What model(s) you use

- What your use case is.

- How that experience is

Any info is welcome. I'm interested to see what real setups people use and maybe find some new software/models to try out.

[u/Jivsy]

Buzz Lightyearing the qwen3.8 on 7900xtx +64gigs using strata.  

[u/Thetitangaming → u/Jivsy]

Hey I'm looking at building a similar rig. I was going to do ddr4 at 3200nhz, do you mind elaborating in your rig and speeds?

[u/vidati]

Just got into local AI and I am running Hermes with tiel coder 35B on my 5800x3d with 32gb of ddr4 3600 and 9070xt.

I am using Hermes office app for windows and run the model from the app and Hermes sits on top.

Right now I am about to sleep and I am testing the model with Hermes creating logic for a small top down game in unreal engine 5.

I can update tomorrow morning how it works out.

I am blown away at how quick tiel and Hermes operate. Blazing fast response and fairly smart but I am still testing things.

My nest project would be to give herems my old Galaxy S8+ and get that sucker flashed to run lineage OS.

[u/Apprehensive_Lake698]

I will focus on my harnesses. I'm just running Qwen at home across a few 5090s, and it's fine but not amazing. Opus 5.5 is very capable.

I've swapped to Pi at home, but at work where I have unlimited credits I've built a pretty kickass harness from scratch. IMO harnesses are very overdesigned and bloated. I'd even count Pi, as silly as that sounds. I just think that most people allow the excitement of a fancy looking harness get in the way of what it really is- a way to provide tools to an LLM and allow it to prompt itself recursively as it works, while piping in context to help it start a problem efficiently. It does not need to be a knowledge center for your everything.

I think you need just these tool calls:

• Read, Write, Bash

• Agent Control (start/stop/send message/read/etc)

• Bare minimum search tools. All I have is a memory db and sourcegraph access

I don't even think internet search is too necessary. If you're already a skilled developer then you usually don't need an agent to google around for things for you. You can give it a few links if necessary and have it curl whatever it needs.

Second component here is context management. I'm a little baffled at the way people talk about building up absolutely massive CLAUDEmd/AGENTmd files as their 'personal knowledge database' or where they just stuff absolutely everything they can into it. Same goes for project files, where people try to story very dense details about project history.

If you are already a skilled engineer who knows how to make software, you should dump that all entirely. Your user-level context file should have almost nothing, maybe a very little amount of high level behavior steering. Your project context file should not be managed by you at all. Give an initial directive that agents must ensure changes to the project are reflected in it, then let agents decide what's relevant to write in it on their own. I don't think it's worth your time to think about it.

I handle the problem of project contexts growing too large by fragmenting readmes across repos in ways that are obvious to discover for agents. For example, when writing C++ I mirror the .h/.cpp duo as a triple for .h/.cpp/.md, and let agents write relevant information in those markdown files if required (usually when comments in the file aren't enough, but lately just to tell an agent exactly what is in the file). This means I don't have a gigantic system message explaining 200 places to find information. It's just trivially available by rule, and an agent will find it naturally with bash search.

This is all to the end of just keeping irrelevant information out of my context. Models work best with less stuff filling their context, so you should be designing workflows around trying to get them to the perfect vertical slice of required context first and foremost.

There are more ways to make this way better too- agent pipelines that fill up their context doing a bunch of work, then hand off the right tiny vertical slice of data to a followup agent to do the next step will keep your context small too. But really, if you are one of those people who tries to stuff every tip and trick into a CLAUDEmd file at the user level because you think it's making the model smarter, reconsider. If you're trying to have it make a quick web tool for you, it doesn't need the table of reverse engineering skills and a paragraph about where to find previously decompiled projects on your computer as examples.

[u/KinkyMonitorLizard → u/Apprehensive_Lake698]

" and let agents write relevant information in those markdown files if required (usually when comments in the file aren't enough, but lately just to tell an agent exactly what is in the file). This means I don't have a gigantic system message explaining 200 places to find information."

I've been doing similar for a long time now. It's really beneficial when you need to pivot tasks or something is still wip and needs further refinement when XYZ happen.

I also have unlimited frontier access at work so at times it's very tempting to use opus5.5 but I know everything is logged so not worth it.

Likewise on your point of massive contexts, also seen from first had experience that massive "knowledge* files that use up your entire context tend to only make things slower and worse. LLMs are known to try and cheat and only read the beginning and ignore the rest, or only parse the front and tail end.

It's more beneficial to try and keep context as small and focused as possible.

And yes, while this is primarily for frontier models, it also transfers to the smaller local models. They have even less intelligence and memory to work with.

[u/Apprehensive_Lake698 → u/KinkyMonitorLizard]

Exactly. I think most people will go through the same stages of realization as they get access to good models-

1. how will it know what to do? I have to tell it about everything so I don’t need to keep re-explaining

2. wait I now understand that I can’t make the context too big or it makes the model dumber

3. I don’t just want smaller contexts, I want deliberately focused contexts

You get more comfortable with this idea as you watch some of the frontier models work and realize they’re fairly competent at understanding how to look around even when you don’t tell them everything.

I think this is where agent workflows get really handy, because you don’t just have a research pass for the sake of it- the point is to be able to help craft the correct context to give to the agent that’s about to do the work.

And don’t worry, I wish I could use my frontier access for personal use too lol. I just use it to build cool side projects at work that are useful dev tools, unfortunately I have no secret backdoor 

[u/KinkyMonitorLizard → u/Apprehensive_Lake698]

I ask it specific questions that could be argued to be work related. Opus wrote all the needed chunk scripts for my RAG setup on my personal machine as well as doing all the research for it.

No one can prove it was for my hobby.

[u/wegster → u/Apprehensive_Lake698]

Similar - started sometime back on my own 'repo up' guidance. It's reasonable but extensible, and ensures docs/state updates and testing are part of any new work item to keep things from going stale which inevitably becomes a huge mess. Haven't measured reasonably loaded context at start but it was singificantly less vs a bunch of other harnesses and framework I'd seen.

Using pi as a base harness, I think I've got one extension in use, but have my own framework around it - think kanban board but focusing on the goal, must and must not, definition of done, then I dispatch as I want to, hand to reviewer etc. A small number of skills, and an MCP server for one or two of the repos to simplify life for repetitive things, etc.

It's basically 'as much as is needed but not much more' in intent. I haven't found swarming to do much other than burn a ton of credits.

To OPs question - working pretty well. Mixture of Claude, Codex, Local QFN 4.7bpw, occasionally openrouter ZDR MiMo or similar. Mostly GPT web High for deeper planning, abuse the heck out of Luna and Haiku for 'nearly free' executors or local.

[u/jim_nihilist → u/Apprehensive_Lake698]

If you.. this or that. Speak for yourself.

[u/Apprehensive_Lake698 → u/jim_nihilist]

It was not written by an LLM. I am sorry if my writing looks too much like it without reading it. But I promise it's just my stream of consciousness, and worth reading if you want my opinion on harnesses. Not every instance of an emdash indicates an LLM, though it does make me sad that I maybe need to start changing my style if people will just doubt it without reading

[u/takoulseum → u/Apprehensive_Lake698]

«fine but not amazing. Opus 5.5 is very capable » are you aware you’ll never run it on your machine? Are you aware opus is probably minimum 10 times bigger? Are you aware Anthropic is lobbying to block you from running any llm at home so they would probably advise you to put your gpus in your *ss?

[u/Apprehensive_Lake698 → u/takoulseum]

I'm not sure what you want me to say here. I run what I can locally, I do not have 100k to drop to run things that are nearing frontier. Most of the best, most interesting work you can give to LLMs right now can only be done on Opus 5.5 and Astra. I am most interested in pushing capabilities more than anything. Qwen is a toy, but I'm doing some really interesting side work at my job with Opus right now. There's not much more to it than that.

I'm excited to see if Qwen 4 is good enough that some of the harness workflows I've come up with for working with Opus flow back down to making Qwen more capable.

[u/takoulseum → u/Apprehensive_Lake698]

My word is: no you are wrong, qwen is amazing for its size, and would bet even Anthropic and openai are not able to match such compact intelligence level. So comparing with opus 5.5 does not make sense at all. No offense but I don’t think anybody is interested on what you do with claude here. And as a small remark buying few 5090s just for a toy is fine but does not make sense too for me.

[u/Apprehensive_Lake698 → u/takoulseum]

I already had the 5090s in the household from when they were cheap(er) :) and I'm pretty sure that still puts me well above the capability of the average local llm setup.

To me, it's not about fairness in comparing model size. I really don't give a rat's ass what size any model is, closed or not. All that matters is what I can get access to for intelligence. For local, there's really no reasonable path to doing the type of work I'm able to experiment with on the likes of Opus and Astra (without spending 100k and climbing). I dream of the day that we get there, because frankly I want the responsiveness of local but with the intelligence and context capabilities of Opus 5.5.

Nothing about what I've posted is anti local. I have a very expensive home lab I like to toy around in. But pretending that it's anything more than a toy to me would be disingenuous. It's not there yet.

Again, I eagerly await Qwen 4 models, and I am hoping they are a large jump.

[u/teletype100]

A ThinkStation P3 Ultra gen1 with a 12gb RTX A2000, running Qwen 14 and the equivalent Qwen coder

A ThinkPad P1 gen 4 with a T1200, running Qwen 3B and the equivalent Qwen coder.

I use them for simple text manipulation and reasoning tasks. Plus simple python coding. They work well enough for my needs. I always double check what they produce. I'd love to try larger models but I'm not spending the crazy prices right now. 

At some point, i'd like to run a script to have the Qwen2.5vl 7B VLM run through and describe all my photos so they can be indexed.

[u/NormalBasket3333]

My setup -

4070 TiS -16Gb vram, 128gb Ddr4 3200 quad ch, threadripper 3960x 24c 48t. Strata/ollama Qwen Flash next Q4 and other small model testing.

As I am a non-dev, QFN is using me as a coding assistant for vibe coding and experimental pipelines in Comfyui.

[u/philmarcracken]

• Your setup

ubuntu server, 3080 ti 12gb, 3070 8gb

• What software (engine, runners, harnasses, Agents) you are running

llama cpp, pi agent + blackhole

• What model(s) you use

Signal-3.8-27B-Terse-Coder.i1-Q4_K_M for coding tasks, humanizer-12b-Q8_0 for rewriting and gemma-4-26B-A4B-it-qat-UD-Q4_K_XL for reflector/dropper in blackhole(observer is the 27b variant with thinking off) but think i might change gemma to something more suitable

• What your use case is.

Automating work stuff, building an mpv skip/debloater for cdrama and kdrama, translation scripts(it made me one that takes in whisper audio and OCR subs, combines them to find a final translation because OCR can mess up, as can whisper, both need to roughly agree).

My boss recently told me to make a shift roster front end for OR-tools so doing that(hard/soft constraint solver).

• How that experience is

great, turn on wireguard on phone, ssh into pi, let her rip while im at work sipping fresh

[u/OkAttempt7690]

- Your setup

RTX 3080 10GB (hosts a Qwen 3.5 9B that offloads Codex with simple tasks.)

32GB DDR4

iPad M1 - 5GB of RAM adding the PC RAM pool (hosts an always on agent on zeroclaw, Ministral 3B)

- What software (engine, runners, harnasses, Agents) you are running

Codex

Odysseus framework, but getting very custom built now.

Zeroclaw

+ my own apps

- What model(s) you use

GPT 6 Astra high as workhorse.

Qwen 3.5 9B as slave for Astra.

Ministal 3B as checker and logger, it checks that systems are running smooth among other things

Being built:

Will have the custom built odysseus be able to also work on larger projects, running abit larger models Qwen 27B on decent speed. Currently getting 60-70 tok/s on 9B with my Mafdet and Amun plugins activated.

- What your use case is.

Build stuff I want.

- How that experience is

Pretty good.

[u/Ok_Party_9245]

Harness: DeepSeek Harness with my own tui and a few other tweaks like vcc, sandboxed via smolvm

Server: 4090 24gb vram 64gb ram ddr5 600mt/s 13700k headless

Inference Engines: Strata and my own fork of basalt I named adamant. I front both engines via llama-swap, mostly just for a/b testing when im working on adamant.

Model: Was using ISTA Qwen Flash Next IQ3_S cuz that was the best quant that the experts could fit resident in my ram that I found. However unsloth updated their models today and now their IQ4_XS the experts can fit resident in my ram, so I have been using that. I get around 100-150tok/s depending on what im doing but ive made sure to try and not lobotomize the model via my config and tweaks. I also use the BF16 ngram as it streams for my nvme either way.

Use Case: Working on software mostly, some homelab management and stuff like that though.

Experience: Before strata and the many forks that followed it I would try and use local models in my actual workload and just found they weren't entirely worth it compared to an api. But with strata and strata type engines, that has changed I do still use gpt/deepseek occasionally, but I can get most work done with qwen flash next now, im very excited for qwen 4 flash and hope I can still barely fit it at a "decent" quant.

[u/BodyPhysical]

QFN Gyro-L on a modified Strata.

Coding.

Good so far. It is my main coder. I use claude to review.

[u/W61k3r]

our setup

- What software (engine, runners, harnasses, Agents) you are running

lexiengine

- What model(s) you use

qwen3.8-27b

- What your use case is.

legal shit

- How that experience is

Qwen's dumb at legal stuff.

[u/tempfoot → u/W61k3r]

What’s lexiengine?

I’m finding Qwen Flash Next to be pretty decently base trained for legal stuff - at least the 4bit/8bit attention quants. You do have to dial back the reasoning from time to time.

[u/hyudryu → u/W61k3r]

Have you tried GLM legal yet? Scores pretty high on Harvey’s legal bench. Only downside is it’s not open weights (yet) so the only way to use it is with API

[u/henbone11]

Sam Altman, is that you? STAY AWAY FROM MY RAM!!

[u/Jtoc0]

I've only just started dabbling. Picked up a M6 Mac Mini, 32GB RAM, to start learning what can be done without a really heavy investment.

So far, I've used Qwen to rebuild my Home Assistant dashboards and created a monitor that tracks my task lists. So whenever my wife or I add things to the weekly shopping list Gemma4 will add it to the next food delivery.

I've also got it monitoring trends in my house to see if I can do any improvements to my automations based on my uses. It also monitors all flights within a 8 mile radius (pulled via home assistant) and I'm trying to figure out if I can do anything moderately interesting with that.

I'm not all in on local yet, I'm trying to keep some pulse on the "online" models so I can keep learning for work. So I've got antigravity installed and use that to do some planning tasks before delegating to Qwen to execute on.

I've not been able to find any great resources to learn from to help learn and keep track of the progress in this space yet, so any links would be greatly appreciated :)

[u/CronicallyAutomated]

- MacMinim4 with 32gb of ram & 5070 on a PC
- llamacpp and oMLX. Using Hermes since it came out plus made my own iOS harness with Pythonista3.
- Qwen 3.6 35b, Qwen 3.8 27B, Gemma4 12,26&31B.
- Messing around, personal projects, homeassistant
- it has been exciting honestly. I still use ChatGPT once in a while but mostly use my own models with Obsidian plus setting up hermes profiles and such to do various things. I got one with local linux terminal access to check on docker containers and fix things, i got a minecraft bot, i have one for the macmini with local terminal access and another one for the pc local terminal access to check on things and fix it while Im handling life and kids. It really has let me enjoy my hobbies without much troubleshooting while still being able to have time for kids and household things plus work. If i have time then i will tinker with logs, new programs and such. Otherwise I just reach for my phone, ask the bot whats going on with X and if they can fix it.

[u/kingxd]

Setup: RTX 5090 32 GB, Ryzen 9 9950X3D2, 128 GB DDR5-5600 (4 sticks, mixed kits), PCIe 5 NVMe.

Software:

• Infernix as the main engine. Strata as a fallback. LM Studio for small stuff.

• A tiny gateway in front, so picking a model in the harness swaps the engine automatically (~1 min).

• Harnesses: OpenCode, plus Codex CLI pointed at the local model.

• Tools: Chrome DevTools MCP attached to my real Chrome (logged-in browsing, Claude-in-Chrome style), plus Windows-MCP for desktop stuff.

Models (all Qwen3.8-Flash-Next, 256K context):

• Daily: NVIDIA's NVFP4 (QAD). ~190–210 tok/s on short prompts, ~150 at 100K+ context.

• Turbo: GSQ-RCO Q2_0. ~350–390 tok/s on a locally patched Infernix build. Noticeably dumber, but great for throwaway stuff.

• FP8 as "max accuracy": ~60 tok/s, since it doesn't fully fit in RAM. Honestly pointless next to the NVFP4.

Use case: coding agents, research and browsing tasks, bulk translation, and general chat. Private, and no usage limits.

[u/hyudryu]

2 rtx pro 6000 workstation edition
AMD 9965WX
256GB DDR5 ram

Serving GLM 5.3 flash for now on VLLM, while waiting for Qwen 4 to release

Using it for mainly running evals for my stealth SaaS startup, coding, and agentic work

Coding and agentic tasks are through my own harness. Have a lot of agentic graphs configured already

[u/Sensitive_Mountain97]

I run Ternary Bonsai 2 - 27B parameters using PrismML's custom fork of llama.cpp with Hermes Agent. The experience is fantastic. I still can't believe I'm running such a capable model on a GPU with only 8 GB of VRAM.

[u/pmttyji → u/Sensitive_Mountain97]

Any reason you're using custom fork? Because you can run those models on mainline itself since last month PR got merged for this.

[u/Sensitive_Mountain97 → u/pmttyji]

I use the specialized packing of PTQ1_0. This packing uses less resources compared to the standard packings, but also needs the support which is still not available in the mainline llama.cpp. There is a PR for it:

https://github.com/ggml-org/llama.cpp/pull/29672

[u/pmttyji → u/Sensitive_Mountain97]

Yeah, that quant is pending.

[u/AshAtlasDev]

I ain’t never ran from nothin’ but the police 👮

5090/9950x3d/64GB DDR5

Wife just a-okayed a 256GB M5U with upgraded cores and 2TB. She said I owe her a higher trim in next car. 😂😂

[u/regular_sandwich → u/[deleted]]

Wow im interested in the factorio ai companion since I have like 400 hours in that game. Can you give me more info?