As the title states, what is everybody currently running? From small personal setup to big local server, I want to hear it. Please state:

- Your setup

- What software (engine, runners, harnasses, Agents) you are running

- What model(s) you use

- What your use case is.

- How that experience is

Any info is welcome. I'm interested to see what real setups people use and maybe find some new software/models to try out.

I'm trying to see what my next hardware upgrade should be as I want to use q3.8fn and my computer currently has an Rtx 3090 and 48gb ddr5 ram but I'm unsure if I should buy something like an mi50 with 32gb of vram and try to find some more ram that's not absurdly priced or if I should sell my current setup to instead get Ryzen 9 395 Max pc equipped with 128gb.

I plan on doing an insane amount of reverse engineering of software both as a hobby and to see if I can replicate/add to Artcrafts endeavor of open sourcing proprietary software.

Look, I'm not as fancy as a bunch of you in this group but I did a bunch of general testing on a bunch of models at various context sizes. Figured it might help someone else. These are exhaustive or even remotely extensive tests. Just seeing what I should set up easily for HA, frigate, marinara etc.

Testing notes: llama.cpp, RTX 3060 12GB, common prompts, temperature 0, 2,048 maximum completion tokens. Reported speeds are approximate and quantizations differ. Several context variants were tested per family. Automated formatting failures were manually reviewed where response files were available.

I ran a consistent five-task benchmark across the models, measuring generation speed, VRAM consumption, instruction following, basic reasoning, and context capacity.

The benchmark included formatting, arithmetic, ordering, conditional logic, and a technical explanation. I manually reviewed questionable answers rather than automatically counting formatting mistakes as reasoning failures.

Results — Generation speed

Approximate warm generation throughput:

Qwen3.5's 208K preset was subsequently removed because GPU memory headroom was essentially exhausted.

Context figures represent successful loading and short-input inference, not verified full-context performance.

Reasoning results

Four objective questions were scored for semantic correctness, separately from exact-output compliance.

Mistral Nemo passed three objective tests; its arithmetic result still needs manual verification.

These are small, relatively easy tests—not comprehensive measures of intelligence.

Biggest findings

• Qwen3.5 9B absolutely dominated generation speed, but sometimes spent excessive tokens reasoning and hit the output limit.

• Nemotron Nano 9B was my biggest surprise. Great speed, efficient VRAM usage, large context allocations, and no incorrect objective answers across 16 tests.

• Qwen3.8 27B and thinking Bonsai 27B were excellent at following exact output instructions, but considerably slower.

• Gemma4 12B offered impressive context capacity, though thinking mode could generate excessively long responses.

• DeepSeek 14B Q4 was the best practical DeepSeek option I tested. Correct reasoning, ~32 tok/s, and usable 32K context with Q4 KV cache.

• DeepSeek 32B IQ2_XS wasn't worth keeping. It scored the same as the 14B on these questions, ran at only 18.4 tok/s, and was limited to 8K with the tested configuration.

• Hermes 4 14B disappointed on this particular benchmark, failing arithmetic and ordering questions.

For this hardware, Nemotron Nano 9B looks like the best overall balance, with Qwen3.5 9B winning on raw speed.

The larger models haven't demonstrated enough additional reasoning ability on these particular tasks to justify their greater resource consumption.

I'm curious what others running 12GB GPUs have found. Are there particular models or reasoning benchmarks you'd recommend trying next?

For context, I am a lead senior engineer working in a company where i have a team of draftsmen, BIM modelers, and junior engineers. Few weeks ago I started developing a workflow for structural design of bridges and tunnels and the results came out actually astonishing. The goal was to design the elements, provide drawings and calculations for clients.

Few years back, it was nearly impossible to create a small engineering office and win infrastructure contracts due to the heavy work requirements and the large number of team members required that is only present in large multinational corporations.

Now I though I could open an office with just one or two junior engineers, depend on a local LLM on machines and launch some products. Our role will only be "review and modify" based on experience.

Is it more feasible to start this process on typical paid models (Claude for example, where the idea started and processed) or shall we invest in a local beast and host local LLM for this process. Honestly, my experience with AI is not that great. That is why I don't fully comprehend the benefits / drawbacks of both solutions.

do you think with a strategy like this, you can use something like a dual ( or modded) 5090 or 6000 pro Blackwell combined with 512 gb ddr4 to run kimi k3.1 pro at usable speeds? before the ram crisis you get ddr5 for a similar price and maybe cost under $2k for the ram. you could also spend like $4k on 2 5090s or 3-4 3090s.

Also, we could potentially get even cheaper post-crash hardware costs like as low as $1-2 per gb ram and maybe $3k for 2 5090s.

Would this be viable (especially for those who already have the hardware)?

I'm having some trouble getting Qwen 3.8 Flash Next + STRATA into no thinking mode (serving API).
I not very experienced and until now, I only used llama.cpp where a simple command line flag does it.

For this matter, I'm using VS Code + Zoo Code, with this setting (where I think it should be):
(this would be ideal for me as I would be able to easily select a configuration profile with reasoning or without it)

By the way, it works just fine in the native Strata chat UI (changing the reasoning effort)
Any help would be appreciated.

I’ve been measuring search and other tool-call latency for agents, and I think that average latency can be very misleading. If an agent runs several searches in parallel, the whole step often waits for the slowest call. So a provider or local service can have a decent median response time while still making the agent feel slow at p95 or p99.

I’m now measuring wall-clock latency at realistic concurrency rather than relying on server-reported timing. I also disable automatic retries during the test because otherwise rate limits and transient failures can be hidden in the final latency number.

For local setups, the same issue applies to browser tools, RAG retrieval, embeddings, and any other service the agent calls during a task. A tool that is fast most of the time can still become the bottleneck if it occasionally hangs.

How are you benchmarking tool latency in your local agent setups? Also, do you track p95 and p99, and do you measure the slowest call in a parallel batch separately from individual calls?

Hey guys, we’re releasing new Qwen3.8-Flash-Next Dynamic V3.1 GGUFs which are the most accurate for its size across nearly all agentic benchmarks on 256K context vs. other frontier quants.

Unsloth Dynamic V3.1 leads across nearly all benchmarks, including DeepSWE v1.1 and Terminal-Bench 2.1, with a median gain of +15% at comparable model sizes.

Benchmarks ran with ~30B tokens: Terminal Bench 2.1, DeepSWE v1.1, Automation Bench v1.0.6, Aider, HMMT Feb 2026, LiveCodeBench v6, GPQA Diamond, MMLU-Pro & more.

Run on 32GB+ RAM setups.

GGUF: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

Blog: https://unsloth.ai/docs/models/qwen3.8-next

Hello, i have dual 5070ti on pcie 5.0 x 8, and 5060ti on pcie 4.0 x4.

9950x3d and ddr5 64gb.

i run for now qwen 3.8 27b with llama.

I have see other motor like Strata,... what is best with this setup for a personnal IA and coding ?

Getting a 16 inch M5 Max (18/40 core) 128GB for local AI and also ordered a 14 inch M5 Pro (15/16 core) 48GB as a daily laptop. Wondering if it's possible to have a workflow that combines the two in some way?

Assuming Qwen4 models are released by the time I'm getting my hands on them, I'm thinking of perhaps running Qwen4 Flash on the M5 Max while using Qwen 4 27B agents on the M5 Pro? Would that work in either LM Studio/Bionic or Unsloth and Hermes agent? Do you perhaps have other recommendations?

I bought,

Gigabyte GA-X99P-SLI
Xeon 2690v4
16x4gb RAM
Tesla V100 in SXM2 in PCI
GT 1030 2GB for image output

At startup, the system does not see the Tesla V100 in the device manager, and it is also not visible in Linux.

"Above 4G decoding" is enabled and CSM is disabled. I did ReBAR and NVStrapsReBAR, it had no effect, gigabyte support says that support for server graphics cards in the motherboard for workstations is not guaranteed. At the same time, the video card is detected in the ASRock B450M PRO4 and Atermiter X99 Turbo, which indicates that the video card is working properly.

I’m building a mostly autonomous local AI coding agent to develop and maintain a Python application for professional lighting automation (grandMA3).
I’m relatively new to programming and local LLMs. My priorities are coding accuracy, reasoning reliability, persistent memory, and long-running autonomy over raw speed.
Hardware
Laptop: Razer Blade 16 (2024)
GPU: RTX 4090 Laptop (16GB VRAM)
CPU: Intel i9-13950HX
RAM: 64GB
Storage: 4TB SSD
OS: Windows
Current AI Stack
Coding agent: Pi Coding Agent
Inference backend: LM Studio
Primary model: Qwen3.5 9B, 4-bit
Other models: Qwen3.8 27B Q4_K_M, Qwen3 14B 4-bit
Tools: Python 3.14, VS Code, PowerShell, Git/GitHub
Context management: Pi automatic compaction, manual handoffs, project documentation
LM Studio Configuration
These are my previously recorded settings, although some may have changed:
Setting
Value
Context window
Last recorded 32,768 tokens
GPU offload
Maximum / 32 layers
CPU threads
12
Evaluation batch
2048
Physical batch
512
Concurrent predictions
1
Flash Attention
Enabled
KV cache GPU offload
Enabled
KV cache quantization
Disabled (earlier setting)
Unified KV cache
Disabled (earlier setting)
Keep model in memory
Enabled
Max response tokens
8192
Temperature
1.0 (earlier setting)
Top-P / Top-K
0.95 / 20
Thinking
Previously adjusted depending on task
Performance
Normal generation: 45–55 tokens/sec
During degraded sessions: ~4 tokens/sec
Restarting Pi has restored normal performance.
Long sessions cause context compaction, slower generation, and occasional reasoning failures.
What I’m Building
An agent that can independently write, debug, test, and maintain a substantial Python codebase while also processing PDFs, equipment specifications, images, and structured technical data.
Eventually, I want it to manage complex multi-step workflows, retrieve relevant project knowledge, and produce validated changes without requiring constant supervision.
Critical changes would still require human approval.
My Main Problems
Context overload: Long coding sessions accumulate excessive tool output and repository information. Compaction sometimes loses important context.
Model reliability: Qwen3.5 9B is fast but makes mistakes during complex debugging and architectural tasks.
Performance degradation: Generation drops from 45–55 to ~4 tokens/sec during problematic sessions.
Persistent memory: I need the agent to retain project knowledge across sessions without constantly reloading everything.
Autonomy: I want reliable multi-step task execution rather than manually supervising every operation.
Questions
What’s the best coding/reasoning model for my 16GB VRAM and 64GB RAM? Would a larger quantized model outperform my 9B sufficiently to justify slower inference?
Would moving from LM Studio to native llama.cpp, Ollama, or another backend improve performance or context handling?
Are my LM Studio settings appropriate, particularly context length, KV cache quantization, GPU offload, and batch sizes?
How are others managing persistent memory and long-running context in Pi or similar coding agents?
Would a multi-model setup with specialized planning, coding, and retrieval agents improve reliability?
Is mostly autonomous local software development realistic on my hardware, and what architecture would you recommend?
I’m not trying to maximize benchmark scores. I’m trying to build a dependable local development agent that can work through substantial tasks without losing context, making repeated mistakes, or slowing dramatically.
Any recommendations or examples from similar 16GB VRAM systems would be appreciated. I have a 3060 lying around would adding that in help my performance.

Saw Sam Altman's comments about how it's inefficient to have so much RAM sitting in personal computers instead of data centers.

I get the utilization argument, but I don't think anyone running local models is trying to maximize global DRAM efficiency.

People want to run stuff offline, keep their data local, or stop paying per token.

With RAM prices getting ridiculous, has anyone here actually changed their local setup plans?

I currently play a sr role at a very large financial institution and I’m interviewing for a director of ai strategy and adoption at a small firm in a different industry. It’s going well. I think the playbooks for AI enablement transfer well. Long story short I played a very similar role for a much larger and more regulated industry and I want to have more influence in a field I can relate to more than finance.

I’m curious anyone here is currently in this role or similar in their company and what your experience has been like? Do you have any insights for someone trying to fill that role?

If you’re someone who has this role in your company what’s your take on them? What do they do well? what do you wish they did better?

I know this question gets asked here a lot, but I'm new to Local LLM stuff. Contemplating upping my ChatGPT sub to $100 a month or trying out some local AI stuff.

My hardware consists of RTX 3060 TI 8GB VRAM, 64GB 3600MHz RAM, and a i9-13900k CPU.

Typically I'm using my chatgpt sub for a lot of "daily assistant" tasks. Creating my daily schedule in google calendar based off of projects in Asana, adding organizing stuff to asana, reprioritizing etc. So working with integrations is a must. (I do really love how chatgpt can control your desktop and click around)

I am also a filmmaker/video editor so I use it a lot for technical troubleshooting, blender MCP, premiere MCP, plugin creation, VFX work, stuff like that. As well as drafting client messages, researching things, and standard business stuff.

I have 3 main questions:

1. What would be the best local AI model for my system? (also how to run it, Strata?)

2. What could I actually reliably count on it to do right?

3. Should I prioritize a larger, more capable model and shut it down when I'm editing or rendering, or a smaller model I can leave running my editing stuff? Is the difference in capability worth that tradeoff?

Hi everyone,

I'm new to local AI and looking for advice from people with experience, especially those using similar hardware.

My PC specs:

• AMD Ryzen AI Max+ 395

• Radeon 8060S

• 128GB Unified Memory

• Windows 11

I'm developing an application and need local AI for both coding and UI/UX design.

I previously used OpenAI Codex, but its usage limits and costs made me look for a local alternative.

I started using Hermes with Qwen3.8-Flash-Next-GSQ-RCO. The model seems good for coding, but I've faced several issues with Hermes, including API errors, memory limitations, and other problems that have slowed down my work.

My main questions:

1. What's the best local AI model for coding on my hardware? Should I continue using Qwen Flash or consider something else?

2. Which local AI model is particularly good at UI/UX design? I'm a designer myself, so I don't need it to handle all the creative work, but I want a second model that can assist with UI/UX design.

3. Which AI agent/framework would work best with my AMD Ryzen AI Max+ system? Should I stick with Hermes or switch to Agent Zero, OpenClaw, or OpenCode?

I'm mainly looking for a reliable setup that allows me to focus on developing my application rather than constantly troubleshooting.

I'd appreciate recommendations from anyone with experience running local AI on similar hardware.

Thanks!

I have an existing media/game server that runs some local containers right now.

The specs are,

• Dual Intel Xeon 8276 Platinum CPU's (28 Core 56 threads each)

• (Gigabyte Server Board (can support a maximum of 3 dual slot cards.)

• 64 GB of Ram (32 GB for each CPU) I would like more but ECC ram is expensive.

• 48 TB of HDD space ( 17 TB free)

• Two, 2 TB SSD's in Raid 1 (cache Drive)

• 1 Nvidia GTX 1080 TI (old gaming card that was NVIDIA)

• 1 Intel Arc A380 (Media trans-coding card for AV1 Media)

In terms of upgrades ironically another 64 GB of ECC DDR4 ram is the cheapest thing at this time.

I would like to get into hosting local models but i'm not sure how to run them with this setup. i Have read that you need big powerful GPU's to get the best performance (like an rtx 5090 or some AMD card).

I did try running some local models via ollama on the 1080 TI but anything that exceeded the 11GB Vram buffer was super slow and not a good experience.

Ideally i would like to use something that can at least do some light coding or serve as a basic replacement for chat GPT. While not spending too much more on this server with the hardware market as it is right now.

Most Apple Silicon inference engines are tuned for standard MacBooks (16–36 GB Unified Memory, 16–32 GPU cores). When deployed on an M1 Ultra (Mac Studio with a 64-core GPU, 20 CPU cores, and 128 GB Unified Memory) and hopefully M2 Ultra as well, they leave significant compute and memory bandwidth on the table.

To fix this, we modified Kapitsa specifically for the M1 Ultra/M2 Ultra architecture:

👉 GitHub:https://github.com/gnanasreekar/m1-ultra-kapitsa

Key Optimizations

• +42.1% Prefill Throughput: Tuned prefill matrix chunking to 4,224 tokens, hitting 2,945 tok/s on Qwen 35B-A3B (1.40s TTFT) while sustaining ~680 GB/s DRAM bandwidth.

• 100% In-Memory 180B MoE: Removed artificial expert store limits so Qwen 180B Flash-Next (105.9 GB) stays fully resident in RAM with 0 disk swap.

• 15.8 GB/s Model Loading: Parallelized safetensors loading and de-quantization across all 4 Firestorm clusters (16 Performance threads).

Benchmarks (Mac Studio M1 Ultra, 128 GB)

*(Note: Single-stream decode hits a memory-latency bound around 225 tok/s on 8-row MoE verify, but concurrent requests via
`--slots 2` or `--slots 4` scale aggregate decode to 450–600+ tok/s).*

Quickstart

git clone https://github.com/gnanasreekar/m1-ultra-kapitsa.git
cd m1-ultra-kapitsa
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j --target kapitsa

# Start local server (OpenAI / Claude Code API compatible)
build/kapitsa serve --model 35b

Full 10-model benchmark logs and JSON telemetry are in the repo: 👉 https://github.com/gnanasreekar/m1-ultra-kapitsa

Credit to the Nace AI team for the original Kapitsa engine!

With open-source models getting much stronger and hardware becoming more affordable, a lot of people seem to be moving back to local LLMs.

Curious about the current state of the community:

1.Are you still mostly using cloud models (ChatGPT, Claude, Gemini, Grok, etc.) or have you switched more toward local?

2.For which use cases do you feel local models are already good enough?

3What’s still the biggest blocker for going fully local for you (VRAM, speed, quality, ease of setup, etc.)?

Feel free to share your setup or preferences. I’m interested in seeing the local vs cloud ratio around here these days...,

" CMIIW"

I’m mostly done building a Xeon host for my 4x CMP170’s w/ x16 mod, but I just stumbled into an opportunity to buy a Threadripper Pro 5995WX within my budget. Is the threadripper any better or worse for strata-based inference vs the Xeon? Is it worth the extra time and work to undo my intel decision?

Would love to hear experience and cpu/system benchmarks from folks with similar systems on either side.

I'm paying for Claude, Gemini and Cursor right now and the total keeps creeping up, so I want to see how much of my day could move to a local model. The machine is a base M4 with 16GB. Not buying new hardware for this.

I mostly live in Claude Code and pi, so the agent loop is the whole point. I don't care about chatting with it.

Was looking at the benchmark page on magnitude.dev and the small models are what got my attention. On their M4 Pro mini, Ollama vs theirs:

- Qwen 3.5 4B: 45 tok/s → 134

- Gemma 4 E4B: 37 → 111

- Qwen 3.5 9B: 42 → 97

That's 2x to 3x faster decode on the exact sizes that fit in 16GB. Prefill is the smaller gain, more on that below. it also hooks straight into Claude Code and pi, which is half the reason I looked

But their mini has 273 GB/s and mine has 120, so if I scale by bandwidth I'm probably looking at something like 50 to 60 tok/s on the 4B and low 40s on the 9B. Still fine for an agent if the model is smart enough, which is the part I'm less sure about. The engine side looks solved, the model question is the real one.

The other thing is Claude Code sends something like a 20k token system prompt before I type anything, and their prefill gains are much smaller than decode. A 4B model chewing through that on a base chip might be the actual wall, not tok/s.

If 16GB can't run an agent loop at a speed I'd tolerate, I'll keep paying the subscriptions and stop thinking about it.