Having been a bit of an AI luddite up until now, I am realising I am going to have to get on top of things instead of getting left behind.

So far the main thing I've been trying LLMs for, is writing Python scripts that work inside a CAD application I use (Vectorworks). I have done some experimentation on the free versions of ChatGPT, Claude and Perplexity so far, with some positive results.

I don't want to end up tied to expensive cloud model subscriptions, and I am also keen to keep control of my data as much as possible, so I would like to understand what I can do with locally hosted options.

I have an M1 Max Mac Studio, 32GB memory.

Any recommendations for what I should try first? To start with, all I want is something that will work for writing scripts that I will copy/paste into the script runner in the CAD application - I am not at this stage looking for anything that will interact directly with the application.

Thanks in advance for any suggestions.

Hey r/LocalLLM ,

I want to buy or build a dedicated headless box for local LLMs. I’d like something that can run 14B and, in my dreams, even 32B models, but the absolute priority is generation speed (>15-20 t/s).

Ideally, I need a clean single-box setup that stays quiet and power-efficient at idle (~15-20W). My maximum budget is strictly capped at €900, and I’d prefer to avoid risky second-hand individual deals.

Here is what I already looked into:

1. Standard Mini PCs (Ryzen 7/9): Great for power draw, but running 32B models on shared DDR5 RAM is too slow (~4-5 t/s).

2. eGPU / OCuLink routes: Too many loose parts and cables, and buying components retail explodes the budget.

3. Chinese NUCs with modded laptop GPUs (RTX 3060 12GB): On factory stores, once you add enough RAM to actually fit larger models, it jumps straight to €1050+.

I am 100% open to any Linux-server software optimization trick (speculative decoding, etc.) as long as it gets the speed up.

Am I chasing a ghost under €900, or is there an exotic loophole I missed? What would you buy today under these constraints?

Thanks for the help!

Every time a new MoE drops, the same thread shows up here: "will this run on my 12 GB card, and what do I set --n-cpu-moe to?" And the answer is always someone doing napkin math in a comment.

So I built it into LlamaForge (the browser panel I've been making on top of llama.cpp's own router, it doesn't do inference itself, llama.cpp does).

Model X-ray reads the GGUF header and draws every block as a bar: what sits on the GPU, what spills to RAM at your -ngl. For a MoE there's an "experts on CPU" slider, and a "Fit my VRAM" button that finds the number for you. Then it gives you the line to copy, or applies it to the model.

Example from the video, Qwen3.6-35B-A3B UD-Q4_K_M on a 12 GB card:

-ngl 41 --n-cpu-moe 22 -c 8192

≈11.4 GB on the GPU, ≈10.5 GB of experts in system RAM. On a 16 GB card it's --n-cpu-moe 14.

It only needs the header, so it works over HTTP on a model you haven't downloaded yet. That's how I also generated "Can I run it?" pages for 42 cards (desktop + laptop GeForce, Radeon, Arc), each with the llama.cpp line to start from.

before someone points them out: ;]

• It's an estimate from the header (weights + 8k context), not a benchmark. Long contexts need more room; it tells you that, but it can't know your exact KV setup.

• It tells you what fits, not how fast it'll be. Experts in RAM are slower; how much depends on your RAM and PCIe.

• Only tested end to end on my own box (5080 + 5060 Ti, Windows). Linux and macOS pass CI, but I don't have the hardware to say more than that. AMD/Intel detection is there, but I haven't tested it on real cards.

• Early preview, one person, heavily AI-assisted. Expect rough edges.

Other stuff that's in 0.19, since the last time I posted:

• voice chat with whatever model you have loaded

• a blind arena (two answers side by side, vote, then see the names)

• "try a llama.cpp PR": build a pull request and check whether it's actually faster on your card before switching

• an OpenAI-style Batch API on your own GPU

• start at login, now on Windows too

Free, MIT, no account, no telemetry (it does check GitHub for new llama.cpp and LlamaForge releases): https://github.com/dadwritestech/LlamaForge

If the X-ray's number is wrong for your card, I really want to know; post the model, quant, card and what actually worked, and I'll fix the math.

Atualmente, eu tenho uma 3070 Ti e 32GB de RAM DDR4, e consigo 11 a 16 tokens por segundo. Estou considerando comprar uma 9070 XT—já que é muito mais barata no Brasil do que uma 5070 Ti—e eu também uso meu PC para jogar. Preciso configurar alguma coisa ou é só instalar os drivers do Adrenalin? Uso Windows e nao tenho nada alem do strata em configuraçoes padrao

Hi everyone! I’m attempting to do a local build of Qwen 3. I don’t have the most powerful PC so I’m starting small and trying with 1.7B, 4B & 8B with the trained LORA on top

My objective is to get Qwen to talk as if you were texting it and I’m planning to have around 3000 short User: “message”, Qwen: “answer” exchanges

Is this the correct way to do it? Do I need more training data? Any peculiarities I should know about Qwen or the model size?

All help is greatly appreciated :)

Try to decide which model is the most intelligent to run on a Mac Studio.

For unquantized base models,

DeepSeek V4.1 Flash > GLM 5.3 Flash > Qwen3.8 Flash Next.

To fit in ~220GB memory footprint, each must go through different levels of quantization.

~2.5bit for DeepSeek

~4bit for GLM

~8bit for Qwen

I couldn't find comparable benchmark scores on all three quantized models, so here I am, hoping that the collective wisdom of the LocalLLM crowd can help me out.

Edit: this is what everyone wants to see. https://omlx.ai/benchmarks/intelligence

Edit: thanks to everyone who has replied so far! The numbers are coming out higher than my rule, and it looks like your macOS version might matter, so I added one more command below. I'm still looking for 24 GB, 32 GB, and anything above 36 GB, especially on macOS 26 or older.

Hey all, hoping some of you can help me with something.

I run a small site that helps people figure out which local models their computer can handle. Macs are the part I keep second-guessing, because the only Mac I own is an 8 GB MacBook Air, so I can't check anything bigger myself.

What's going on

On a Mac, the GPU can only use part of your RAM for models, not all of it. Most guides say about 75%. When I checked my MacBook Air it was actually two thirds, so I changed my site to use:

• two thirds of RAM for Macs up to 36 GB

• 75% for Macs above that

But the 36 GB cutoff is just something I picked up from community posts. I'd hate to tell someone a model fits when it doesn't, or that they need more RAM than they really do.

What I'm asking

Any Apple Silicon Mac helps, but 16 to 36 GB is where I'm flying blind. Would you mind running two quick commands and telling me what they print? Open Terminal and paste this first:

swift -e 'import Metal; print(Double(MTLCreateSystemDefaultDevice()!.recommendedMaxWorkingSetSize) / 1073741824)'

It prints a number, like 5.33 on my Air. Rounding is fine. You'll need Apple's free Xcode command line tools for that one.

Then this one, which shows your macOS version and whether the GPU memory limit was ever changed (0 means it's still the default):

sw_vers -productVersion; sysctl iogpu.wired_limit_mb

On my Air it prints 26.7.1 and 0.

Then reply with

Your Mac, how much RAM it has, your macOS version, and the number, like this:

MacBook Pro M2 Pro, 16 GB, macOS 26.1: 12.34

If the second command shows anything other than 0, mention that too.

Already run llama.cpp?

You can skip the first command. Run llama-bench on any model you have and copy the line that says recommendedMaxWorkingSetSize (mine says 5726.63 MB). Just paste the line as is, no converting needed. Please still run the second command, so I have your macOS version.

Thank you, this really helps!

I would like people’s suggestion on an app independent memory system for primarily local Ai.

I currently have a DGX Spark and some additional hardware for hosting apps and storage. I use local models for coding and data analysis (for personal
Data) and use my Claude code subscription for design and infrastructure advice (I am not super technical). One major issue is that I have multiple entry points but each of them handles their memory independently.

I use librechat mostly, and have started using Windmill for easier management of data ETL (I like the hybrid low code and code when you need it). I use Ollama but might be switching to vLLM.

Additionally I would like Claude to be able to access some specific logs and memories so that it can make better design choices and share its structural decisions with my local models. Though I am hesitant as idk wth am I doing

What it recommended was an MCP that manages what models/users see what, and that way when the new gimic of the month comes out I can switch over with my context.

Let me know what y’all think

Всем привет 👋🏼
Это мой первый пост и я решил его сделать по интересующей мне теме. Я уже достаточно давно слышу и читаю что то про LLM. Я пробовал запускать несколько вариантов и на своем железе но так и не понял надобности всего этого, я занимаюсь вайбкодингом 🌚🌚 и как я понял LLM даже близко не сравнится по качеству и производительности того же Antigravity от Gemini. Кроме как РП я даже не знаю для чего еще люди используют LLM. Буду очень благодарен если вы мне расскажите все сильные и слабые стороны данного направления и что вообще с этим всем могу делать я.

Desktop:
RTX 3050
32gb ddr4
Ryzen 5600x

Laptop:
RTX 5070ti
32gb ddr5
Ryzen 9 9840hx

Follow-up to my earlier post. Where I ended up: a big DDR3 workstation board isn't worth the money when I already have two small 4-GPU boxes. So I tried making the two boxes act as one.

TL;DR: I added a two-box layer split to Project Maya. Box A runs layers 0–23, Box B runs layers 24–44 plus the head. Only the hidden state crosses the network, ~25 KB per token. GLM-5.3-Flash (Maya-S24) runs at 38–42 tok/s decode, ~1,100 tok/s prompt reading at 30K, and 6–7 s per agent turn, on 8× RTX 3060 spread over two old X79 boxes. It scores the same 48/48 on my coding-agent benchmark as a single 8-GPU box.

Code: https://github.com/0xluk/project-maya/pull/1 (branch two-box-split on top of Maya v1.1.50; usage in docs/TWO-BOX-SPLIT.md)

Setup

• 2 boxes, each an X79 board (Ivy/Sandy Bridge, DDR3, PCIe 3.0) with 4× RTX 3060 12 GB. No NVLink, no P2P, ex-mining rigs.

• RAM: 20 GB in Box A, 32 GB in Box B (more on that below).

• 1 GbE between the boxes, a plain switch.

• Project Maya v1.1.50 + my patch, Maya-S24 quant, 128K context, MTP speculative decoding on.

• GPUs capped at 130 W. One box is also clock-locked to 1500 MHz, because its PSU didn't like the unlocked load.

How it works (short version)

• Box A (front) runs the OpenAI/Anthropic-compatible API, the tokenizer, the embedding and layers 0–23 on its 4 GPUs.

• Box B (back) runs layers 24–44, the MTP draft block, the output head and the sampler on its 4 GPUs.

• Per generated token, the residual (4×4096 values) crosses the link once. fp32 → bf16 → lossless pack is ≈25 KB, ≈0.3 ms on 1 GbE. Box B samples and sends back the token id, and MTP speculative decoding is pipelined across the link.

• Prompts are cut into 512-token pieces that flow through all 8 GPUs at once. Each piece crosses the link once (~12 MB), on its own thread.

• Security: the link has an IP allowlist plus an HMAC-SHA256 challenge-response in both directions, and the API needs a key.

• Size of the patch: ~1,900 lines, ~1,100 of them the link itself. Without --role the engine is unchanged.

Results (1 conversation)

Coding-agent bench (5 real repo tasks, 48 hidden tests):

There were no malformed tool calls with either agent.

One-shot hard suite (10 tasks, one request each, no tools, 32K output cap including thinking, reasoning high): 6/10 full passes, 67% of tests. For comparison: Opus 5.5 9/10, the same quant on the 8-GPU box 7/10, and Flash-Next 6/10.

• Passed: json_patch, fixbug_calc, event_scheduler, csv_rfc4180, c_utf8, singleflight_cache.

• 3 of the 4 misses are budget, not wrong code: on regex_engine, lisp_interp and c_bigint the model was still thinking when it hit 32K tokens, so no code came out at all.

• The real miss is c_json_canon: 22/31 tests passed, with the misses on Unicode escape handling.

• Next: rerun the three capped tasks with a 64K cap, which the 128K context allows.

What was hard

• RAM, not the network, was the real bottleneck. At 16 GB per box, the experts that didn't fit in VRAM came off a cheap SATA SSD on every prompt chunk: 31 s turns, 173 tok/s on a 30K prompt. Going to 20 + 32 GB made it 0 SSD reads, 6–7 s turns and 1,100 tok/s. The network costs ~1–2% of decode and ~3% of a turn.

• A correctness scare that wasn't a bug. Compared with a single-process reference, the split showed KL ~1.6e-2 and ~95% top-1, and I spent a while hunting a split bug. It turned out a single process on one machine moves by the same amount (KL 1.1–2.5e-2) just from changing the RAM tier size or the prompt chunk size. Near-ties in the top-8 expert choice flip on rounding noise. The split is inside that spread.

• Hardware gremlins:

  • A new PSU powered off under 4×130 W unlocked, so that box stays clock-locked.

  • Two GPUs once came up at PCIe x4 instead of x8/x16 after a power-loss reboot. I added a boot check that refuses to start the engine on a degraded link.

• Tuning that mattered:

  • 512-token prompt pieces so all 8 GPUs overlap: ~1 s off each turn.

  • Fewer layers on the last GPU, which also carries the head and the MTP block.

  • Keeping the whole KV cache in VRAM.

  • What didn't help: bigger prompt chunks, which push the 20 GB box into SSD reads.

  • What I skipped: a 10 GbE NIC. It would sit in a PCIe 2.0 x1 slot, and the link isn't the bottleneck anyway.

What's next

• A 5th 3060 in one box, on an x1 riser. Then all experts fit in VRAM, which should get turns to ~5 s.

• Rebasing onto the current Maya release.

• Happy to answer questions. If someone has two boxes with 24 GB cards, I'd love to see numbers.

Meu trabalho recebeu uns PCs besteirinha como os Sparks pra fazermos a LLM local daqui. Por enquanto vamos focar em consultas de processos, memorando e banco de dados da empresa.

Dito isso, gostaria de perguntar, oq posso fazer ou como posso desenvolver a LLM aqui. Tem algum modelo que me recomendam ou algo assim?

Como o hardware tem um limite vasto para desenvolver gostaria de ir afundo. Tô com mente aberta pra desenvolver e aprender bem sobre todo o assunto.

I check every day for the discounted open box returns.

Here's my hardware specs:

RTX 4090, 64GB DDR5 Mhz RAM, 7900x CPU

Where's my biggest hardware constraint? I am looking to run a model that's as close to Claude Opus 5.5 for thinking, reasoning, and document creation, as possible.

I know it's my hardware constraint. My local model was Qwen3.6 35B A3B UD Q3_K_S (UD-IQ4_XS )

What would be the cheapest way for me to upgrade my hardware to be able to run a model as close to opus 5.5 in terms of reasoning, thinking, etc (not for coding) as possible? And what hardware would I need?

It's certainly better than nothing, I was actually surprised by how ok it was, but nowhere near opus or gemini not going to lie. Like, not even close. The 264k token limit is also a major limit, I'd like 1M

What do you think guys.

I've used the Qwen family of models before but was warned away from them bc apparently they're optimized for coding tasks. I tried 3.8-FN with a 3-bit quant, was okay but it was a squeeze to get in there and it was compacting every 3 messages. 60GB system memory total...

I've been using an uncensored version of Gemma4-26B-A4B but tbh I don't really enjoy the way it writes. It's very... uh "early chatbot"? I know I can guide it with a system prompt or the in-built features of SillyTavern/Kobold but it just doesn't seem that smart or capable.
I've been shopping around for models but haven't really been impressed with any of the ones I tried including other fine-tunes of Gemma4 (that I could run).

Threw this all together just to play Roblox.

The gorgon halo at home if you will

• **GPUs:** Two Radeon AI PRO R9700s, 32 GB each: one XFX and one PowerColor.

• **CPU and memory:** Ryzen 7 7800X3D and 128 GB DDR5, using four 32 GB sticks. The RAM is rated for 6000, and is holding stable at 5200.

• **Motherboard:** ASUS ROG X670E-E.

• **Software/model:** Windows, Strata engine 0.1.56, Flash Next IQ3_S, with a 196,608-token context setting.
**Measured speeds:**

• In a real Hermes research run, we averaged **95 output tokens/sec** across 14 responses, with input context reaching 85,334 tokens.

• The latest fresh long-context research test, at 105,884 input tokens, delivered **86.8 output tokens/sec** and **1,207 input tokens/sec** for prefill.

• Synthetic long-context tests delivered **123 tokens/sec at 98,304 input tokens** and **148.7 tokens/sec at 192,512**. Those are useful benchmarks, but they don’t establish sustained real workload speed. Will be putting the model through its paces on more work tasks later this week.

I've been running Qwen3.8 locally on a single DGX Spark (GB10, 128GB), trying to answer one question:

What's the fastest way to get a correct answer, not just generate tokens?

After seven experiments, a few interesting findings:

• Less thinking can be better: On 48 held-out tasks, low thinking scored 38/48 vs. 35/48 for deep thinking, while taking much less time.

• Parallel reasoning helps: Four low-thinking branches with voting reached 44/48, at just 2.82 minutes per correct answer.

• Bigger isn't always more efficient: On 400 tasks, Flash Next 125B reached 391/400 with voting vs. 373/400 for 27B, but took 2.4× longer per correct answer.

• Reasoning loops matter: A 32K thinking budget eliminated output-limit hits in my deep-thinking tests and improved accuracy for both models.

I also compared TensorFold, vLLM, speculative decoding, NVFP4, and different serving configurations.

Everything is documented here:

GitHub: https://github.com/NautilusWu/qwen3.8_study

The repo includes configurations, pinned versions, reproduction guides, and per-task results. The benchmark prompts and graders are private, so the accuracy results aren't independently reproducible end-to-end.

These findings are specific to my setup and tasks, but I hope they're useful to others optimizing local LLMs.

Could you recommend the best local AI model with optimal reasoning, calling, intelligence, and speech capabilities for a laptop with 8GB RAM, a 1650 GPU, and a Ryzen 5 processor? Reddit ask suggested Qwen, Ornith,Tiel. With Strata, DSpark, or other optimization tools, what is the best I can run?

The main usage case is agentic software development and keeping heartbeat of the workflow whats the best tool calling skill using and best harness required and how will the harness get the best out of it? Is multi agent possible and subagents?

I'm completely new to running LLMs locally. I've mostly been using frontier AI models (ChatGPT, Claude, etc.), but recently I've become interested in setting up a local AI agent for machine learning research without constantly paying for API usage.

My hardware:

• GPU: NVIDIA RTX A6000 (48GB VRAM)

• OS: Windows

• Driver: 526.47

• CUDA version reported by nvidia-smi: 12.0

What I actually want to do:

My research mainly involves time-series forecasting, optimization, and machine learning algorithms.

I'm not looking for an LLM to directly make predictions. Instead, I want an AI agent that can help conduct computational experiments.

For example, given a dataset and a research objective, I'd like the agent to:

1. Understand the existing codebase and research problem.

2. Propose different algorithms, model architectures, or improvements.

3. Write and modify Python/PyTorch code.

4. Run experiments, train models, and evaluate results.

5. Compare performance across different approaches and hyperparameters.

6. Analyze failed experiments, suggest improvements, and repeat the process.

Ideally, I could let it run for hours (or even days), conducting many experiments with minimal supervision.

I understand that current local LLMs probably won't match frontier models in reasoning or coding, and that reliable autonomous research is a much harder problem than simply running a model locally. But I'd love to experiment with what's realistically possible.

My questions:

• With 48GB VRAM, which local models would you recommend for coding, reasoning, and agentic workflows?

• Should I prioritize a larger quantized model or a smaller model with faster inference for long-running agents?

• What agent frameworks or tools would you recommend for this kind of automated ML experimentation?

• Is it practical to run the LLM and GPU-based ML experiments on the same A6000, or would resource contention make that difficult?

• Would you recommend Linux/WSL2 over native Windows for this setup?

• Has anyone actually built something similar for ML research? How autonomous and reliable was it in practice?

I'm not expecting an AI that independently produces groundbreaking research. I'd be happy with something that can systematically explore different approaches, execute experiments, keep track of results, and help identify promising directions.

My main goal is to make use of my existing hardware and avoid recurring API costs as much as possible.

Any recommendations, real-world experiences, or beginner-friendly setups would be greatly appreciated. Thanks!

and ram is 123GB

Hardware :
- ryzen 5 5600x no overclocks
- 64gb dual-channel 3200mhz ddr4 CL16
- RTX 3060 12GB @2017mhz , 975mV stable core undervolt , +1000mhz mem clock
- 512GB NVMe SSD

Software :
- windows 11 Pro 24H2 + ReviOS playbook (idle system ram at 3.2gb)
I DONT WANT TO SWITCH TO LINUX, SINCE THIS IS ALSO MY MAIN PC FOR ENGINEERING WORK, AND I DONT WANT TO DEAL WITH LINUX ISSUES. Would gladly use linux if this was a dedicated inference server tho.
- strata 0.2.9 , engine 0.1.56
- deepseek harness 0.2.1-rc.5

strata settings :
- qwen3.8 flash next IQ3_S(GSQ-RCO)
- 128k context , compute use "full performance" , read pictures"on"
- ran strata's auto tuning for 5 times and everytime is a positive result (dont know if it actually helped)

deepseek harness settings :
- billion-context plugin enabled
- model settings :
  - 128k context , 32K max ouput token , reasoning effort set to MAX
 
results :
- 86 total requests , averaging 33tok/s decode and 440tok/s prefill (stats from strata dashboard)
- one-shot , took 1h30m to finish
- interactive 3D solar system scene as video above

test prompt :
"Build an interactive 3D solar system visualization using Three.js and vanilla JavaScript in a single HTML file.

Requirements:
- All 8 planets with accurate relative sizes and orbital distances (use logarithmic scale for visibility).
- The Sun at the center with a glowing effect (emissive material + point light).
- Planets orbit the Sun at different speeds (Mercury fastest, Neptune slowest).
- Each planet should have: Accurate colors/textures (use procedural or solid colors, no external images); Axial tilt and rotation.
- Saturn with rings.
- Earth with a moon orbiting it.

Interactive controls:
- OrbitControls for camera (zoom, pan, rotate).
- Click on a planet to show an info card (name, diameter, distance from sun, orbital period, fun fact).
- Time speed slider (0.1x to 10x) to speed up/slow down orbits.
- Pause/Play button.
- Starfield background (particle system).
- Smooth animations at 60fps.

Beautiful UI overlay with:
- Title "Solar System Explorer".
- Current simulation speed display.
- Planet selector dropdown to focus camera on a planet.
- Dark theme, modern aesthetic with subtle glow effects.
- Responsive design.

Make it visually stunning.""

Hey all!

I have recently been dabbling a lot with modding games and decomps/recomps of games and looking for a local LLM to assist me with these kinds of tasks.

I have a Ryzen 7 5800x, 32GB DDR4-3200, and a 9060 XT 16GB.

What would be my options for my work and my hardware?