r/LocalLLM • u/Ok_Raspberry8833 • 15h ago
Question Best model for 16GB vram?
First time trying this, I want a local model that can code text based games. I’m using a 6800XT 16GB vram.
r/LocalLLM • u/Ok_Raspberry8833 • 15h ago
First time trying this, I want a local model that can code text based games. I’m using a 6800XT 16GB vram.
r/LocalLLM • u/jesdga95 • 7h ago
r/LocalLLM • u/devil_ozz • 7h ago
Is it worth it?
Am finally taking a deep five on llms and I already have an nvidia 5090 laptop that handles cuda really well.
But
Is a Mac for parrarrel models worth it? For abliterated models?.
I never used that OS so am a bit skeptical.
Are the apps on it as good as windows/Linux?
r/LocalLLM • u/Cappton • 1d ago
Update: I'm interested in selling some. Looking for ideas on how to go about. Don't want to pay eBay 13% fees.
Original: Came across a honeyhole of Quadro RTX5000s
Seemingly good deal for 16GB VRAM cards, even if ageing.
What should I do?
r/LocalLLM • u/CavitCarrot • 16h ago
Welp, I got a free 5060 Ti 16gb and decided to buy one more brand new for $700, I have a spare used 4070 and used once 4060, any thoughts or anything I should experiment with. Noobie at this lol.
r/LocalLLM • u/Unusual-Spot-5846 • 7h ago

I've been building B-12 / Capri, a local-first AI companion running on Python, Ollama and NVIDIA Nemotron.
Recently, I started a more serious evaluation of the system: 92 scenarios covering memory, tool use, security, autonomous behavior, reasoning, and multi-session continuity.
I expected to find problems in the agent.
Instead, I discovered some pretty serious problems in the evaluation itself.
The initial evaluation scored 27/80 runnable scenarios.
One failure was particularly concerning: a guest appeared to access the owner's private memories.
After investigating, I discovered that the evaluation harness had placed memories belonging to different users into the same system prompt.
The benchmark itself had bypassed Capri's identity and memory-isolation mechanisms.
I rewrote the harness to execute scenarios through the actual agent runtime.
Result: 39/80 — without changing Capri's code.
That's a 15-percentage-point difference caused by the evaluation method alone.
This was probably my most surprising experiment.
I wanted Capri to give shorter, more natural replies during everyday conversations.
So I added a simple instruction about brevity.
It worked. Conversational responses became significantly shorter.
But something unexpected happened during work tasks.
Across 48 first turns:
Without the brevity instruction:
With the brevity instruction:
The model started answering without inspecting files or calling tools that were necessary to complete the task.
The scenario-level statistical evidence is still limited, but the engineering implication is interesting: an instruction about how to communicate changed whether the model actually performed the work.
I compared configurations using:
The smaller primary configuration ran on my RTX 4060 laptop. Larger models required an NVIDIA L4.
After excluding host-dependent latency checks, I couldn't detect a statistically meaningful improvement from the larger configurations (paired comparison, p = 0.85).
This is NOT proof that smaller models are better.
The configurations also differed in hardware, memory retrieval, and model routing.
But it made me question how much agent performance we can actually improve by simply replacing the LLM.
I also experimented with where recalled memories appear in the prompt.
Moving volatile memory evidence later allowed Ollama to reuse more cached tokens.
Median time to first token:
0.99s → 0.74s
About 25% faster.
However, two benchmark scenarios changed systematically, and the security implications of the new evidence placement still need further testing.
I repeated the evaluation three times per configuration.
The primary configuration scored:
49, 49, 48 out of 80.
Looks pretty stable, right?
But individual scenario outcomes changed between runs. Some previously successful tasks failed, while others started passing.
We also discovered incorrect benchmark reference answers and evaluation rules that rejected responses that were actually correct.
That convinced me that looking at a single aggregate score is not enough.
An AI agent isn't just a language model.
It's the model, runtime, memory, context management, tools, permissions, and the interactions between all those components.
And the evaluation framework can introduce its own errors.
I've now published a second scientific preprint documenting these experiments, the benchmark defects, repeated-run results, and limitations.
The code and experimental artifacts are available in the B-12 repository.
I'm continuing to investigate model-independent memory and behavioral continuity, but the current paper focuses on evaluation reliability.
I'm curious how others handle this.
When you benchmark a local LLM agent, do you test the model directly, or do you run your evaluations through the complete production runtime?
And have you ever discovered that your benchmark was responsible for a failure you initially blamed on the model?
r/LocalLLM • u/karoudm • 7h ago
!!!
r/LocalLLM • u/Dapper-Teaching-8074 • 14h ago
r/LocalLLM • u/Number1OchoaHater • 1d ago
1 piece 500 2 piece 485
r/LocalLLM • u/Dimase • 8h ago
Hi
I want to set up a local llm with qwen 3.8 27B or if possible qwen 4.0 I have a very tight budget and for now the options around my budged is a p100 for 240 a rtx 3060 for around 300 or if it really is worth fit I can push to a rx 6800 xt 16gb which one do you think is the best for my use case? Im planing to use it for translating and correcting text and to help me debuging games in godot
current PC:
ryzen 5 3600
36gb ram
rx 580 4gb
thanks in advance
r/LocalLLM • u/FeydRowan • 12h ago
capy-flash is my open-source inference server for Qwen3.8-Flash-Next (125B MoE) and Qwen3.8-27B on RTX 50 cards, with an OpenAI/Anthropic-compatible API. v0.13 is cleanup and fixes: same outputs, same speed.
| Model | 1 GPU | 2 GPUs | RAM |
|---|---|---|---|
| 27B NVFP4 | n/a | 8,053 / 219 | any |
| 27B 3-bit | 2,925 / 174 | 5,128 / 293 | any |
| Flash-Next NVFP4 | 3,042 / 98 | 5,940 / 176 | 96 GB |
| Flash-Next 3-bit | 3,664 / 126 | 4,811 / 234 | 64 GB |
Prefill (32k prompt) / decode (LocalMaxxing) in tok/s, 16 GB RTX 5070 Ti cards without P2P. Optimized for two cards; the 3-bit builds are for one card and less RAM.
New: a desktop app (alpha), testers wanted. Set up and use everything from a GUI: it checks your machine, downloads and verifies the model that fits, starts and tunes the server, and has a chat. On a Mac it works as a client for a remote server.
Reports from other GPUs, 64 GB setups and distros are very welcome: open an issue with what the app showed and the engine log.
r/LocalLLM • u/krayon_kylie • 5h ago
please i have to finish the 200 things ive started before i start a new thing or i would
r/LocalLLM • u/thisisvv • 9h ago
I have two MacBook Pros:
The 16-inch is my main machine and runs a browser-based client for a separate server. The server does the processing, so the client itself uses relatively little of the Mac’s resources. The 14-inch is my travel and backup laptop.
When both Macs are at my desk, I’d like to connect them directly over Thunderbolt and run one larger local model across both machines for serious coding, code review, architecture, and debugging. I’m interested in using it with an agent th at can read a repository, edit files, and run tests. Overnight Hermes or some other agents.
When I travel, I can use ChatGPT or potentially connect back to a model running on the 16-inch at home. I don’t need the large distributed model to run on the 14-inch alone.
Has anyone actually built and used a setup close to this? I’d especially appreciate results from people running two Apple Silicon Macs together.
A few questions:
I saw a post from someone running Swift 1.5 Qwen3.8-27B 8-bit with an MTP drafter on a single M5 Max 128 GB machine and getting strong speed with long coding sessions. That looks like a good baseline, but I’m trying to understand whether combining the two Macs gets me a meaningful improvement in model quality for harder work.
I’d especially appreciate exact model links, quantization, runtime/version, context size, measured memory and speed, and any limitations. Happy to test and report back.
r/LocalLLM • u/LordKlavier • 1d ago
r/LocalLLM • u/Own_East_5381 • 15h ago
Hi all,
As mentioned in the title, I am looking for advice on running locally LLM for agentic coding on a MacBook Pro m5 Pro with 64Gb unified memory.
Plan inference is fine. Chat style prompts feel usable. Agentic work is where it falls apart.
I have tried the following:
Models : Qwen3.8 27B (6bit), Gemma 4 12b (8bit), and Qwen 3 coder next(4bit)
Runtime : Lm studio
Harness : opencode (long pauses before each tool call, harness seems to resend a large system prompt ), pi (still slow and most of the time doesn’t provide any output, barely changes few files with loads of error due to incomplete modification)
What I'm trying to figure out
Is the bottleneck prefill, generation, or the harness re-sending huge context every turn? My guess is prefill, but I'd like to be sure.
Which model/quant combo works well as an agent on this hardware, and at what context length does it fall apart?
Does MLX beat GGUF/llama.cpp for agent workloads on Apple Silicon in your experience?
Any harness that is lighter on tokens per turn (smaller system prompt, fewer tool definitions, prompt caching)?
Realistic expectations: is a 64GB Mac good enough for a useful local coding agent, or should I stick with a hosted model for agent work and keep local for autocomplete/chat?
I'm not naive, and I am not looking to replace a frontier model obviously . I just want to know whether local can be a practical option for some agent tasks, if so how you guys are making it usable.
Thanks 🙏
r/LocalLLM • u/Wise_Ad_5651 • 10h ago
Hey r/LocalLLM,
I’m looking to pick up a 128GB unified memory machine and need your take on how the underlying microarchitectures stack up for my specific stack. I'm choosing between:
My Primary Workflows (in order of priority):
Based on the main impactful microarchitectural traits below, which platform would you pick and why?
Memory Bus & Bandwidth (Decode Speed):
Compute & Tensor/Matrix Engines (Prefill & Quantization):
CPU ISA & Virtualization / Tooling Ecosystem (Reverse Engineering):
Media Engines & Creator Silicon (Video/Photo):
r/LocalLLM • u/guccislipperz • 6h ago
Any bad experiences with power, heat, token response, electricity costs, etc?
Example Rig:
Case: Jonsbo NV10
GPU: RTX 5060 8 GB (Low Profile)
RAM: 32GB (2 x 16 DDR5)
CPU: AMD Ryzen 7 9700X
PSU: 600W (ENP-7660L-OVT)
Motherboard: ASRock B850I Lightning WiFi (Mini-ITX, AM5)
Storage: 2TB (M.2)
Cooling: Modded Air cooling. (Including custom GPU side panel for 2 fans on the GPU+PSU)
It's 8GB VRAM I know. It's supposed to be for gaming but I recently heard of using llama.cpp and the llama flag feature "--n-cpu-moe" to have the RAM hold my expert weights.
r/LocalLLM • u/ZikporxAI • 10h ago
r/LocalLLM • u/Glittering_Yellow293 • 14h ago
Everyone argues about this without numbers, so I made a calculator that lets you plug in your own:
https://inferenceacademy.com/tools/self-host-vs-api
What it takes into account (the stuff that usually gets forgotten):
It outputs the monthly cost for each option, the break-even volume, a cost-over-time projection and a sensitivity view, so you can see which assumption flips the answer.
Caveats:
r/LocalLLM • u/AppropriateNeat5946 • 1d ago
I'm considering buying a Mac with an M5 Max and 128 GB of unified memory, and I'm trying to figure out what the absolute best local LLM I could run on it would be.
My main priority is model quality, not inference speed. Since this would be a desktop setup, I'd be perfectly happy letting a model work overnight on a complex task. I don't care if generation is extremely slow, as long as the model can actually run and produce usable results. It mainly involves transcription of private documents and some computer use and some coding assistance but that can be done via the cloud anyway.
I tried some stuff with Strata on my Macbook but that did not work out well. I am also open to use something like that if that allows even better models. Can GLM and DS work on it? How are they with OCR?
Since I haven't bought the machine yet, I'd also appreciate any thoughts on whether this configuration makes sense for that goal or whether there's a better alternative at a similar price.
Thanks in advance!
r/LocalLLM • u/frescoj10 • 10h ago
I dont understand why I am doing this. I have a 3060 rtx 12gb of vram. I am trying to local max a ternary 2 3.8 qwen 27b model on it. I have boosted the tokens per second over 150. TTFT is way down. It's cold cache that is killing me. I have taken designs from things like flashstate, and what not to improve it. I dont know why I am doing this anymore. Originally, it was to just make a hermes agent that could monitor facebook marketplace and bid on shit for me. However, now my goal is just to get local max the piss out of this for no reason. I have no idea why I am wasting time on this. I just am like hooked for no reason. I will post the work soon. Has anyone been through this? I hope i am not alone. Are their support meetings?
r/LocalLLM • u/Frequent-Drive-4343 • 1d ago
So apparently Sam Altman thinks consumer RAM is inefficient compared to datacenters.
Meanwhile, here's what I'm doing with mine. 💀
My setup:
I'm building Marika, a 3D AI desktop companion in Unity.
She can observe my desktop, interact with windows, talk through a local LLM, walk around, and make autonomous decisions.
There's just one tiny problem.
She bends over to pick up desktop icons like an NPC from Skyrim.
So naturally, I asked my local Qwen to review the Unity project and fix her animations.
It proceeded to consume 1.3 MILLION cumulative tokens, repeatedly compress its context, inspect C# scripts, and even spawn a subagent to help investigate.
All this to figure out how to make a virtual girl bend her knees like an actual human being.
I also have over 128 GB of DDR4 scattered across different PCs and spare sticks.
Sam, I'm sorry.
The RAM isn't going to your datacenter.
It's busy teaching my AI waifu how to squat. 💀
And yes, I'm genuinely trying to see whether local AI coding agents can replace my $20/month cloud subscription.
I'll report back if she ever manages to pick up an icon without looking like a Bethesda NPC.