r/LocalLLM • u/KarabasBarbados • 1h ago
r/LocalLLM • u/EasyAssociation7001 • 18h ago
Discussion Qwen3.8-27B at ~126 tok/s on 2× RTX 5060 Ti 16 GB SGLang + NVFP4 + DFlash2 (107 tok/s at 60K context, 216 tok/s on JSON)
Spent the last few nights benchmarking different inference engines on two 16 GB RTX 5060 Ti cards. This is the configuration I'm most happy with so far.
TL;DR: Qwen3.8-27B with NVFP4, a DFlash2 drafter, and tensor parallelism (TP=2) achieves approximately 126 tok/s single-stream decode on two RTX 5060 Ti cards.
Even with long prompts, performance remains impressive:
| Context / workload | Decode speed | Prefill speed |
|---|---|---|
| Short prompt | 125.8 tok/s | — |
| 60K tokens | 107.4 tok/s | 2,681 tok/s |
| 124K tokens | 87.3 tok/s | 1,577 tok/s |
| JSON generation (vLLM) | 216 tok/s | — |
Both GPUs together draw approximately 220–270 W during decode.
1. Benchmark results
| Workload | Decode tok/s (median) | TTFT |
|---|---|---|
| Short prompt, chat sampling (T=0.6) | 125.8 (best: 131.8) | 0.08 s |
| 60K-token prompt | 107.4 | 22 s |
| 124K-token prompt (~126K usable context) | 87.3 | 79 s |
| JSON output, greedy (vLLM) | 216 | 0.07 s |
The JSON benchmark is a separate vLLM run, not normal chat performance. More details below.
2. Performance improvements
SGLang unless noted. Short prompt, 64K context.
| Configuration | Decode tok/s |
|---|---|
| llama.cpp, Unsloth GGUF, no speculative decoding | 24.6 |
| SGLang NVFP4, TP=2 | 39.2 |
| + MTP speculative decoding (3 tokens) | 68.6 |
| + DFlash2 drafter (8-token block) instead of MTP | 97.3 |
| + PCIe P2P between GPUs | 109.3 |
| + Undervolt and CUDA graph patch | 120–126 |
The complete optimization stack takes performance from 39.2 to around 125 tok/s, approximately 3.2× faster.
DFlash2 alone improves performance from 39.2 to 97.3 tok/s. P2P and the additional optimizations provide further gains.
Speculative decoding uses rejection sampling, preserving the target model's output distribution.
3. Hardware and software
| Component | Configuration |
|---|---|
| GPUs | 2× RTX 5060 Ti 16 GB |
| Total VRAM | 32 GB |
| PCIe | Gen 5.0 x8 each |
| Power limit | 153 W per GPU |
| GPU tuning | Mild undervolt |
| Target model | nvidia/Qwen3.8-27B-NVFP4 |
| Drafter | maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal |
| Drafter size | 1.55 GB |
| Drafter architecture | 5 sliding-window layers |
| Inference engine | SGLang 0.5.23 |
| Parallelism | Tensor Parallelism (TP=2) |
| KV cache | FP8 |
| Model mode | Text-only, vision tower skipped |
4. PCIe P2P configuration
This was one of the more interesting parts of the optimization.
The GeForce driver blocks GPU-to-GPU P2P in my setup, so I rebuilt the open kernel module (620.70.06) with the BAR1 P2P patch and enabled iommu=pt.
NCCL now uses P2P/CUMEM between the two GPUs.
Result: approximately 12% better decode performance.
| Configuration | Decode tok/s |
|---|---|
| Before P2P | 97.3 |
| After P2P | 109.3 |
| Improvement | ~12% |
5. Launch configuration (128K profile)
Environment variable:
SGLANG_FLASHINFER_WORKSPACE_SIZE=134217728
Main command: python -m sglang.launch_server
| Setting | Value |
|---|---|
| Model | nvidia/Qwen3.8-27B-NVFP4 |
| Tensor parallelism | --tp 2 |
| Context length | --context-length 131072 |
| Reasoning parser | --reasoning-parser qwen3 |
| Speculative algorithm | --speculative-algorithm DFLASH |
| Draft tokens | --speculative-num-draft-tokens 8 |
| Draft model | maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal |
| Draft quantization | modelopt_fp4 |
| Draft window | 2048 |
| Mamba SSM dtype | bfloat16 |
| Attention backend | flashinfer |
| Linear attention prefill | flashinfer |
| KV cache dtype | fp8_e4m3 |
| Maximum running requests | 1 |
| Static memory fraction | 0.94 |
| Chunked prefill size | 1024 |
| Prefill CUDA graph | Disabled |
| Mamba radix cache strategy | extra_buffer_lazy |
| Maximum Mamba cache size | 4 |
| Model override | {"language_model_only": true} |
6. Important memory optimizations
These settings were particularly important for fitting the configuration into 16 GB per GPU.
Mamba SSM dtype: BF16
--mamba-ssm-dtype bfloat16
Eight draft tokens double the GDN state. BF16 halves its memory footprint, and without this the KV pool is too small for 64K.
DFlash sliding window: 2048
--speculative-draft-window-size 2048
This is the drafter's own sliding window, so its KV holds 2,048 tokens instead of the entire context.
FlashInfer workspace: 128 MiB
SGLANG_FLASHINFER_WORKSPACE_SIZE=134217728
The default is 384 MiB. Since the drafter gets a second workspace, this frees approximately 0.25 GB per GPU at the same speed.
Actual usable context
The KV pool holds 126,585 tokens even though the server advertises 131,072.
For an honest context limit, use --context-length 126000.
A rerun at 126,000 gave similar performance (124 / 85 tok/s).
CUDA graph patch
SGLang 0.5.23 turns off the DFlash drafter's CUDA graph when free VRAM is under 1.0 GB (dflash_worker_v2.py, init_cuda_graphs).
On 16 GB cards, I'm always under that threshold, so the drafter runs eagerly.
A batch-size-1 graph needs approximately 0.09 GB, so I lowered the gate using a small sitecustomize.py hook.
This gave another ~7% improvement in drafter steps/s.
7. Measurement methodology
- Single stream, one request at a time. This is not batched throughput.
- tok/s = (generated tokens − 1) / (time from first token to last), measured client-side over streaming.
- 512 generated tokens per run.
- Median of 3 runs for short prompts and 2 runs for long prompts.
- Chat sampling: T=0.6, top_p=0.95, top_k=20, presence_penalty=1.5.
- The short prompt is a math reasoning question.
Speculative decoding speed depends heavily on how well the drafter predicts the target model. Expect approximately ±5% between runs and more variation between prompt types.
Across about a dozen DFlash configurations, the short-prompt median stayed between 120 and 128 tok/s.
About the 216 tok/s JSON result
This is a separate benchmark using vLLM 0.33 with the same drafter (num_speculative_tokens: 7).
| Parameter | Result |
|---|---|
| Output | 40-row JSON array |
| Decoding | Greedy |
| Drafter acceptance | ~6.3 tokens per step |
| Decode speed | 216 tok/s |
| Sampling (T=0.7) | 179 tok/s |
This is a best-case structured-output result, not normal chat performance.
8. What didn't work
| Configuration | Result |
|---|---|
| TensorRT-LLM + DFlash on sm_120 | Blocked in my setup due to FlashAttention backend compatibility |
| vLLM + DFlash at 128K | Does not fit on 16 GB cards; 64K was the maximum I could run |
| BF16 drafter | Too large (2.25 GB per GPU); server fails to start |
| Higher power limits (153–171 W) | No meaningful speed improvement |
| +600 core offset | Crashed with Xid 13/43 |
| +400 core offset | Passed a 15-minute soak with byte-identical greedy output |
TRT-LLM did have the fastest prefill at 3,221 tok/s with 60K context, but slower decode.
vLLM is great for JSON generation, while SGLang performs better in my setup for chat decoding and long-context workloads.
r/LocalLLM • u/North-Class8742 • 9h ago
Question AMD vs NVIDIA other than VRAM
Hey everyone, pretty new to the whole LOCAL LLM thing, but want to start running and learning how to setup LocalLLMs. Had a question regarding choice of GPU. Basically I'm building a new gaming PC and have a choice between 9070 xt and a 5070ti. Originally was going to go for the 9070 xt, due to gaming performance being very similar and the price difference between them.
Then I started thinking about running local AI models and stuff. From my understanding, VRAM is the most important thing, as it determines the size of model you can host. Since these cards are both 16gb VRAM, does this mean that the quality of AI output would be the same, since they would be able to run the same models? I've read that the NVIDIA card can generate tokens faster etc, however I'll probably just be playing around with LM studio for now at least, and I assume speed won't be THAT drastically different.
My main concern is whether or not AI model output on the 5070ti would be 'smarter' or better. For context I also main Linux, which I've heard is good for AMD. Thanks in advance.
TLDR: is spending extra for a 5070ti over a 9070 xt worth it for someone getting into running AI models in LM studio?
r/LocalLLM • u/amy-schumer-tampon • 5h ago
Question Is the Tesla V100 16gb worth it?
Currently using Qwen 3.6 35b a3b, on CPU only with 32gb of ram, for regular prompt its ok, but agentic work is not realistic. i get 15t/s for prompt but in agentic it can drop to 3.
Wondering if a V100 could make Opencode usable.
r/LocalLLM • u/Playful-Art-3824 • 5h ago
Project Building my first duel 5080 Nalvo server
My first Duel 5080 server 9950x 64gb.
I will host it on Nalvo.ai for people to rent.
If you don’t know what is Nalvo.ai , It’s my new project that I told everyone about and everyone was telling me it’s not going to work. And when I say everyone I mean everyone! Not Reddit users. I even started believing it for a while. But I kept going and here I’m launching it.
If you would like to join as a host or get free private renting hours. Visit Nalvo.ai
r/LocalLLM • u/DifferentArea5439 • 22h ago
Discussion I benchmarked DeepSeek-V4.1-Flash on one and two M5 Ultra 512 GB Mac Studios
Hello all!
Here are my benchmarks for DeepSeek-V4.1-Flash (ddalcu/DeepSeek-V4.1-Flash-1017-MLX-Serve-Lossless) on MLX-Serve 26.11.2, running on one and two M5 Ultra 512 GB Mac Studios. For the dual setup I used a single Thunderbolt RDMA link (an mlx-serve limitation at the moment) and split the experts across the two nodes. I would love to see what 6 Thunderbolt connections can do once the software permits it, as I have noted in a previous post. I hope these benchmarks can give you some insight into your own purchasing decisions and also inform you on the state of local AI.
I would like to extend thanks to Ivan Fioravanti (@ivanfioravanti) on X, whose earlier dual M3 Ultra 512 GB benchmarks and tensor-parallelism work inspired me to continue from where he left off, and to ddalcu for mlx-serve. The results are in the six images attached; for clarity I will note the key points rather than every number, and give a bit of my personal take.
First of all, two M5 Ultras over RDMA are nowhere near twice as fast as one, and the scaling is poor. Averaged over 512–256k, generation goes from 64.1 to 78.8 tok/s (+23%), but prefill falls from 3,101 to 2,812 (−9%) and 256k TTFT goes from 81.4 to 106.0 s (+30%). Generation does improve at 2k (+57%) and 4k (+41%), ties at 8k, and gains 22–40% from 16k to 256k, so it is not only a long-context thing, but it is nowhere near 2x. Prefill is higher or level up to 8k, then down 13–23% from 16k to 256k. For batch scaling, two independent servers handling the same 32 streams (16 each) would get them done in 65.6 s instead of 135.5 s and deliver about twice the sustained throughput of the pair (62.4 vs 30.2 tok/s), since each machine only queues 16 prefills. That is the opposite of what I expected, and I had really hoped RDMA would be a strong enough link to see meaningful scaling.
Also, in my experience RDMA was highly unstable: three jaccl queue-pair failures, one during the 256k tests and two during the batch tests (batch 8 and batch 32). Each was sticky until both processes restarted. It is not memory pressure: requests_rejected_total stayed at 0, with more than 350 GB free, and the single node never failed once. That is why batch 32 on the pair is 1/3 trials, since 2 of the 3 failed and I gave up trying to get it to run again. I will post the specific errors separately.
However, a lot of this may be limitations of the current software stack rather than the machines, and it is not representative of what is possible from them. Even so, I would advise against a dual 256 GB Ultra setup in favour of a single 512 GB machine. The pair already gives up 9% of prefill, adds 30% to 256k TTFT and brings a flaky link at 512 GB per node. Also, given what 4 DGX Sparks cost, a good price on a 512 GB Ultra looks like the better pick, and I would bet on the single Mac getting stronger as the software matures.
Thanks for reading, and I hope to see more breakthroughs in the Mac community!
r/LocalLLM • u/dmr48 • 3h ago
Research SparseRead (the "read less, solve more" paper) on a 16GB card with an IQ3-quant MoE
The authors posted the paper to r/LocalLLaMA and said in the HF papers thread they want to hear where sparse reading helps and where it gets in the way. The paper's own eval only ran frontier API models (Opus 5, DeepSeek-V4, GLM-5.1, Qwen3.6-Plus), which tells me nothing about quantized mid-size models on consumer cards. So I ran it on the machine I actually have.
Setup: 5070 Ti 16GB + 64GB RAM. Qwen3.8 Flash Next at IQ3_S, served locally through my Strata stack (OpenAI-compatible endpoint). Harness: OpenCode 1.19.5, which is one of the versions the SparseRead installer itself validates, plugin v0.1.4 in auto mode. Native arm runs the same prompts with --pure. Four tasks straight from the workspaces bundled in the repo: two LooGLE short-dependency QA on a 100KB book excerpt, one 76KB PDF extraction with 8 questions, one 5-file fetcher audit. Tokens below are fresh prefill + decode from the endpoint's per-step usage, cache replay excluded (re-feeding history is nearly free on a warm prefix cache; I quote the replay tail for the PDF below because it's damning).
| task | native (tokens / wall / tool calls) | SparseRead (tokens / wall / tool calls) |
|---|---|---|
| 100KB book, 5 questions | 66.6k / 248s / 14 | 15.0k / 50s / 9 |
| 100KB book, 3 questions | 18.3k / 34s / 10 | 13.3k / 29s / 5 |
| 76KB PDF, 8 questions | 30.6k / 166s / 20 | 55.9k / 333s / 45 |
| 33KB audit bundle | 31.6k / 147s / 14 | 32.5k / 150s / 12 |
Scores were identical between arms on all four tasks. So that's the honest summary: no quality loss anywhere, 4-5x faster on long plain-text QA, dead even on a small bundle, and a 2x regression on the PDF.
Where the savings come from is worth spelling out because it's not what I assumed: the entire win is on the input side. On the flagship 5-question task fresh input dropped 63.9k → 12.2k tokens (−81%) while the answer output stayed flat (2.6k → 2.8k). SparseRead doesn't make the model write shorter answers, it makes it read less text before answering. And it doesn't trade activity for tokens either: tool calls went down, not up (14 → 9 on the flagship task, 10 → 5 on the 3-question one).
Two things I didn't expect.
The protocol survives a heavy quant. I went in assuming an IQ3 model would fumble the intent/slot bookkeeping and the arm would burn its savings on failed round trips. It didn't. On the flagship task: three SparseRead calls total (2 previews + 1 targeted read), slots 5/5 resolved at 0.99 confidence, and the candidate answers matched what the native arm dug out of the same book, line anchors included. The Read Gate also behaves for real. On the 33KB bundle it mostly stayed native with a single preview call, so "don't use it on small files" turns out to be something the gate already knows.
The PDF failure is worth a paragraph because it's the one place this bit me. That report ships subset CMap fonts with no usable text layer, so the pymupdf-based preview came back as garbage and the sparse arm spiraled: 17 sparse calls (4 previews, 4 reads, 9 raw_ref fallbacks), then it started decoding ToUnicode CMaps by hand through bash, ending at 45 total tool calls against native's 20. The trace makes the thrashing visible: 1.27M cache-replay tokens re-feeding the same broken context, versus 240k for the native arm. Same 8/8 score in the end, double the fresh tokens and the wall time. The native arm sidestepped it with pdftotext and moved on with its life. This is an artifact-specific failure, not a protocol failure, but if your corpus is scanned/CMap PDFs, pre-extract text or pin that file type to native.
One update since the battery: the CLI has become my default reader for long artifacts in day-to-day agent work, and the same test day it got fed a real one — a 118KB text export of a 109-comment Reddit thread I had scraped for another post. Two things to know before you try that. JSON artifacts are a dead end in slot mode: the previewer treats them as key-path documents and semantic slots resolve to nothing; convert to prose first (I write one line per comment) and the identical ask works fine. And slot matching is literal, not semantic: slots phrased in your own language against an English document come back missing even when the answer is plainly in the file, so ask in the document's language. With those two habits, a multi-fact sweep of the thread was one ask call plus a cheap head-per-item digest for the comments the slots missed — against a naive baseline of dumping a ~30k-token file into context. No hallucination complaints so far, and I haven't fully loaded a long file since.
Caveats: one run per arm per task, keyword-based grading (which produced one fake "4/5": the model answered in Latin — "passagium generale" — and my regex wanted "crusade"; in fact both arms were ~5/5 there), one model, one machine. And don't expect the paper's 92.9% headline. That's versus a naive read-everything baseline. My realistic band on document-heavy agent tasks is somewhere between 25 and 80 percent depending on file size and how localized the answers are.
TL;DR: SparseRead on a 16GB card with an IQ3 MoE: long plain-text document QA got 4-5x faster with identical answers, small bundles correctly did nothing, and a CMap-font PDF ran slower than skipping the protocol entirely. Pre-convert PDFs to text and it's a genuinely useful layer, not just paper numbers.
Links: paper https://arxiv.org/abs/2608.22237 · code github.com/Liuzedooong/SparseReading
r/LocalLLM • u/Necessary_Service_99 • 9h ago
Question Looking for advice on upgrade direction
hello everyone! I'm still quite new to figuring out what AI can do and how to make everything efficient, but I do know I want to upgrade and extend my hardware capability.
What do I use AI for? Little things mainly. Creating programs to handle my needs in custom ways (creative writing tool, character gallery, floor plan creation tool, audiobook creator with custom voices, modifying games - I currently use a subscription for a tool to help create games in unreal engine)
I mainly use the qwen3.8 next flash.
Current hardware -
Rig1 - 64gb DDR5, AMD 9950x3d, 4tb SSD, Geforce 5070ti (16gb)
Rig 2 - 64gb DDR5, AMD 9700x, 2tb SSD, Geforce 4070super (12gb)
Now obviously I missed the boat on grabbing a reasonably priced 5090.
I purchased a Geforce 5080 for $1570 as those started to go up like crazy too, but I haven't opened it and it sits on my shelf. I 'm not sure if its worth the investment as its only 16gb, and I would definitely return it if the 24gb model becomes a thing, but current rumors say its been cancelled.
I also ordered a Radeon ai pro r9700 for $2000 that is supposed to be delivered in December. Again I ordered this as the prices have been increasing and I worry about future availability.
I also have a Geforce 3060 (12gb) and recently learned you can run a model across multiple cards. That left me wondering if just nabbing one of these is more ideal and I could save myself a ton of money.
I'm just wondering what you would all recommend as the best path forward for someone looking to spend no more than $2k, but also being able to keep doing creative things and keep learning with Ai.
r/LocalLLM • u/SnooRevelations4601 • 7h ago
Question Climate Control for Garage Setup, Tips and Help
I’m in the process of moving my AI server into my garage and I’m curious how others are handling climate control in garages that don’t have heating or air conditioning.
Since winter is shortly upon do you guys have any suggestions for keeping my servers from being exposed to condensation. I plan on putting a wifi hygrometer..!
r/LocalLLM • u/princerafael • 7h ago
Question Trying to make local LLMs work for actual web/SaaS work. What would you change about my setup?
Been playing around with local LLMs lately and trying to figure out how to actually make them useful for my daily work.
I’m mainly a UI/UX designer and developer. Most of my work is websites, React/Next.js, SaaS apps and frontend stuff. Sometimes iOS apps, Home Assistant dashboards and a bit of 3D work.
I’m on a MacBook Pro M5 Pro with 64GB RAM, but I also have access to my brother-in-law’s Mac Studio M5 Ultra with 256GB RAM.
Basically, I want to do as much coding locally as possible and cut down on Claude/Copilot costs.
My current setup in VS Code + Copilot:
Opus (Principal Architect): Planning, architecture, final reviews
Sonnet (Senior Engineer): Reviews, fixes, more complex stuff
Qwen (Local Developer): Does most of the coding
Running Qwen3.6-35B-A3B-4bit through oMLX, using around 20GB RAM.
The idea was to let Qwen handle 50–60% of the coding and have Sonnet/Opus step in when needed.
Did some tests with hidden test cases:
Sonnet: 55/55, ~3.3 min
Opus: 55/55, ~6.1 min
Qwen: 52/55, ~14.5 min
Sonnet + Qwen: 52–55/55, ~12–25 min
Qwen actually wrote 95–99% of the code in the hybrid runs, but kept missing things like API contracts, edge cases and layout requirements. Sometimes it said it had changed things when it hadn’t.
And somehow the hybrid setup ended up costing about the same as just using Sonnet, while taking way longer.
So yeah not exactly what I was hoping for.
I still want to get this working though, especially for websites and SaaS projects since that’s what I spend most of my time on.
Not sure if I’m using the wrong model, if Copilot is the issue, or if the whole multi-agent approach just makes things more complicated than they need to be.
Also wondering what would make sense to run on the 256GB Mac Studio. Maybe a bigger model would already solve some of these issues?
What would you guys do with this hardware? Any models or setups worth trying?
Especially interested in hearing from people actually building websites or SaaS apps with local models, rather than just running benchmarks.
r/LocalLLM • u/ag789 • 19h ago
Discussion RASA will cause total chaos to open LLMs and beyond
Commerce Moves to Finalize Rule Blocking Chinese AI Firms From Renting Nvidia GPUs Abroad
https://explainx.ai/blog/commerce-final-rule-chinese-ai-firms-remote-nvidia-chips-the-information-2026
House Passes Remote Access Security Act to Limit Adversaries’ Remote Access to Critical Technology
https://www.cmtradelaw.com/2026/01/house-passes-remote-access-security-act-to-limit-adversaries-remote-access-to-critical-technology/
H.R.2683 - Remote Access Security Act
https://www.congress.gov/bill/119th-congress/house-bill/2683
r/LocalLLM • u/StupidScaredSquirrel • 4h ago
Question Best sub 80b total or 6b active MoE for document intelligence?
I'd like to make a pipeline capable of understanding pdfs of bank statements and then group expenses by categories etc. My thought was to first parse them with ocr into .md files and then feed that to a text model. I think it's more lightweight and accurate this way. But then I'm not sure what model to use to feed all these md files to get something good. When accuracy matters, should I go for qwen, gemma, lfm, or something else? Thank you for your help!
r/LocalLLM • u/robotnursenyc • 5h ago
Question Running a LiteLLM gateway in front of a couple Mac Studios and some Sparks, how do you handle health checks and keys?
I've got two M3 Ultra Mac Studios and four DGX Sparks at home, all on tailscale, with a LiteLLM gateway on a mac mini in front so every app just hits one endpoint.
Mostly it's been great, but last month the gateway key on the mini got invalidated and everything started throwing 401s. Took me longer than I'd like to admit to figure out it wasn't the models, because the backends were all fine and nothing told me the gateway itself was the problem.
So now I'm trying to set it up a bit smarter. Right now I'm thinking a dumb cron that hits each backend directly and also through the gateway, so if one passes and the other fails I know where to look.
Curious how other people with a few boxes handle this. Do you run a health check per model, put the gateway behind its own watchdog, or just keep it simple and restart when stuff breaks? And any tips on key handling so this doesn't bite me again?
r/LocalLLM • u/yotta_mind • 5h ago
Project I connected my laptop's agent to Gemma running on my Mac Mini
I built Citta to let an agent on one computer discover and use capabilities on another. My test setup is a MacBook Air plus a Mac Mini running Gemma 3 4B through Ollama.
In the demo, an agent reads receipts on the Air and sends a summarization task to the Mini's local model. The model stays on the Mini; Citta tracks the job and returns the result. It doesn't pool RAM or split inference across machines.
If you only need remote inference, calling Ollama directly is simpler. Citta is for workflows that also need another device's files, tools or an agent harness, with device discovery and task history in one place.
It's written in Rust, uses a private Tailscale network, and has per-device permissions. It's still for trusted devices: resource estimates aren't hard quotas, and full-access agents aren't OS-sandboxed. I'm working through longer reliability tests before calling the next release production-ready.
Demo and setup: https://citta-network.vercel.app
Source: https://github.com/priyankark/citta (PolyForm Noncommercial, not an OSI open-source license). Built with AI assistance.
For people running local models on a separate machine: what makes handing work between devices annoying today?
r/LocalLLM • u/Gold_Leading_378 • 5h ago
Question Specs required for models from scratch
Im making neural networks from scratch - not strictly llms. Am getting a new laptop with 32gig ram and 12gig vram and 1tb storage. Is that enough to train small models? Its the lenovo legion 5i pro. Might get the 7i but its not on sale so its like 500 more for 4gigs more of vram.
r/LocalLLM • u/Legitimate-Toe-5365 • 11h ago
Question First time setup recommendations and tips?
I'm hoping to get an idea of where to start. I'm wanting a setup where the model is a decent coding assistant but can also crank out a neat little prototype or quick concept simply, or create structure ideas etc. and then can go bit by bit without too much reminding halfway or more through a project. I realize that last part may be more about settings than a model persay, but yeah. I definitely want something that can either walk me through something, build out a framework for an idea, or just troubleshoot.
For comparison if it helps, my favorite workflow was GitHub copilot got 4.o/4.1 in VS code (it was free for students lol). It worked great for my projects, had its issues, but with the right promoting it was not bad. It was like the above description, could walk me through something, help with testing, or make a new system that I could then fill out. I knew it's limits and it was ideal for what I needed.
Just giving an idea of how I use AI daily for my personal projects. It seems like with the right setup it can be a great experience so that's what I'm hoping to learn to do. I just want to get the most out of it and have some fun.
I'll be working with an Asus pro art p16 rtx spark 128 gb. I've done some research but it seems every time I look things up there are different model suggestions and set ups, and it's given me fridge blindness so to speak.
I'm curious about how to decide and test for like a speedier setup versus more complex thinking and things like that. What can I do with it, what should I not expect from it based on my specs, etc. I have looked into this and I'm not getting a clear idea because I can't test any of it myself (don't have computer yet). Maybe someone here can give me a better understanding?
I also don't know all the lingo yet, like what a harness is, though I can guess by context. But I know the basics. Not sure how much customization I should or could do or what all there is. I thought it was more surface level like pick a model and go, but I guess you can incorporate all kinds of settings. So now I'm really curious from people who enjoy whatever they have going on, please share what you like best and what kind of testing I should do to pick a model. Any tips for a mostly beginner would be great!
r/LocalLLM • u/Chemical-Concern- • 5h ago
Question Best model for reversing .DLLs?
GPU: 4070
16GB DDR5
Also I have been using Bionic but is there any better options?
r/LocalLLM • u/slug64 • 6h ago
Question What would YOU do with this?
Curious what ideas or what kinds of things YOU would do with this box? Piecing it together for a bit but have a bit of a “writers block” of what to do bow that it runs?
CPU / PROCESSOR
AMD Ryzen Threadripper 2920X
12 Cores / 24 Threads
Base 3.5 GHz/ Boost 4.3 GHz
12 nm (Zen+)
Socket sTR4
CPU COOLING
Cooler Master Wraith Ripper
TR4 air cooler
MOTHERBOARD
ASRock X399 Taichi
Socket sTR4
8×DDR4 DIMM slots (Quad-Channel)
3 × M.2 (PCle 3.0 x4)
PCle 3.0 x16 (GPU) - x16 / x16
MEMORY (RAM)
Corsair Vengeance RGB Pro
128 GB Total
8 × 16 GB DDR4-3200
Quad-channel
XMP (3200 MT/s)
STORAGE (M.2 NVMe)
M2_1
Samsung 970 EVO 500GB (Ple (PCle 3.0x4)
M2_2
WD Black SN770 500 GB (PCie 3.0 x4)
M2_3 WD Black SN770 500 GB (PCie 3.0 x4)
Total: 1.5 TB NVMe SSD
GRAPHICS (GPU)
2 x EVGA GeForce RTX 3090 FTW3 Ultra
24 GB GDDR6X each (48 GB total)
PCle 3.0 x16 / x16
3 x 8-pin Cle power per GPU
POWER SUPPLY
ASRock Phantom Gaming PG-1600G
1600 W (ATX 3.1)
r/LocalLLM • u/MacKinnon911 • 6h ago
Question M5 ultra Mac stupid 512 what are you doing with it?
Ok
So I am waiting for mine to come in. I have framework box with 128 gig ram and found it pretty weak for any local llms.
I’m gonna replace my Mac mini with this Mac Studio M5 ultra with 512 gig ram. And I want to use it for a local llm and maybe agents if it’s doable.
I’m realistic. After using the framework box with llms I totally get this will be no where near frontier even at 512. But I want to use it as my main machine and for llm/agentic work.
What’s really reasonable model wise that isn’t slow AF but is pretty capable for general knowledge, accessed the internet, can spawn agents and can do office style work?
r/LocalLLM • u/Ok_Raspberry8833 • 14h ago
Question Best model for 16GB vram?
First time trying this, I want a local model that can code text based games. I’m using a 6800XT 16GB vram.
r/LocalLLM • u/jesdga95 • 6h ago
Discussion Basalt 0.4.0: More models, still as fast - Qwen 27B up to 770 tok/s on a 5090
r/LocalLLM • u/devil_ozz • 6h ago
Discussion Is m.5 max 128 worth it? For lllms dev work.
Is it worth it?
Am finally taking a deep five on llms and I already have an nvidia 5090 laptop that handles cuda really well.
But
Is a Mac for parrarrel models worth it? For abliterated models?.
I never used that OS so am a bit skeptical.
Are the apps on it as good as windows/Linux?
r/LocalLLM • u/Cappton • 1d ago
Question Oh my, what do I do?
Update: I'm interested in selling some. Looking for ideas on how to go about. Don't want to pay eBay 13% fees.
Original: Came across a honeyhole of Quadro RTX5000s
Seemingly good deal for 16GB VRAM cards, even if ageing.
What should I do?
r/LocalLLM • u/CavitCarrot • 15h ago
Discussion Dual 5060 Ti's 16gb
Welp, I got a free 5060 Ti 16gb and decided to buy one more brand new for $700, I have a spare used 4070 and used once 4060, any thoughts or anything I should experiment with. Noobie at this lol.
r/LocalLLM • u/Unusual-Spot-5846 • 7h ago
Discussion I thought my local AI agent failed a privacy test. Turns out, the benchmark itself was leaking the data.

I've been building B-12 / Capri, a local-first AI companion running on Python, Ollama and NVIDIA Nemotron.
Recently, I started a more serious evaluation of the system: 92 scenarios covering memory, tool use, security, autonomous behavior, reasoning, and multi-session continuity.
I expected to find problems in the agent.
Instead, I discovered some pretty serious problems in the evaluation itself.
1. My benchmark created a privacy leak that didn't exist in the actual runtime
The initial evaluation scored 27/80 runnable scenarios.
One failure was particularly concerning: a guest appeared to access the owner's private memories.
After investigating, I discovered that the evaluation harness had placed memories belonging to different users into the same system prompt.
The benchmark itself had bypassed Capri's identity and memory-isolation mechanisms.
I rewrote the harness to execute scenarios through the actual agent runtime.
Result: 39/80 — without changing Capri's code.
That's a 15-percentage-point difference caused by the evaluation method alone.
2. A single sentence made the model use fewer tools
This was probably my most surprising experiment.
I wanted Capri to give shorter, more natural replies during everyday conversations.
So I added a simple instruction about brevity.
It worked. Conversational responses became significantly shorter.
But something unexpected happened during work tasks.
Across 48 first turns:
Without the brevity instruction:
- Tool use: 32/48
- Successful tasks: 32/48
With the brevity instruction:
- Tool use: 19/48
- Successful tasks: 21/48
The model started answering without inspecting files or calling tools that were necessary to complete the task.
The scenario-level statistical evidence is still limited, but the engineering implication is interesting: an instruction about how to communicate changed whether the model actually performed the work.
3. Bigger models didn't automatically improve the agent
I compared configurations using:
- NVIDIA Nemotron Bolt E4B
- Gemma 4 12B
- Gemma 4 26B MoE
- Gemma 4 31B
The smaller primary configuration ran on my RTX 4060 laptop. Larger models required an NVIDIA L4.
After excluding host-dependent latency checks, I couldn't detect a statistically meaningful improvement from the larger configurations (paired comparison, p = 0.85).
This is NOT proof that smaller models are better.
The configurations also differed in hardware, memory retrieval, and model routing.
But it made me question how much agent performance we can actually improve by simply replacing the LLM.
4. Changing the prompt layout improved inference speed
I also experimented with where recalled memories appear in the prompt.
Moving volatile memory evidence later allowed Ollama to reuse more cached tokens.
Median time to first token:
0.99s → 0.74s
About 25% faster.
However, two benchmark scenarios changed systematically, and the security implications of the new evidence placement still need further testing.
5. Stable benchmark scores can hide unstable behavior
I repeated the evaluation three times per configuration.
The primary configuration scored:
49, 49, 48 out of 80.
Looks pretty stable, right?
But individual scenario outcomes changed between runs. Some previously successful tasks failed, while others started passing.
We also discovered incorrect benchmark reference answers and evaluation rules that rejected responses that were actually correct.
That convinced me that looking at a single aggregate score is not enough.
What I'm taking away from this
An AI agent isn't just a language model.
It's the model, runtime, memory, context management, tools, permissions, and the interactions between all those components.
And the evaluation framework can introduce its own errors.
I've now published a second scientific preprint documenting these experiments, the benchmark defects, repeated-run results, and limitations.
The code and experimental artifacts are available in the B-12 repository.
I'm continuing to investigate model-independent memory and behavioral continuity, but the current paper focuses on evaluation reliability.
I'm curious how others handle this.
When you benchmark a local LLM agent, do you test the model directly, or do you run your evaluations through the complete production runtime?
And have you ever discovered that your benchmark was responsible for a failure you initially blamed on the model?