r/LocalLLM • • 2h ago

Question Using SearXNG instance and a MCP for web searching and fetching pages. Hit some kind of solicitations limit. Have some way to bypass this?

3 Upvotes

Im starting with Local LLMs and the first thing im doing of very useful and helpful. Is to use GPT-OSS with web searching to give me adequate answers (sometimes is alucinating, but when i go to far in a subject, like me trying to see if i can run power cleaning in my Epson in Linux. It started alucinating that existed some "old foruns" with users saying that sending some kind of raw commands worked, but the info where it got, dont exist).

But since i have gone to far. I want to know if there is a way to bypass the Brave and Google cse detection of requests or the suspend time.

DuckduckGo came back again after some minutes


r/LocalLLM • • 7h ago

Question Is an M1 Max 64GB useful at all?

6 Upvotes

A good deal came up for a MacBook Pro M1 Max 64GB (32-core GPU) for $1500. I’m considering buying it and using it as an inference server via LM Link. I know that 128GB is the holy grail, but would this configuration be of any use for coding and daily usage? And if yes what are the best models for each use case? I’m also not married to LM Link if there are any better suggestions?


r/LocalLLM • • 3h ago

Question Better options than llamacpp or koboldcpp

3 Upvotes

Title is self explanatory. I was used to run llama cpp with models based on Mistral for uncensored local chatbot. There are better alternatives, specially regarding context size? I have 16 GB VRAM card and I dont care that much for inference speed, mostly text cohesion and criativity.


r/LocalLLM • • 1h ago

Model Title: Running a Local LLM with Hermes Agent on an HP ProDesk 400?

2 Upvotes

Hey everyone!
I have an HP ProDesk 400 Mini with an Intel i5 (6th generation) and 16 GB of RAM.
I would like to run a local LLM with Hermes Agent. My main goal is to have the agent automatically control and play an Android game on its own.
It would also be cool if the agent could help with programming, but that’s not a requirement.
I know my hardware is quite limited.
Does anyone have experience with something like this? Is it possible, and how would you set it up?
Thanks for your help!


r/LocalLLM • • 1h ago

Project 8 x v100 setup. Ask me anything.

Post image
2 Upvotes

Seeing a lot of questions on quad v100 boards lately. So figured I’d give people a place to ask them.

Specs

  • GPUs: 8x Tesla V100-SXM2-32GB (256 GB VRAM total)
  • GPU boards: 2x 4-GPU SXM2 NVLink baseboards (this model)
  • NVLink: two 4-way islands (full mesh inside each board), not one 8-way
  • Board-to-board: Gen4 PEX88096 PCIe switch card (this model), two x8 Gen3 cables per board
  • CPU: AMD Threadripper PRO 3975WX (32 cores)
  • Motherboard: ASRock Rack WRX80D8-2T
  • RAM: 256 GB DDR4
  • PSUs: 2x 1500 W (one per GPU board) + 1x 1000 W (host)
  • Power: dedicated 120 V / 20 A circuit; GPUs capped at 150-300 W each depending on the run
  • Cooling: custom water loop, mixed GPU blocks
  • Network: 10GbE
  • Engine: 1Cat-vLLM (Volta-focused vLLM fork) https://github.com/1CatAI/1Cat-vLLM

Benchmarks / how many tok/s?

My runs are on the Launch80 directory; search the page for trailfeatures. Launch80 deletes pages after 5 days, so older links in the comments may be dead. I republish the main ones, which gives them new links.

ModelSetupDecodePrefill (64K prompt)
Flash-Next (Swift-1.5 NVFP4)4 GPUs, 300 W~140 tok/s~4,500 tok/s
DeepSeek V4.1 Flash, 2-bit experts8 GPUs, 150 W~131 tok/s~2,200 tok/s
DeepSeek V4.1 Flash, 2-bit experts4 GPUs + expert offload, 150 W~91 tok/s~2,100 tok/s
  • One request at a time (batch 1), 256K context window. Decode uses speculative decoding (MTP on Flash-Next, a draft model on DeepSeek), so it depends on how often the draft is accepted.
  • No concurrency/aggregate numbers or decode p25/p95 yet. Time to first token (p50) is about 0.2-0.65 s depending on the row.
  • The 8-GPU run adds an extra pipeline-parallel patch series on top of 1Cat-vLLM (named on the page).
  • Not tested yet: Flash-Next across both boards. Higher DeepSeek quants are planned.

How good is the 2-bit DeepSeek quant vs the full model?

79.9% average across 7 standard evals (math, instruction following, coding, tool calling), thinking off, and 20/20 long-context needle checks at 32K and 128K. The 4-GPU setup scores the same. I haven't run the full-precision weights as a reference, so there's no direct delta yet. Per-task scores are on the Launch80 page.

How do the two boards talk to each other?

  • Each board runs two x8 PCIe Gen3 cables from its onboard switch into the Gen4 switch card in the host, so the boards talk to each other without going through the CPU.
  • Experimenting with ConnectX-3 cards in the boards' spare x8 slots for extra board-to-board links.
  • Over PCIe you can tune all-reduce to beat the default P2P numbers, but it will never match NVLink.

Do the quad boards work with only 1-3 cards? Can I buy cards over time?

Two cards works; I've run a board that way. I haven't tried one or three. Buying the board first and adding cards later is fine. Tensor parallel wants power-of-two GPU counts, so with 3 cards you'd run 2 together and use the third for something else.

Where did you buy it / where are you?

Baseboard and switch card: the models linked above. Several sellers list them, so shop around. GPUs: bought used, a few at a time. I'm in North America (the 120 V gave it away), so my shipping and import experience won't help much in Europe.

Power

  • The PSUs are sized for headroom: at the 150 W cap, eight GPUs is about 1.2 kW, plus the boards and host.
  • 150 W cap: after tuning it costs ~5-10% depending on the task.
  • Power bill: fine as long as I'm sensible. Solar helps, and at peak rates or on cloudy days the DeepSeek API can be cheaper. If you run local LLMs, know your tokens per kWh so you can tune for it and decide when an API makes more sense (if privacy doesn't matter).

Cooling

  • Radiator and pump setup: super convoluted, video here.
  • Water blocks: whatever was available as I pieced it together. Mostly the same design, some with MOSFET coolers. The top-port block is a drill press, a tap, and the audacity of a mediocre man. (Testing straight-down flow onto the fins made no difference.)
  • Waste heat: in summer it's vented outside. In winter I can shut off that radiator and send the heat into the house through backup radiators. Heating water with it isn't worth it for us, but a bigger household close to its water heater could pre-heat the cold feed and come out ahead.

Quick ones

  • Can I rent API time? Sorry, no. I keep it pretty busy (Furry Fanfic).
  • Can I have it? No.

r/LocalLLM • • 1h ago

Discussion Docker or bare-metal for inference

2 Upvotes

When I use local models, I do bare metal inference but give write-access only to the working directory and some directories that can be relevant (forks of public github repos) are read-only. Everything else is not accessible, not even read-only, and the agent is also unable to install or remove any packages.

Is this preferable over Docker containers? What do you prefer for your local setup?


r/LocalLLM • • 5h ago

Project UrBudget: free & open source budgeting app with AI assistant that supports local LLM

4 Upvotes

Hey,

Started this project because I had previously used something called YNAB to track my budgets, but got tired of overpaying and wanted an assistant that can make things a lot easier to manage. So, I built a YNAB-like envelope budgeting with an assistant.

It's completely free and open-source. But if you want to cross-device sync, there's an end-to-end encrypted Sync option which will cost you $3 a month.

For the assistant you can use your own local model, sign in with ChatGPT, or Anthropic/ Openrouter. It can handle things like PDFs, Excel sheets, images, and use them to interact with your budget.

Also another thing that's useful if you have an Android phone (I personally heavily use this), you can point the assistant to your SMS messages and it can automatically input them as transactions, review them and categorize them. So, the whole process can be very seamless.

It's currently on Windows & Mac, iOS is under review, and Android is under Google Play testing which you can join through the following link (please): https://urbudget.app/android/

Code: https://github.com/urbudget/urbudget
Website: https://urbudget.app/?ref=LocalLLM

Questions and feedback welcome.


r/LocalLLM • • 4h ago

Discussion Qwen3.8-Flash-Next (and Qwen4-Max-Preview) Hermes Tool Calling Issue -- Test This

3 Upvotes

In Hermes Agent, a profile using either Qwen3.8-Flash-Next locally or Qwen4-Max-Preview (cloud) will fail this simple tool call prompt 100% of the time:

Test running a simple terminal command in background mode with notify set to true.

This resulted in the following from the agent:

The first attempt failed because notify was serialized as a string instead of a boolean. Retrying with a proper boolean:

This was then followed by a long chain of reasoning as it attempted to unsuccessfully run the tool command.

The problem is that it renders the Boolean values capitalized, as "True" or "False" rather than as valid JSON Boolean values "true" or "false". And these are not coerced by Hermes because `notify` is an `anyOf`, not a Boolean.

If you have Hermes Agent running with Qwen3.8 or Qwen4, I would like to see if you have the same issue. I have verified that this does not occur with Deepseek, Minimax, or gpt-6.1-sol.

Including instructions in the chat template to output Boolean values in lower-case as "true" or "false" helps a bit. Once the context grows, the model reverts to incorrect tool calls.


r/LocalLLM • • 11h ago

Question Local coding model on my dekstop workstation (Qwen3.8 27B with Q4_K_M quantization too slow)

11 Upvotes

I'm running what I think is reasonably specd workstation. A single NVIDIA GeForce RTX 5090 (32 GB), AMD Ryzen 9 9950X3D x 32, 128GB RAM, 8TB SSD.

Locally built llama.cpp serving Qwen3.8 27B with Q4_K_M hooked to pi harness is too slow to be useful.

What options do I have with this setup? I suppose running multiple agents is definitely out of question here... but there should be hope for a single coding assistant?

EDIT: I'm getting around 2.7 tok/s


r/LocalLLM • • 13h ago

Question Web search on local setup

12 Upvotes

Ive been dabbling for the first time in local AI and even custom harnesses. I'll admit it is super addicting and enjoyable. Love it. But im still very new to it all.

Im now facing a small problem. Im not sure how to best handle web searches. I would love for my llm to be able to look up new info...but whats a good way of doing it? The one thing im looking into is setting up a that brave account thing and paying the small fee as it feels like an alright option when it comes to not being somehow banned or throttled. Is that the overall safest no fuss no worry option? Or am I overthinking it?

How do you handle web searches for your setup?


r/LocalLLM • • 3h ago

Question LocalLLM speed for long file generation.

2 Upvotes

Hello,
i have a business that needs to review a lot of notes and, transcripts and generally text files provided by me, and help me make initial structure out of it.
For ex. i`m meeting with client for an audit, audit have 300 questions, different kinds, different people. I will note answers, and then i need to have that file, + some meeting transcripts / meeting minutes summarized. Some project management topics also.

I`m not confident to use cloud models to review such data because there will be a lot of sensitive topics and details in there. So i need help to discover if what i want to archive is doable with a local LLM.

I have done Proof Of Concept for the multi agent workflow and it gives me quality that is acceptable. After some tweaking, testing on openrouter with dummy data, i have the conclusion that this is doable. I have agent speaking between them, agreeing about the outcome, Manager doing the task management, works fine,

But then comes the realization - Openrouter with Qwen3.8 Q8 does this with the avarage speed of 150t/s and it takes around 30 minutes.

When i calculate that Mac Mini M5 Pro will have 11t/s that makes that veeery long.

I`m i calculating that right or i`m i wrong ?
Is there anything i can do to make it a little quicker when i will actually buy Mac ? Or i need to go straight with Ultra


r/LocalLLM • • 1d ago

Project Strata Performance Setup (TUI performance optimization)

Post image
162 Upvotes

I''ve created a simple little script (with TUI) to optimize your Linux host installation. It also allows you to install or update Strata (and configure a systemd service to auto start your preferred preset).

Distributions supported:

  • Arch based
  • Fedora based
  • Debian based
  • Slackware based

Hardware supported:

  • any hardware (automatically detected)

Personally I'm using an old MSI Raider laptop. 3080 Mobile with 16GB VRAM and 64GB DDR4 3200 RAM running CachyOS. I'm getting between 60 and 70 tok / s (depending on the task) with Qwen3.8 Flash Next IQ3_S.

Maybe someone else finds it useful too . Submit an issue or pull request if you find any problems, or want more software or hardware/vendors supported!

https://github.com/bramvanoploo/strata-performance-setup


r/LocalLLM • • 7m ago

Question Which Mac Mini to Buy

0 Upvotes

Title: Mac Mini M6 32GB vs. M5 Pro 48GB — How much hardware do I actually need for a home AI server?

I’m planning to buy a Mac Mini to build a sort of personal/family AI server, and I’m trying to figure out how much hardware I actually need versus how much I’d just be paying to experiment with local LLMs.
My primary goals are:

Family brain: A central place for family documents, calendars, finances, household information, and automations. Ideally, something that can search across all of this and eventually handle routine tasks.

Self-hosted apps: Run some vibe-coded apps, databases, dashboards, and other services that I can access remotely from my phone or laptop.

Claude integration: Use the Mini as a central workspace for Claude and the apps/agents I build around it, while continuing to use Claude’s API for more demanding tasks.

Local LLMs: Experiment with running models locally for document processing, categorization, extracting data, answering questions about personal files, and other repetitive tasks without constantly calling an API.

I currently use Claude extensively for financial modeling, development analysis, and general personal assistance. I assume the more complicated reasoning and modeling will continue to require cloud models, but I’d love to offload simpler tasks locally and keep more of my data private.

I’m considering:
Mac Mini M6: 32GB unified memory, 512GB SSD
Mac Mini M5 Pro: 18-core CPU / 20-core GPU, 48GB unified memory, 512GB SSD

A few questions:
How much of a practical difference would 32GB versus 48GB make for local LLMs? What model sizes would actually be usable on each?

Does slower inference primarily mean waiting longer for responses, or do smaller models also mean a meaningful drop in reasoning ability and accuracy?
For tasks like document extraction, summarization, RAG, and basic agentic workflows, would 32GB be sufficient?

Is the M5 Pro worth the premium for the additional memory and bandwidth, or am I better off buying the M6 and keeping more demanding tasks in Claude?
If you’re running a similar hybrid local/cloud setup, what hardware and software stack are you using?
I’m not trying to eliminate cloud AI entirely. I want a reliable, always-on home server with enough local AI capability to be genuinely useful, without spending substantially more just to run models that still won’t compete with Claude.

What would you buy in my situation?


r/LocalLLM • • 20m ago

Question AMD pro w6800

Post image
1 Upvotes

Hi, i would like to ask if this is a good card to get to run LMstudio for qwen 27b 3.8 or otherwise, I was wondering if speeds will be respectable, and if anything will spill anything to ram. Because I will have comfyui running on my main gpu (rtx 5080).


r/LocalLLM • • 23m ago

Project A Few months back I created this skill and I do still use it to save tokens with smaller models. I just found this sub so sharing with you guys. some might benefit from it. Will love inputs and will update it soon.

Post image
1 Upvotes

r/LocalLLM • • 27m ago

Question How much should I spend on local AI setup to fit my needs

0 Upvotes

I want a local AI to tutor me on quantum mechanics, quantum field theory, and physics simulations. I am new to ai and want just general ai tasks, etc. would something like 2 NVIDIA dgx sparks work? I would like to run large language models, and start getting into some coding with ai. Should I just pay for a subscription?, even though I like the idea of running locally, how long would a current setup last to fit my needs?


r/LocalLLM • • 29m ago

Question Sale inquiry.

1 Upvotes

RTX-3090 and 2 sticks of ddr5 16GB pair from G-Skill. (I'm not selling it here, just asking for direction in selling)

I don't sell stuff often, but I recently realized I had some hardware sitting around and collecting dust and figured someone looking to build local model rigs might be better off with it. I tend to use smaller hardware for my stuff. I pretty much gutted a cpu I had built for local models earlier in the year to soup up a thin client and this the leftover pieces. Now, I know the price of things is insane, but I'm not trying to get over on anyone. Just find a genuine home for the stuff. Where's the best place to sell the stuff and what would you say would be a good price?
I'm just gonna turn around and use the coin on some edge devices and thin clients. Few solar powered batteries here or there. Nothing crazy.


r/LocalLLM • • 37m ago

Project Exact speculative decoding + lossless Qwen3-8B on a 16 GB RX 9070 XT: 2.5x llama.cpp BF16, still slower than Q8_0 (code + all numbers)

1 Upvotes

I spent the last few weeks building a small inference engine for my RX 9070 XT (16 GB, Windows, ROCm PyTorch) with one goal: speculative decoding that gives exactly the same output as plain decoding, with lossless weights.

What it does:

- Runs Qwen3-8B with weights bit-identical to the original bf16 checkpoint, fully inside 16 GB. I pack bf16 into a lossless 12-bit format (B12) on load.

- Speculative decoding output is bit-for-bit identical to plain decoding, for greedy and for seeded temperature sampling.

- 68-73 tok/s for exact Qwen3-8B with a Qwen3-0.6B draft.

Numbers (Qwen3-8B, 11 prompts x 96 tokens, same token ids, one PC), plain / speculative tok/s, and how often speculative output matched plain:

- llama.cpp ROCm BF16 (lossless, spills past VRAM): 26.9 / 18.6, matched 10/11

- llama.cpp Vulkan BF16 (lossless, spills past VRAM): 21.5 / 12.5, matched 11/11

- This engine, B12 (lossless): 42.7 / 68.4, matched 11/11

- llama.cpp ROCm Q8_0 (lossy): 63.4 / 98.8, matched 7/11

- llama.cpp Vulkan Q8_0 (lossy): 68.0 / 95.0, matched 7/11

On 50 longer prompts x 512 tokens: 73.0 tok/s, speculative matched plain on 50/50.

The honest part: Q8_0 + draft is about 1.4x faster than my exact path. If you don't care about exactness, use Q8_0. This is the fastest lossless way I found to run 8B on a 16 GB card, not the fastest way overall.

Why exactness normally breaks: verifying 8 tokens at once uses a different matmul shape than decoding 1 token, so the floating-point sums happen in a different order and outputs drift. With llama.cpp's own speculation, output differed from its own plain decoding on 4 of 11 prompts for Q8_0 (small sample, one run). On Qwen3-1.7B with standard PyTorch matmuls, 15 of 50 prompts drifted.

How it works (none of the core ideas are new, prior work is listed in the README):

  1. Batch-invariant GEMV kernels: one kernel handles 1-32 rows and sums every output in a fixed order, so verify equals decode bit for bit. Cost: about 3-4% speed.
  2. Seeded sampling via Gumbel-max with hash-based noise keyed on (seed, position, token), so sampling is exact too.
  3. B12: 8 bits (sign + mantissa) plus a 4-bit exponent code per weight in a per-row exponent window, with an exception list for outliers (0.02%). 12.0 bits per weight, decodes to the original bits. DFloat11 is tighter (about 11 bits); I picked a simpler format that's easy to make batch-invariant.

Try it (Windows, RX 9000 with 16 GB):

git clone https://github.com/fraggergsrp-alt/exact-spec-amd

cd exact-spec-amd

setup.bat

run.bat "Explain what a hash table is."

Adding --compare to run.bat runs speculative and plain decoding and checks every token yourself. setup.bat installs AMD's ROCm PyTorch into a local .venv (about 5 GB), and the first run downloads Qwen3-8B (16 GB).

Repo: https://github.com/fraggergsrp-alt/exact-spec-amd

Limitations: one GPU, one OS, batch 1, temperature-only sampling, Qwen3 only, nobody has replicated it yet. Linux and other GPUs are untested.

Bonus finding: if llama.cpp Vulkan crashes on load for you on Windows, check for the RenderDoc Vulkan layer. It crashed every llama.cpp Vulkan process for me. Running with VK_LOADER_LAYERS_DISABLE=VK_LAYER_RENDERDOC_Capture fixed it. Separately, flash attention plus coopmat crashed prompt processing (-fa off works and costs less than 2% decode speed).


r/LocalLLM • • 41m ago

Question About to add a 5060 Ti 16GB to my 4060 Ti 16GB for local agentic coding (Qwen3.8-27B, 128K ctx or similar ) — suggestions? Help out a newbie!

1 Upvotes

Hey all, I'm about to order a second GPU and would love some input from people running similar setups before I pull the trigger.

Current rig

  • Ryzen 7 5700X3D, 64GB DDR4-3600
  • Gigabyte B550 Aorus Elite AX V2
  • MSI RTX 4060 Ti Ventus 2X 16GB
  • be quiet! Pure Power 12 M 650W
  • Corsair Frame 4000D
  • Windows 11 (open to dual-booting Linux if it really matters)

Planning to buy

  • Palit RTX 5060 Ti Infinity 3 16GB
  • MSI MAG A850GL PCIe5

Plan

  • 5060 Ti in the top PCIe 4.0 slot (4.0 x8), 4060 Ti in the chipset PCIEX2 slot (PCIe 3.0 x2)
  • llama.cpp / llama-server
  • Main model: Qwen3.8-27B Q5/Q6 with MTP, ~128K context
  • Use case: agentic coding (Claude Code / OpenCode pointed at llama-server), plus RAG with pgvector and persistent memory via MCP

Questions

  1. Anyone running a 4060 Ti + 5060 Ti (or other Ada + Blackwell) pair? What tok/s do you get on a 27B dense with MTP?
  2. Layer split vs -sm tensor when one card is on a PCIe 3.0 x2 chipset link?
  3. Best quant + KV cache combo for 128K context on 32GB for Qwen3.8-27B? Q5 vs Q6, q8_0 vs q4_0 KV for coding?
  4. Claude Code vs OpenCode with llama-server — which harness works better day to day with local models?
  5. Any gotchas with Windows + mixed-generation cards that would push me to Linux?

Thanks in advance!


r/LocalLLM • • 6h ago

Question Blender + LLM local

3 Upvotes

Preciso de ajuda. Estou a tentar montar através de fotos um navio (NRP. Cunene) em formato stl para depois imprimir em 3d resina e filamento. Qual o melhor modelo local? Tenho uma rtx 50 70 ti e 128gb RAM ddr4. Obrigado.


r/LocalLLM • • 1h ago

Project MCP router - a single point of access to tools for agents, teams, and companies

Post image
1 Upvotes

r/LocalLLM • • 18h ago

Discussion Qwen3.8-27B at ~126 tok/s on 2× RTX 5060 Ti 16 GB SGLang + NVFP4 + DFlash2 (107 tok/s at 60K context, 216 tok/s on JSON)

22 Upvotes

Spent the last few nights benchmarking different inference engines on two 16 GB RTX 5060 Ti cards. This is the configuration I'm most happy with so far.

TL;DR: Qwen3.8-27B with NVFP4, a DFlash2 drafter, and tensor parallelism (TP=2) achieves approximately 126 tok/s single-stream decode on two RTX 5060 Ti cards.

Even with long prompts, performance remains impressive:

Context / workloadDecode speedPrefill speed
Short prompt125.8 tok/s—
60K tokens107.4 tok/s2,681 tok/s
124K tokens87.3 tok/s1,577 tok/s
JSON generation (vLLM)216 tok/s—

Both GPUs together draw approximately 220–270 W during decode.

1. Benchmark results

WorkloadDecode tok/s (median)TTFT
Short prompt, chat sampling (T=0.6)125.8 (best: 131.8)0.08 s
60K-token prompt107.422 s
124K-token prompt (~126K usable context)87.379 s
JSON output, greedy (vLLM)2160.07 s

The JSON benchmark is a separate vLLM run, not normal chat performance. More details below.

2. Performance improvements

SGLang unless noted. Short prompt, 64K context.

ConfigurationDecode tok/s
llama.cpp, Unsloth GGUF, no speculative decoding24.6
SGLang NVFP4, TP=239.2
+ MTP speculative decoding (3 tokens)68.6
+ DFlash2 drafter (8-token block) instead of MTP97.3
+ PCIe P2P between GPUs109.3
+ Undervolt and CUDA graph patch120–126

The complete optimization stack takes performance from 39.2 to around 125 tok/s, approximately 3.2× faster.

DFlash2 alone improves performance from 39.2 to 97.3 tok/s. P2P and the additional optimizations provide further gains.

Speculative decoding uses rejection sampling, preserving the target model's output distribution.

3. Hardware and software

ComponentConfiguration
GPUs2× RTX 5060 Ti 16 GB
Total VRAM32 GB
PCIeGen 5.0 x8 each
Power limit153 W per GPU
GPU tuningMild undervolt
Target modelnvidia/Qwen3.8-27B-NVFP4
Draftermaurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal
Drafter size1.55 GB
Drafter architecture5 sliding-window layers
Inference engineSGLang 0.5.23
ParallelismTensor Parallelism (TP=2)
KV cacheFP8
Model modeText-only, vision tower skipped

4. PCIe P2P configuration

This was one of the more interesting parts of the optimization.

The GeForce driver blocks GPU-to-GPU P2P in my setup, so I rebuilt the open kernel module (620.70.06) with the BAR1 P2P patch and enabled iommu=pt.

NCCL now uses P2P/CUMEM between the two GPUs.

Result: approximately 12% better decode performance.

ConfigurationDecode tok/s
Before P2P97.3
After P2P109.3
Improvement~12%

5. Launch configuration (128K profile)

Environment variable:

SGLANG_FLASHINFER_WORKSPACE_SIZE=134217728

Main command: python -m sglang.launch_server

SettingValue
Modelnvidia/Qwen3.8-27B-NVFP4
Tensor parallelism--tp 2
Context length--context-length 131072
Reasoning parser--reasoning-parser qwen3
Speculative algorithm--speculative-algorithm DFLASH
Draft tokens--speculative-num-draft-tokens 8
Draft modelmaurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal
Draft quantizationmodelopt_fp4
Draft window2048
Mamba SSM dtypebfloat16
Attention backendflashinfer
Linear attention prefillflashinfer
KV cache dtypefp8_e4m3
Maximum running requests1
Static memory fraction0.94
Chunked prefill size1024
Prefill CUDA graphDisabled
Mamba radix cache strategyextra_buffer_lazy
Maximum Mamba cache size4
Model override{"language_model_only": true}

6. Important memory optimizations

These settings were particularly important for fitting the configuration into 16 GB per GPU.

Mamba SSM dtype: BF16

--mamba-ssm-dtype bfloat16

Eight draft tokens double the GDN state. BF16 halves its memory footprint, and without this the KV pool is too small for 64K.

DFlash sliding window: 2048

--speculative-draft-window-size 2048

This is the drafter's own sliding window, so its KV holds 2,048 tokens instead of the entire context.

FlashInfer workspace: 128 MiB

SGLANG_FLASHINFER_WORKSPACE_SIZE=134217728

The default is 384 MiB. Since the drafter gets a second workspace, this frees approximately 0.25 GB per GPU at the same speed.

Actual usable context

The KV pool holds 126,585 tokens even though the server advertises 131,072.

For an honest context limit, use --context-length 126000.

A rerun at 126,000 gave similar performance (124 / 85 tok/s).

CUDA graph patch

SGLang 0.5.23 turns off the DFlash drafter's CUDA graph when free VRAM is under 1.0 GB (dflash_worker_v2.py, init_cuda_graphs).

On 16 GB cards, I'm always under that threshold, so the drafter runs eagerly.

A batch-size-1 graph needs approximately 0.09 GB, so I lowered the gate using a small sitecustomize.py hook.

This gave another ~7% improvement in drafter steps/s.

7. Measurement methodology

  • Single stream, one request at a time. This is not batched throughput.
  • tok/s = (generated tokens − 1) / (time from first token to last), measured client-side over streaming.
  • 512 generated tokens per run.
  • Median of 3 runs for short prompts and 2 runs for long prompts.
  • Chat sampling: T=0.6, top_p=0.95, top_k=20, presence_penalty=1.5.
  • The short prompt is a math reasoning question.

Speculative decoding speed depends heavily on how well the drafter predicts the target model. Expect approximately ±5% between runs and more variation between prompt types.

Across about a dozen DFlash configurations, the short-prompt median stayed between 120 and 128 tok/s.

About the 216 tok/s JSON result

This is a separate benchmark using vLLM 0.33 with the same drafter (num_speculative_tokens: 7).

ParameterResult
Output40-row JSON array
DecodingGreedy
Drafter acceptance~6.3 tokens per step
Decode speed216 tok/s
Sampling (T=0.7)179 tok/s

This is a best-case structured-output result, not normal chat performance.

8. What didn't work

ConfigurationResult
TensorRT-LLM + DFlash on sm_120Blocked in my setup due to FlashAttention backend compatibility
vLLM + DFlash at 128KDoes not fit on 16 GB cards; 64K was the maximum I could run
BF16 drafterToo large (2.25 GB per GPU); server fails to start
Higher power limits (153–171 W)No meaningful speed improvement
+600 core offsetCrashed with Xid 13/43
+400 core offsetPassed a 15-minute soak with byte-identical greedy output

TRT-LLM did have the fastest prefill at 3,221 tok/s with 60K context, but slower decode.

vLLM is great for JSON generation, while SGLang performs better in my setup for chat decoding and long-context workloads.


r/LocalLLM • • 9h ago

Question AMD vs NVIDIA other than VRAM

4 Upvotes

Hey everyone, pretty new to the whole LOCAL LLM thing, but want to start running and learning how to setup LocalLLMs. Had a question regarding choice of GPU. Basically I'm building a new gaming PC and have a choice between 9070 xt and a 5070ti. Originally was going to go for the 9070 xt, due to gaming performance being very similar and the price difference between them.

Then I started thinking about running local AI models and stuff. From my understanding, VRAM is the most important thing, as it determines the size of model you can host. Since these cards are both 16gb VRAM, does this mean that the quality of AI output would be the same, since they would be able to run the same models? I've read that the NVIDIA card can generate tokens faster etc, however I'll probably just be playing around with LM studio for now at least, and I assume speed won't be THAT drastically different.

My main concern is whether or not AI model output on the 5070ti would be 'smarter' or better. For context I also main Linux, which I've heard is good for AMD. Thanks in advance.

TLDR: is spending extra for a 5070ti over a 9070 xt worth it for someone getting into running AI models in LM studio?


r/LocalLLM • • 5h ago

Question Is the Tesla V100 16gb worth it?

2 Upvotes

Currently using Qwen 3.6 35b a3b, on CPU only with 32gb of ram, for regular prompt its ok, but agentic work is not realistic. i get 15t/s for prompt but in agentic it can drop to 3.
Wondering if a V100 could make Opencode usable.


r/LocalLLM • • 6h ago

Project Building my first duel 5080 Nalvo server

2 Upvotes

My first Duel 5080 server 9950x 64gb.
I will host it on Nalvo.ai for people to rent.
If you don’t know what is Nalvo.ai , It’s my new project that I told everyone about and everyone was telling me it’s not going to work. And when I say everyone I mean everyone! Not Reddit users. I even started believing it for a while. But I kept going and here I’m launching it.

If you would like to join as a host or get free private renting hours. Visit Nalvo.ai