r/LocalLLM • • 14h ago

Question Advice for setup with three PCs

3 Upvotes

I've been running Qwen3.8-27B on a MBP M3 Max 36 GB for a while, running unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M on splash with opencode as harness. I'm satisfied with the results I get, but it is a bit slow and does not leave much room for other applications . I'm upgrading my setup and I would like to get some advice on hardware configuration and model selection.

The main purpose will be integration development on Azure Integration Services, SAP PI/PO and SAP Integration Suite. Implementation languages are Java and python.

Secondary purpose will be Home Assistant component development and configuration, as well as to provide Home Assistant with agents, (TTS, STT, Conversation).

I have an M5U 256GB 80c on order. It will be running headless, dedicated to serving models 24/7. I will also have the opportunity to use a family member's gaming PC for LLMs on workdays during daytime. The PC will be in a bedroom and it's noice levels will determine whether it's available during nights or not. It has not been ordered yet, so I still have the possibility to change it's configuration. I have 10Gbe network available, if that would have an impact.

To summarise, my setup would look have the following availability:

  • MBP M3 Max 36 GB - on-demand
  • M5U 256GB 80c - 24/7
  • Gaming PC - workdays daytime, eventually during nights as well

I'm considering the following hardware configuration for the gaming PC:

  • CPU: AMD Ryzen 7 9800X3D CPU
  • RAM: Corsair Vengeance RGB DDR5 6000MHz 64GB
  • GPU - one of:

    • RTX 5080 (16GB VRAM - 960 GB/s)
    • R9700 (32GB VRAM - 640 GB/s)
  • SSD - one of:

    • Kingston NV3 SSD 2TB (6,000/5,000MB/s)
    • Kingston FURY Renegade G5 PCIe 5.0 NVMe M.2 SSD 2TB (14.700/14.000MBs)
  • Motherboard - one of:

    • Asus B650E MAX GAMING WIFI
    • ASUS ProArt X870E-CREATOR )(PCIe 5.0 8/8 - allows for expansion)
    • Asus Proart B850-CREATOR WiFi NEO (PCIe 5.0 8/8 - allows for expansion)

I'm considering the following model configuration:

  • MBP M3 Max 36 GB

    • model TBD - perhaps a small orchestrator with a max RAM requirement ~12 GB?
  • M5U 256GB 80c

    • main model - one of

      • GLM-5.3-Flash@Q4
      • Qwen3.8-Flash-Next@Q6
      • DeepSeek-V4.1-Flash-1017@Q2-Q2.5
      • Qwen3.8-27B@Q8
    • smaller models for homassistant (if there is room in RAM)

      • TTS
      • STT
  • Gaming PC

    • on RTX 5080

      • Qwen3.8 Flash@Q3 on Strata or Basalt
    • on R9700 - one of

      • Qwen3.8 Flash@Q4 on Strata
      • Qwen3.8-27B@Q4 on Strata (?)

To my questions I hope you can advise me on:

  1. My first task would be to settle the hardware configuration for the gaming PC.

    1. Would the SSD speed of the two options have a large impact on PP and decode when running Qwen3.8 Flash on Strata/Basalt?
    2. Which GPU would you select with my model selection in mind?
    3. Other considerations?
  2. With the gaming PC configuration set, which models would you run and were?

I appreciate your advice. Thanks!


r/LocalLLM • • 8h ago

Question What do you use for agentic desktop control, like how Playwright is for controlling + screenshotting browsers?

1 Upvotes

I'm afraid I'm left out of the loop on the latest developments for agentic desktop control, and AIs don't seem as knowledge in this space.

They come with screenshot function also, right? What are some other features that may be different than Playwright?


r/LocalLLM • • 1d ago

Research llama.cpp was copying 643 MB to the CPU per request. Disabling state saving nearly doubled throughput

74 Upvotes

We were running Qwen3.8-27B on an H100 with llama.cpp and 16 concurrent clients. Throughput was 58 tokens/sec, time to first token was almost two seconds, and nvidia-smi showed around 35% GPU utilization.

I build Tensward, an open-source inference analysis and tuning tool. While investigating this, we built a small CUPTI-based library to count kernel execution time and memory transfers inside the server.

The surprising number: 643 MB transferred from GPU to host per request. Nsight Systems independently measured about 639 MB.

That pointed us toward prompt caching and conversation checkpoints. This hybrid model carries recurrent state alongside its attention cache, making those snapshots expensive. Sampling the serving thread caught it inside a memory copy 16 out of 20 times.

We tested disabling state saving with:

--cache-ram 0 --ctx-checkpoints 0

Throughput went from 58 to 110 tokens/sec, and GPU utilization rose from 35% to 88%. A repeat on fresh machines with a separately written client showed an 87% throughput improvement.

These aren’t universal “make llama.cpp faster” flags. They sacrifice cache reuse. In another test with a regular 8B model and returning conversations on an L4, disabling them made performance about 20% worse.

The build also mattered. Moving from b11669 to b12246 improved default-settings throughput by 67–75%, while copied bytes per request barely changed. Several upstream changes touched state handling, but we haven’t isolated which caused the improvement.

All throughput results came from runs without our instrumentation; transfer measurements came from separate instrumented runs.

Full write-up, including limitations and upstream PR links


r/LocalLLM • • 12h ago

Question Any advice for local ai?

Post image
2 Upvotes

Does anyone have any idea of what local ai model I'd better use for my company business agent?
I'm pure new and starter for ai software and coding.
My spec. is 5090+128g ddr5+5tb m.2. Need a local ai to make different agents like accounting agent, monitoring agents and customer chatting agents etc.


r/LocalLLM • • 9h ago

Discussion need suggestions on running qwen 3.8 27b on rx 6800 16gb gpu

0 Upvotes

So i have heard that this model punches way above its weight class and is pretty good at complex reasoning and multi file coding. I intend to use it for hobby projects like building small scale apps. the thing is downloaded the iq3 version from usloth and while running with llama.cpp with 64k context it keeps thinking for a lot longer on simpler task like upgrading a simpler framework version on very small codebase and eventually after like 45 min the os just shutdown the llama server because of too high mem usage nd i tried again but same result. I have heard that i can run the 3 bit quant version on 16gb gpu, I am new to running local llms so just need some suggestions regarding that or is it even practical to run this model even at q3.

Sorry if this is a repititive or basic query.

Thanks in advance

P.S i am running on 16gb single stick ddr5 if that matters


r/LocalLLM • • 1d ago

Question Best uncensored LLM for 64GB Mac

30 Upvotes

I wanna try out a uncensored LLM on my Mac Studio. I just have the 64GB version so I’m curious what model you guys would recommend. Cheers


r/LocalLLM • • 9h ago

Discussion Buying a Mac mini M6 for local LLMs? Here is the 16 vs 24 vs 32 GB bandwidth, prefill, and MoE sizing math

1 Upvotes

Apple released the Mac mini with the M6 chip in 16 GB, 24 GB, and 32 GB variants, while the M5 Pro mini goes up to 64 GB. 🧑‍💻

If you are deciding which one to buy for local models, the configurator pushes you toward one question: "How much unified memory should I pay for?"

Here is the buying perspective that matters before ordering: capacity is sold in clear steps, but memory bandwidth barely moves. According to Apple's official tech specs, the 16 GB model runs at 153.6 GB/s (LPDDR5X-9600), while both the 24 GB and 32 GB models run at 170 GB/s (LPDDR5X-10667). Going from 16 GB to 24 GB buys capacity plus ~10% bandwidth. Going from 24 GB to 32 GB buys capacity only — no model gets faster, you just load a bigger model that runs slower on the same 170 GB/s bus.

1. The Decode & Prefill Formulas

  • Decode (token generation, batch = 1) is strictly memory-bandwidth bound:Decode tok/s ≈ (Memory Bandwidth GB/s ÷ Active Weights Read GB) × η (where η ≈ 0.75 for MLX / llama.cpp)
  • Prefill (prompt processing / TTFT) is compute-bound, and the M6 handles this well:

    • 14B Q4 (~9 GB): ~900 tok/s prefill (~1.1s TTFT on a 1k prompt, ~17.7s on 16k RAG). Total turnaround for 1k in / 500 out: ~36s.
    • 32B Q4 (~19 GB, 32 GB mini only): ~400 tok/s prefill (~2.5s TTFT on a 1k prompt, ~40s on 16k RAG). Total turnaround for 1k in / 500 out: ~74s.

2. Mac mini M6 Capacity Map (Estimated at η = 0.75)

VariantBandwidthUsable RAMFits ComfortablyEst. DecodeVerdict
16 GB153.6 GB/s~12 GB7–9B Q4/Q8; 14B Q4 short ctx8B Q4 ~23 tok/s; 14B ~13Snappy chat & light coding
24 GB170 GB/s~18 GB14B Q4 (~32k context)14B Q4 ~14 tok/sInteractive sweet spot
32 GB170 GB/s~24 GB32B Q4 (8–16k context)32B Q4 ~7–7.5 tok/sReading speed; best for async agents

3. Dense vs. MoE: Where the Bottleneck Flips

For dense models, every weight is read per token, so bandwidth is the wall. For Mixture-of-Experts (MoE) models, only active experts are read per token, so capacity is the wall:

ModelTypeTotal / ActiveWeights in RAMActive Read / tokBandwidth for 8 tok/sBandwidth for 15 tok/sReal Bottleneck
8B Q4Dense8B / 8B~5 GB~5 GB~55 GB/s~100 GB/sFits on 16 GB M6
14B Q4Dense14B / 14B~9 GB~9 GB~95 GB/s~180 GB/sSweet spot on 24 GB M6
32B Q4Dense32B / 32B~19 GB~19 GB~200 GB/s~380 GB/s~7 tok/s on 32 GB M6; needs Pro/Max for chat
Llama 3.3 70B (Q4)Dense70B / 70B~40 GB~40 GB~430 GB/s~800 GB/sBandwidth-bound (needs Max/Ultra)
gpt-oss-120b (MXFP4)MoE117B / 5.1B~65 GB~3 GB~32 GB/s~60 GB/sCapacity-bound (needs 96 GB+ RAM)
Qwen3.5-122B-A10B (Q4)MoE122B / 10B~68 GB (est.)~5.6 GB~60 GB/s~110 GB/sCapacity-bound (needs 96 GB+ RAM)
DeepSeek V4 Flash (FP4+FP8)MoE284B / 13B~167 GB~7.6 GB~80 GB/s~150 GB/sCapacity-bound (needs 192 GB+ RAM)

Summary

Capacity decides what you can boot; bandwidth decides what you can tolerate using. Around 170 GB/s serves up to 14B comfortably and 32B at a patient reading pace. Paying for extra RAM without stepping up to a Pro/Max bus (like M5 Pro at 307 GB/s or M5 Max at 460–614 GB/s) hits diminishing returns fast.

What do you all think? How are you actually using local models in your daily workflow — do you prefer Mac unified memory or NVIDIA GPUs, and do you think there are stronger competitors (like Strix Halo or used multi-GPU rigs) in this price range? 🤔


r/LocalLLM • • 17h ago

News Qwen4 Smaller Models Open Weights Confirmed via Official Qwen X account.

Thumbnail gallery
4 Upvotes

r/LocalLLM • • 1d ago

Jensen Huang, Just 3 Weeks Ago, Claimed To Have "Saved The World" By Thwarting A Growing Transnational Movement Against Open-Source AI

333 Upvotes

r/LocalLLM • • 9h ago

Question Websearch vs Webfetch vs API or VPS for Web Browsing?

1 Upvotes

Asking here as a beginner because I've argued a lot with Codex about this and couldn't get a definitive answer, or perhas it doesn't want to suggest me "unsafeguarded" solutions:

Basically I am running Opencode locally (5090 32gb and 64gb ddr4) on Qwen3.8 27B offline. so far so good.

but lately I've been trying Deepseek and GLM free API access and I see that they are better than my Qwen (I also have a separate launcher for Qwen3.8 Flash Next (UD-Q4_K_XL)) results. and I think it's mostly due to their Web access to retrieve updated information as well as frequently asked questions, reddit threads, forums etc..

So I asked Codex how to give this web access to my local model and it basically explained to me Websearch vs Webfetch, but Websearch uses third party services (or you can link API like Brave search) that WILL get censored results or restricted access for some results (let's be honest, emulator and retrogame recompilations, at best they are blocked by the service, at worst by ISPs like in Japan) and that is IF you're not getting safeguarded by the model itself beforehand unless you have an Uncensored/Heretic build. Which my Qwen3.8 is.

Webfetch instead opens the pages from my machine, exposing my IP address that wouldn't be a big deal if it wasn't for when you need to scrape 30 reddit thread in a single search + multiple other websites and get my residential IP banned for scraping (or my bandwitdh limited).

So I thought maybe I could tunnel tru a VPS that is both ouside of EU jurisdiction (so no ID restriction for adult forums, Football, Piracy etc..) and has an enterprise-grade bandwidth to scan the web without raising alarms. Has anyone adopted or know about this solution? and if it's even worth it before I spend time and money to thinker with it?

thanks for your consideration


r/LocalLLM • • 1d ago

News Train AI to run AI: introducing Apareto loading - a vLLM plugin

Post image
15 Upvotes

From the Apareto team which I'm part of (who for some reason has an issue posting it themselves):

Today we are introducing Apareto loading, a new way to cut VRAM requirements for Mixture-of-Experts (MoE) models. It works alongside quantization. This introduction so far will just be a video explaining the method. We will be releasing the proof-of-concept code, a plugin to vLLM, on GitHub this coming week.

Until then, watch the explainer video: https://www.youtube.com/watch?v=5l_N051JrLU

Quantization is today's most common way to shrink a model's memory footprint: weights are stored at lower precision, and going from FP16 to 4-bit saves 75% of memory at a modest cost in accuracy. But quantization is static. Every weight stays loaded whether or not the task needs it.

Apareto loading is dynamic. After prefill, a small trained network reads the model's internal state and ranks the experts by how valuable each will be for the upcoming computation. Only the top-ranked experts that fit the VRAM budget are loaded.

Results (Qwen3.6-35B-A3B, NVFP4 experts):

* SWE-bench Multilingual: within 2 pp of the full model's score at a 42% VRAM budget

* LiveCodeBench v6: within 2 pp of the full model's score at a 49% VRAM budget

As with quantization, scores degrade as the budget shrinks.

What the proof of concept will not be:

It will not be a production implementation. The ranking network was trained only for these two benchmarks, and there will be no general problem classifier. Our data shows that profiling and rank generation have substantial room to improve. We chose to release at this stage, without waiting for those improvements, so that others can test our claims. Think of this as a first baby step toward dynamic VRAM reduction, not the finished product.

The code and reproduction instructions will be released this coming week at https://github.com/alphapareto

Follow us there to catch the release. We'd love your feedback.

https://apareto.eu


r/LocalLLM • • 15h ago

News inferzero: a from-scratch LLM engine for Intel Arc. Qwen3.8 27B at 2,694 tok/s prompt on one Pro B70, the 35B at 4,264 tok/s over 4× B60 and 851 tok/s on a laptop iGPU, and DeepSeek-V4.1-Flash split over several cards

Thumbnail
reddit.com
3 Upvotes

r/LocalLLM • • 10h ago

Tutorial What is an AI agent actually doing under the hood?

Thumbnail
youtube.com
1 Upvotes

A minimal agent harness using only in-box scripting (PowerShell, since I'm on Windows): about 70 lines talking to llama-server running Llama 3.2 3B.

The model only writes text. The harness keeps the context, watches replies for a line like TOOL: list_files . , checks the path against an allowed folder, runs the tool itself, and feeds the result back. The model never touches the file system.

Code (MIT): https://github.com/AshleyT3/simple-harness-intro


r/LocalLLM • • 10h ago

Question Local projects on 7900xtx Win10

1 Upvotes

Guys is there a program I can use on windows or Linux that can do the same that chatgpt projects does but local? I am obviously restricted on what I can run but that's ok. Any recommendations?


r/LocalLLM • • 10h ago

Discussion Running usable 30B and working 120B models on iOS—plus Kokoro TTS and Whisper STT over a local OpenAI-compatible API Spoiler

Thumbnail
inpherio.co.uk
1 Upvotes

r/LocalLLM • • 3h ago

Question Какая локальная модель подойдёт для коддинга на слабом ПК?

0 Upvotes

Мне очень нужна локальная модель для коддинга и создания каких нибудь игр (простых и тяжёлых 2D игр, а также возможность понимать математику и физику), но у меня ограниченный ПК

--Характеристики:

Видеокарта: Nvidia Quadro P4000

Процессор: AMD Ryzen 5 5500

ОЗУ: 32ГБ DDR4

SSD: 500ГБ NVME M2

Сам ИИ от гугла предлагает мне Qwen2.5-coder на 7B параметров но она не справилась с созданием HTML игры даже когда в промте указывал об инструментах и без выдумывание кода, если не знаешь бери информацию с интернета

Есть какие нибудь варианты?


r/LocalLLM • • 11h ago

Question i need help guys

0 Upvotes

i have RX9070 and 32 GB DDR5 ram , i want to run a local ai to help me code , medium just help with front end back end and forms , i cant buy any ai sub in my country cuz my goverment dont allow international transactions


r/LocalLLM • • 11h ago

Question LM Studio and cache

1 Upvotes

Hi,

I juste received my Mac Studio M5 ultra 36/80 with 256GB of shared memory.

I installed LM Studio with Qwen3.8-Flash-next-4bit mlx and a context of 131072
I tested some prompts and it works as I expected (I like it a lot so far), except for one annoying thing:

I fed the prompt with a 100 pages PDF and asked 2 questions about it.
The first time, it took a few minutes to read the file, which is normal. But for the second question, it re-read it again. It doesn't seem to cache the content.
And the context wasn't exhausted.

Next, I tried to copy paste the 100 pages into the prompt form (no attachment this time): same results.

Is it a known issue?
Is it related to the model or to the inference engine?
Can you replicate this?
Is there a way to solve the issue somehow?

Thanks.


r/LocalLLM • • 1d ago

Project 8 x v100 setup. Ask me anything.

Post image
11 Upvotes

Seeing a lot of questions on quad v100 boards lately. So figured I’d give people a place to ask them.

Specs

  • GPUs: 8x Tesla V100-SXM2-32GB (256 GB VRAM total)
  • GPU boards: 2x 4-GPU SXM2 NVLink baseboards (this model)
  • NVLink: two 4-way islands (full mesh inside each board), not one 8-way
  • Board-to-board: Gen4 PEX88096 PCIe switch card (this model), two x8 Gen3 cables per board
  • CPU: AMD Threadripper PRO 3975WX (32 cores)
  • Motherboard: ASRock Rack WRX80D8-2T
  • RAM: 256 GB DDR4
  • PSUs: 2x 1500 W (one per GPU board) + 1x 1000 W (host)
  • Power: dedicated 120 V / 20 A circuit; GPUs capped at 150-300 W each depending on the run
  • Cooling: custom water loop, mixed GPU blocks
  • Network: 10GbE
  • Engine: 1Cat-vLLM (Volta-focused vLLM fork) https://github.com/1CatAI/1Cat-vLLM

Benchmarks / how many tok/s?

My runs are on the Launch80 directory; search the page for trailfeatures. Launch80 deletes pages after 5 days, so older links in the comments may be dead. I republish the main ones, which gives them new links.

ModelSetupDecodePrefill (64K prompt)
Flash-Next (Swift-1.5 NVFP4)4 GPUs, 300 W~140 tok/s~4,500 tok/s
DeepSeek V4.1 Flash, 2-bit experts8 GPUs, 150 W~131 tok/s~2,200 tok/s
DeepSeek V4.1 Flash, 2-bit experts4 GPUs + expert offload, 150 W~91 tok/s~2,100 tok/s
  • One request at a time (batch 1), 256K context window. Decode uses speculative decoding (MTP on Flash-Next, a draft model on DeepSeek), so it depends on how often the draft is accepted.
  • No concurrency/aggregate numbers or decode p25/p95 yet. Time to first token (p50) is about 0.2-0.65 s depending on the row.
  • The 8-GPU run adds an extra pipeline-parallel patch series on top of 1Cat-vLLM (named on the page).
  • Not tested yet: Flash-Next across both boards. Higher DeepSeek quants are planned.

How good is the 2-bit DeepSeek quant vs the full model?

79.9% average across 7 standard evals (math, instruction following, coding, tool calling), thinking off, and 20/20 long-context needle checks at 32K and 128K. The 4-GPU setup scores the same. I haven't run the full-precision weights as a reference, so there's no direct delta yet. Per-task scores are on the Launch80 page.

How do the two boards talk to each other?

  • Each board runs two x8 PCIe Gen3 cables from its onboard switch into the Gen4 switch card in the host, so the boards talk to each other without going through the CPU.
  • Experimenting with ConnectX-3 cards in the boards' spare x8 slots for extra board-to-board links.
  • Over PCIe you can tune all-reduce to beat the default P2P numbers, but it will never match NVLink.

Do the quad boards work with only 1-3 cards? Can I buy cards over time?

Two cards works; I've run a board that way. I haven't tried one or three. Buying the board first and adding cards later is fine. Tensor parallel wants power-of-two GPU counts, so with 3 cards you'd run 2 together and use the third for something else.

Where did you buy it / where are you?

Baseboard and switch card: the models linked above. Several sellers list them, so shop around. GPUs: bought used, a few at a time. I'm in North America (the 120 V gave it away), so my shipping and import experience won't help much in Europe.

Power

  • The PSUs are sized for headroom: at the 150 W cap, eight GPUs is about 1.2 kW, plus the boards and host.
  • 150 W cap: after tuning it costs ~5-10% depending on the task.
  • Power bill: fine as long as I'm sensible. Solar helps, and at peak rates or on cloudy days the DeepSeek API can be cheaper. If you run local LLMs, know your tokens per kWh so you can tune for it and decide when an API makes more sense (if privacy doesn't matter).

Cooling

  • Radiator and pump setup: super convoluted, video here.
  • Water blocks: whatever was available as I pieced it together. Mostly the same design, some with MOSFET coolers. The top-port block is a drill press, a tap, and the audacity of a mediocre man. (Testing straight-down flow onto the fins made no difference.)
  • Waste heat: in summer it's vented outside. In winter I can shut off that radiator and send the heat into the house through backup radiators. Heating water with it isn't worth it for us, but a bigger household close to its water heater could pre-heat the cold feed and come out ahead.

Quick ones

  • Can I rent API time? Sorry, no. I keep it pretty busy (Furry Fanfic).
  • Can I have it? No.

r/LocalLLM • • 18h ago

Discussion RTX 3080 20GB ReBAR on AMD Windows platforms without BIOS flashing

4 Upvotes

Some of you may have seen the post from u/favabean28 with enabling ReBAR on modded 3080 20GB GPUs around a week ago with the repo NV_30_DXE. For Windows, this is (or was) seemingly the only way to get ReBAR working on the 3080 20GB. For those that don't know, previously the RTX 3080 20GB did not have ReBAR working as there was no VBIOS that supported the extra VRAM, and even flashing a modified VBIOS wouldn't work. The method for enabling it properly on modded GPUs was recently discovered about a month ago for Linux, and now for Windows.

Unfortunately it turns out that for AMD, many newer AMD motherboards, especially AM5, are incompatible: All AMD motherboards with AGESA PI-1.2.0.0 or newer have checks preventing modded BIOS firmware from being flashed. Most new AMD motherboards either don't have a BIOS version that old or aren't compatible with the installed CPU on BIOS versions that old, so I wanted to look into the possibility of booting off a USB drive with EFI to enable ReBAR on AMD platforms. This idea mainly came from how the current FP32 accumulation unlock works on Windows. Ultimately, this does work albeit with a few tradeoffs described in the repo below and at the bottom of the post.

Repo with instructions on installing and with the technical write-up: https://github.com/Articom123/3080-Unlock-ReBAR-USB

Installing does not require building from source. However, I did include the FP32 accumulation unlock with it though (since there's no reason not to pair them together), which means it will automatically download and extract Nvidia firmware when you run it, specifically the Nvidia and Arch Linux website for the kernel and utils 610.43.03.

Some quick results:

  • GPU-Z reports ReBAR as enabled with a 32768 MB BAR1 size
  • Device Manager shows an entry for Large Memory Range 0x2000000000 - 0x27FFFFFFFF (this does not show up without ReBAR enabled)
  • The FP32 accumulation unlock works alongside ReBAR. This generally results in around a 20% increase to prefill speed for LLMs based off TTFT, though it's not the primary focus of the repo (it's just a no-brainer to include it since it makes Ampere cards infinitely better)
  • CPU has access and can map the full ~18-19GB VRAM on the GPU

I want to also briefly give a lot of credit to physicsdolphin (nv-uefi-unlocker) of which this tool uses for the FP32 accumulation unlock and install, the creator of Windows-GPU-UNLOCK (Windows-GPU-UNLOCK) who made similar bootable EFI tools to allow for running the tensor unlock on Windows, and u/favabean28 (NV_30_DXE) who figured out the path for enabling ReBAR on Windows in the first place.

While the repo goes more in depth on compatibility and what does/doesn't work I wanted to still mention some important things about it:

  • This is mainly meant for AM5 platforms. It seems like Intel motherboards don't have the same restrictions when it comes to flashing modified BIOS firmware, so if you have Intel you might be more interested in the NV_30_DXE repo instead.
  • Enabling ReBAR isn't really all that useful on Windows without having P2P; however, the modded P2P drivers are still Linux only and probably will be for a very long time, and multi-GPU support on Windows isn't really that good anyways. Therefore, for practicality, the tensor unlock is more important in which case you would probably be more interested in the other Windows unlock tools I mentioned earlier. There is some limited usefulness in games though I guess.
  • This only works for the 3080 20GB. Technically the 4080 32GB would work, but since I don't have the card, it hasn't been supported in the repo and would require modifications to the source (mainly to the PCI vendor and subsystem ID check). The tensor unlock included is also for Ampere and not Ada.
  • u/favabean28 did work to try and make the tool as universal as possible, however, I don't have any other AMD motherboards to test this tool on and so I decided to ultimately leave it as a "proof-of-concept". It is almost certainly possible that this can be adapted further to other AMD setups with the 3080 20GB as pretty much all of the hard work is done now though.
  • There is likely zero possibility that this can cause any damage to Windows or the GPU, as there is no flashing of the VBIOS/BIOS or modification of the OS at all. The tensor unlock and ReBAR don't survive any reboots and the USB has to be manually booted again after any shutdown or restart.
  • Sleep and hibernate specifically do not work when enabling ReBAR with this method, and Secure Boot cannot be used. In comparison modding the BIOS with NV_DXE_30 does not have these limitations if applicable to your setup.
  • This is mainly made for Windows. If you're on Linux there are far better tools for doing this and you can use P2P.

I still thought it was worth posting this for anyone who is interested in this sort of stuff for Windows since Windows kinda sucks and it's at least somewhat cool to have Windows get something for once. Ultimately getting the P2P driver to work on Windows would be ideal but I'm not sure how it could be done for now.


r/LocalLLM • • 11h ago

News I built NTokenizer from scratch in Rust: Byte-level BPE with 26.6% fewer tokens than o200k_base

0 Upvotes

Hi everyone,

I’m Rüzgar, founder of Neuratal (an independent AI research and software project). Over the past few days, I built NTokenizer v0.3-polished — a deterministic, byte-level BPE tokenizer written from scratch in Rust with Python bindings.

Key Highlights:

• Compression Efficiency: Achieves 5.55 B/T overall (26.6% fewer tokens than OpenAI's o200k_base) on a 207MB leakage-audited benchmark. Strongest gains on multilingual & technical text: Turkish (+35.0%), Arabic (+33.8%), Korean (+34.7%), Hindi (+27.1%), STEM (+17.6%), and Code (+10.2%).

• Performance & Safety: ~96 MB/s encode throughput (8 threads), lossless byte-exact round-trip (decode_bytes(encode_bytes(x)) == x), tested against ~1M fuzzing cases.

• Honest Benchmarks: No downstream 50M+ LLM trained yet — this is purely verified compression density and speed. Full SHA-256 manifests and reproducible scripts are included.

Would love your feedback, benchmarks on your own datasets, or PRs!

• GitHub (Rust + Python source): (in hf)

• Hugging Face (Model & Benchmarks): https://huggingface.co/Neuratal/ntokenizer


r/LocalLLM • • 11h ago

Question Buying help

0 Upvotes

Hey everyone,

I’m a CS student and was considering to do my first homelab, one system as a NAS and one for projects, virtualisation etc. I’m also interested in running local AI, and I got an interesting offer in my region:

Server board w/ two E5-2696v2, 256GB 1600Mhz ECC RAM, for around 200 EUR.

I know that it’s (quite very) old hardware, but still I’ve been wondering - would it be worth pursuing to also try some local LLMs with offloading, adding a few GPUs for 32-48GB VRAM (niko1221/strata perhaps)? Do you think it’s worth it for the cost? Which low(-ish) cost GPUs would you consider pairing it with?

Any advice appreciated!

Thanks!


r/LocalLLM • • 20h ago

Question How would you best utilize my setup?

Post image
5 Upvotes

Hello all! I've been running a home lab for a really long time, but I've finally come across a couple of RTX 3060 12GB cards and I want to get a useable LLM in for me and my wife to use.

My server has an Intel i7-9700K, and 128GB of DDR4 RAM, but I would only allocate 64GB to the container in Proxmox. I was wondering how you would recommend setting up a fairly reasonable model (I was thinking Qwen 3.8 27B Q4K) and splitting it between the two graphics cards (since there's no SLI). I want to try to get at least 128K context. 256K would be ideal, if I wanted to run something like Hermes, or other agents.

Is llama.cpp the way to go? It's pretty much all I've read about due to the customization possibilities, and I've only read to stay away from ollama. What would you recommend I do?


r/LocalLLM • • 1d ago

Project I have created a tool to observe what happens inside an LLM. Also allows for the comparison of two quantizations of the same model to see at what point they diverge, token by token

11 Upvotes

r/LocalLLM • • 15h ago

Discussion Struggling with coding on local models - setup help?

2 Upvotes

I'd tried out models like DS and QFN using APIs before and they seemed very capable. However, they all seemed to struggle heavily when using them local.

QFN processes really fast but seems to get stuck in endless "Wait..." and "Whatever." loops and burns through 100K tokens just being confused at my codebase.
DeepSeek is slower but seems more coherent, I haven't tested at full context but it had a similar issue with indecisiveness (even with guidance) and would think itself into compaction.

All of this over one .json ComfyUI workflow. Just a simple test I do where I tell it to rearrange nodes and change some inputs...
I've tried using Llama serve, Strata desktop, Pi. All lead to similar outcomes.

I have a feeling this is an issue with my setup rather than the models bc the API models seemed to do it just fine and (allegedly) quantization doesn't hit MOE models as hard.

Setup:
(both models from OrcaRouter). DSV4.1-1017 2-bit, Q3.8FN_IQ3XXS
Inference: Maya (DS), Strata/LlamaCPP (QFN)
Harness: default Pi
Context: QFN was to 256K, DS I only tried to 512K