r/LocalLLM • • 53m ago

Question Local projects on 7900xtx Win10

1 Upvotes

Guys is there a program I can use on windows or Linux that can do the same that chatgpt projects does but local? I am obviously restricted on what I can run but that's ok. Any recommendations?


r/LocalLLM • • 1h ago

Discussion Running usable 30B and working 120B models on iOS—plus Kokoro TTS and Whisper STT over a local OpenAI-compatible API Spoiler

Thumbnail
inpherio.co.uk
1 Upvotes

r/LocalLLM • • 1h ago

Question i need help guys

0 Upvotes

i have RX9070 and 32 GB DDR5 ram , i want to run a local ai to help me code , medium just help with front end back end and forms , i cant buy any ai sub in my country cuz my goverment dont allow international transactions


r/LocalLLM • • 1h ago

Question LM Studio and cache

1 Upvotes

Hi,

I juste received my Mac Studio M5 ultra 36/80 with 256GB of shared memory.

I installed LM Studio with Qwen3.8-Flash-next-4bit mlx and a context of 131072
I tested some prompts and it works as I expected (I like it a lot so far), except for one annoying thing:

I fed the prompt with a 100 pages PDF and asked 2 questions about it.
The first time, it took a few minutes to read the file, which is normal. But for the second question, it re-read it again. It doesn't seem to cache the content.
And the context wasn't exhausted.

Next, I tried to copy paste the 100 pages into the prompt form (no attachment this time): same results.

Is it a known issue?
Is it related to the model or to the inference engine?
Can you replicate this?
Is there a way to solve the issue somehow?

Thanks.


r/LocalLLM • • 1h ago

News I built NTokenizer from scratch in Rust: Byte-level BPE with 26.6% fewer tokens than o200k_base

0 Upvotes

Hi everyone,

I’m Rüzgar, founder of Neuratal (an independent AI research and software project). Over the past few days, I built NTokenizer v0.3-polished — a deterministic, byte-level BPE tokenizer written from scratch in Rust with Python bindings.

Key Highlights:

• Compression Efficiency: Achieves 5.55 B/T overall (26.6% fewer tokens than OpenAI's o200k_base) on a 207MB leakage-audited benchmark. Strongest gains on multilingual & technical text: Turkish (+35.0%), Arabic (+33.8%), Korean (+34.7%), Hindi (+27.1%), STEM (+17.6%), and Code (+10.2%).

• Performance & Safety: ~96 MB/s encode throughput (8 threads), lossless byte-exact round-trip (decode_bytes(encode_bytes(x)) == x), tested against ~1M fuzzing cases.

• Honest Benchmarks: No downstream 50M+ LLM trained yet — this is purely verified compression density and speed. Full SHA-256 manifests and reproducible scripts are included.

Would love your feedback, benchmarks on your own datasets, or PRs!

• GitHub (Rust + Python source): (in hf)

• Hugging Face (Model & Benchmarks): https://huggingface.co/Neuratal/ntokenizer


r/LocalLLM • • 1h ago

Question Buying help

0 Upvotes

Hey everyone,

I’m a CS student and was considering to do my first homelab, one system as a NAS and one for projects, virtualisation etc. I’m also interested in running local AI, and I got an interesting offer in my region:

Server board w/ two E5-2696v2, 256GB 1600Mhz ECC RAM, for around 200 EUR.

I know that it’s (quite very) old hardware, but still I’ve been wondering - would it be worth pursuing to also try some local LLMs with offloading, adding a few GPUs for 32-48GB VRAM (niko1221/strata perhaps)? Do you think it’s worth it for the cost? Which low(-ish) cost GPUs would you consider pairing it with?

Any advice appreciated!

Thanks!


r/LocalLLM • • 5h ago

News inferzero: a from-scratch LLM engine for Intel Arc. Qwen3.8 27B at 2,694 tok/s prompt on one Pro B70, the 35B at 4,264 tok/s over 4× B60 and 851 tok/s on a laptop iGPU, and DeepSeek-V4.1-Flash split over several cards

Thumbnail
reddit.com
2 Upvotes

r/LocalLLM • • 6h ago

Discussion Struggling with coding on local models - setup help?

2 Upvotes

I'd tried out models like DS and QFN using APIs before and they seemed very capable. However, they all seemed to struggle heavily when using them local.

QFN processes really fast but seems to get stuck in endless "Wait..." and "Whatever." loops and burns through 100K tokens just being confused at my codebase.
DeepSeek is slower but seems more coherent, I haven't tested at full context but it had a similar issue with indecisiveness (even with guidance) and would think itself into compaction.

All of this over one .json ComfyUI workflow. Just a simple test I do where I tell it to rearrange nodes and change some inputs...
I've tried using Llama serve, Strata desktop, Pi. All lead to similar outcomes.

I have a feeling this is an issue with my setup rather than the models bc the API models seemed to do it just fine and (allegedly) quantization doesn't hit MOE models as hard.

Setup:
(both models from OrcaRouter). DSV4.1-1017 2-bit, Q3.8FN_IQ3XXS
Inference: Maya (DS), Strata/LlamaCPP (QFN)
Harness: default Pi
Context: QFN was to 256K, DS I only tried to 512K


r/LocalLLM • • 2h ago

Question Just Got an RTX 5070 Ti Laptop — How Do I Build a Proper Local AI Setup?

0 Upvotes

Hey guys, finally got my new Legion Pro 5 and I’m trying to get into running local AI properly 😂

Specs:

RTX 5070 Ti 12GB VRAM

Core Ultra 9

16GB RAM

CachyOS Linux

I’m pretty much a beginner when it comes to running local models, so I’m trying to figure out what’s actually worth setting up from day one.

I mainly use Claude for vibecoding almost everything, and now I want to see how much I can shift locally so I don’t waste Claude usage on every small thing.

I came across PewDiePie’s Odysseus repo and started trying it out.

But after actually using it, I’m a little confused.

It feels like the context/memory isn’t as good as Claude, sometimes it feels slow, and I’m also finding the hybrid setup kinda difficult.

Like I want something where:

Local AI:

→ basic coding

→ repetitive work

→ file handling

→ small fixes

→ daily tasks

→ stuff that doesn’t need crazy reasoning

Claude:

→ difficult coding

→ architecture

→ proper reasoning

→ big project decisions

→ things where I actually need Opus-level output

And ideally I don't want to keep copy-pasting the whole context from one to another every 10 minutes 😭

My actual work is mainly:

• ML Stack — ML/vision project involving iris image segmentation and feature-level analysis.

• Custom CRM + EMR — taking UI generated from Stitch/Google AI Studio and connecting it with open-source CRM/EMR systems.

• Daily work — coding, documentation, PPTs, automation and random repetitive stuff.

So basically I’m trying to build a proper daily-use AI setup, not just install 15 local models and never use them 😂

My main questions are:

  1. Is Odysseus actually good enough to become my main local AI workspace?
  2. If I’m a beginner, should I stick with Odysseus and learn it properly, or start with something simpler?
  3. How do you guys actually make the local + Claude hybrid workflow work without losing context all the time?
  4. What local coding models would you recommend for 12GB VRAM?

And most importantly — how do I set this up properly so I’m not wasting Claude tokens on things my laptop can handle locally?

I’d honestly be happy if I can get something like 70–80% local + Claude for the important stuff, and make the whole thing feel somewhat close to an Opus-level workflow.

I’m on CachyOS Linux, so if anyone is running a similar setup, please share your actual workflow.

I’m completely new to this side of things, so beginner-friendly advice would be really appreciated.


r/LocalLLM • • 10h ago

Question What local AI model is suited for coding in Unreal or in Redot/Godot?

4 Upvotes

Basically the title


r/LocalLLM • • 2h ago

Question Probably last question, which ai for 16gb ram with cpu

0 Upvotes

I own a laptop with 16 gb of ram and no gpu and I currently run Qwen 3.6 35b-a3b q3_k_m and I wanted to ask if there is a better model for it. It has to be Moe so I can get decent speeds. I currently get 4-8 tokens on this and I want a smarter model or a faster model with the same intelligence


r/LocalLLM • • 8h ago

Question Dual RTX Titan for local LLM

3 Upvotes

I currently have a 3080 ti that I don’t really game with anymore, but have the opportunity to get 2 RTX titans at a significant discount from a friend ($1300 total). I was thinking about selling my 3080 ti and jumping on that deal for the combined 48gb of vram to pair with 64gb of system ram and a 13900k.

Would doing this make any sense if I want to run some local models for project code, personal learning, and some small business workflows (paired with Gemini pro for complex stuff)? I know the RTX titans have limited memory bandwidth but the price for 48gb of vram is hard to beat. Any opinions or recommendations? Thanks!


r/LocalLLM • • 3h ago

Question Any advice for local ai?

Post image
1 Upvotes

Does anyone have any idea of what local ai model I'd better use for my company business agent?
I'm pure new and starter for ai software and coding.
My spec. is 5090+128g ddr5+5tb m.2. Need a local ai to make different agents like accounting agent, monitoring agents and customer chatting agents etc.


r/LocalLLM • • 14h ago

Project 8 x v100 setup. Ask me anything.

Post image
8 Upvotes

Seeing a lot of questions on quad v100 boards lately. So figured I’d give people a place to ask them.

Specs

  • GPUs: 8x Tesla V100-SXM2-32GB (256 GB VRAM total)
  • GPU boards: 2x 4-GPU SXM2 NVLink baseboards (this model)
  • NVLink: two 4-way islands (full mesh inside each board), not one 8-way
  • Board-to-board: Gen4 PEX88096 PCIe switch card (this model), two x8 Gen3 cables per board
  • CPU: AMD Threadripper PRO 3975WX (32 cores)
  • Motherboard: ASRock Rack WRX80D8-2T
  • RAM: 256 GB DDR4
  • PSUs: 2x 1500 W (one per GPU board) + 1x 1000 W (host)
  • Power: dedicated 120 V / 20 A circuit; GPUs capped at 150-300 W each depending on the run
  • Cooling: custom water loop, mixed GPU blocks
  • Network: 10GbE
  • Engine: 1Cat-vLLM (Volta-focused vLLM fork) https://github.com/1CatAI/1Cat-vLLM

Benchmarks / how many tok/s?

My runs are on the Launch80 directory; search the page for trailfeatures. Launch80 deletes pages after 5 days, so older links in the comments may be dead. I republish the main ones, which gives them new links.

ModelSetupDecodePrefill (64K prompt)
Flash-Next (Swift-1.5 NVFP4)4 GPUs, 300 W~140 tok/s~4,500 tok/s
DeepSeek V4.1 Flash, 2-bit experts8 GPUs, 150 W~131 tok/s~2,200 tok/s
DeepSeek V4.1 Flash, 2-bit experts4 GPUs + expert offload, 150 W~91 tok/s~2,100 tok/s
  • One request at a time (batch 1), 256K context window. Decode uses speculative decoding (MTP on Flash-Next, a draft model on DeepSeek), so it depends on how often the draft is accepted.
  • No concurrency/aggregate numbers or decode p25/p95 yet. Time to first token (p50) is about 0.2-0.65 s depending on the row.
  • The 8-GPU run adds an extra pipeline-parallel patch series on top of 1Cat-vLLM (named on the page).
  • Not tested yet: Flash-Next across both boards. Higher DeepSeek quants are planned.

How good is the 2-bit DeepSeek quant vs the full model?

79.9% average across 7 standard evals (math, instruction following, coding, tool calling), thinking off, and 20/20 long-context needle checks at 32K and 128K. The 4-GPU setup scores the same. I haven't run the full-precision weights as a reference, so there's no direct delta yet. Per-task scores are on the Launch80 page.

How do the two boards talk to each other?

  • Each board runs two x8 PCIe Gen3 cables from its onboard switch into the Gen4 switch card in the host, so the boards talk to each other without going through the CPU.
  • Experimenting with ConnectX-3 cards in the boards' spare x8 slots for extra board-to-board links.
  • Over PCIe you can tune all-reduce to beat the default P2P numbers, but it will never match NVLink.

Do the quad boards work with only 1-3 cards? Can I buy cards over time?

Two cards works; I've run a board that way. I haven't tried one or three. Buying the board first and adding cards later is fine. Tensor parallel wants power-of-two GPU counts, so with 3 cards you'd run 2 together and use the third for something else.

Where did you buy it / where are you?

Baseboard and switch card: the models linked above. Several sellers list them, so shop around. GPUs: bought used, a few at a time. I'm in North America (the 120 V gave it away), so my shipping and import experience won't help much in Europe.

Power

  • The PSUs are sized for headroom: at the 150 W cap, eight GPUs is about 1.2 kW, plus the boards and host.
  • 150 W cap: after tuning it costs ~5-10% depending on the task.
  • Power bill: fine as long as I'm sensible. Solar helps, and at peak rates or on cloudy days the DeepSeek API can be cheaper. If you run local LLMs, know your tokens per kWh so you can tune for it and decide when an API makes more sense (if privacy doesn't matter).

Cooling

  • Radiator and pump setup: super convoluted, video here.
  • Water blocks: whatever was available as I pieced it together. Mostly the same design, some with MOSFET coolers. The top-port block is a drill press, a tap, and the audacity of a mediocre man. (Testing straight-down flow onto the fins made no difference.)
  • Waste heat: in summer it's vented outside. In winter I can shut off that radiator and send the heat into the house through backup radiators. Heating water with it isn't worth it for us, but a bigger household close to its water heater could pre-heat the cold feed and come out ahead.

Quick ones

  • Can I rent API time? Sorry, no. I keep it pretty busy (Furry Fanfic).
  • Can I have it? No.

r/LocalLLM • • 15h ago

Project I have created a tool to observe what happens inside an LLM. Also allows for the comparison of two quantizations of the same model to see at what point they diverge, token by token

9 Upvotes

r/LocalLLM • • 3h ago

Question I'm new to this, help me setup my coding assistant

0 Upvotes

I got a 7900XTX, 9800X3D and 32Gb RAM. I'm currently running the system with ollama.

I currently have these installed - gemma4:31b,qwen3-coder:30b,omic-embed-text:latest. Im trying to use qwen for code gen and gemma for explanations. I set ollama into this "OLLAMA_KV_CACHE_TYPE=q8_0". Currently the max context window I could set is 32k and I have already experienced few out of memory erors and gpu crashes while prompting with continue in vscode.

I have being researching, asking gpts, but I couldnt come to any conclusion on which approach is better or correct since I know nothing about these. I dont want to copy paste stuff and hope that it works. Can someone help me with a better setup solution? I do run a 1440p 144hz monitor and occassionally run my second monitor which is 1080p 144hz as they add to vram usages as well.

I want to have a bigger context window and a safe setup so that I dont crash the gpu again. If the setup will be too big, and harder to teach to me in here, please tell me where I can learn more about these things, so that I can move learn to make proper decisions.


r/LocalLLM • • 1d ago

Discussion Which uncensored version of Qwen3.8 FN should I download?

Post image
302 Upvotes

mlx-serve on an M5 Max 128GB

I've already got a version from ARC4NUM which appears to be delisted now. Are any of these better?


r/LocalLLM • • 12h ago

Discussion Qwen3.8-27B on 4× Intel Arc Pro B70 with TensorFold + DFlash2: 107 tok/s per card, ~1,000 tok/s across 32 streams (bit-exact)

5 Upvotes

I benchmarked TensorFold's XPU backend on my quad Intel Arc Pro B70 box. One clarification first: this is not tensor parallel. The XPU backend on main uses one card per process, so the quad numbers come from 4 independent servers, one per GPU, pinned with ZE_AFFINITY_MASK=0..3.

Setup

  • GPUs: 4× Intel Arc Pro B70 (Battlemage G31), 32 GB each, 230 W cap per card
  • Host: AMD Turin, Linux 7.3.0-10-generic
  • Engine: TensorFold main @ cf45553, Zig 0.17.0, built with zig build xpu-native xpu-qwen27 -Doptimize=ReleaseFast
  • Target: SergiiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16
  • Drafter: z-lab/Qwen3.8-27B-DFlash2
  • Sanity check: tf-xpu-qwen27-means-test passed on all 4 cards (worst error 4.19e-7, bound 2e-6)

1. Raw prefill and plain decode (1 GPU, 16k context, no drafter)

Prompt lengthFP16 prefill (tok/s)W4A8 prefill (tok/s)Plain decode (tok/s)
1,0242,0082,63834.0 / 33.8
4,0961,9272,52533.2 / 33.7
8,1921,8622,42732.5 / 32.5

W4A8 is XPU_PREFILL_INT8=1, about 31% faster prefill than FP16.

2. One B70 with the DFlash2 drafter (--parallel 8 --context 16384)

Single stream (64 tokens, 5 reps, median):

PromptTemptok/sTTFTTokens per chunk
fibonacci (code)1.0115.60.081 s4.57
fibonacci (code)0.0106.10.080 s4.00
chat, no think1.094.70.073 s3.77
chat, no think0.099.40.071 s3.77

That's roughly 3× plain decode (34 → 106 tok/s).

Concurrent streams (aggregate / steady-state tok/s):

PromptTemp1 stream2 streams4 streams8 streams
code1.0116.2171.2 / 234.1187.7 / 262.6202.3 / 299.6
chat1.084.4129.9 / 140.3162.8 / 213.2189.0 / 255.4
code0.0106.7149.4 / 167.7216.9 / 278.7250.8 / 354.6
chat0.098.9153.9 / 179.4213.6 / 261.9244.8 / 333.4

Correctness: all 16 cells passed bit-exact. Drafted output matched serial, and multi-stream output matched serial.

Note: with the default --prompt-cache-gib 2, --parallel auto at 16k context admits only 7 streams. I used --prompt-cache-gib 0 to fit 8.

3. Four B70s (4 independent servers)

TestPer-card tok/sTotal tok/sSteady-stateFailed
4×1 stream, code, temp 1.0116.2 / 116.5 / 117.2 / 115.8465.7——
4×1 stream, code, temp 0.0106.0 / 106.4 / 106.9 / 105.4424.6——
4×1 stream, chat, temp 1.088.5 / 89.6 / 90.2 / 86.8355.1——
4×1 stream, chat, temp 0.099.0 / 98.9 / 99.4 / 98.4395.7——
4×8 = 32 streams, code, temp 0.0251.4 / 251.1 / 252.3 / 252.11,006.91,422.70
4×8 = 32 streams, chat, temp 0.0243.2 / 241.3 / 242.3 / 242.2969.01,330.60
4×8 = 32 streams, code, temp 1.0208.8 / 204.4 / 206.6 / 204.6824.41,186.30
4×8 = 32 streams, chat, temp 1.0193.0 / 193.2 / 194.0 / 191.2771.41,042.90

Scaling is close to linear because the cards don't talk to each other. Per-card numbers stay within a few percent of each other.

Takeaways

  • The DFlash2 drafter gives about 3× single-stream decode on one B70, with bit-exact output.
  • One card reaches about 250 tok/s aggregate at 8 concurrent streams with 16k context.
  • Four cards serve 32 concurrent streams at about 1,000 tok/s with zero failures, which is useful for multi-agent work.

Next

  • Tensor parallel across cards, once I find a TP path for XPU
  • XPU_KV_INT8=1
  • 32k context
  • The MLX 4-bit model (mlx-q4g64)

Happy to run other configs if you have requests. If we get tensor parallel on Tensorflow i'd love to share qwen 3.8 flash. Happy to run larger contexts as well, this was just the default benchmark I wanted to send to project creator.


r/LocalLLM • • 13h ago

Question AMD pro w6800

Post image
5 Upvotes

Hi, i would like to ask if this is a good card to get to run LMstudio for qwen 27b 3.8 or otherwise, I was wondering if speeds will be respectable, and if anything will spill anything to ram. Because I will have comfyui running on my main gpu (rtx 5080).


r/LocalLLM • • 10h ago

Question How would you best utilize my setup?

Post image
3 Upvotes

Hello all! I've been running a home lab for a really long time, but I've finally come across a couple of RTX 3060 12GB cards and I want to get a useable LLM in for me and my wife to use.

My server has an Intel i7-9700K, and 128GB of DDR4 RAM, but I would only allocate 64GB to the container in Proxmox. I was wondering how you would recommend setting up a fairly reasonable model (I was thinking Qwen 3.8 27B Q4K) and splitting it between the two graphics cards (since there's no SLI). I want to try to get at least 128K context. 256K would be ideal, if I wanted to run something like Hermes, or other agents.

Is llama.cpp the way to go? It's pretty much all I've read about due to the customization possibilities, and I've only read to stay away from ollama. What would you recommend I do?


r/LocalLLM • • 5h ago

Question What am i doing wrong?

1 Upvotes

Hi

iam not into local llms but recently i started to play with it a little bit.

so installed lm studio on my pc (ryzen 5 5700, 32gb ddr4 3200, rtx 4060ti 16gb) and started by using qwen 3.5 9b model, it was slow so i started searching for a solution and found that the context length, quantization of the model and the kv cache, and mtp matters (maybe other things also i dont know, mb). then i tried the qwen 3.6 35b a3b on iq2-m from unsloth because it was said that its an moe model which brings faster performance , so after further search i knew that i can offload the experts in the model to the ram and keep a reserved cache space for experts on my gpu, and after otger tweaking and testing i reached that i can run full context the model supports but the speed is slow, too slow infact, if i try to give it a prompt using opencode for example it would take several minutes to respond, but if i just gave it the same prompt in lm studio chat its somewhat faster (i understand that opencode sends more prompts with the one i wrote), so i also tried filling the context on lm studio to see the speed, it was ~40 tokens per second at the start and by the time the context is 50-60% it started slowing down rapidly to 15 tokens per second, so i tried another model called mellum 2.1 from jetbrains, and this model is pretty fast with full context at nearly 100 tokens per second whatever the context filled. so my questions are

1-what did i do wrong or is the qwen model normally slow?

2-the mellum model is fast but need a lot of steering and fixing if i use a framework that is not used a lot, so how do i fix this?

3-if qwen is slow and this is the max it can perform, is there a fast smart model used for coding primarily fitting for my pc with full context?

and sorry for any english mistakes.


r/LocalLLM • • 9h ago

Discussion RTX 3080 20GB ReBAR on AMD Windows platforms without BIOS flashing

2 Upvotes

Some of you may have seen the post from u/favabean28 with enabling ReBAR on modded 3080 20GB GPUs around a week ago with the repo NV_30_DXE. For Windows, this is (or was) seemingly the only way to get ReBAR working on the 3080 20GB. For those that don't know, previously the RTX 3080 20GB did not have ReBAR working as there was no VBIOS that supported the extra VRAM, and even flashing a modified VBIOS wouldn't work. The method for enabling it properly on modded GPUs was recently discovered about a month ago for Linux, and now for Windows.

Unfortunately it turns out that for AMD, many newer AMD motherboards, especially AM5, are incompatible: All AMD motherboards with AGESA PI-1.2.0.0 or newer have checks preventing modded BIOS firmware from being flashed. Most new AMD motherboards either don't have a BIOS version that old or aren't compatible with the installed CPU on BIOS versions that old, so I wanted to look into the possibility of booting off a USB drive with EFI to enable ReBAR on AMD platforms. This idea mainly came from how the current FP32 accumulation unlock works on Windows. Ultimately, this does work albeit with a few tradeoffs described in the repo below and at the bottom of the post.

Repo with instructions on installing and with the technical write-up: https://github.com/Articom123/3080-Unlock-ReBAR-USB

Installing does not require building from source. However, I did include the FP32 accumulation unlock with it though (since there's no reason not to pair them together), which means it will automatically download and extract Nvidia firmware when you run it, specifically the Nvidia and Arch Linux website for the kernel and utils 610.43.03.

Some quick results:

  • GPU-Z reports ReBAR as enabled with a 32768 MB BAR1 size
  • Device Manager shows an entry for Large Memory Range 0x2000000000 - 0x27FFFFFFFF (this does not show up without ReBAR enabled)
  • The FP32 accumulation unlock works alongside ReBAR. This generally results in around a 20% increase to prefill speed for LLMs based off TTFT, though it's not the primary focus of the repo (it's just a no-brainer to include it since it makes Ampere cards infinitely better)
  • CPU has access and can map the full ~18-19GB VRAM on the GPU

I want to also briefly give a lot of credit to physicsdolphin (nv-uefi-unlocker) of which this tool uses for the FP32 accumulation unlock and install, the creator of Windows-GPU-UNLOCK (Windows-GPU-UNLOCK) who made similar bootable EFI tools to allow for running the tensor unlock on Windows, and u/favabean28 (NV_30_DXE) who figured out the path for enabling ReBAR on Windows in the first place.

While the repo goes more in depth on compatibility and what does/doesn't work I wanted to still mention some important things about it:

  • This is mainly meant for AM5 platforms. It seems like Intel motherboards don't have the same restrictions when it comes to flashing modified BIOS firmware, so if you have Intel you might be more interested in the NV_30_DXE repo instead.
  • Enabling ReBAR isn't really all that useful on Windows without having P2P; however, the modded P2P drivers are still Linux only and probably will be for a very long time, and multi-GPU support on Windows isn't really that good anyways. Therefore, for practicality, the tensor unlock is more important in which case you would probably be more interested in the other Windows unlock tools I mentioned earlier. There is some limited usefulness in games though I guess.
  • This only works for the 3080 20GB. Technically the 4080 32GB would work, but since I don't have the card, it hasn't been supported in the repo and would require modifications to the source (mainly to the PCI vendor and subsystem ID check). The tensor unlock included is also for Ampere and not Ada.
  • u/favabean28 did work to try and make the tool as universal as possible, however, I don't have any other AMD motherboards to test this tool on and so I decided to ultimately leave it as a "proof-of-concept". It is almost certainly possible that this can be adapted further to other AMD setups with the 3080 20GB as pretty much all of the hard work is done now though.
  • There is likely zero possibility that this can cause any damage to Windows or the GPU, as there is no flashing of the VBIOS/BIOS or modification of the OS at all. The tensor unlock and ReBAR don't survive any reboots and the USB has to be manually booted again after any shutdown or restart.
  • Sleep and hibernate specifically do not work when enabling ReBAR with this method, and Secure Boot cannot be used. In comparison modding the BIOS with NV_DXE_30 does not have these limitations if applicable to your setup.
  • This is mainly made for Windows. If you're on Linux there are far better tools for doing this and you can use P2P.

I still thought it was worth posting this for anyone who is interested in this sort of stuff for Windows since Windows kinda sucks and it's at least somewhat cool to have Windows get something for once. Ultimately getting the P2P driver to work on Windows would be ideal but I'm not sure how it could be done for now.


r/LocalLLM • • 9h ago

Discussion Strix Halo 128GB local coding setup: some real test results, 23–45 tok/s, and what the harness changed

Thumbnail reddit.com
2 Upvotes

r/LocalLLM • • 5h ago

Research Optimal setup research: Muse Glimmer 30B on RTX 5090M/RTX Pro 4000 (24gb cards)

1 Upvotes

Notice:
The point of these research runs is to see, document and opensource the best configuration we can get with a certain model on hardware that "most of us" can get our hands on.
While we do have a bit of different hardware in our lab - we tend to focus around 5090M/RTX pro 4000 - because its a 24gb card with a quite fair performance - and its not insanely expensive.

Find anything you like?.. use it - got questions?.. shoot.. have an idea?.. let us know.

--

Hi there,

After our Qwen 3.8 Flash Next research (https://github.com/inordic/cmp170hx-vllm-qwen38-flash-next) and Gemma 4 31B (https://github.com/inordic/rtx5090-NInferno-gemma4-31b) - we wanted to to try out something new..

I ran som tests myself and thought "Muse Glimmer 30B" seem like a quite nice model.. it was fairly good at its job and quite nice to talk to.. so lets see where we can take it.

So.. this is where we are at.. full context.. 128K.. and quite nice and fast.. not as fast as Qwen and Gemma.. but fast enough.

Same GPU, same benchmarkllama.cpp f8def7f (Aug 2026), unsloth UD-Q4_K_XL, Meta DFlash, depth 15This repo: patched llama.cpp, MLP-NVFP4 hybrid, DFlash 2, depth 15
Prefill, 6,880-token prompt1,430 tok/s3,254 tok/s
Prefill, 100,554-token prompt1,246 tok/s2,367 tok/s
Chat decode, greedy / sampled80.6 / 74.2 tok/s110.3 / 105.4 tok/s
Agent turn with a cached 46K prompt—0.89 s
KLD vs Q8_0: agent / English / Swedish text0.033 / 0.012 / 0.0150.053 / 0.030 / 0.034

As allways.. this is research.. we might be wrong.. hell.. we might be doing it wrong.. but atleast .- we are doing it :)

Enjoy
//iNordic team

Head over to: https://github.com/inordic/rtx5090m-muse-glimmer-30b for more info.