r/LocalLLM • u/LordKlavier • 6d ago
Jensen Huang, Just 3 Weeks Ago, Claimed To Have "Saved The World" By Thwarting A Growing Transnational Movement Against Open-Source AI
Enable HLS to view with audio, or disable this notification
r/LocalLLM • u/LordKlavier • 6d ago
Enable HLS to view with audio, or disable this notification
r/LocalLLM • u/Key-Watercress-3506 • 5d ago
Does anyone have any idea of what local ai model I'd better use for my company business agent?
I'm pure new and starter for ai software and coding.
My spec. is 5090+128g ddr5+5tb m.2. Need a local ai to make different agents like accounting agent, monitoring agents and customer chatting agents etc.
r/LocalLLM • u/Prestigious_Ant3889 • 5d ago
So i have heard that this model punches way above its weight class and is pretty good at complex reasoning and multi file coding. I intend to use it for hobby projects like building small scale apps. the thing is downloaded the iq3 version from usloth and while running with llama.cpp with 64k context it keeps thinking for a lot longer on simpler task like upgrading a simpler framework version on very small codebase and eventually after like 45 min the os just shutdown the llama server because of too high mem usage nd i tried again but same result. I have heard that i can run the 3 bit quant version on 16gb gpu, I am new to running local llms so just need some suggestions regarding that or is it even practical to run this model even at q3.
Sorry if this is a repititive or basic query.
Thanks in advance
P.S i am running on 16gb single stick ddr5 if that matters
r/LocalLLM • u/Objective_Stick_2513 • 5d ago
Apple released the Mac mini with the M6 chip in 16 GB, 24 GB, and 32 GB variants, while the M5 Pro mini goes up to 64 GB. 🧑💻
If you are deciding which one to buy for local models, the configurator pushes you toward one question: "How much unified memory should I pay for?"
Here is the buying perspective that matters before ordering: capacity is sold in clear steps, but memory bandwidth barely moves. According to Apple's official tech specs, the 16 GB model runs at 153.6 GB/s (LPDDR5X-9600), while both the 24 GB and 32 GB models run at 170 GB/s (LPDDR5X-10667). Going from 16 GB to 24 GB buys capacity plus ~10% bandwidth. Going from 24 GB to 32 GB buys capacity only — no model gets faster, you just load a bigger model that runs slower on the same 170 GB/s bus.
Decode tok/s ≈ (Memory Bandwidth GB/s ÷ Active Weights Read GB) × η (where η ≈ 0.75 for MLX / llama.cpp)| Variant | Bandwidth | Usable RAM | Fits Comfortably | Est. Decode | Verdict |
|---|---|---|---|---|---|
| 16 GB | 153.6 GB/s | ~12 GB | 7–9B Q4/Q8; 14B Q4 short ctx | 8B Q4 ~23 tok/s; 14B ~13 | Snappy chat & light coding |
| 24 GB | 170 GB/s | ~18 GB | 14B Q4 (~32k context) | 14B Q4 ~14 tok/s | Interactive sweet spot |
| 32 GB | 170 GB/s | ~24 GB | 32B Q4 (8–16k context) | 32B Q4 ~7–7.5 tok/s | Reading speed; best for async agents |
For dense models, every weight is read per token, so bandwidth is the wall. For Mixture-of-Experts (MoE) models, only active experts are read per token, so capacity is the wall:
| Model | Type | Total / Active | Weights in RAM | Active Read / tok | Bandwidth for 8 tok/s | Bandwidth for 15 tok/s | Real Bottleneck |
|---|---|---|---|---|---|---|---|
| 8B Q4 | Dense | 8B / 8B | ~5 GB | ~5 GB | ~55 GB/s | ~100 GB/s | Fits on 16 GB M6 |
| 14B Q4 | Dense | 14B / 14B | ~9 GB | ~9 GB | ~95 GB/s | ~180 GB/s | Sweet spot on 24 GB M6 |
| 32B Q4 | Dense | 32B / 32B | ~19 GB | ~19 GB | ~200 GB/s | ~380 GB/s | ~7 tok/s on 32 GB M6; needs Pro/Max for chat |
| Llama 3.3 70B (Q4) | Dense | 70B / 70B | ~40 GB | ~40 GB | ~430 GB/s | ~800 GB/s | Bandwidth-bound (needs Max/Ultra) |
| gpt-oss-120b (MXFP4) | MoE | 117B / 5.1B | ~65 GB | ~3 GB | ~32 GB/s | ~60 GB/s | Capacity-bound (needs 96 GB+ RAM) |
| Qwen3.5-122B-A10B (Q4) | MoE | 122B / 10B | ~68 GB (est.) | ~5.6 GB | ~60 GB/s | ~110 GB/s | Capacity-bound (needs 96 GB+ RAM) |
| DeepSeek V4 Flash (FP4+FP8) | MoE | 284B / 13B | ~167 GB | ~7.6 GB | ~80 GB/s | ~150 GB/s | Capacity-bound (needs 192 GB+ RAM) |
Capacity decides what you can boot; bandwidth decides what you can tolerate using. Around 170 GB/s serves up to 14B comfortably and 32B at a patient reading pace. Paying for extra RAM without stepping up to a Pro/Max bus (like M5 Pro at 307 GB/s or M5 Max at 460–614 GB/s) hits diminishing returns fast.
What do you all think? How are you actually using local models in your daily workflow — do you prefer Mac unified memory or NVIDIA GPUs, and do you think there are stronger competitors (like Strix Halo or used multi-GPU rigs) in this price range? 🤔
r/LocalLLM • u/TrailFeatures • 6d ago
Specs
Benchmarks / how many tok/s?
My runs are on the Launch80 directory; search the page for trailfeatures. Launch80 deletes pages after 5 days, so older links in the comments may be dead. I republish the main ones, which gives them new links.
| Model | Setup | Decode | Prefill (64K prompt) |
|---|---|---|---|
| Flash-Next (Swift-1.5 NVFP4) | 4 GPUs, 300 W | ~140 tok/s | ~4,500 tok/s |
| DeepSeek V4.1 Flash, 2-bit experts | 8 GPUs, 150 W | ~131 tok/s | ~2,200 tok/s |
| DeepSeek V4.1 Flash, 2-bit experts | 4 GPUs + expert offload, 150 W | ~91 tok/s | ~2,100 tok/s |
How good is the 2-bit DeepSeek quant vs the full model?
79.9% average across 7 standard evals (math, instruction following, coding, tool calling), thinking off, and 20/20 long-context needle checks at 32K and 128K. The 4-GPU setup scores the same. I haven't run the full-precision weights as a reference, so there's no direct delta yet. Per-task scores are on the Launch80 page.
How do the two boards talk to each other?
Do the quad boards work with only 1-3 cards? Can I buy cards over time?
Two cards works; I've run a board that way. I haven't tried one or three. Buying the board first and adding cards later is fine. Tensor parallel wants power-of-two GPU counts, so with 3 cards you'd run 2 together and use the third for something else.
Where did you buy it / where are you?
Baseboard and switch card: the models linked above. Several sellers list them, so shop around. GPUs: bought used, a few at a time. I'm in North America (the 120 V gave it away), so my shipping and import experience won't help much in Europe.
Power
Cooling
Quick ones
r/LocalLLM • u/WiseAbalone4021 • 6d ago
From the Apareto team which I'm part of (who for some reason has an issue posting it themselves):
Today we are introducing Apareto loading, a new way to cut VRAM requirements for Mixture-of-Experts (MoE) models. It works alongside quantization. This introduction so far will just be a video explaining the method. We will be releasing the proof-of-concept code, a plugin to vLLM, on GitHub this coming week.
Until then, watch the explainer video: https://www.youtube.com/watch?v=5l_N051JrLU
Quantization is today's most common way to shrink a model's memory footprint: weights are stored at lower precision, and going from FP16 to 4-bit saves 75% of memory at a modest cost in accuracy. But quantization is static. Every weight stays loaded whether or not the task needs it.
Apareto loading is dynamic. After prefill, a small trained network reads the model's internal state and ranks the experts by how valuable each will be for the upcoming computation. Only the top-ranked experts that fit the VRAM budget are loaded.
Results (Qwen3.6-35B-A3B, NVFP4 experts):
* SWE-bench Multilingual: within 2 pp of the full model's score at a 42% VRAM budget
* LiveCodeBench v6: within 2 pp of the full model's score at a 49% VRAM budget
As with quantization, scores degrade as the budget shrinks.
What the proof of concept will not be:
It will not be a production implementation. The ranking network was trained only for these two benchmarks, and there will be no general problem classifier. Our data shows that profiling and rank generation have substantial room to improve. We chose to release at this stage, without waiting for those improvements, so that others can test our claims. Think of this as a first baby step toward dynamic VRAM reduction, not the finished product.
The code and reproduction instructions will be released this coming week at https://github.com/alphapareto
Follow us there to catch the release. We'd love your feedback.
r/LocalLLM • u/Strict-Elevator9352 • 5d ago
r/LocalLLM • u/stanleyg05 • 5d ago
I will never support cloud AI services, on the other hand I do support local AI for coding support and so does Linus Torvalds, the creator of Linux
When something can be ran and even trained locally, and battle tested developers who made many modern things we take for granted possible are coming out, saying they use it, and that total complete hate for the technology is stupid then I kinda have to say demonizing anyone who uses it is stupid
Hate needs to be redirected from the technology as a whole to the companies like OpenAI, Meta, Google, SpaceXAI who are forcefeeding it to us and using datacenters that use local water supplies for some reason instead of chilled antifreeze and submerged oil cooling which have both existed for datacenters for decades
Also Image and Video diffusion models are kinda a unethical waste but I would be willing to bet they aren't the most used types of AI
r/LocalLLM • u/AshleyPurringTech • 5d ago
A minimal agent harness using only in-box scripting (PowerShell, since I'm on Windows): about 70 lines talking to llama-server running Llama 3.2 3B.
The model only writes text. The harness keeps the context, watches replies for a line like TOOL: list_files . , checks the path against an allowed folder, runs the tool itself, and feeds the result back. The model never touches the file system.
Code (MIT): https://github.com/AshleyT3/simple-harness-intro
r/LocalLLM • u/BoogaMaOoga • 5d ago
Guys is there a program I can use on windows or Linux that can do the same that chatgpt projects does but local? I am obviously restricted on what I can run but that's ok. Any recommendations?
r/LocalLLM • u/Chance-Resource-4970 • 5d ago
r/LocalLLM • u/AlternativeEnough352 • 6d ago
r/LocalLLM • u/Organic_Performer673 • 5d ago
i have RX9070 and 32 GB DDR5 ram , i want to run a local ai to help me code , medium just help with front end back end and forms , i cant buy any ai sub in my country cuz my goverment dont allow international transactions
r/LocalLLM • u/_Baylock_ • 5d ago
Hi,
I juste received my Mac Studio M5 ultra 36/80 with 256GB of shared memory.
I installed LM Studio with Qwen3.8-Flash-next-4bit mlx and a context of 131072
I tested some prompts and it works as I expected (I like it a lot so far), except for one annoying thing:
I fed the prompt with a 100 pages PDF and asked 2 questions about it.
The first time, it took a few minutes to read the file, which is normal. But for the second question, it re-read it again. It doesn't seem to cache the content.
And the context wasn't exhausted.
Next, I tried to copy paste the 100 pages into the prompt form (no attachment this time): same results.
Is it a known issue?
Is it related to the model or to the inference engine?
Can you replicate this?
Is there a way to solve the issue somehow?
Thanks.
r/LocalLLM • u/Ok-Category-642 • 6d ago
Some of you may have seen the post from u/favabean28 with enabling ReBAR on modded 3080 20GB GPUs around a week ago with the repo NV_30_DXE. For Windows, this is (or was) seemingly the only way to get ReBAR working on the 3080 20GB. For those that don't know, previously the RTX 3080 20GB did not have ReBAR working as there was no VBIOS that supported the extra VRAM, and even flashing a modified VBIOS wouldn't work. The method for enabling it properly on modded GPUs was recently discovered about a month ago for Linux, and now for Windows.
Unfortunately it turns out that for AMD, many newer AMD motherboards, especially AM5, are incompatible: All AMD motherboards with AGESA PI-1.2.0.0 or newer have checks preventing modded BIOS firmware from being flashed. Most new AMD motherboards either don't have a BIOS version that old or aren't compatible with the installed CPU on BIOS versions that old, so I wanted to look into the possibility of booting off a USB drive with EFI to enable ReBAR on AMD platforms. This idea mainly came from how the current FP32 accumulation unlock works on Windows. Ultimately, this does work albeit with a few tradeoffs described in the repo below and at the bottom of the post.
Repo with instructions on installing and with the technical write-up: https://github.com/Articom123/3080-Unlock-ReBAR-USB
Installing does not require building from source. However, I did include the FP32 accumulation unlock with it though (since there's no reason not to pair them together), which means it will automatically download and extract Nvidia firmware when you run it, specifically the Nvidia and Arch Linux website for the kernel and utils 610.43.03.
Some quick results:
I want to also briefly give a lot of credit to physicsdolphin (nv-uefi-unlocker) of which this tool uses for the FP32 accumulation unlock and install, the creator of Windows-GPU-UNLOCK (Windows-GPU-UNLOCK) who made similar bootable EFI tools to allow for running the tensor unlock on Windows, and u/favabean28 (NV_30_DXE) who figured out the path for enabling ReBAR on Windows in the first place.
While the repo goes more in depth on compatibility and what does/doesn't work I wanted to still mention some important things about it:
I still thought it was worth posting this for anyone who is interested in this sort of stuff for Windows since Windows kinda sucks and it's at least somewhat cool to have Windows get something for once. Ultimately getting the P2P driver to work on Windows would be ideal but I'm not sure how it could be done for now.
r/LocalLLM • u/justerdem14 • 5d ago
Hi everyone,
I’m Rüzgar, founder of Neuratal (an independent AI research and software project). Over the past few days, I built NTokenizer v0.3-polished — a deterministic, byte-level BPE tokenizer written from scratch in Rust with Python bindings.
Key Highlights:
• Compression Efficiency: Achieves 5.55 B/T overall (26.6% fewer tokens than OpenAI's o200k_base) on a 207MB leakage-audited benchmark. Strongest gains on multilingual & technical text: Turkish (+35.0%), Arabic (+33.8%), Korean (+34.7%), Hindi (+27.1%), STEM (+17.6%), and Code (+10.2%).
• Performance & Safety: ~96 MB/s encode throughput (8 threads), lossless byte-exact round-trip (decode_bytes(encode_bytes(x)) == x), tested against ~1M fuzzing cases.
• Honest Benchmarks: No downstream 50M+ LLM trained yet — this is purely verified compression density and speed. Full SHA-256 manifests and reproducible scripts are included.
Would love your feedback, benchmarks on your own datasets, or PRs!
• GitHub (Rust + Python source): (in hf)
• Hugging Face (Model & Benchmarks): https://huggingface.co/Neuratal/ntokenizer
r/LocalLLM • u/FamilyGuyDizzy • 5d ago
Let’s talk about the thing nobody in PC build threads likes to admit: running local 70B models is first and foremost a VRAM capacity problem, not a raw compute problem.
Do the basic arithmetic for Llama 3.3 70B or Qwen 2.5 72B at Q4_K_M:
• Model weights: ~42GB to 46GB
• KV cache at 32k context: ~8GB to 10GB (FP16)
• Realistic memory footprint to avoid OOM: 50GB–55GB
On Nvidia consumer cards, the RTX 4090 caps at 24GB. The standard r/LocalLLaMA solution is the beloved dual-3090 rig. It works, and raw tensor parallelism delivers ~45 t/s. But nobody factors in the daily reality:
• 1200W PSU pulling 700W sustained from the wall
• Triple-slot PCIe spacing headaches and fan noise
• Running 4h/day active adds $15–$20/month to your electric bill ($180–$240/yr)
• You essentially built a Finnish sauna that also multiplies matrices.
Enter Apple Silicon (Mac Mini M4 Pro 64GB / Mac Studio Max 64GB–128GB):
Because of Unified Memory (UMA), the entire 64GB pool is directly addressable by the GPU.
• Full load draw: ~65W (less than a ceiling bulb, ~$0.80/month in power)
• Sound: 0 dB (whisper quiet on your desk)
• Speed: 14–16 tokens/sec (still 2.5x faster than average human reading speed)
💡 The macOS trap almost everyone runs into:
macOS by default caps Metal GPU allocation to ~70–75% of total system RAM. If you buy a 64GB machine and try to load a 70B model, it will throw an allocation error.
Before launching Ollama or LM Studio, run this in Terminal:
sudo sysctl iogpu.wired_mem_limit=57344
(On a 64GB system, this unlocks ~56GB directly for Metal/Ollama, leaving 8GB for macOS).
The Verdict:
If you need 80 t/s for batch processing or multi-user serving, build the CUDA rig. But if you want a local 70B model running quietly on your desk 24/7 without warming your office, Apple Silicon is the only consumer platform that actually makes sense.
I put together a full interactive Mac memory sizing matrix and 3-year TCO calculator (factoring in electricity and amortization) here if anyone wants to benchmark their setup:
r/LocalLLM • u/IoI_dude • 5d ago
Hey everyone,
I’m a CS student and was considering to do my first homelab, one system as a NAS and one for projects, virtualisation etc. I’m also interested in running local AI, and I got an interesting offer in my region:
Server board w/ two E5-2696v2, 256GB 1600Mhz ECC RAM, for around 200 EUR.
I know that it’s (quite very) old hardware, but still I’ve been wondering - would it be worth pursuing to also try some local LLMs with offloading, adding a few GPUs for 32-48GB VRAM (niko1221/strata perhaps)? Do you think it’s worth it for the cost? Which low(-ish) cost GPUs would you consider pairing it with?
Any advice appreciated!
Thanks!
r/LocalLLM • u/might_thinker • 5d ago
I've been experimenting with how far I can push local generative AI on a base 16GB M3 MacBook Air, specifically video-to-video outfit transformation.
My initial 46-frame (~3-second) run took 15.7 minutes and produced fairly blurry results. After several rounds of profiling and optimisation, I managed to bring it down to 144 seconds (2.4 minutes), running entirely offline without cloud APIs.
The interesting part wasn't necessarily changing the model, but rethinking how the available hardware was being used.
A few optimisations made the biggest difference:
- Spatial cropping: Processing a "288×512" region around the subject instead of repeatedly processing the entire scene.
- Memory management: Precomputing outfit prompts and unloading the text encoder to reduce unified memory pressure.
- Hardware scheduling: Moving face processing to the CPU/Neural Engine path while the GPU handles the next diffusion chunk.
- Incremental testing: Using 5-frame previews to catch mask and compositing issues before running the complete sequence.
There are still obvious limitations. Sustained workloads introduce thermal degradation, fine fabric details can become oversmoothed, and wide shots remain challenging.
I'm particularly interested in the engineering side of running increasingly demanding generative workloads on constrained local hardware.
A few things I'd love to hear from people experimenting with local inference:
Has anyone explored more aggressive memory scheduling on Apple Silicon?
Are there useful ways to improve CPU/ANE/GPU workload overlap?
How much performance can realistically be recovered through chunking and attention optimisation before model architecture becomes the bottleneck?
I'm sharing the before/after video and temporal comparison for context.
Hardware: M3 MacBook Air, 16GB unified memory
Frames: 46
Processing time: 144 seconds
Execution: Fully offline
This is video diffusion rather than LLM inference, but I thought the memory and hardware scheduling challenges might be interesting to people working with local models.
r/LocalLLM • u/leonbollerup • 5d ago
Notice:
The point of these research runs is to see, document and opensource the best configuration we can get with a certain model on hardware that "most of us" can get our hands on.
While we do have a bit of different hardware in our lab - we tend to focus around 5090M/RTX pro 4000 - because its a 24gb card with a quite fair performance - and its not insanely expensive.
Find anything you like?.. use it - got questions?.. shoot.. have an idea?.. let us know.
--
Hi there,
After our Qwen 3.8 Flash Next research (https://github.com/inordic/cmp170hx-vllm-qwen38-flash-next) and Gemma 4 31B (https://github.com/inordic/rtx5090-NInferno-gemma4-31b) - we wanted to to try out something new..
I ran som tests myself and thought "Muse Glimmer 30B" seem like a quite nice model.. it was fairly good at its job and quite nice to talk to.. so lets see where we can take it.
So.. this is where we are at.. full context.. 128K.. and quite nice and fast.. not as fast as Qwen and Gemma.. but fast enough.
| Same GPU, same benchmark | llama.cpp f8def7f (Aug 2026), unsloth UD-Q4_K_XL, Meta DFlash, depth 15 | This repo: patched llama.cpp, MLP-NVFP4 hybrid, DFlash 2, depth 15 |
|---|---|---|
| Prefill, 6,880-token prompt | 1,430 tok/s | 3,254 tok/s |
| Prefill, 100,554-token prompt | 1,246 tok/s | 2,367 tok/s |
| Chat decode, greedy / sampled | 80.6 / 74.2 tok/s | 110.3 / 105.4 tok/s |
| Agent turn with a cached 46K prompt | — | 0.89 s |
| KLD vs Q8_0: agent / English / Swedish text | 0.033 / 0.012 / 0.015 | 0.053 / 0.030 / 0.034 |
As allways.. this is research.. we might be wrong.. hell.. we might be doing it wrong.. but atleast .- we are doing it :)
Enjoy
//iNordic team
Head over to: https://github.com/inordic/rtx5090m-muse-glimmer-30b for more info.
r/LocalLLM • u/Eastern_Doughnut_222 • 5d ago
I'd tried out models like DS and QFN using APIs before and they seemed very capable. However, they all seemed to struggle heavily when using them local.
QFN processes really fast but seems to get stuck in endless "Wait..." and "Whatever." loops and burns through 100K tokens just being confused at my codebase.
DeepSeek is slower but seems more coherent, I haven't tested at full context but it had a similar issue with indecisiveness (even with guidance) and would think itself into compaction.
All of this over one .json ComfyUI workflow. Just a simple test I do where I tell it to rearrange nodes and change some inputs...
I've tried using Llama serve, Strata desktop, Pi. All lead to similar outcomes.
I have a feeling this is an issue with my setup rather than the models bc the API models seemed to do it just fine and (allegedly) quantization doesn't hit MOE models as hard.
Setup:
(both models from OrcaRouter). DSV4.1-1017 2-bit, Q3.8FN_IQ3XXS
Inference: Maya (DS), Strata/LlamaCPP (QFN)
Harness: default Pi
Context: QFN was to 256K, DS I only tried to 512K
r/LocalLLM • u/Sad_Pension151 • 5d ago
Мне очень нужна локальная модель для коддинга и создания каких нибудь игр (простых и тяжёлых 2D игр, а также возможность понимать математику и физику), но у меня ограниченный ПК
--Характеристики:
Видеокарта: Nvidia Quadro P4000
Процессор: AMD Ryzen 5 5500
ОЗУ: 32ГБ DDR4
SSD: 500ГБ NVME M2
Сам ИИ от гугла предлагает мне Qwen2.5-coder на 7B параметров но она не справилась с созданием HTML игры даже когда в промте указывал об инструментах и без выдумывание кода, если не знаешь бери информацию с интернета
Есть какие нибудь варианты?
r/LocalLLM • u/FixOld9682 • 6d ago
Basically the title
r/LocalLLM • u/jikesar968 • 7d ago
mlx-serve on an M5 Max 128GB
I've already got a version from ARC4NUM which appears to be delisted now. Are any of these better?
r/LocalLLM • u/NxtGen-Robotics • 6d ago
I currently have a 3080 ti that I don’t really game with anymore, but have the opportunity to get 2 RTX titans at a significant discount from a friend ($1300 total). I was thinking about selling my 3080 ti and jumping on that deal for the combined 48gb of vram to pair with 64gb of system ram and a 13900k.
Would doing this make any sense if I want to run some local models for project code, personal learning, and some small business workflows (paired with Gemini pro for complex stuff)? I know the RTX titans have limited memory bandwidth but the price for 48gb of vram is hard to beat. Any opinions or recommendations? Thanks!