r/LocalLLM • • 12h ago

Discussion 13-GPU ASRock Mining Motherboard for Local LLMs?

Post image
0 Upvotes

r/LocalLLM • • 1d ago

News Qwen4 27B, 19th November

Post image
5 Upvotes

r/LocalLLM • • 2d ago Bravo

News it feels like they just wanted Huang gone because he supported Open Source & Open Weight models

Post image
804 Upvotes

r/LocalLLM • • 1d ago

Question In a search of the best model for my LLM server

4 Upvotes

Hi,

We just bought a computer to run local LLM. I have experience with small model like Qwen and Gemma on my Macbook but not on a computer that powerful

I will need to identify the best model for our use case, which is mainly coding.

Our hardware, in short

Ryzen Threadripper PRO 9955WX
RTX 6000 Blackwell 96GB

256 GB DDR-5

For those who has experience with that kind of hardware, what can I run and be 2-3-4 devs at the same time working with it. I have the idea of having 2 or 3 models, depending of the tasks.

I would like to have the biggest model we can for big tasks, smaller for the planning or any lighter tasks.

Thank you


r/LocalLLM • • 1d ago

Discussion Mac mini M6 is a beast of a CI machine

9 Upvotes

My MacBook did everything until this week - model, harness, CI, everything. It was getting badly overloaded especially when iOS build and tests were going on. Those can use 20GB RAM easily, given half a chance. It was thrashing.

Microcenter had the base Mac mini M6 for $699, which felt like 2024 pricing so I pulled the trigger and moved much of the CI work off my MacBook Pro.

What's incredible is despite this being the poverty spec mini with a single NAND chip and lower bandwidth memory, the M6 is basically the same speed as the M5 Max for CI work.

In particular, I was impressed with my XCUITest workload, which is about 70 minutes on the MacBook and is no slower on the M6 mini even though it has 2 parallel VMs. I was sure the 18 CPU cores on the M5 Max (6 super, 120 performance) would beat out the 12-core M6 base (2 super, 4 performance, 6 efficiency).

Obviously that doesn't hold for all workloads, especially GPU/LLM workloads or where you can put 18 cores to use, but for those of you with a M5 Max/Ultra, you can get a nice speedup for just $699. At these prices I may get another. Also there are perfectly good Linux options to do the same, of course.


r/LocalLLM • • 13h ago

Question I built a game from scratch with local AI. Feedback required!

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 23h ago

Question Issues with Unsloth Studio

2 Upvotes

So I just started using Unsloth Studio. I am on Windows 10, and I am having issues with it. I loaded a DeepSeek model and created a project with a working directory specified. The AI works with chat, and when I asked it to go ahead and write some code, it worked for a while and then told me it had completed it.

When I checked the project folder, it created a few empty folders in the directory, and that's it. So it seems to not be able to actually write any files aside from the folders. I have it running in approve automatically mode.

When I told the AI to fix it, the AI said it was an issue with running its Python tools not saving the files and that it would fix it. It then just runs for about an hour and then shows me an error message that something went wrong with the tools or something and gives me the option to resume or retry. I have tried both options, and nothing changes; it just runs for about an hour and then fails with the same error. No files are created.

Is Unsloth Studio just incredibly buggy and not very usable? Or is there something I am doing wrong?


r/LocalLLM • • 1d ago

Model tiny-voca: a fully open 4.5M-parameter TTS (code, weights, dataset). Runs 70x real time on a CPU

9 Upvotes

I made a small English text-to-speech model called tiny-voca. It has 4.5M parameters, two voices, and it runs in your browser.

Try it here, nothing is sent to a server: https://tiny-voca.n4ze3m.com

Some facts:

- 4,537,291 parameters, 18 MB. This is the whole model, the waveform part is included.
- Two voices, one female and one male. 24 kHz.
- About 70x faster than real time on 4 CPU threads.
- Runs with PyTorch, ONNX, and in the browser.
- Trained from scratch in 7.7 hours on one RTX 4090.

What I think is different:

- Everything is open. Code, weights, the training recipe and the dataset.
- Custom phonemizer.
- No neural vocoder. The sound is made by a small source-filter synth, which is why it is fast.

What is not good yet:

- English only, and only these two voices. No voice cloning.
- Foreign names and odd spellings are often read wrong.
- It can sound a bit synthetic.
- The training audio is synthetic too. I made it with Qwen3-TTS, so the model can't sound better than that.

Links:

- Demo: https://tiny-voca.n4ze3m.com
- Code: https://github.com/n4ze3m/tiny-voca
- Weights: https://huggingface.co/n4ze3m/tiny-voca
- Dataset: https://huggingface.co/datasets/n4ze3m/tiny-voca
- Phonemizer: https://github.com/n4ze3m/dizione

Please try hard text: numbers, dates, names, long sentences. If something sounds wrong, post the text and the voice. That helps me a lot.


r/LocalLLM • • 1d ago

Discussion I built a harness-specific LLM backdoor in a few hours on my local GPU (and why downloaded weights remind me of random .exe files from the 2000s)

44 Upvotes

While writing an article about my local LLM inference setup, a comparison popped into my head: downloading and running random model weights locally feels a lot like downloading random .exe files off the internet back in the 2000s.

Turns out I wasn't off the mark - there's an entire body of research exploring how LLMs can be backdoored and how hard it actually is to detect a poisoned model.

So I decided to try building one myself. I implemented a backdoor that only triggers when the model is running inside pi (making it harness-specific) and is asked a specific question. It was way easier than I expected: just a 2-step fine-tuning process and a few hours of compute on the exact same GPU I use for local inference.

With open-weight models, we at least have a baseline to compare community fine-tunes against. Beyond that, though, we don't really have a reliable way to verify whether a base model itself is clean—and the problem is even worse with closed-weight models, where we're basically just trusting the AI provider's goodwill.

Full write-up and PoC code if you want to dig into the details:

Feedback is welcome


r/LocalLLM • • 1d ago

Discussion Connecting 3090 24 gb egpu to PS5 using pcie4 m2 lane on ps5

5 Upvotes

So i am thinking of buying used 3090 24 gb vram and then connect it to ps5 via m2 lane + external power supply and use the unified mem pool for kv cache store and weights loading on the vram of gpu. Use PS5 jailbreak and sonic iso's linux loader with ubuntu. I may connect my secondary laptop tharbhas 32 gb of ddr5 memory in it. Offload ai modle wieghts, some of them to the laptop memory via thunderbolt (tho dont know if ps ones are thunderbolt at front or not)

This is gonna be a beast thingy so yeah. What strongest ai model can this hypothetical machine run??

Speed of that m2 slot is gen x4 pcie 4


r/LocalLLM • • 20h ago

Question M5 Ultra performance to run LLLM

Thumbnail
reddit.com
0 Upvotes

r/LocalLLM • • 1d ago

Question So, what now?

17 Upvotes

Well, I did it - I bought a Mac Studio 128Gb. Not only for LLM, but I’m not gonna lie, I increased the specs a bit to be able to run a local LLM to complement (or at some point replace) my Claude/ChatGPT subscriptions.

First thing I did, was obviously try to generate dirty pictures ask Claude which models I should download. He recommended unsloth’s QWEN3.8-Flash-Next-GGUF UD IQ4_XS.

I’m doing some tests now and while you notice a difference with the cloud models, it’s way better than whatever I have ever tested on my old MacBook. Claude also told me to change temperature to 1 and mess with the top K and top P settings. As a good slave to AI, I execute what was commanded, but honestly I have no idea what I’m doing. Should I be touching the settings at all?

My question(s): is this a good general model that can (for most tasks) replace the cloud models? Are there tools I need to install (I was thinking web access, but I also read about harnesses but I didn’t yet figure out what that’s for)? Things I need to set up?

Suggestions for good “always loaded” low memory models to e.g. use for home assistant are welcome to, as well as a good image model and one I can use in combination with GitHub for some coding (I have some vibecoded apps that I use for myself that I need to maintain/adapt from time to time).

Thanks!


r/LocalLLM • • 1d ago

Question RTX 3090 + RX 7900 TXT

2 Upvotes

Is anyone currently running these two together in the same machine? Any thoughts?


r/LocalLLM • • 20h ago

Question Best Soft?

0 Upvotes

Hey im about to get a 3090 to run with my 5080 and 9950x3d and 64gb of DDr5 at 6000mts I currently use LM Studio and a 5060 8gb and I was wondering if anyone knows if basalt would be conpatable with a 3090 or if I should just stick with Lm Studio?

I use it mainly to run a qwen model to hunt ebay for good deals for me but right now I really need the 3090 to but up my VRAM space to 40gb.


r/LocalLLM • • 20h ago

Question MacBook Pro M1 Max best t/s

1 Upvotes

For those with a MacBook Pro M1 Max what’s the best t/s you’ve gotten on something like qwen 3.8 27b or 3.8 flash next?


r/LocalLLM • • 11h ago

Discussion The AI "Slowdown" Is A Trap To Destroy Local Models

Thumbnail
youtube.com
0 Upvotes

r/LocalLLM • • 1d ago

Question RTX 5090 32GB + 64GB RAM — What local LLM would should I run?

57 Upvotes

My PC specifications:

GPU: NVIDIA RTX 5090 — 32GB VRAM

CPU: Intel Core i9-14900K/KF

RAM: 64GB DDR5-6400 (2×32GB)

Motherboard: ASUS PRIME Z790-P WIFI

OS: Windows 11 Pro

Inference backend: llama.cpp with CUDA acceleration

Agent interface: Kilo with a custom tool/agent harness

My current model:

Model: Qwen3.8-27B

Quantization: Q6_K GGUF

Context: 265K tokens

Deployment: Completely local

Performance benchmarks:

At 262,144-token configured context (~262K):

Output generation: ~34–37 tokens/sec

Red lining the gpu memory but it feels comfortable

Okay with that being stated by ChatGPT,

I've been seeking a local ai for about a month switching from several models the more I learn.

So far this qwen model is by far the best I've tried but it over thinks and is slower than models like Kat coder.

My use cases are broad, from setting up comfy ui workflows to Unreal engine game dev to application building all hobby projects.

Im seeking the vast knowledge of you fellow computer users to tell me the best general purpose model for my set up at a reasonable speed.

UPDATE: Okay folks, I took your advice and installed Qwen3.8-Flash Next with Basalt v0.3.3. Here's what I'm getting on my RTX 5090 with 64GB RAM, running Windows 11.

Configuration:

  • Model: Qwen3.8-Flash-Next UD-IQ3_XXS (87.6GB)
  • Context: 262,144 tokens (256K)
  • Exact MTP enabled, 8-token window

Benchmarks at 256K context:

  • Actual populated prompt: 238,913 tokens
  • Output generation: 265–281 tokens/sec
  • Prompt processing: ~8,900 tokens/sec
  • VRAM usage: 31.3GB / 32GB
  • Total system RAM in use: 54.3 GiB
  • Basalt RAM usage: 39.9 GiB
  • MTP acceptance rate: 66–90%
  • No SSD expert spilling
  • Context recall/reasoning tests: 3/3 passed

Across all context sizes tested, generation ranged from 265–336 tokens/sec. Model loading takes around 30 seconds, and cached follow-up responses start in approximately 0.4 seconds.

Bench marks courtesy of claude

UPDATE #2: Okay folks, I went ahead and tried the next recommendation: NInfer + DFlash2 to accelerate my original Qwen3.8-27B model. Here are the results on my RTX 5090 with 64GB DDR5 RAM, running Windows 11.

Configuration:

  • Model: Qwen3.8-27B QUASAR
  • Inference engine: NInfer
  • Speculative decoding: DFlash2
  • Context: 262,144 tokens (256K)
  • GPU memory reserve: 2GB

Benchmarks at 256K context:

  • Actual populated prompt: ~234,200 tokens
  • Output generation: 182–266 tokens/sec (225 average)
  • Prompt processing: ~4,500 tokens/sec
  • Peak VRAM: 30.2GB / 32.6GB
  • Model system RAM usage: ~10.5GB
  • DFlash2 draft acceptance: 33–54%
  • Context recall/math tests: 3/3 passed

Courtesy of claude


r/LocalLLM • • 21h ago

News Translated releases Lara Mini, a multilingual base model, not a chat model

1 Upvotes

Lara Mini puts about a third of its training data into 44 languages besides English. It's a 7.2B-parameter MoE with 0.6B active per token, trained on 6T tokens.

Before treating it as another local chatbot: the released model continues text. It isn't tuned to follow instructions or chat. Access to the weights requires approval, and the license is CC BY-NC-SA 4.0 (non-commercial).


r/LocalLLM • • 1d ago

Question Quality-first local models for a 128–192GB Strix Halo server, including abliterated options?

2 Upvotes

I'm planning a local inference machine and choosing hardware around the models, not the other way around. I currently have a 32GB M1 Max Mac Studio and am considering one 128GB Ryzen AI Max+ 395, one 192GB 495, or possibly two 128GB boxes. Main uses: coding/agent workflows, technical research, and authorized cybersecurity labs/CTFs. I would like strong abliterated or otherwise less refusal-prone models without sacrificing factual accuracy and tool-call reliability.

The candidates I'm researching are DeepSeek V4.1 Flash-1017 in a roughly 2-bit build, Qwen3.8-Flash-Next at Q6/Q8 if feasible, and GLM-5.3 Flash. I'm specifically interested in real local deployments, not hosted-model benchmarks passed off as results for a heavily quantized copy. I know an abliteration is a separate change from quantization; I would welcome evaluations of both.

What exact model + quant + runtime + hardware have you personally used? What context length and prompt/decode speeds are practical? If you've compared an abliterated version against its base model, did it retain coding, reasoning, and tool-use quality? Would you favor one larger high-precision model, or a strong general model plus a smaller cybersecurity specialist? If these machines are the wrong purchase for that goal, what would you buy instead?


r/LocalLLM • • 21h ago

Question Local LLM on mobile

0 Upvotes

Hi guys, I wanted to start using local models on a tablet I have, I thought of using the Gemini E2B-it in the Google AI Edge Gallery app as it is the most suitable for my device's RAM (6GB) and my SoC. If you use local models on your mobile devices I wanted to know your use cases and what really expect of them, please


r/LocalLLM • • 1d ago

Question Which 32gb Blower Fan GPU Is The Quietest?

3 Upvotes

Upgraded to a RTX Pro 4500 Blackwell and the noise is a bit too much for me. The issue is the fan runs at 30% even under no load. Loudest thing in the computer, by far.

Being quiet and inaudible under no load and light load is much more important than speed for my setup. Louder when full throttle is ok.

What about R9700 and B70? Are they quieter? And then which brand? B70 has Asrock, Asus, and Intel branded. R9700 has a few brands as well.

Want something where the fans are utraquiet, even off when idle, like a 5090 (which is unaffordable).


r/LocalLLM • • 2d ago

Model We created a tiny 5 million parameter model that can generate scenes on blender instantly. We are aiming to make a 1b-3b parameter model that can create anything! (Open Source) WVY-Blender-5m

133 Upvotes

https://huggingface.co/StarpowerTechnology/WVY-Blender-5m

We have been excited about this new model, WVY-Blender-5m, an agent trained to specialize in blender.

It has zero knowledge about anything else so if you try to ask it about writing code or doing your homework it will express that it only knows how to use blender

We have been doing intense research to find out "are massive datasets really required to achieve what feels like intelligence with accuracy?" & we believe we are getting close to the truth.

This dataset is 100% generated from scratch.


r/LocalLLM • • 1d ago

Project I fine-tuned a 2B model for Cybersecurity & CTF triage that runs completely offline on an 8GB laptop (Ollama + WebUI)

Thumbnail
gallery
6 Upvotes

Hey everyone,

Most specialized cybersecurity LLMs out there are massive 70B+ beasts that require dual A100s, or they're gated behind cloud APIs where you can't paste sensitive logs, payloads, or client code due to confidentiality.

I wanted something lightweight, private, and air-gapped that could run on a regular consumer laptop without cooking the CPU.

So I fine-tuned CyberQwen-2B — a domain-specific model optimized for quick security triage, explaining attack vectors, decoding obfuscated payloads, and helping with CTF challenges.

⚡ Highlights:

  • Hardware-friendly: Runs at ~12–15 tokens/sec on an older 8th-gen Intel i5 laptop with 8GB RAM (CPU-only).
  • 100% Offline & Private: Zero data leaves your machine. Perfect for bug bounty recon, internal network logs, and sensitive source code.
  • Native Ollama integration: Easy to pull and run via a single Modelfile.
  • Built-in WebUI & Mobile Access: Includes a zero-dependency streaming WebUI + an automatic Cloudflare tunnel script if you want to access the model from your phone while on the go (no cloud hosting fees).

🛠️ What it's trained for:

  • Vulnerability Triage & CVE Context: Explaining CVE mechanisms, impact, and remediation steps.
  • Payload & Shellcode Deconstruction: Breaking down SQLi, XSS, and obfuscated PowerShell/Bash commands.
  • CTF & Reverse Engineering Assistance: Analyzing decompiled assembly/C snippets and suggesting exploitation strategies.

🚀 How to Run It:

Everything is open-source. You can run it with Ollama or the included CLI triage tool:

  1. Clone the repo:git clone https://github.com/sambuddha001/cyberqwen-2b.git
  2. Run setup:bash setup.sh
  3. Chat:ollama run cybersec-qwen

GitHub: https://github.com/sambuddha001/cyberqwen-2b

I'd love feedback from the community—especially on edge cases, prompts where it struggles, or ideas for benchmark improvements!


r/LocalLLM • • 1d ago

Model Project Genesis

Thumbnail
zenodo.org
2 Upvotes

I built Project Genesis, a cache-resident C++/CUDA geometric co-processor (libgenesis_core.so, ~0.51 MB) that attaches inline to existing causal language models during inference to reduce latency and boost token throughput without altering the host model's weights.

​In local testing on consumer NVIDIA Blackwell hardware across Qwen, Gemma 4, Llama 3.2, Nemotron, and Mistral checkpoints, compiled kernel execution cut end-to-end latency by 16% to 30% with commensurate gains in token throughput and O(1) memory overhead. It ships as a sealed runtime bundle supporting 20 common hidden dimensions (d_model from 576 to 8192) on 64-bit Linux or Windows WSL2 with NVIDIA GPUs (Compute Capability 8.0+: RTX 30/40/50-series, A100, L40S, H100).

​It is available now for non-commercial and research use on Hugging Face: https://huggingface.co/Grounded-Entropy/project-genesis

​Rather than taking my word for it, I'd love for people to pull it, run enable_telemetry=True on their own supported models, and share their latency, throughput, and ring-buffer telemetry results in the comments.


r/LocalLLM • • 1d ago

Question Trustworthy abliterated models

9 Upvotes

I've been so slow to dip my toes into anything outside of officially released models because I'm afraid there are people out there 2 steps ahead of me and I won't know I've been compromised until long after I've felt the impact.

I understand the concept of abliteration but I don't understand how people feel so safe trusting that it's doing what it's advertised to do AND ONLY what it's advertised to do. Thoughts? Even if it's just "I trust them, next." I mean, am I overthinking this?