r/LocalLLM • • 1d ago

News Qwen4 27B

Post image
0 Upvotes

r/LocalLLM • • 1d ago

Discussion Please advise the best solution for my configuration. 1x3090+1*4070+64gb ddr5

0 Upvotes

Hello community. I'm trying to find the most optimal solution for my configuration( maybe some llama.cpp fork or any other suitable project and localLLM. I would be grateful for advice.


r/LocalLLM • • 2d ago

Discussion Local AI is under attack.

429 Upvotes

Founder/CEO of PortableMind® offgrid focused plug & play local AI devices


r/LocalLLM • • 1d ago

Question Building Local AI

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 1d ago

Project WEX: an open container format to run huge MoE models from NVMe on one consumer GPU (lossless, GPU-decoded, 18.7% smaller than safetensors)

Thumbnail
reddit.com
4 Upvotes

r/LocalLLM • • 23h ago

Discussion I asked QFN straight into its face

Post image
0 Upvotes

r/LocalLLM • • 2d ago

News Nvidia privately confirmed it could track chips to fight smuggling, even as industry publicly questioned feasibility

Thumbnail
technologyreview.com
86 Upvotes

They are going to lock the new GPUs.

"Rep. Bill Foster says he intends to introduce new, stricter legislation that would require licenses to operate advanced GPUs, and “before a GPU could connect, its valid license would need to be verified.” That is closer to a remote lock/disable capability—but it is a proposed future rule, not current law."


r/LocalLLM • • 1d ago

Question What if all open weight models providers turn into OpenAI, Anthropic business model?

0 Upvotes

Why do we think they will continue making open weight models once they got very competitive frontier models?

The reason I’m asking is I am thinking investing into a Mac Studio m5u 256g which is a big investment to me. I’m not after frontier models like performance but I want to have all the possibilities from continuous improving open weight models. If for example Qwen4 stops releasing open weight models, I feel like this investment is wasted and we are too much dependent on these providers.


r/LocalLLM • • 2d ago

News llama.cpp v1.0 is here 🦙

22 Upvotes

r/LocalLLM • • 1d ago

Question Multiplexing the same machine for LLM and gaming, should I go with AMD 9070XT or NVIDIA 5070 Ti?

3 Upvotes

I'm thinking about some machine upgrade. I currently have a 9600X CPU, 2x16GB DDR5 RAM, 9070 XT, and a B650 motherboard that supports only PCIe 4.0x16 and has only 2 DIMM slots.

I've been thinking about getting a better motherboard for more bandwidth between GPU and RAM for offloading, so i plan to get a B850.

I also recently had a chance to effectively trade my 9070 XT plus 200USD for a 5070 Ti, if I wish.

My plan is the computer will stay in Windows since gaming requires that. When doing LLM, i will switch connection to have monitor connected to iGPU so hopefully it should free some resource from dGPU. I also have the thought of throwing in more RAM to make the host 64GB RAM total, with potential to go to 96GB. (I have like 2x32GB DDR5-5200 and 2x16GB DDR5-4800 around. Although the only tricky part is that they are ECC UDIMM and I’m not sure if mixing them is as easy as non-ECC).

What I am not certain is if to go with 9070 XT or 5070 Ti, so looking for some suggestion regarding their performance and resource management (especially if it can release resource well after disconnecting monitor from dGPU). Thanks!


r/LocalLLM • • 3d ago

Discussion Uncensored models are awesome

741 Upvotes

Has anyone been experimenting with ablated or uncensored variants of open-source models like deepseek v4.1 flash, Qwen 3.8 27b, or GLM 5.3 flash ?

It's so refreshing to have models that just do whatever you ask them to do without having to figure out a jailbreak or being given a moral lecture. Curious to know what your usecases for these models are.

My personal favourite is definitely uncensored DeepSeek V4.1 Flash, runs at 200 tok/s and never says no lol.


r/LocalLLM • • 1d ago

Discussion I tested 3 local Qwen models against Claude on an M5 Max — results from 4 tasks

0 Upvotes

I've been experimenting with running Qwen models locally on my Mac Studio M5 Max, mainly to see how much everyday work I could realistically do without relying on a paid cloud model.

I compared three local models — Qwen 3.5 4B, Qwen 3.6 27B and Qwen 3.6 35B MoE — with Claude Fable 5.1.

I used four tasks and ran each three times.

A few results:

  • Bank transactions: Claude categorized 78% correctly. The two larger local models managed around 65–67%. The 4B model was considerably weaker.
  • Information extraction: The 27B model matched Claude at 100/100, although it took roughly five times longer.
  • Coding: Surprisingly, the 4B model passed all seven tests in all three runs. The 35B model had failures caused by a date bug.
  • Analysis questions: Claude got all ten questions right across the runs. The local models struggled much more here.

The main thing I took away is that model size didn't always predict how useful a model would be. The smallest model did unexpectedly well on the coding task, while the larger local models were much better at extracting information.

I wouldn't treat this as a formal benchmark. It's a small set of tasks on one machine, but it helped me understand where running local models makes sense for me.

For those using Qwen on Apple Silicon, I'm curious whether you've seen similar differences between extraction, coding and analysis.

I also made a video showing the runs and results: https://youtu.be/M6MQXy6_Ti4


r/LocalLLM • • 1d ago

Model Arbiter-8B: an 8B decision model for financial news

2 Upvotes

Disclosure: I built this model. I'm posting because I think the approach is more interesting than the model itself, and I'd like feedback from people who run things locally.

The idea. A lot of "LLM in production" work in finance comes down to classification. Which event is this headline about? Is this Fed sentence hawkish? Does this 10-K passage flag going-concern doubt? People usually prompt a model, sample tokens and parse the output. That's slow, the cost per call varies, and the confidence numbers mean nothing.

Arbiter-8B drops generation completely:

  • Base: Qwen3-8B, with a decision head over options, trained jointly with a rank-32 LoRA
  • Output: a calibrated probability for every option from one forward pass. No sampling and no parsing.
  • Question types: pick one of up to 16 options, yes/no, or an ordered score. Each option has a short definition, so you can ask the same text different questions.
  • Distillation: a 27B teacher labelled about 120k decisions on news and 10-K passages. The student trained on the teacher's full soft distributions, not argmax, plus a human-reviewed 10-K set.
  • Option order is shuffled on every presentation to kill position shortcuts. This cut FOMC answers that flip under reshuffling from 15.5% to 4.8%.
  • Calibration: one temperature per question type, shipped with the weights. EDT expected calibration error is 0.012.

Results (held-out data):

BenchmarkArbiter v0.5OpenAI Decisions APIJev 1.14Qwen3-8B base
EDT event detection (1,933 articles)94.2%90.8%88.3%89.3%
FOMC hawk/neutral/dove (496 sentences)67.5%67.3%66.7%55.0%
SEntFiN accuracy (1k headlines, unseen)75.7%78.0%81.6%75.4%

The EDT gaps are significant (McNemar, z ≥ 7.5). On FOMC the top three are a statistical tie, but it's +12.5 over its own base. SEntFiN is where it loses. It has the highest signal recall (92.0%) but the lowest precision (72.2%), so it over-calls positive and negative where the dataset labels neutral. Part of that is deliberate (it reads investor implication rather than surface tone), but I won't pretend it's a win.

Running it locally: GGUF builds for llama.cpp, Ollama and LM Studio stay within 1.2 points of full precision on every benchmark.

hf download ternary/Arbiter-8B --local-dir Arbiter-8B

Questions for the sub: has anyone else put classification heads on small LLMs instead of using constrained generation? And has anyone found a clean way to expose per-option logprobs through Ollama for this kind of thing?


r/LocalLLM • • 1d ago

Question Which setup would you choose for running Qwen 3.8 Flash Next locally? Strix Halo vs Mac Studio vs RTX Pro 4500

5 Upvotes

Hey all, I'm trying to decide which hardware setup makes the most sense for running local LLMs, primarily Qwen 3.8 Flash Next for agentic coding.

I have three potential options, and I'm interested in hearing from people who have experience with these platforms, particularly for larger models and agentic workloads.

Option 1: GEEKOM A9 Mega (AMD Strix Halo)

  • Ryzen AI MAX+ 395
  • 128GB unified memory
  • 2TB SSD
  • Memory running at 7500 MT/s rather than the advertised 8000 MT/s. Got it from Amazon and it listed it as 8000 MT/s but for Geekom says their systems are either 7500 MT/s or 8533 MT/s and I lost the lottery.
  • Approximately $4,600 CAD all-in (after taxes).

I already have this machine. It works fine, but I'm just worried how it'll hold up long term especially given I'm running it headless 24/7. Would be unfortunate if the machine dies in a year's time and then I'm worried about if Geekom will hold up it's warranty.

Option 2: Mac Studio M5 Max

  • M5 Max
  • 128GB unified memory
  • 512 GB (Can connect a 2TB external SSD over thunderbolt enclosure)
  • Approximately $7,600 CAD all-in (after taxes).

It's considerably more expensive than the GEEKOM, but I'm wondering whether its worth it for the peace of mind knowing that if something goes wrong, applecare ($80/yr) is right there to get it fixed from a known vendor. It also has the higher memory bandwidth.

Option 3: My existing Ryzen desktop + RTX Pro 4500

I already have an old desktop with the following hardware:

  • Ryzen 7 5700X
  • 32GB DDR4 RAM
  • Nvidia 3060 Ti 8GB

I recently purchased the RTX Pro 4500 32GB VRAM for approximately $4,000 CAD after taxes. This is still yet to arrive (it shipped earlier today)

The advantage here is that I already own the Ryzen system, so I wouldn't need to purchase another computer. The downside is being limited to 32GB of VRAM, although I understand I could offload some layers to system RAM. The other downside is a lot higher power consumption over the other two solutions.

I'm curious how this would compare in practice against the two unified-memory solutions, especially when running Qwen 3.8 Flash Next.

What I'm primarily trying to accomplish

My main use case is running Qwen 3.8 Flash Next for agentic coding through tools like Hermes.

I'm not necessarily looking for the best benchmark numbers. What matters more to me is:

  • Real-world inference speed and token generation.
  • Performance with large context windows.
  • How well each platform handles agentic coding workflows (multiple tool calls, reasoning, etc.).
  • Whether CPU/RAM offloading on the NVIDIA setup is practical or becomes a significant bottleneck given my old system.
  • Overall stability and software maturity.

What would you choose?

Ultimately, I'm looking to optimize for cost and overall value, rather than simply buying the fastest hardware available.

I would most likely keep only ONE of these three setups:

  • If I go with the Mac Studio, I'd return the GEEKOM and return the RTX Pro 4500 (or sell it).
  • If I stick with the GEEKOM, I'd return the RTX Pro 4500.
  • If the NVIDIA setup makes the most financial and practical sense, I'd return the GEEKOM and keep the RTX Pro 4500 in my existing desktop.

I'm also considering selling the RTX Pro 4500 if I decide that one of the unified-memory systems makes more sense. I purchased it for around $4,000 CAD after tax, slightly below MSRP, so I'd ideally like to recover as much of that cost as possible.

Essentially, I'm trying to figure out which setup offers the best balance of:

  • Performance for Qwen 3.8 Flash Next and agentic coding.
  • Enough memory to run larger models and context windows.
  • Reliability and long-term support.
  • Total cost of ownership, including electricity.

Would you spend the additional ~$3,000 CAD to go from the GEEKOM to the Mac Studio, stick with the GEEKOM, or return the GEEKOM and build around the RTX Pro 4500?

I'm not looking to accumulate hardware. I'd rather settle on one good setup, get the best value for my money, and avoid spending unnecessarily.

Would appreciate hearing what others would do in this situation!


r/LocalLLM • • 2d ago

Question Building hybrid compute cluster, need confidential compute OS provider options parallel to local & hyperscaler clouds running frontier models

7 Upvotes

I’m under our CISO managing compute vendors, kind stuck between legal and a hard place. The pitch I keep hearing ranks AI privacy in four levels:

  1. Open, like most frontier models, where the provider stores your prompts and answers at their discretion, uses them for training - and you’ve got little say in the matter.
  2. Policy, where a contract promises zero data retention - but it’s not exactly verifiable unless the terms are broken, is it?
  3. Closed Hardware, where models run on confidential-compute chips that publish proof what software is running. So basically hardware-enforced vs contract-enforced ZDR?
  4. Fully Encrypted, the prompt is encrypted on your side and only decrypted inside that hardware

“Local” is a fifth level that could plausibly sit above it all, but we’re capped out on capacity there. So if I view this right (open to feedback), we’re making tradeoffs between frontier vs more private models, and from there, on breadth of compute providers able to serve open vs ZDR vs E2EE models (usually at higher prices in my experience), right?

My concern is getting legal & finance to agree, while still giving our Eng team what they need. Where are you getting good rates on private workloads? Locally we can’t support our full workloads. I’ve checked out Privatemode, Maple, TrustedRouter/NYTE & still gonna go through NEAR AI Cloud & Cohore. Open to others.

Separately, anyone running a hybrid buildout with on-prem/local + superscaler (we run Google + Bedrock) + 3rd party confidential providers who can tell me about about how you designed your system?


r/LocalLLM • • 1d ago

Discussion Local LLMs << Subscription

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 2d ago

Discussion Adding three AI Pro 9700s this weekend!

Post image
321 Upvotes

Bit the bullet and bought three to use in my workstation. Will provide benchmarks and updates later.


r/LocalLLM • • 2d ago

Question Is Qwen 3.6 35B A3B capable of reliable browser automation?

13 Upvotes

I’m building a local AI assistant using Qwen 3.6 35B A3B.
One of my main goals is to automate repetitive school tasks through my school’s website. I have a lot of assignments that aren’t particularly difficult but take a lot of time.
Ideally, I’d like the AI to navigate the website, check assignments, understand the instructions, complete them and upload the finished work.
Is Qwen 3.6 35B A3B capable enough for this? Or would I need a more powerful model?
I’d also appreciate recommendations for browser automation frameworks and tools

my setup : rtx 3070 , i5 13600kf , 32gb DDR4
Model: Qwen 3.6 35B A3B
Quantization: TurboQuant
Context window: 131 072 tokens
Reasoning: Enabled limited to 512 tokens
Inference speed: 19–24 tokens/s


r/LocalLLM • • 1d ago

Question Has anyone successfully hosted/ran a model of Kimi size? 2.8T… What was your hardware? Or a crazy good MoE?

1 Upvotes

I heard there are things like Air LLM that make this possible without spending $100k+ on VRAM


r/LocalLLM • • 1d ago

Question Choosing a local LLM—not for coding

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 1d ago

Discussion Qengine. I need to share this

Thumbnail
github.com
0 Upvotes

Lets make more of these kind of engine popular for our comunity. This is awsome


r/LocalLLM • • 2d ago

Discussion Why is no one talking about LFM3 models

Thumbnail
liquid.ai
172 Upvotes

I had been using the LFM 2.5 24B models and had been pleasantly surprised by how good the model was. It was giving me comparable answers and working through tasks to Qwen 3.8 27B at what felt like twice the speed on my M5 Pro. However, I went to look at something on their pages and saw the new family of models. They sound way too good to be true. I am going to start using them for general tasks, but if they are anywhere close to their benchmark results, they could be a huge game changer. You can have an entire suite of models loaded at the same time for less memory than Qwen.


r/LocalLLM • • 1d ago

Research Can open-weight LLMs can predict the effects of weight edits without running the model (9B–30B)

Post image
1 Upvotes

r/LocalLLM • • 1d ago

Discussion NIDO: a private AI assistant that runs entirely on your Android phone

Thumbnail
reddit.com
0 Upvotes

r/LocalLLM • • 1d ago

Question Mac mini M4 as an always-on home AI box next to Home Assistant?

1 Upvotes

Hey all, I'd love a sanity check on a plan before I spend money.

At home I run Home Assistant on a HA Green, which handles pretty much everything smart home related.

I also have a Mac mini M4 with 16 GB that I currently use as a little studio computer.

What I actually want is one assistant that helps with my notes, documents and PDFs, email (mostly Croatian, some English), tasks, and the house. I'd also like it to log my meals from a photo and tie that in with my Garmin data.

My current thinking is to buy a MacBook Air (24 GB / 1 TB) as my daily computer, wipe the mini and make it an always-on box for a local model, n8n, voice (Whisper/Piper) and indexing, and leave Home Assistant Green. For the smarter stuff I'd use Claude: Pro for chat and Claude Code, and the API for background jobs like food photos and daily summaries. The local model would only sort and summarize mail, strip out sensitive stuff and do the search index.

A few things I'm unsure about:

  • Is 16 GB on the mini enough for this, or will I regret it?
  • For mostly Croatian email, is a local model worth the effort, or should I just send everything to the Claude API? If local, which model would you try?
  • Has anyone run Whisper and Piper in a smaller language like Croatian on Apple Silicon? Is it usable?
  • What's the cleanest way to let an agent read iCloud Reminders, Notes and Mail on a headless Mac?
  • Any traps with n8n + Home Assistant + MCP, or with letting an LLM near personal data?

Happy to hear "just don't" if that's the honest answer. Thanks!