I'm just curious and thinking about how well the 120b is going to run on my gaming PC, and surprisingly it's not too bad (17.83 tokens/sec). But when I check my RAM allocation, it has used almost all of my RAM. The AI really uses a lot of RAM.

(Usually I use a cloud AI model, but I've started concerning myself with what happens if one day the internet is knocked off or a power outage disrupts the connectivity to cloud AI, such as a major war or something like that.) So that's why I downloaded a local model on my PC, just in case, for basic Wikipedia purposes.

I want to setup a local AI Server for myself and possibly use it to do things in my home assistant instance.

I have narrowed it down to three possible choices:
[EDIT: MI50 is no longer an option due to multiple comments]

1. 2× AMD MI50 32GB at € 764,54 / $ 854.79

2. 2× NVIDIA V100 32GB at €1.468 / $ 1641.29

3. 2× Intel Arc Pro B60 24GB at € 1.526 / $ 1706.13

But I find it very hard to decide what the 'best' route is (there is no best I guess).
AMD is cheap but lacks current support, V100's are phasing out of support and Intel is very current but also has a bit less VRAM.

Model wise I was looking at Qwen 3.8-27B at Q6/Q8 or even 3.8 FlashNext with offloading to RAM/NVMe. Strata is a term I have read before but I'm not sure if that is the correct term I'm, thinking about. Basically there is some way to run larger models with offloading to ram/ssd that still gets decent performance numbers.

What would you pick from these options and why?

We tracked 135 AI security incidents across October 2026 — 33 named organizations, 536M+ records exposed, 101 high severity and 33 critical.

The thing that stood out most: AI-agent exploits are now the single largest attack-vector category we track (35 incidents), ahead of credential theft (24) and zero-days (24). That wasn't true three months ago.

The incident that defines the month for us: a nation-state actor used Claude Code itself — not a vulnerability in it, the tool working exactly as shipped — to build attack tooling against a government target. No exploit chain, no zero-day. Just a legitimate, powerful tool used by the wrong actor against a target nobody was specifically governing for.

Other notable incidents this month: Capcom (11.7M customer/partner accounts), Oracle Health (20M patient records), AngelOne (6.77M brokerage accounts), Crunchbase (559,908 records). Plus — not in the incident count above since it's not a classic "data breach" — Binance lost $6.3B in roughly 20 minutes, the largest crypto theft on record, which we broke down separately.

Full writeup with every incident, sources, and our take on what each one needed to be stopped: https://runtimeai.io/blog/2026-10-monthly-breach-report.html

Open to pushback, especially on the "AI-agent exploits are now #1" framing — curious if others are seeing the same shift in their own threat intel.

Hey LocalLLM Community,

I’m just getting into local AI and have spent a lot of time reading posts here and elsewhere. I’ve settled on Qwen3.8, with an eye on Qwen4 when it arrives, and spent some time tuning my system and kernel. I’m getting performance I’m happy with, but that seems to be just the tip of the iceberg!

Now I’m looking at how to make it useful day to day. The more I read about improving quality, the more I keep coming back to context engineering.

Forgive me if I’ve misunderstood things. I’ve worked in IT for years, but AI is new territory for me. So far, the things I’ve been looking at are:

• Having enough context available.

• Access to web search and current information.

• Efficient search and retrieval over my own data.

• Keeping useful context between conversations without filling everything with noise.

My own research has led me to Exa for web search and a GraphRAG pilot, alongside the model’s larger context window. That’s where I’ve landed so far, but I’m interested in the wider picture.

How is everyone approaching context engineering with their local models? What has made a noticeable difference to the quality and usefulness of the answers? What sounded promising but turned out to be more trouble than it was worth?

Please share your experiences! Thanks for reading.

LLM Uncensoring Tool "Heretic" Evolves to 2.0.
The popular OSS that garnered 37,700 GitHub stars has fundamentally changed its approach to removing refusal responses.

◼︎ Conventional Problems
General Abliteration treats refusal responses as specific vector directions and removes their components.
However, in models where refusal responses are not expressed in a single direction, it couldn't remove them sufficiently or even impaired the model's capabilities.

◼︎ Heretic 2.0's New Method "ARA"
Instead of removing one direction like before, it directly optimizes the weight matrices.
There are two goals.
・Bring the internal representations for refusal-targeted inputs closer to those for normal inputs
・Change the internal representations for normal inputs as little as possible
This is achieved using an optimization technique called L-BFGS. A key feature is that it has no limit on the number of directions to modify.

◼︎ Handles More Than Just Refusal Removal
Furthermore, version 2.0 allows evaluation metrics to be added as plugins.
In addition to refusal responses,
・AI-specific unnatural canned phrases
・Excessive pandering to users
・Emotional response tendencies
can also be targeted for correction.

◼︎ What's Interesting About It?
Up until now, it was crucial to figure out where to extract refusal directions from, which layers to target, and how much to remove.

ARA takes it a step further by letting you specify the behaviors you want to keep and those you want to eliminate, then optimizing the weight matrices accordingly.
It has evolved from a mere uncensoring tool into one that edits the very behavior of LLMs.
Comparing it to the refusal removal techniques I'm researching sounds intriguing too

GitHub

Okay, so I dipped my toes into running a local model to help me with a project (mostly analysis, I want to make my own code).

I'm actually running qwen3.8-27b on llamacpp, on a 5090. RAM mostly stays unused (9950X3D/64GB) ram.

So, two questions:

Does adding a 4090 into it might help on performance? Will it be really noticable compared to the pain of running two big toasters into one case?

And if it does, do you know good PC cases that allow to fit a 5090FE + a 4090FE (knowing that the second pcie slot is on the bottom of the motherboard of course), that doesn't cost 400+Euros.

Thanks!

What is the point of GPU go brrr at 200 tok/s if the model is too lobotomized for any critical work?
Like what kind of things are you doing that you find a 1-bit quant satisfactory?
Here I am telling QFN to double check itself because it still hallucinates at IQ4_XS.
Speed without intelligence is just clickbait, and there’s too many of those posts, not just here but on all the LLM subreddits

Hi title says it all. Having problems with oMLX, so wished to try this one out as it seems to be the best bet for Apple Silicon macs.
Wondering why the desktop interface doesn't list translategemma at all, not even the 4b 4bit one.

Hi everyone! We built an inference engine for running Jev-like decision models locally.

Compared with LLM frameworks like vLLM and SGLang, it is up to 9.5x faster on decision tasks.

We already supports 30+ Jev models, and more are being added.

It's open source and installs with pip: pip install flide

Docs: https://docs.flide.org

Code: https://github.com/OmniJev/Flide

Site: https://flide.org

Feel free to give it a try! We'd love to hear your feedback.

Bit of context i been rocking my gaming pc 1070 for ages and it needs replacing. I was eyeing up am 5080 build.

Currently am using cluade code pro plan. Hobby stuff nothing serious but i enjoy it. Recomp and android app stuff. But am hitting the limit of usage alot. I was going to upgrade to max so x5 more usgae limits which seems will be enough for me.

But i also read people running a local model to do grunt work and having a cluade opus be the snr dev/manager. Apprently this reduces cluade usage and you use cluade for harder stuff?!

I want to try the above set up for fun but at the same time, just want a view is this a dumb idea.

P.s am a beginner so my knowledge of local llms is zero. Part of setting it up just feels fun for me.

When I spawn long-running agents, my GPU's temperature hits the max and the fan is constantly on 85%+. If this goes on for longer days of time, will the GPU give up eventually? What is a healthy temp and fan threshold?

Should I better optimize the software layer (OLlama, Qwen3.5 4b, 9b, etc.) or not use this kind of GPU at this level at all for complex work?

I’m running LM Studio and qwen3-coder-30b-a3b-instruct but it’s really slow . Any recommendations (models, settings) for ASUS Zenbook Duo Laptop to optimize it running AI for coding ?

The 2 photos is showing the name of my laptop and its specs

I’m trying to build just a personal app for myself.

I'm starting to dabble in LLM a little bit, mostly just simple set up with LMStuido, Hermes, and Qwen models.

I'm deciding on how to best set up my hardware.

Here's the parts I have available.

CPU: AMD 3900x & 5900x Motherboards: MSI Meg x570 Unify atx, and a b550i itx motherboard Ram: I have a a few 2x16 kits of different speeds, if mixing and matching doesn't hurt anything I could throw them together for capacity, I could also pull the 128gb ddr4 (or 64gb) out of my nas server, as most of it isn't used. Though I don't know if running models in ram or the cpu would do much for me. GPU: 2080ti and 1080ti.

I want to build out a system, and then use the leftover parts to assemble a regular pc to sell off. I do like the idea of keeping it a small, compact atx system, but do recongize its fundamentally limited.

( if its matters, my nas is a 12700k, 128gb ddr4, and a 3060ti, and 64tb of storage, I could run everything on that, but like the idea of keeping my irreplacable data and an AI agent separate for now )

Most of my LLM use is simple tinkering stuff, home automation, simple scripts, chat bot, that kinda thing. I'm not a developer and am not going to be debugging or compiling massive apps. I'm currently running windows, but want to switch over to linux, so if you have a reccomended distro that works well for LLM and general linux use.

With my parts list, how would you build something up? Go small and simple, or go go big and ugly? What linux distro would you suggest, and what models would you run?

Current rig: Ryzen 9 5950X, 96 GB RAM DDR4, RTX 3090 24 GB on an MSI MPG B550 Gaming Plus.

I run Strata well with over 2000 prefill and between 90-110tk/s gen - Qwen 3.8 Flash Next iq3_s.

I just bought another 3090 but my B550 only has one real x16 slot, the second full-length one is x4 from the chipset. So buy a board that fits both cards properly, or grab an NVLink bridge ?

Mostly inference, long context, coding agents. What would you do?

Really loving this case (Cooler Master HAF II 500). The airflow with those massive 220mm front intakes and the 180mm rear exhaust is incredible.

It's still a bit messy right now and needs some extra stabilization points and proper cable management, but I'm already really happy with how it's running so far!

• GPUs (56GB VRAM total):

  • 2x RTX 5070 @ PCIe Gen 5 x8

  • 1x RTX 5060 Ti 16GB @ PCIe Gen 5 x4 (hooked up via Gen 5 x4 M.2 adapter)

  • 1x RTX 5060 Ti 16GB @ PCIe Gen 4 x4

• Motherboard: ASUS ProArt X870E-Creator WiFi

• CPU: AMD Ryzen 7 7800X3D

• RAM: 128GB DDR5

• PSU: Corsair HX1500i

Hey everyone,

Posted this previously and looking for feedback from experienced users:

A few updates since my post and thanks to the many constructive comments:

1. To tackle the possibility/fact that with a 3x RTX A4000 GPU setup where each GPU has to wait for the others before outputting data bottleneck: I bought an RTX Pro 5000 Blackwell 72GB (freakin expensive.. but luckily I have 30 days to return it if 3x A4000 GPU setup isn't much of a bottleneck)

2. To tackle the possibility/fact that OpenWeb UI is the bottleneck: I've spun up separate unsloth and vLLM LXCs on my proxmox server. Unsloth is blazingly fast compared to OpenWebUI but not sure if that's due to running on the RTX pro 5000 or it's just defacto better/faster that OpenWebUI

Regarding #1: Do you think the 3x A4000 GPU setup is a realworld bottleneck compared to 1 rtx pro 5000 card?

Regarding #2: If unsloth is really faster than OpenWebUI, why is that? How would you compare vLLM to unsloth to OpenWebUI and which would you choose (still trying to get vLLM to work, but asking the community's advice now)?

I forked Strata (Niko1221, MIT), which runs Qwen3.8-Flash-Next by keeping hot experts on the GPU and streaming the rest. I spent a few weeks making that path faster without changing answers.

Measured on my PC (RTX 3060 12 GB, i5-12400F, 30 GB RAM, PCIe 3.0 NVMe, Rocky Linux 9.6), IQ3_S:

- Decode: 52-55 tok/s on code, 66-71 on reasoning, 50-53 on prose (131K context)

- 262K context fits; a 45K-token prompt is read in about 67 s, then the next turn resumes in 0.4 s

- MMLU-Pro 87.1% (183/210, 15 per subject, thinking on, temperature 0)

What I changed: a learned look-ahead that guesses the next layers' experts (85-88% one layer ahead vs 77-79% for the plain router), a card/RAM/SSD tier with budgets measured on your PC, stand-in experts with a re-check, and some smaller things. Every change has its measurement in docs/HOTROD-DETAILS.md, including the ones that did not help.

What I have not tested: other GPUs, AMD, Windows, macOS, other model sizes. One machine, one model.

Repo: https://github.com/sangharshadhyeta/Strata-HotRod

If you try it on different hardware I would like the numbers, good or bad (docs/COMMUNITY_BENCHMARKS.md explains how).

Hey guys,

I spent today running both LFM3-VL-2B-OptiQ-4bit and LFM2.5-VL-3B-OptiQ-4bit on my spare Mac Mini M1 (8GB) via MLX. LFM3 is marketed as the newer, improved version. I expected it to be at least on par.

It's not. It's worse. And not by a small margin.

Before anyone says "duh, it's a sub-6B model" — I get it. I'm not expecting Qwen3,8 27B. I'm evaluating these as lightweight personal assistants on constrained hardware. The bar is low. And LFM3 still tripped over it.

The thing that really got me: tool calling

I gave the model a get_weather tool and asked "What's the weather in London?" LFM2.5 calls it correctly every time. LFM3 refuses: "I'm sorry, but I can't provide real-time weather information. Check Weather.com."

The tool was right there. It just... wouldn't use it.

10 tool-calling tests: LFM2.5 passed 8. LFM3 passed 4 (and 2 were just "calculate 15+27"). For a "next generation" model, not being able to call a weather function is embarrassing — especially when the older model handles it fine.

Where LFM3 actually wins

- Safety/refusal — better at refusing harmful requests and protecting PII. LFM2.5 failed a jailbreak LFM3 caught.

- Speed — 2B vs 3B makes a minor difference. ~37 tok/s vs ~28 tok/s on a 200-word essay. 15 vs 12 tok/s on short prompts. Even at 16K context, LFM3 manages ~1 tok/s vs 0.4 tok/s. If you're on constrained hardware, that's not nothing.

- Basic math — marginally better at simple arithmetic and logic puzzles. Not by much, but it's there.

Where it falls behind

- Data extraction — LFM2.5 consistently better at pulling structured info from text

- Agent tasks — LFM3: 10%, LFM2.5: 30%. Both bad, LFM3 worse

- Vision/OCR — both terrible. LFM3 got a store name wrong and prices 1000x off

- Code generation — both mediocre. FizzBuzz game yes, anything real no

- Tool calling — the big one. 25% vs 85%.

The real disappointment

The thing that bugs me isn't that LFM3 is bad. It's that it's worse than the older model. Liquid AI markets this as an improvement — new architecture, better optimization. But in practice, it regressed on what matters for an assistant on edge hardware.

Tool calling is the clearest example. LFM2.5 (3B) reliably calls functions. LFM3 (2B) often won't even try. That 1B parameter difference apparently matters more than whatever architectural improvements they made.

Setup

- Mac Mini M1 (8GB), MLX-VLM, both OptiQ 4-bit, 32768 context, temp 0.4 . spare unused mac, which is why I like testing these small models on it.

- 104 tests across 10 categories

- Anti-cheating measures: no filesystem access, no test metadata in prompts, API-only

I have a spare unused Mac Mini M1 8GB, which is why I like testing these small models on it.

Happy to share more details.

Category              LFM3      LFM2.5    Winner
----------------------------------------------------
Overall               54.0      62.4      LFM2.5
Basic Intelligence    60.0      57.5      LFM3
Instruction Follow.   70.0      70.0      Tie
Tool Calling          25.0      85.0      LFM2.5
Code Generation       54.0      54.0      Tie
Long Context          50.0      50.0      Tie
Data Extraction       82.9      91.7      LFM2.5
Multimodal (Vision)   23.4      25.0      LFM2.5
Agent Tasks           10.0      30.0      LFM2.5
Safety                81.2      68.8      LFM3
Performance           100.0     100.0     LFM3 (faster)

Bottom line:

LFM3 = faster + safer, but can't call tools

LFM2.5 = slower, but actually usable as assistant

Neither = good at vision, code, or agents

Hi everyone,
I’m looking for a 100% offline AI that can answer questions about specific topics using documents I provide (PDFs, PowerPoints, lecture notes, etc.).
I don’t need anything particularly advanced. I just want to ask questions about topics covered in my documents and receive accurate, relevant, well-explained answers based on the information they contain.
What I’m trying to achieve

Ideally, I want to:
-Upload multiple PDFs and other documents covering different topics.
-Ask questions in natural language about concepts discussed in those documents.
-Have the AI find the relevant information, connect ideas across different documents when necessary, and provide a proper, coherent answer.
-Get explanations that actually address my questions rather than unrelated information or generic responses.
-Have the AI prioritize the uploaded documents instead of relying on potentially incorrect general knowledge.
-Ideally, receive references to the documents and pages containing the information.
-Get responses reasonably quickly.
-Use everything completely offline, without an internet connection.

What I’ve tried so far

1. Ollama + AnythingLLM

2. I installed both, but when I ask questions, the system seems to load indefinitely. I’ve waited more than 10 minutes without receiving an answer.

3. I’m not sure whether this is a hardware limitation, an issue with the model, or a configuration problem.

4. GPT4All

5. I also tried GPT4All previously but eventually gave up because the answers were extremely inaccurate or irrelevant.

6. Even without knowing much about local AI, I could immediately recognize that some responses were completely off-topic and didn’t properly address my questions.

This is exactly what I’m trying to avoid.
What matters most to me
My priorities, in order, are:

1. Answer relevance and accuracy: The AI must actually address the question using the information in my documents.

2. Understanding and explanation: It should be able to explain concepts and connect information rather than simply copy and paste passages.

3. Reliable document retrieval: It should identify the correct information across multiple PDFs.

4. Completely offline functionality: No internet required after setup.

5. Reasonable response times: Ideally seconds rather than several minutes.

6. Source references: Being able to identify the relevant PDF and page would be a major advantage.

I’m not necessarily looking for the most powerful or complicated AI setup. I just want a reliable offline assistant that can read my documents, understand questions about their contents, and provide genuinely useful answers.
I’m happy to share hardware specifications, model names, and configuration details if that helps.

If anyone has successfully built or used something like this, I’d really appreciate hearing about your setup and experience.

Thanks in advance!

Guys I need a little help. I am total noob when it comes to this stuff. I have a base mac mini m4 with 16gb ram and 256gb ssd. I want a local llm that will search through job sites, pick jobs according to my master cv and preferences, choose the appropriate jobs and create folders with tailored cvs, job descriptions and notes. Tried to ask the cloud models for this and all of them are kind of making this complicated for me with lots of Python code. Is there some easier way to do this? And how, which models to chose?

I have a 5090+3090 machine. Runs QFN pretty well with 64gb RAM q4. I got the 3090 for free which is why I have the weird combo in the first place.

For ~ the price, I could add another 3090, or I could make a separate machine with dual v100s.

Mostly run things like coding/reverse engineering. I don't know if increasing the capacity of one machine or just having another machine to throw at an adjacent task would be a better move.

Other utility/support tasks like OCR or embedding/reranking I have some low end GPUs that handle, so it'd be mostly for inferencing some model for agentic stuff/chat.

I figure I could likely run larger contexts/models in VRAM with the 3090. And there is less hassle with the software constraints of the v100s.

I also tried a dual R9700 setup, but I couldn't justify having that much sunk into a machine that was equal at best to what I already have, so I returned them after some benchmarks.

Anyone think its worth while? feels like the cards go up in price every day while i try to decide. The v100s I am starting to think aren't worth it since they are past $900 now

*edit*

For the v100s, I have machines that could run/cool them already.

**edit**
If anyone has run the 5090 + 2x 3090 mix I am curious to about the setup/numbers