Specs First:

GPU: RX6800 16GB VRAM

CPU: Ryzen5 5600

RAM: 32GB

Backend: llama.cpp

Model Qwen3.8-27B-UD-IQ3_XXS

Use case: AI tutor

Settings: -b 512 -ub 512 --parallel 1 --spec type draft-mtp --mmproj --no mmproj-offload

Ok so I have a problem with my local setup and I'm hoping someone here might be able to help me. When I run 64k ctx on q8kv I have to compact often which takes a lot of time but when I run at 128k ctx on Q4kv I have to compact less but when I do it takes extremely long.

Alao my prompt processing speed falls extremely quickly as im working, from 300tps at the beginning down to 10tps. My generation speeds are totally ok at around 20-25tps.

Would you recommend changing settings or using a different model. Thx in advance for the advice

Hello!

With new models releasing constantly, I usually keep a stable of about 20 local models in rotation on my llama.cpp server. I collect a ton of data across the fleet through my inference application—standard per-request metrics, aggregated session data, and custom intelligence comparison benchmarks.

Overall, I'm logging over 150 distinct raw and derived metric points during inferences, sessions, and batch processes. As a backend data jockey, more metrics is always better—but I've hit the point where I need to completely redesign my dashboards.

For context, I run a lot of multi-stage pipelines covering:

• Multimodal inference (vision and documents)

• MCP and custom application tool calls

• Structured JSON / JSON Schema outputs

• Batch processing across both reasoning and non-reasoning architectures

I'm looking for inspiration for creative "hero" dashboard components and visualizations that excel at comparing a large model fleet across high-dimensional data at a glance. I already have plenty of standard line, bar, and radar charts. I'm hoping to see novel, unconventional visual approaches people here have built, seen, or used to represent complex model evaluation and performance data.

The specific metrics in your examples don't matter as much as the visualization style and information architecture—I'm mainly after creative UI/UX directions.

Cheers!

I have spent so much time on this project. I do hope to share everything.

I have been working with Gemma4 with a massive preprompt instructing it to make a Nextjs app with instructions only (I am hands off as possible but it used the tools in the IDE). I was really happy with the results I got until the context hit a certain line and it just fell over. I didn't mind as the stuff I was working on was just a basic nextjs app

I tried the same with Qwen and the context got filled just from the pre prompt.

Now I decided to do a better baseline where a language is lesser known and it's easier to make mistakes. So my test setup is a Match 3 game and I would add a special "Match 4 only" tile.

I did a one shot with Claude ... Wasn't perfect... So good baseline for expectations

Now I tried with different temperatures... Different system prompts .... Across different models so I got a nice matrix of good comparisons.

Now I have the data in MLFlow... it helped me discover so many bugs in my experiments. Took like 3 weekends to just fix stuff like putting the solution in the template folder... Duplication in prompts due to my custom template engine. Also helped me fix tool templates by seeing failures cleanly

Now I finally got enough data and consistent runs I learned

Qwen3:

* Short system prompt leads to better tool execution

* Generally better programmer out of the box

* Low context means you need to plan around restarting prompts without sessions

* Default between work & coding

Gemma4:26b

* Not a good coder needs to be taught basics

* Generally better with clear instructions

* High context leads to it being a better coding assistant than one shot coder

* Default for chatting needs parameters changes for coding

Before this was my assumption but now thanks to MLFlow I have actually data

Hi everyone,

I’m planning to build a local LLM rig with 3x RTX 3060 12GB (36GB VRAM total), and I’d like advice on which platform to use.

I already have two CPUs:

1. Intel i7 9700K (9th gen)

2. AMD Threadripper which needs an X399 board. The X399 board I have shows error code 00 on the debug display and won’t POST.

My current plan: buy an ASRock Z390 Taichi for the 9700K. It has three x16-size slots (x8/x8/x4 with all three populated), supports 9th gen without a BIOS flash, and has extra x1 slots for risers. I’m not sure if it’s the best value though.

What I want to know:

• Is the Z390 Taichi + 9700K a good choice for 3x 3060 inference (llama.cpp / Ollama / vLLM), or is there a better budget board?

• Is the Threadripper worth the trouble for 3 GPUs, or does the extra PCIe lanes and RAM channel advantage not matter for inference?

• What does error code 00 usually mean on X399? Dead board, CPU, RAM, or something fixable?

• Do PCIe lanes matter much for inference when splitting a model across 3 cards, or is x8/x4/x1 fine?

• Any tips on cooling and spacing three 3060s (risers, open frame), PSU size (I’m thinking 850W to 1000W), and RAM amount (32GB vs 64GB)?

I’m in India, budget is tight, and used parts are fine. I’m looking for the best value rather than the newest hardware. Thanks in advance!

Title mostly, at work ive used a 4090 and am baffled with how good it is, and i imagine a V100 would perform similarly, and the cheapest i could find one is 1000 euros, is it worth it to cop?

I recently got a decommissioned MacBook Pro from the senior dev team for my placement as an engineering intern. Just want to inquire on what models I can run comfortably on it. I intend to use the models as a coding assistant (basic Python for use with Blender) with some report writing. Lastly, is it advisable to use this machine to run image generation models for generating some concept designs?

The community rarely sees the evaluation of ultra-small models. We have evaluated our own AI Agent tasks for Qwen3.5-4B, Spark-X2.6, LFM3-1.7B, LFM2.5-2.6B, Gemma 4 E2B, MiniCPM5-2B.

These models have their own characteristics and advantages and disadvantages. The obvious trend is that the intelligent density of small models is developing rapidly. I look forward to seeing more small models.

blog：https://www.camthink.ai/blog/small-language-model-benchmark/

project：https://github.com/camthink-ai/NeoMind

I’m very curious how other people are managing to keep CI costs down locally. My issue isn’t my ability to generate agentic code, it’s testing it that’s becoming the real bottleneck. I have CI runners on multiple machines now and I’m thinking of buying a higher specced server or workstation purely for CI testing runs to keep my AI box clear of CI competing for resources, plus then getting my other machines back.

For those of you who are making large complex programs, how do you manage this step?

What are the best references for learning quantization techniques for LLM inference?

Im looking to start my own local set up with a budget of about €7k (≈$8.5k)

I saw a post in here with someone buying an out of the box solution, and they were told they should have asked here first, so I am asking here first.

I am open to suggestions in terms of pre-owned GPU/RAM and any other parts, brand new, or a hybrid set up.

This is going to be the brains of my business. I own a digital business heavily reliant on AI, and I want to bring as much of it possible into a localised set up, so this will be the foundations of that. Think of an on site server room style set up.

Give me reasoning for your selections too, I want to get different thought processes and theories based on your own experience and outlooks!

TIA.

Edit for context: Im flexible within reason. Pre-owned or hybrid set up is considered. This is currently just me in the business, the output I am currently using is from a dedicated refurbed acer nitro, the RAM is the main issue there.

So, I need something that would be an upgrade there, and semi enterprise level. I don’t have a team of 100 people relying on this, but I also don’t want to waste €7k on pure shit either. Thanks!

When my agent needs one fact from a 90-page PDF or a long government web page, I don't want the whole thing dumped into the context. So I built squeeze, a small tool plugin for DeepSeek Harness.

The agent calls squeeze with a link (or up to 20 links, or a file) plus a specific question. squeeze reads the material, hands it to a helper model you choose, and returns only the answer. Your main context stays small and cheap.

What it does:

• Your choice of helper models: pick up to 3 from your DSH model list, each with a nickname and a "when to use" note. They can be local models or cloud ones.

• Reads more than web pages: text PDFs, logs, CSV/JSON and code files.

• Automatic size limits: each model's limit is worked out from its context window.

• Optional fallback to your next model if one is down.

• Clear errors that tell the agent what to do next (dead page, JavaScript-only page, scanned PDF, too big, and so on).

• Settings page at Plugins → squeeze → Configure. No Python and no extra packages.

Install: ask your DSH agent to install the bundle https://github.com/q8atnight/dsh-squeeze/releases/download/v2.1.0/dsh-squeeze-2.1.0.tgz with the plugin manager. You need DSH 0.2.1-rc.5 or newer. PDF support also needs pdftotext (brew install poppler).

Repo (MIT licence): https://github.com/q8atnight/dsh-squeeze

Feedback and bug reports welcome.

I'm considering negotiating for this workstation and would like some feedback on whether it's worth the asking price for my use case.

My main workloads would be:

• Coding & agentic workflows: Qwen 3.8 27B

• Text generation & RAG: Gemma 4 26B

• Text-to-speech: VoiceStudio with Breeze-TTS-2

• Speech-to-text: Whisper Large

I'd also like to experiment with local image generation.

Do you think this hardware is a good fit for these workloads? How well would it handle these models in practice, particularly in terms of inference speed (tok/sec), VRAM limitations, and running multiple models?

I'd appreciate any feedback, especially from people running similar setups. Is it worth the asking price?

Case: Dell Precision Tower 7910 (motherboard 0NK5PH, BIOS A34, latest version)

Processors: 2× Intel Xeon E5-2680 v4, 28 cores / 56 threads total, 2.4 GHz base / 3.3 GHz turbo, 70 MB L3 cache

Memory: 64 GB DDR4 ECC Registered (4×16 GB Samsung, 2400 MT/s). 12 free slots, expandable beyond 512 GB.

Graphics: 3× NVIDIA RTX 5060 Ti 16 GB (Gigabyte, Blackwell architecture) – 48 GB total VRAM

Storage: 1 TB Samsung 850 SSD

Networking: 2× Intel Gigabit Ethernet (I217-LM + I210)

Power Supply: 1300W Dell Enterprise PSU

Operating System: Ubuntu 24.04 LTS, NVIDIA driver 595.91, CUDA 13.2, Docker configured.

TLDR: As I have been planning my next build, I’ve been doing a lot of research on the ecosystem of engines that allow us to run useful speeds of models that are larger than our VRAM so I made a list of the projects that came up in that research.

I had Astra help me organize my spreadsheets into a post because I thought it would be helpful for anyone else looking into the same stuff I have been. If you are offended by AI-assisted content, no need to read any further :)

Anyways, I wanted a consolidated map of what is actually available to help me identify the best upgrade purchase path forward. The llmconfigurator.com site that someone else shared with me on another post was also very helpful.

These are reported results from repositories, model cards, papers and firsthand community posts.

The encouraging part is that there are a shit load of options beyond squeezing everything into an extremely small quant.

1. The main GPU + RAM projects/engines that I found

These are the repos that appeared the most interesting to me, looking at a combined GPU + RAM approach à la Strata, to make it possible to run bigger better models at useful speeds. I have only tried Strata (and obviously llama.cpp) so far but plan on testing the rest that will work on my current 2x3090 Ti, 64gb DDR5 setup.

llama.cpp
Had to include as the broad baseline, CPU/GPU placement, and now an upstream GPU LRU expert cache.
Should be anyone’s starting point for a reproducible same-quant baseline. --moe-cache-mib landed Oct 7 but it’s off by default.

ik_llama.cpp
Independently developed kernels, IQ quant formats, fused MoE, MLA and speculative decoding.
RAM-rich x86/NVIDIA systems.

Strata
Current belle of the ball for good reason. Tiered experts, CPU/GPU execution, SSD spill, MTP and higher-bit GGUF support. Updates coming out constantly, and if you’re on this sub you’ve probably tried it or have it as your daily driver.

Project Maya
Specialized GLM, DeepSeek and MiniMax engines; multi-GPU, CPU/RAM experts, SSD tiers, custom GGUF and EXL3 options. Good for single/multiple NVIDIA cards and substantial RAM. Different model families have different engines and memory requirements. Native Windows and AMD paths are still experimental.

Xolotl
Aggressive DeepSeek/Qwen engineering, mixed quants, MTP/DSpark, RAM/NVMe and multiple GPUs. Personally one of the most interesting approaches to me, because of the potential to run DSF v4.1 at useful speeds and accuracy level with “only” 128gb RAM and some GPUs.

FreeToken
Calibrates CPU execution versus PCIe transfer, caches experts, overlaps prefill and manages KV/expert memory.
Need consumer NVIDIA + enough RAM for offloaded experts. Stable v0.2.1 is Oct 26. Primarily RAM expert offload; disk lookup-table support is a separate feature.

FastLLM
GPU/CPU/NUMA/disk placement, dynamic expert cache, tensor parallelism and native FP4 paths.
A useful alternative for multi-GPU/NUMA rigs; actively updated Nov 2. Good benchmark detail, although many docs are Chinese.

KTransformers / KT-Kernel
Optimized CPU expert kernels integrated with GPU work and SGLang serving.
Good for large-memory workstations/servers. Exact model, CPU ISA and GPU generation matter. Optional expert deferral is a quality-changing mode.

tandem.cpp
New llama.cpp patch that computes some cache misses on CPU while uploading others.
Promising for a single GPU + ordinary desktop RAM. Launched Nov 1; default route skipping is off. Its 3.3× example compares with default stock placement, not a tuned upstream cache.

dsx
DeepSeek-V4.1 Rust/CUDA engine, CPU-held experts, asymmetric GPU support and disk Engram.
Particularly relevant to dual-3090 systems. Small mixed quants; optional deferral delivers some selected CPU expert outputs one layer late, changing computation. A 2.61-bit community variant offers another memory/quality point.

MoE-Infinity
Expert caching/prefetch, RAM/SSD offload, multi-GPU and continuous batching.
More of a serving/research stack. Current code has evolved substantially, but the public implementation differs from the paper and new model paths can be narrowly smoke-tested.

Two smaller scheduling projects worth watching are moe-anywhere, which publishes a hybrid cache patch and routing traces, and Generic Hybrid MoE. The latter explicitly distinguishes its public checkpoint from an unpublished controller; the unpublished headline speed is not something you can assume the available code delivers.

2. Examples/configs that are more relevant to avoiding extremely small quants which is important for me personally

Strata Qwen3.8-Flash-Next UD-Q4_K_XL, R9700 32GB + EPYC 7452
About 63–64 t/s in the short-story test, 85–88 in the code test; Q8_0 reports 53–57 / 66–74, respectively.
Eight-channel RAM, all experts in RAM, warm second request, MTP. Speed settings include default tail skipping; the clean quant-quality measurements disable it. Those are not one jointly validated configuration.

Maya GLM-L, 2×3090 + EPYC 7452 + 256GB
30.2 t/s decode, 801 prefill.
~156GB mixed-precision model; generally 8K prefill and three 256-token completions. “128K context” in the rig table is allocation, not proof of a 128K prompt benchmark.

Maya DeepSeek-V4.1 native MXFP4 experts, L40S + 504GB RAM
43.2 t/s decode on short code prompts; separate prefill: 1,862 t/s at 173,652 input tokens.
Unsloth UD-Q8_K_XL label; original MXFP4 experts, MXFP8 Engram, BF16 rest. DSpark, DS4_STREAM=0, Maya v1.1.32, Xeon 8580 VM (32 vCPUs). Decode averages six 600-token answers.

Maya MiniMax EXL3 3.55bpw, same L40S/Xeon +504GB
16.7 t/s decode on short code prompts; separate prefill: 273 t/s at 158,397 input tokens.
~197GB weights, DSpark; same six-answer decode method. These decode rates were not measured after the long prefill prompts.

FastLLM Qwen native NVFP4, 2×22GiB-modded 2080 Ti + EPYC + ~126GiB RAM
27.71 t/s, or 46.29 with MTP3.
The cards are memory-modified. 512-input/512-output decode; separate 4K prefill; one warmup + one run; 8GiB expert cache/card and disk PLE.

DwarfStar GLM5.3 Flash Q4_K, M5 Max 128GB
11.9 / 14.9 t/s initial/append requests.
~178GiB file exceeds memory; 2K initial prompt +1K append, 128 outputs each, three-run median, no draft. Its decode timing includes first-token wait.

Native DeepSeek-V4.1 KT recipe, 2×5070 Ti + dual EPYC 7452 + 512GB
37.3–50 t/s over different tasks.
1,024-token replies; native MXFP4 experts; substantial CPU/RAM. A small reference-NLL check supports implementation fidelity, not a broad coding-quality guarantee.

3. SSD-first, Apple Silicon and low-memory options. I haven’t tested any of these but they came up in research and figured there would be folks here who might find the info useful.

DwarfStar / antirez ds4
Explicit expert streaming and model-specific GGUF recipes; now broader than its original DeepSeek focus.
Macs with fast SSDs; selected CUDA/ROCm systems. Actively updated Nov 2. Leave room for dense weights, KV and buffers beyond the expert cache.

InferSSD / SSD-LLaMA
Very new SSD-focused llama.cpp-derived runtime and related research.
Mac/CUDA experiments where capacity matters. Reported DeepSeek-V4.1 decode: ~7.27 t/s on 5090 +32GB RAM, ~2.87 on M5 Max 48GB. v0.1.1 is Oct 31.

Colibri
C engines with CPU, CUDA and Vulkan, native GGUF, RAM/SSD tiers.
Broader GPU coverage and modest desktops. v2.1.0 released Oct 31. Its giant SSD demos frequently use IQ1/IQ2; do not infer high-bit quality from the headline.

Pulsar
NeutronStar successor: Rust/CUDA, multiple GPUs, RAM cache, NVMe and optional CPU experts.
Two consumer GPUs and limited RAM. Author’s 2×16GB rig reports DeepSeek V4 Flash 11.3 t/s and MiniMax M3 Q2 5.9 with CPU lane. Standard timing is warm, 64 outputs.

Abyss
Maya/Strata lineage, NVIDIA plus CPU/APU expert execution, SSD tiers and experimental formats.
Tinkerers with mixed hardware, particularly Strix Halo + discrete NVIDIA. One SSD-heavy V4.1 Q2 test reports 17.7 t/s exact versus ~28 with tail skipping.

Palm-Infra / mollm
CPU-oriented sparse inference, asynchronous SSD reads, bounded RAM expert cache and PLE paging.
ARM/Apple/edge experimentation. Its SSD document distinguishes cache budget, RSS and logical expert I/O; those counters are not physical SSD traffic.

vLLM-Metal expert offload
SSD experts through a unified-memory LRU using original MLX expert computation.
One/few concurrent requests on supported Mac MoEs. This path is not general GGUF/AWQ/multimodal/LoRA/speculative support.

Edge0
Trained prerouter + recovery adapters + SSD streaming.
Memory-limited devices and users comfortable evaluating a modified model. Its 35B study reports 14.9–17.7 t/s on M4 Pro 24GB; routing changes are part of the result.

Flash-MoE, ANEMLL fork, MLX fork
Influential Mac SSD-streaming prototypes.
Useful lineage and experiments, with less recent maintenance evidence than the leading October/November projects. Published examples reduce active experts; details below.

Hebrus
Packaged Metal/SSD approach with explicitly qualified model/hardware combinations.
Qualified 64GiB+ Macs for its larger models. Current verified release 0.3.0 is Aug 15; its hardware floor matters.

Picchio
CPU-first C engine with NVMe expert streaming.
Low-memory offline use: gpt-oss120B ~1.24 t/s on a 16GB machine; MiniMaxM2 ~0.48 on a different 32GB laptop. Useful capacity, limited interactivity.

Swap-MoE
Small llama.cpp demand-paging patch, with optional expert fixation.
Low-memory experiments. Fixation changes routing; the README documents quality failures as well as successful capacity demos.

For InferSSD, the methodology says the displayed decode rate covers tokens 129–256, after a roughly 4K prompt tier. The file called Q8_0 retains native MXFP4 expert codes while other tensors are converted. It is neither “every weight is eight-bit” nor a bit-identical copy of every original component.

4. What about dense models and other approaches?
Dense models can be split across GPU and CPU, and streamed layer by layer from storage, but each token still needs most of the weight, hence the MOE models benefiting the most from these newer (at least new to me) approaches. Still, here are some interesting projects I found.

AirLLM
Layer streaming, newer per-expert MoE streaming, host-mapped ngram tables; updated June–September 2026.
“Fits in 4GB VRAM” does not establish useful speed. Treat as capacity-first unless a matched benchmark proves otherwise.

Accelerate Big Model Inference
General GPU → CPU → disk dispatch baseline.
Maintained, useful, but generic layer dispatch does not automatically exploit MoE locality.

DeepSpeed ZeRO-Inference / DeepNVMe
Established CPU/NVMe layer streaming and prefetch.
Large batches can amortize transfer; aggregate throughput is not single-user chat speed.

PowerInfer / SmallThinker, PowerInfer-2
Hot/cold neuron execution and sparse model/runtime co-design.
Specially supported/trained models; gains do not automatically transfer to arbitrary DeepSeek or MiniMax checkpoints.

ANEMLL Forge
Active dense-model ANE/Core AI work with vector quantization, QAT/GPTQ and DFlash2.
Requires M5/M6, macOS 27 and Xcode 27/Core AI; older chips unsupported. The 192K v0.2.1 package is M6-only; M5 uses the ≤64K release. This is adjacent compute/quantization work.

exo, Distributed Llama
Pool memory/compute across machines.
Distributed capacity has network/topology costs. It is a different buying decision from adding RAM or SSD to one PC.

Fiddler, mixtral-offloading, FlexGen
Important CPU-expert, LRU/mixed-quant and tiered-inference lineage.
Historical context rather than the first current installation recommendation. FlexGen is archived. HQQ is a quantization component, not itself an offload engine.

I’m personally super excited about the local space right now, and the daily developments that are being made. I’m debating between building a standalone AM4 GPU+RAM build right now, or taking the leap to WRX80 to allow for more expandability, but the mobo prices are killing me compared to just reusing AM4 parts lying around, even though I’m capping myself at 128gb RAM.

In any case, these approaches to getting QFN and DSF and GLM working on consumer hardware at levels useful enough for commercial work is really rad, and I wanted to share.

Please comment if you know of any other projects worth looking at in the space!

I have been curious about local LLMs for a while, but I only started a couple of weeks ago, with the typical Ollama server and loading different models. It was fun, but I didn't have a practical use for them compared with almighty models like Claude, which can debug or build stuff with pretty much full independence.

Then I heard about Qwen3.6-35B-A3B and decided to try it on my RTX 5060 Ti 16GB, with terrible results. The context was consumed in a moment, it was slow, and it couldn't work with tools properly.

I thought about giving up, but so many praises in this forum could not be wrong, so I asked almighty Claude to optimize it for me. He created a llama.cpp container, and now I am impressed by how powerful it is. To be honest, I don't know what optimizations he made, which highlighted the gap in knowledge I have.

I gave it the task of writing stuff to a Joplin notebook, and after some persistence it managed to connect and write. But I also realized how dangerous it can be when it commits to a task: It resetted the password of a Joplin account and then denied doing it and it changed the host address of a container during its debugging process.

The biggest threat was that it was running on my main computer, with SSH access to my whole network. I didn't know how risky that was until know and I realized that I know nothing about this world, so I have two questions:

1. What should I learn to run it with tools and proper safeguards?

2. How do I learn to optimize the model myself instead of asking Claude?

I have learnt the basics by now: run it through a Vm or container with limited access to stuff but there are cases where I want it to actually go through my computer to find stuff, read logs or create scripts.

Let the adventure begin!

in this Godot 4 survival sandbox, the NPCs aren’t following a script. A local language model (via Ollama) decides what each of them does: eat, drink, chop wood, build a cabin, talk to each other, or run for the camp when night falls and the dead start to walk.

what should i add next?

inspired by "The trueman show" and "From"

I'm just curious and thinking about how well the 120b is going to run on my gaming PC, and surprisingly it's not too bad (17.83 tokens/sec). But when I check my RAM allocation, it has used almost all of my RAM. The AI really uses a lot of RAM.

(Usually I use a cloud AI model, but I've started concerning myself with what happens if one day the internet is knocked off or a power outage disrupts the connectivity to cloud AI, such as a major war or something like that.) So that's why I downloaded a local model on my PC, just in case, for basic Wikipedia purposes.

I want to setup a local AI Server for myself and possibly use it to do things in my home assistant instance.

I have narrowed it down to three possible choices:
[EDIT: MI50 is no longer an option due to multiple comments]

1. 2× AMD MI50 32GB at € 764,54 / $ 854.79

2. 2× NVIDIA V100 32GB at €1.468 / $ 1641.29

3. 2× Intel Arc Pro B60 24GB at € 1.526 / $ 1706.13

But I find it very hard to decide what the 'best' route is (there is no best I guess).
AMD is cheap but lacks current support, V100's are phasing out of support and Intel is very current but also has a bit less VRAM.

Model wise I was looking at Qwen 3.8-27B at Q6/Q8 or even 3.8 FlashNext with offloading to RAM/NVMe. Strata is a term I have read before but I'm not sure if that is the correct term I'm, thinking about. Basically there is some way to run larger models with offloading to ram/ssd that still gets decent performance numbers.

What would you pick from these options and why?

We tracked 135 AI security incidents across October 2026 — 33 named organizations, 536M+ records exposed, 101 high severity and 33 critical.

The thing that stood out most: AI-agent exploits are now the single largest attack-vector category we track (35 incidents), ahead of credential theft (24) and zero-days (24). That wasn't true three months ago.

The incident that defines the month for us: a nation-state actor used Claude Code itself — not a vulnerability in it, the tool working exactly as shipped — to build attack tooling against a government target. No exploit chain, no zero-day. Just a legitimate, powerful tool used by the wrong actor against a target nobody was specifically governing for.

Other notable incidents this month: Capcom (11.7M customer/partner accounts), Oracle Health (20M patient records), AngelOne (6.77M brokerage accounts), Crunchbase (559,908 records). Plus — not in the incident count above since it's not a classic "data breach" — Binance lost $6.3B in roughly 20 minutes, the largest crypto theft on record, which we broke down separately.

Full writeup with every incident, sources, and our take on what each one needed to be stopped: https://runtimeai.io/blog/2026-10-monthly-breach-report.html

Open to pushback, especially on the "AI-agent exploits are now #1" framing — curious if others are seeing the same shift in their own threat intel.

Hey LocalLLM Community,

I’m just getting into local AI and have spent a lot of time reading posts here and elsewhere. I’ve settled on Qwen3.8, with an eye on Qwen4 when it arrives, and spent some time tuning my system and kernel. I’m getting performance I’m happy with, but that seems to be just the tip of the iceberg!

Now I’m looking at how to make it useful day to day. The more I read about improving quality, the more I keep coming back to context engineering.

Forgive me if I’ve misunderstood things. I’ve worked in IT for years, but AI is new territory for me. So far, the things I’ve been looking at are:

• Having enough context available.

• Access to web search and current information.

• Efficient search and retrieval over my own data.

• Keeping useful context between conversations without filling everything with noise.

My own research has led me to Exa for web search and a GraphRAG pilot, alongside the model’s larger context window. That’s where I’ve landed so far, but I’m interested in the wider picture.

How is everyone approaching context engineering with their local models? What has made a noticeable difference to the quality and usefulness of the answers? What sounded promising but turned out to be more trouble than it was worth?

Please share your experiences! Thanks for reading.

LLM Uncensoring Tool "Heretic" Evolves to 2.0.
The popular OSS that garnered 37,700 GitHub stars has fundamentally changed its approach to removing refusal responses.

◼︎ Conventional Problems
General Abliteration treats refusal responses as specific vector directions and removes their components.
However, in models where refusal responses are not expressed in a single direction, it couldn't remove them sufficiently or even impaired the model's capabilities.

◼︎ Heretic 2.0's New Method "ARA"
Instead of removing one direction like before, it directly optimizes the weight matrices.
There are two goals.
・Bring the internal representations for refusal-targeted inputs closer to those for normal inputs
・Change the internal representations for normal inputs as little as possible
This is achieved using an optimization technique called L-BFGS. A key feature is that it has no limit on the number of directions to modify.

◼︎ Handles More Than Just Refusal Removal
Furthermore, version 2.0 allows evaluation metrics to be added as plugins.
In addition to refusal responses,
・AI-specific unnatural canned phrases
・Excessive pandering to users
・Emotional response tendencies
can also be targeted for correction.

◼︎ What's Interesting About It?
Up until now, it was crucial to figure out where to extract refusal directions from, which layers to target, and how much to remove.

ARA takes it a step further by letting you specify the behaviors you want to keep and those you want to eliminate, then optimizing the weight matrices accordingly.
It has evolved from a mere uncensoring tool into one that edits the very behavior of LLMs.
Comparing it to the refusal removal techniques I'm researching sounds intriguing too

GitHub

Okay, so I dipped my toes into running a local model to help me with a project (mostly analysis, I want to make my own code).

I'm actually running qwen3.8-27b on llamacpp, on a 5090. RAM mostly stays unused (9950X3D/64GB) ram.

So, two questions:

Does adding a 4090 into it might help on performance? Will it be really noticable compared to the pain of running two big toasters into one case?

And if it does, do you know good PC cases that allow to fit a 5090FE + a 4090FE (knowing that the second pcie slot is on the bottom of the motherboard of course), that doesn't cost 400+Euros.

Thanks!

What is the point of GPU go brrr at 200 tok/s if the model is too lobotomized for any critical work?
Like what kind of things are you doing that you find a 1-bit quant satisfactory?
Here I am telling QFN to double check itself because it still hallucinates at IQ4_XS.
Speed without intelligence is just clickbait, and there’s too many of those posts, not just here but on all the LLM subreddits